Academic literature on the topic 'Image structure representation'

Multimedia documents are composed of multiple file format combinations, such as image and text, image and sound, or image, text and sound. The type of multimedia document determines the form of analysis for knowledge architecture design and retrieval methods. Over the last few decades, theories of text analysis have been proposed and applied effectively. In recent years, theories of image and sound analysis have been proposed to work with text retrieval systems and progressed quickly due in part to rapid progress in computer processing speed. Retrieval of multimedia documents formerly was divided into the categories of image and text, and image and sound. While standard retrieval process begins from text only, methods are developing that allow the retrieval process to be accomplished simultaneously using text and image. Although image processing for feature extraction and text processing for term extractions are well understood, there are no prior methods that can combine these two features into a single data structure. This dissertation will introduce a common representation format for multimedia documents (CRFMD) composed of both images and text. For image and text analysis, two techniques are used: the Lorenz Information Measurement and the Word Code. A new process named Jeong's Transform is demonstrated for extraction of text and image features, combining the two previous measurements to form a single data structure. Finally, this single data measurements to form a single data structure. Finally, this single data structure is analyzed by using multi-dimensional scaling. This allows multimedia objects to be represented on a two-dimensional graph as vectors. The distance between vectors represents the magnitude of the difference between multimedia documents. This study shows that image classification on a given test set is dramatically improved when text features are encoded together with image features. This effect appears to hold true even when the available text is diffused and is not uniform with the image features. This retrieval system works by representing a multimedia document as a single data structure. CRFMD is applicable to other areas of multimedia document retrieval and processing, such as medical image retrieval, World Wide Web searching, and museum collection retrieval.

There has been great interest in recent times in the use of elastography for the characterization of human tissue. Digital Image Elasto-Tomography is a novel breast cancer pre-screening technique under development at the University of Canterbury, which aims to identify and locate stiff areas within the breast that require further investigation using images of the surface motion alone. A calibrated array of five digital cameras is used to capture surface motion of the breast under harmonic actuation. The forward problem, that is the resulting motion for a given mechanical property distribution, is calculated using the Finite Element Method. The inverse problem is to find the mechanical properties which reproduce the measured surface motion through numerical simulation. A reconstruction algorithm is developed using a shape based description to reduce the number of parameters in the inverse problem. A parallel Genetic Algorithm is developed for parameter optimization. A geometric method termed Fitness Function Analysis is shown to improve the inclusion location optimization problem. The ensemble of solutions generated using the Genetic Algorithm is used to produce an optimal and a credible region for inclusion location. Successful single frequency phantom reconstructions are presented. An effective way of combining information from multi-frequency phantom data by examining the characteristics of the measured surface motion using data quality metrics is developed and used to produce improved reconstructions. Results from numerical simulation datasets and a two inclusion phantom used to test the optimization of multiple and ellipsoidal inclusions indicate that although two inclusions can be successfully reconstructed, the single inclusions assumption may suffice even in irregular, heterogeneous cases. This assumption was used to successfully locate the stiffest inclusion in a phantom containing multiple inclusions of differing stiffness based on three multi-frequency datasets. The methods developed in phantoms are applied to three in vivo cases for both single and multi-frequency data with limited success. This thesis builds on previous work undertaken at the University of Canterbury. The original contributions in this work are as follows. A new reconstruction algorithm combining a genetic algorithm with fitness function analysis is developed. The most realistic tissue mimicking phantoms to date are used. An ellipsoidal shape-based description is presented, and applied to the first multi-inclusion reconstructions in DIET. This work presents the first reconstruction using meshes created directly from data using a meshing algorithm developed by Jonas Biehler. A multi-frequency cost function is developed to produce the first multi-frequency and in vivo reconstructions using DIET data.

A photograph typically depicts an aspect of the real world, such as an outdoor landscape, a portrait, or an event. The task of creating abstract digital representations of images has received a great deal of attention in the computer vision literature because it is rarely useful to work directly with the raw pixel data. The challenge of working with raw pixel data is that small changes in lighting can result in different digital images, which is not typically useful for downstream tasks such as object detection. One approach to representing an image is automatically extracting and quantising visual features to create a bag-of-terms vector. The bag-of-terms vector helps overcome the problems with raw pixel data but this unstructured representation discards potentially useful information about the spatial and semantic relationships between the parts of the image. The central argument of this thesis is that capturing and encoding the relationships between parts of an image will improve the performance of extrinsic tasks, such as image description or search. We explore this claim in the restricted domain of images representing events, such as riding a bicycle or using a computer. The first major contribution of this thesis is the Visual Dependency Representation: a novel structured representation that captures the prominent region–region relationships in an image. The key idea is that images depicting the same events are likely to have similar spatial relationships between the regions contributing to the event. This representation is inspired by dependency syntax for natural language, which directly captures the relationships between the words in a sentence. We also contribute a data set of images annotated with multiple human-written descriptions, labelled image regions, and gold-standard Visual Dependency Representations, and explain how the gold-standard representations can be constructed by trained human annotators. The second major contribution of this thesis is an approach to automatically predicting Visual Dependency Representations using a graph-based statistical dependency parser. A dependency parser is typically used in Natural Language Processing to automatically predict the dependency structure of a sentence. In this thesis we use a dependency parser to predict the Visual Dependency Representation of an image because we are working with a discrete image representation – that of image regions. Our approach can exploit features from the region annotations and the description to predict the relationships between objects in an image. In a series of experiments using gold-standard region annotations, we report significant improvements in labelled and unlabelled directed attachment accuracy over a baseline that assumes there are no relationships between objects in an image. Finally, we find significant improvements in two extrinsic tasks when we represent images as Visual Dependency Representations predicted from gold-standard region annotations. In an image description task, we show significant improvements in automatic evaluation measures and human judgements compared to state-of-the-art models that use either external text corpora or region proximity to guide the generation process. In the query-by-example image retrieval task, we show a significant improvement in Mean Average Precision and the precision of the top 10 images compared to a bag-of-terms approach. We also perform a correlation analysis of human judgements against automatic evaluation measures for the image description task. The automatic measures are standard measures adopted from the machine translation and summarization literature. The main finding of the analysis is that unigram BLEU is less correlated with human judgements than Smoothed BLEU, Meteor, or skip-bigram ROUGE.

In clinical applications it is often beneficial to use multiple imaging technologies to obtain information about different biomedical aspects of the subject under investigation, and to make best use of such sets of images they need to first be registered or aligned. Registration of multi-modal images is a challenging task and is currently the topic of much research, with new methods being published frequently. Structural representation models extract underlying features such as edges from images, distilling them into a common format that can be easily compared across different image modalities. This study compares the performance of two recent structural representation models on the task of aligning multi-modal biomedical images, specifically Second Harmonic Generation and Two Photon Excitation Fluorescence Microscopy images collected from skin samples. Performance is also evaluated on Brightfield Microscopy images. The two models evaluated here are PCANet-based Structural Representations (PSR, Zhu et al. (2018)) and Discriminative Local Derivative Patterns (dLDP, Jiang et al. (2017)). Mutual Information is used to provide a baseline for comparison. Although dLDP in particular gave promising results, worthy of further investigation, neither method outperformed the classic Mutual Information approach, as demonstrated in a series of experiments to register these particularly diverse modalities.

This research project examines an intriguing type of depictions which can be found within Sarum processional manuscripts, a genre of liturgical books which were produced between the thirteenth and fifteenth centuries. The central focus is a specific example from Norwich which was produced between the late fourteenth and early fifteenth centuries. I propose that their flat, ordered, and geometrically arranged mode of representation can be best understood when considered in relation to the semantic characteristics of notational systems. Their visual form signals that they are not representations of idiosyncratic events which have happened in the past, but are instead authoritative prescriptive layouts. They illustrate what important objects are required for the performance of a ritual and the number, status and position of the participants that will need to be in attendance. Thus, I argue that the viewer is not intended to be a passive witness to a scene taking place in the image, but is instead a presumed participant in a future performance of a specific character. Three arguments are introduced to lend support to the thesis. The first presents historical evidence which illustrates the authoritative role given to these books. The text discusses their widespread use and argues that this authoritative role may have been the result of a deliberate strategy on the part of the individuals seeking to increase the circulation of the books associated with the Sarum Use. The second argument is based on the examination of the relationships between the images and the texts found within the books. It states that the images do not present sufficient information to be considered pictorial instructions, but instead, can convey other meanings. The final argument is that the pictorial images have the semantic characteristics of a notational system. I argue that they are related to one specific system—the musical scores which interleave the images and intermingle with them in the pictorial frame.<br>Arts, Faculty of<br>Art History, Visual Art and Theory, Department of<br>Graduate

La représentation d’image sous une forme hiérarchique a été largement utilisée dans un contexte de classification. Une telle représentation est capable de modéliser le contenu d’une image à travers une structure arborescente. Dans cette thèse, nous étudions les méthodes à noyaux qui permettent de prendre en entrée des données sous une forme structurée et de tenir compte des informations topologiques présentes dans chaque structure en concevant des noyaux structurés. Nous présentons un noyau structuré dédié aux structures telles que des arbres non ordonnés et des chemins (séquences de noeuds) équipés de caractéristiques numériques. Le noyau proposé, appelé Bag of Subpaths Kernel (BoSK), est formé en sommant les noyaux calculés sur les sous-chemins (un sac de tous les chemins et des noeuds simples) entre deux sacs. Le calcul direct de BoSK amène à une complexité quadratique par rapport à la taille de la structure (nombre de noeuds) et la quantité de données (taille de l’ensemble d’apprentissage). Nous proposons également une version rapide de notre algorithme, appelé Scalable BoSK (SBoSK), qui s’appuie sur la technique des Random Fourier Features pour projeter les données structurées dans un espace euclidien, où le produit scalaire du vecteur transformé est une approximation de BoSK. Cet algorithme bénéficie d’une complexité non plus linéaire mais quadratique par rapport aux tailles de la structure et de l’ensemble d’apprentissage, rendant ainsi le noyau adapté aux situations d’apprentissage à grande échelle. Grâce à (S)BoSK, nous sommes en mesure d’effectuer un apprentissage à partir d’informations présentes à plusieurs échelles dans les représentations hiérarchiques d’image. (S)BoSK fonctionne sur des chemins, permettant ainsi de tenir compte du contexte d’un pixel (feuille de la représentation hiérarchique) par l’intermédiaire de ses régions ancêtres à plusieurs échelles. Un tel modèle est utilisé dans la classification des images au niveau pixel. (S)BoSK fonctionne également sur les arbres, ce qui le rend capable de modéliser la composition d’un objet (racine de la représentation hiérarchique) et les relations topologiques entre ses sous-parties. Cette stratégie permet la classification des tuiles ou parties d’image. En poussant plus loin l’utilisation de (S)BoSK, nous introduisons une nouvelle approche de classification multi-source qui effectue la classification directement à partir d’une représentation hiérarchique construite à partir de deux images de la même scène prises à différentes résolutions, éventuellement selon différentes modalités. Les évaluations sur plusieurs jeux de données de télédétection disponibles dans la communauté illustrent la supériorité de (S)BoSK par rapport à l’état de l’art en termes de précision de classification, et les expériences menées sur une tâche de classification urbaine montrent la pertinence de l’approche de classification multi-source proposée<br>Hierarchical image representations have been widely used in the image classification context. Such representations are capable of modeling the content of an image through a tree structure. In this thesis, we investigate kernel-based strategies that make possible taking input data in a structured form and capturing the topological patterns inside each structure through designing structured kernels. We develop a structured kernel dedicated to unordered tree and path (sequence of nodes) structures equipped with numerical features, called Bag of Subpaths Kernel (BoSK). It is formed by summing up kernels computed on subpaths (a bag of all paths and single nodes) between two bags. The direct computation of BoSK yields a quadratic complexity w.r.t. both structure size (number of nodes) and amount of data (training size). We also propose a scalable version of BoSK (SBoSK for short), using Random Fourier Features technique to map the structured data in a randomized finite-dimensional Euclidean space, where inner product of the transformed feature vector approximates BoSK. It brings down the complexity from quadratic to linear w.r.t. structure size and amount of data, making the kernel compliant with the large-scale machine-learning context. Thanks to (S)BoSK, we are able to learn from cross-scale patterns in hierarchical image representations. (S)BoSK operates on paths, thus allowing modeling the context of a pixel (leaf of the hierarchical representation) through its ancestor regions at multiple scales. Such a model is used within pixel-based image classification. (S)BoSK also works on trees, making the kernel able to capture the composition of an object (top of the hierarchical representation) and the topological relationships among its subparts. This strategy allows tile/sub-image classification. Further relying on (S)BoSK, we introduce a novel multi-source classification approach that performs classification directly from a hierarchical image representation built from two images of the same scene taken at different resolutions, possibly with different modalities. Evaluations on several publicly available remote sensing datasets illustrate the superiority of (S)BoSK compared to state-of-the-art methods in terms of classification accuracy, and experiments on an urban classification task show the effectiveness of proposed multi-source classification approach

The emergence of new media tools and social media platforms over the last ten years has created new means of intercultural engagement. On Instagram, there is a growing trend of travel pages and travel bloggers whose aim is to introduce and share with other users the highlights of the destinations they travel to. This also applies to locals in these destinations who wish to portray their country positively and promote it as a tourist destination, particularly in certain countries of the Middle East where there is the added challenge of an ‘unsafe’ image to combat. This research focuses on Lebanon to find out to what extent Instagram can be considered a tool to combat the ‘unsafe’ image of the country, and if users who come across depictions of Lebanon on Instagram perceive the country as a potential tourist destination. Using a mixed methods approach, this thesis combines surveys and semi-structured interviews with Canadian participants to reach its findings. Its theoretical framework makes use of Edward Said’s conception of the ‘other’ (1978), Stuart Hall’s system of representations (1980), Jan Neverdeen Pieterse’s hybridization paradigm (1996), and Eli Pariser’s (2011) echo chamber to analyze its findings. Broadly, findings show that while Instagram can effectively be considered a tool to counter the ‘unsafe’ image of Lebanon, and while the country may be branded as a potential tourist destination to users who come across such depictions of it, algorithm restrictions limit the potential for such depictions to fulfill their potential since they don’t always reach users who perceive Lebanon to be an ‘unsafe’ place.

Des del seu inici, la comunitat investigadora sobre reconeixement de patrons i visió per computador ha reconegut la importància d’aprofitar la informació estructural de les imatges. Els grafs s’han seleccionat com el marc adequat per representar aquest tipus d’informació a causa de la seva flexibilitat i poder de representació capaç de codificar, tant els components, objectes i entitats com les seves relacions. Tot i que els grafs s’han aplicat amb èxit a una gran varietat de tasques -com a resultat de la seva naturalesa simbòlica i relacional- sempre han patit d’algunes limitacions comparats amb mètodes estadístics. Això es deu al fet que algunes operacions matemàtiques trivials no tenen una equivalència en el domini dels grafs. Per exemple, en la base de moltes aplicacions de reconeixement de patrons hi ha la necessitat de comparar objectes. No obstant això, aquesta operació trivial no està degudament definida per grafs quan considerem vectors de característiques. Al llarg d’aquesta recerca, el principal domini d’aplicació està basat en el tema de l’Anàlisi i Reconeixement d’Imatges de Documents. Aquest és un subcamp de la Visió per Computador que té com a objectiu compendre imatges de documents. En aquest context, l’estructura -particularment la representació en forma de graf- proporciona una dimensió complementària al contingut de la imatge. En Visió per Computador la primera dificultat que ens trobem recau en construir una representació significativa de grafs capaç de codificar les característiques rellevants d’una imatge donada. Això es deu al fet que és un procés que ha de trobar un equilibri entre la simplicitat de la representació i la flexibilitat, per tal de representar les diferents deformacions que apareixen en cada domini d’aplicació. Hem estudiat aquest tema en l’aplicació de la recerca de paraules, dividint els diferents traços en grafemes –les unitats més petites d’un alfabet manuscrit&-. També, hem investigat diferents metodologies per accelerar el procés de comparació entre grafs perquè la recerca de paraules o, inclús, de forma més general, l’aplicació en la recerca de grafs, pugui incloure grans col·leccions de documents. Aquestes metodologies han estat principalment dues: (a) un sistema d’indexació de grafs combinat amb un sistema de votació en l’àmbit de nodes capaç d’eliminar resultats improbables i (b) usant representacions jeràrquiques de grafs que duen a terme la majoria de les comparacions en una versió reduïda del graf original, mitjançant comparatives entre els nivells més abstractes i els més detallats. A més a més, la representació jeràrquica també ha demostrat obtenir una representació més robusta que el graf original, lidiant amb el soroll i les deformacions de manera elegant. Per tant, proposem explotar aquesta informació en forma de codificació jeràrquica del graf que permeti utilitzar tècniques estadístiques clàssiques. Els nous avenços en aprenentatge profund geomètric han aparegut com una generalització de les metodologies d’aprenentatge profund aplicades a dominis no Euclidians –com grafs i varietats–, i han promogut un gran interès en la comunitat científica per aquests esquemes de representació. Així doncs, proposem una distància de grafs capaç d’obtenir resultats comparables a l’estat de l’art en diferents tasques aprofitant aquests nous desenvolupaments, però considerant les metodologies tradicionals com a base. També hem realitzat una col·laboració industrial amb la finalitat d’extreure informació automàtica de les factures de l’empresa (amb dades anònimes). El resultat ha estat el desenvolupament d’un sistema de detecció de taules en documents administratius. D’aquesta manera les xarxes neuronals basades en grafs han demostrat ser aptes per detectar patrons repetitius, els quals, després d’un procés d’agregació, constitueixen una taula.<br>La comunidad que investiga el reconocimiento de patrones y la visión por computador ha reconocido la importancia de aprovechar la información estructural de las imágenes. Los grafos se han seleccionado como el marco adecuado para representar este tipo de información a causa de su flexibilidad y poder de representación capaz de codificar los componentes, los objetos, las entidades y sus relaciones. Aunque los grafos se han aplicado con éxito a una gran variedad de tareas –como resultado de su naturaleza simbólica y relacional–, siempre han sufrido algunas limitaciones comparados con los métodos estadísticos. Esto se debe al hecho que algunas operaciones matemáticas triviales no tienen una equivalencia en el dominio de los grafos. Por ejemplo, en la base de la mayoría de aplicaciones de reconocimiento de patrones hay la necesidad de comparar objetos. No obstante, esta operación trivial no está debidamente definida por grafos cuando consideramos vectores de características. Durante la investigación, el principal dominio de aplicación se basa en el Análisis y Reconocimiento de Imágenes de Documentos. Este es un subcampo de la Visión por Computador que tiene como objetivo comprender imágenes de documentos. En este contexto la estructura -particularmente la representación en forma de grafo- proporciona una dimensión complementaria al contenido de la imágen. En Visión por Computador la primera dificultad que nos encontramos se basa en construir una representación significativa de grafos que sea capaz de codificar las características relevantes de una imagen. Esto se debe a que es un proceso que tiene que encontrar un equilibrio entre la simplicidad de la representación y la flexibilidad, para representar las diferentes deformaciones que aparecen en cada dominio de la aplicación. Hemos estudiado este tema en la aplicación de la búsqueda de palabras, dividiendo los diferentes trazos en grafemas –las unidades más pequeñas de un alfabeto manuscrito–. Tambien, hemos investigado diferentes metodologías para acelerar el proceso de comparación entre grafos para que la búsqueda de palabras o, incluso, de forma más general, la aplicación de búsqueda de grafos, pueda incluir grandes colecciones de documentos. Estas metodologías han estado principalmente dos: (a) un sistema de indexación de grafos combinado con un sistema de votación en el ámbito de los nodos capaces de eliminar resultados improbables y (b) usando representaciones jerárquicas de grafos que llevan a término la mayoría de las comparaciones en una versión reducida del grafo original mediante comparativas entre los niveles más abstractos y los más detallados. Asimismo, la representación jerárquica también ha demostrado obtener una representación más robusta que el grafo original, además de lidiar con el ruido y las deformaciones de manera elegante. Así pues, proponemos explotar esta información en forma de codificación jerárquica del grafo que permita utilizar técnicas estadísticas clásicas. Los nuevos avances en el aprendizaje profundo geométrico han aparecido como una generalización de las metodologías de aprendizaje profundo aplicadas a dominios no Euclidianos –como grafos y variedades– y han promovido un gran interés en la comunidad científica por estos esquemas de representación. Proponemos una distancia de grafos capaz de obtener resultados comparables al estado del arte en diferentes tareas aprovechando estos nuevos desarrollos, pero considerando las metodologías tradicionales como base. También hemos realizado una colaboración industrial con la finalidad de extraer información automática de las facturas de la empresa (con datos anónimos). El resultado ha sido el desarrollo de un sistema de detección de tablas en documentos administrativos. Así pues, las redes neuronales basadas en grafos han demostrado ser aptas para detectar patrones repetitivos, los cuales, después de un proceso de agregación, constituyen una tabla.<br>From its early stages, the community of Pattern Recognition and Computer Vision has considered the importance on leveraging the structural information when understanding images. Usually, graphs have been selected as the adequate framework to represent this kind of information due to their flexibility and representational power able to codify both, the components, objects or entities and their pairwise relationship. Even though graphs have been successfully applied to a huge variety of tasks, as a result of their symbolic and relational nature, graphs have always suffered from some limitations compared to statistical approaches. Indeed, some trivial mathematical operations do not have an equivalence in the graph domain. For instance, in the core of many pattern recognition application, there is the need to compare two objects. This operation, which is trivial when considering feature vectors, is not properly defined for graphs. Along this dissertation the main application domain has been on the topic of Document Image Analysis and Recognition. It is a subfield of Computer Vision aiming at understanding images of documents. In this context, the structure and in particular graph representations, provides a complementary dimension to the raw image contents. In computer vision, the first challenge we face is how to build a meaningful graph representation that is able to encode the relevant characteristics of a given image. This representation should find a trade off between the simplicity of the representation and its flexibility to represent the deformations appearing on each application domain. We applied our proposal to the word spotting application where strokes are divided into graphemes which are the smaller units of a handwritten alphabet. We have investigated different approaches to speed-up the graph comparison in order that word spotting, or more generally, a retrieval application is able to handle large collections of documents. On the one hand, a graph indexing framework combined with a votation scheme at node level is able to quickly prune unlikely results. On the other hand, making use of graph hierarchical representations, we are able to perform a coarse-to-fine matching scheme which performs most of the comparisons in a reduced graph representation. Besides, the hierarchical graph representation demonstrated to be drivers of a more robust scheme than the original graph. This new information is able to deal with noise and deformations in an elegant fashion. Therefore, we propose to exploit this information in a hierarchical graph embedding which allows the use of classical statistical techniques. Recently, the new advances on geometric deep learning, which has emerged as a generalization of deep learning methods to non-Euclidean domains such as graphs and manifolds, has raised again the attention to these representation schemes. Taking advantage of these new developments but considering traditional methodologies as a guideline, we proposed a graph metric learning framework able to obtain state-of-the-art results on different tasks. Finally, the contributions of this thesis have been validated in real industrial use case scenarios. For instance, an industrial collaboration has resulted in the development of a table detection framework in annonymized administrative documents containing sensitive data. In particular, the interest of the company is the automatic information extraction from invoices. In this scenario, graph neural networks have proved to be able to detect repetitive patterns which, after an aggregation process, constitute a table.