Tesi sul tema "Graph Attention Networks"

Attention is a substantial mechanism for human to process massive data. It omits the trivial parts and focuses on the important ones. For example, we only need to remember the keywords in a long sentence and the principal objects in an image for rebuilding the sources. Therefore, it is crucial to building an attention network for artificial intelligence to solve the problem as human. This mechanism has been fully explored in the text-based tasks, such as language translation, reading comprehension, and sentimental analysis, as well as the visual-based tasks, such as image recognition, object detection, and action recognition. In this work, we explore the attention mechanism in the multi-modal tasks, which involve the inputs of both text and image, i.e. visual question answering and visual dialog. It involves three vital components in both tasks, the input question (with history for visual dialog), the given image, and the generated answers. Therefore, three kinds of relationships should be investigated step by step to solve the problem. We first build the attention between words and objects for generating the representation of them, followed by the relationship between the representation and the answers if the general word embedding does not work properly, and the relationship between the representation and the attributes of answers comes last for few-shot learning. First, the bilinear graph networks revisit the relationship between the words from question and objects for image in the visual question answering task from a graph perspective. The classical bilinear attention networks build a bilinear attention map to extract the joint representation of words and objects but lack fully exploring the relationship between words for complex reasoning. In contrast, our networks model the context of the joint embeddings of words and objects. Two kinds of graphs are investigated, namely image-graph and question-graph. The image-graph transfers features of the detected objects to their related query words, enabling the output nodes to have both semantic and factual information. The question-graph exchanges information between these output nodes from image-graph to amplify the implicit yet important relationship between objects. These two kinds of graphs cooperate with each other, and thus our resulting model can model the relationship and dependency between objects, which leads to the realization of multi-step reasoning. Then, our novel image-question-answer synergistic network values the role of the answer for precise visual dialog. We extend the traditional one-stage solution to a two-stage solution. In the first stage, candidate answers are coarsely scored according to their relevance to the image and question pair. Afterward, in the second stage, answers with high probability of being correct are re-ranked by synergizing with image and question. Finally, we propose to learn the representations of attributes from the answers with enough data, which are later composed to constrain the learning of the few-shot ones. We generate the few-shot dataset of VQA with a variety of answers and extract their attributes without any human effort. With this dataset, we build our attribute network to disentangle the attributes by learning their features from parts of the image instead of the whole one.

This thesis deals with rotation estimation of 3D model of human jaw. It describes and compares methods for direct analysis od 3D models as well as method to analyze model using rasterization. To evaluate perfomance of proposed method, a metric that computes number of cases when prediction was less than 30° from ground truth is used. Proposed method that uses rasterization, takes  three x-ray views of model as an input and processes it with convolutional network. It achieves best preformance, 99% with described metric. Method to directly analyze polygonal model as a sequence uses attention mechanism to do so and was inspired by transformer architecture. A special pooling function was proposed for this network that decreases memory requirements of the network. This method achieves 88%, but does not use rasterization and can process polygonal model directly. It is not as good as rasterization method with x-ray display, byt it is better than rasterization method with model not rendered as x-ray.  The last method uses graph representation of mesh. Graph network had problems with overfitting, that is why it did not get good results and I think this method is not very suitable for analyzing plygonal model.

In recent years, deep learning has made a significant impact in various fields – helping to push the state-of-the-art forward in many application domains. Convolutional Neural Networks (CNN) have been applied successfully to tasks such as visual object detection, image super-resolution, and video action recognition while Long Short-term Memory (LSTM) and Transformer networks have been used to solve a variety of challenging tasks in natural language processing. However, these popular deep learning architectures (i.e., CNNs, LSTMs, and Transformers) can only handle data that can be represented as grids or sequences. Due to this limitation, many existing deep learning approaches do not generalize to problem domains where the data is represented as graphs – social networks in social network analysis or molecular graphs in chemoinformatics, for instance. The goal of this thesis is to help bridge the gap by studying deep learning solutions that can handle graph data naturally. In particular, we explore deep learning-based approaches in the following areas. 1. Graph Attention. In the real-world, graphs can be both large – with many complex patterns – and noisy which can pose a problem for effective graph mining. An effective way to deal with this issue is to use an attention-based deep learning model. An attention mechanism allows the model to focus on task-relevant parts of the graph which helps the model make better decisions. We introduce a model for graph classification which uses an attention-guided walk to bias exploration towards more task-relevant parts of the graph. For the task of node classification, we study a different model – one with an attention mechanism which allows each node to select the most task-relevant neighborhood to integrate information from. 2. Graph Representation Learning. Graph representation learning seeks to learn a mapping that embeds nodes, and even entire graphs, as points in a low-dimensional continuous space. The function is optimized such that the geometric distance between objects in the embedding space reflect some sort of similarity based on the structure of the original graph(s). We study the problem of learning time-respecting embeddings for nodes in a dynamic network. 3. Brain Network Discovery. One of the fundamental tasks in functional brain analysis is the task of brain network discovery. The brain is a complex structure which is made up of various brain regions, many of which interact with each other. The objective of brain network discovery is two-fold. First, we wish to partition voxels – from a functional Magnetic Resonance Imaging scan – into functionally and spatially cohesive regions (i.e., nodes). Second, we want to identify the relationships (i.e., edges) between the discovered regions. We introduce a deep learning model which learns to construct a group-cohesive partition of voxels from the scans of multiple individuals in the same group. We then introduce a second model which can recover a hierarchical set of brain regions, allowing us to examine the functional organization of the brain at different levels of granularity. Finally, we propose a model for the problem of unified and group-contrasting edge discovery which aims to discover discriminative brain networks that can help us to better distinguish between samples from different classes.

To combat fake news, researchers mostly focused on detecting fake news and journalists built and maintained fact-checking sites (e.g., Snopes.com and Politifact.com). However, fake news dissemination has been greatly promoted by social media sites, and these fact-checking sites have not been fully utilized. To overcome these problems and complement existing methods against fake news, in this thesis, we propose a deep-learning based fact-checking URL recommender system to mitigate impact of fake news in social media sites such as Twitter and Facebook. In particular, our proposed framework consists of a multi-relational attentive module and a heterogeneous graph attention network to learn complex/semantic relationship between user-URL pairs, user-user pairs, and URL-URL pairs. Extensive experiments on a real-world dataset show that our proposed framework outperforms seven state-of-the-art recommendation models, achieving at least 3~5.3% improvement.

The Vehicle Routing Problem (VRP) is one of the most intensively studied combinatorial optimization problems in the Operations Research (OR) community. Its relevance is not only related to the various real-world applications it deals with, but to its inherent complexity being an NP-hard problem. From its original formulation more than 60 years ago, numerous mathematical models and algorithms have been proposed to solve VRP. The most recent trend is to leverage Machine Learning (ML) in conjunction with these traditional approaches to enhance their performance. In particular, this work investigates the use of ML-driven components as destroy or repair methods inside the Large Neighborhood Search (LNS) metaheuristic, trying to understand if, where, and when it is effective to apply them in the context of VRP. For these purposes, we propose NeuRouting, an open-source hybridization framework aimed at facilitating the integration between ML and LNS. Regarding the destroy phase, we adopt a Graph Neural Network (GNN) assisted heuristic, which we hybridize with a neural repair methodology taken from the literature. We investigate this integration both on its own and as part of an Adaptive Large Neighborhood Search (ALNS), performing an empirical study on instances of various sizes and against some traditional solvers.

Starting in 2027, the high-luminosity Large Hadron Collider (HL-LHC) will begin operation and allow higher-precision measurements and searches for new physics processes between elementary particles. One central problem that arises in the ATLAS detector when reconstructing event information is to separate the rare and interesting hard scatter (HS) interactions from uninteresting pileup (PU) interactions in a spatially compact environment. This problem becomes even harder to solve at higher luminosities. This project relies on leveraging the time dimension and determining a time of the HS interactions to separate them from PU interactions by using information measured by the upcoming High-Granularity Timing Detector (HGTD). The current method relies on using a boosted decision tree (BDT) together with the timing information from the HGTD to determine a time. We suggest a novel approach of utilizing a graph attentional network (GAT) where each bunch-crossing is represented as a graph of tracks and the properties of the GAT are applied on a track level to inspect if such a model can outperform the current BDT. Our results show that we are able to replicate the results of the BDT and even improve some metrics at the expense of increasing the uncertainty of the time determination. We conclude that although there is potential for GATs to outperform the BDT, a more complex model should be applied. Finally, we provide some suggestions for improvement and hope to inspire further study and advancements in this direction which shows promising potential.
Från och med 2027 kommer \textit{high-luminosity Large Hadron Collider} (HL-LHC) att tas i drift och möjliggöra mätningar med högre precision och utforskningar av nya fysikprocesser mellan elementarpartiklar. Ett centralt problem som uppstår i ATLAS-detektorn vid rekonstruktionen av partikelkollisioner är att separera sällsynta och intressanta interaktioner, så kallade \textit{hard-scatters} (HS) från ointressanta \textit{pileup}-interaktioner (PU) i den kompakta rumsliga dimensionen. Svårighetsgraden för detta problem ökar vid högre luminositeter. Med hjälp av den kommande \textit{High-Granularity Timing-detektorns} (HGTD) mätningar kommer även tidsinformation relaterat till interaktionerna att erhållas. I detta projekt används denna information för att beräkna tiden för enskillda interaktioner vilket därmed kan användas för att separera HS-interaktioner från PU-interaktioner. Den nuvarande metoden använder en trädregressionsmetod, s.k. boosted decision tree (BDT) tillsammans med tidsinformationen från HGTD för att bestämma en tid. Vi föreslår ett nytt tillvägagångssätt baserat på ett s.k. uppvaktande grafnätverk (GAT), där varje protonkollision representeras som en graf över partikelspåren och där GAT-egenskaperna tillämpas på spårnivå. Våra resultat visar att vi kan replikera de BDT-baserade resultaten och till och med förbättra resultaten på bekostnad av att öka osäkerheten i tidsbestämningarna. Vi drar slutsatsen att även om det finns potential för GAT-modeller att överträffa BDT-modeller, bör mer komplexa versioner av de förra tillämpas. Vi ger slutligen några förbättringsförslag som vi hoppas ska kunna inspirera till ytterligare studier och framsteg inom detta område, vilket visar lovande potential.

La congestion routière constitue un défi majeur pour les zones urbaines, car le volume de véhicules continue de croître plus rapidement que la capacité globale du réseau routier. Cette croissance a des répercussions sur l'activité économique, la durabilité environnementale et la qualité de vie. Bien que des stratégies visant à atténuer la congestion routière ont connu des améliorations au cours des dernières décennies, de nombreux pays ont encore du mal à la gérer efficacement.Divers modèles ont été développés pour aborder ce problème. Cependant, les approches existantes peinent souvent à fournir des prédictions en temps réel et localisées qui peuvent s'adapter à des conditions de trafic complexes et dynamiques. La plupart de ces approches s'appuient sur des horizons de prédiction fixes et manquent de l'infrastructure intelligente nécessaire à la flexibilité. Cette thèse comble ces lacunes en proposant une approche intelligente, décentralisée et basée sur l'infrastructure pour l'estimation et la prédiction de la congestion routière.Nous commençons par étudier l'Estimation du Trafic. Nous examinons les mesures de congestion possibles et les sources de données requises pour différents contextes pouvant être étudiés. Nous établissons une relation tridimensionnelle entre ces axes. Un système de recommandation basé sur des règles est développé pour aider les chercheurs et les opérateurs du trafic à choisir les mesures de congestion les plus appropriées en fonction du contexte étudié.Nous passons ensuite à la Prédiction du Trafic, où nous introduisons notre approche DECOTRIVMS. Cette dernière utilise des panneaux intelligents à messages variables pour collecter des données detrafic en temps réel et fournir des prédictions à court terme avec des horizons de prédiction variables.Nous avons utilisé des Réseaux de Graphes avec Attention en raison de leur capacité à capturer des relations complexes et à gérer des données structurées en graphes. Ils sont bien adaptés pour modéliser les interactions entre différents segments routiers étudiés.Nous avons aussi employé des méthodes d'apprentissage en ligne, spécifiquement la Descente de Gradient Stochastique et la Descente de Gradient Adaptative. Bien que ces méthodes ont été utilisées avec succès dans divers autres domaines, leur application à la prédiction de la congestion routière reste sous-explorée. Dans notre thèse, nous visons à combler cette lacune en explorant leur efficacité dans le contexte de la prédiction de la congestion routière en temps réel.Enfin, nous validons l'efficacité de notre approche à travers deux études de cas réalisées à Mascate, Oman, et à Rouen, France. Une analyse comparative est effectuée, évaluant divers modèles de prédiction, y compris les Réseaux de Graphes avec Attention, les Réseaux de Graphes Convolutionnels et des méthodes d'apprentissage en ligne. Les résultats obtenus soulignent le potentiel de DECOTRIVMS, démontrant son efficacité pour une prédiction précise et efficace de la congestion routière dans divers contextes urbains
Traffic congestion presents a critical challenge to urban areas, as the volume of vehicles continues to grow faster than the system’s overall capacity. This growth impacts economic activity, environmental sustainability, and overall quality of life. Although strategies for mitigating traffic congestion have seen improvements over the past few decades, many cities still struggle to manage it effectively. While various models have been developed to tackle this issue, existing approaches often fall short in providing real-time, localized predictions that can adapt to complex and dynamic traffic conditions. Most rely on fixed prediction horizons and lack the intelligent infrastructure needed for flexibility. This thesis addresses these gaps by proposing an intelligent, decentralized, infrastructure-based approach for traffic congestion estimation and prediction.We start by studying Traffic Estimation. We examine the possible congestion measures and data sources required for different contexts that may be studied. We establish a three-dimensional relationship between these axes. A rule-based system is developed to assist researchers and traffic operators in recommending the most appropriate congestion measures based on the specific context under study. We then proceed to Traffic Prediction, introducing our DECentralized COngestion esTimation and pRediction model using Intelligent Variable Message Signs (DECOTRIVMS). This infrastructure-based model employs intelligent Variable Message Signs (VMSs) to collect real-time traffic data and provide short-term congestion predictions with variable prediction horizons.We use Graph Attention Networks (GATs) due to their ability to capture complex relationships and handle graph-structured data. They are well-suited for modeling interactions between different road segments. In addition to GATs, we employ online learning methods, specifically, Stochastic Gradient Descent (SGD) and ADAptive GRAdient Descent (ADAGRAD). While these methods have been successfully used in various other domains, their application in traffic congestion prediction remains under-explored. In our thesis, we aim to bridge that gap by exploring their effectiveness within the context of real-time traffic congestion forecasting.Finally, we validate our model’s effectiveness through two case studies conducted in Muscat, Oman, and Rouen, France. A comprehensive comparative analysis is performed, evaluating various prediction techniques, including GATs, Graph Convolutional Networks (GCNs), SGD and ADAGRAD. The achieved results underscore the potential of DECOTRIVMS, demonstrating its potential for accurate and effective traffic congestion prediction across diverse urban contexts

In this work I present several studies, which might appear rather heterogeneous for both experimental questions and methodological approaches, and yet are linked by a common leitmotiv: spatial attention. I will address issues related to the assessment of attentional asymmetries, in the healthy individual as in patients with neurological disorders, their role in various aspects of human cognition, and their neural underpinning, driven by the deep belief that spatial attention plays an important role in various mental processes that are not necessarily confined to perception. What follows is organized into two distinct sections. In the first I will focus on the evaluation of visuospatial asymmetries, starting from the description of a new paradigm particularly suitable for this purpose. In the first chapter I will describe the effects of multitasking in a spatial monitoring test; the main result shows a striking decreasing in detection performance as a function of the introduced memory load. In the second chapter I will apply the same paradigm to a clinical population characterized by a brain lesion affecting the left hemisphere. Despite a standard neuropsychological battery failed to highlight any lateralized attentional deficit, I will show that exploiting concurrent demands might lead to enhanced sensitivity of diagnostic tests and consequently positive effects on patients’ diagnostic and therapeutic management. Finally, in the third chapter I will suggest, in light of preliminary data, that attentional asymmetries also occur along the sagittal axis; I will argue, in particular, that more attentional resources appear to be allocated around peripersonal space, the resulting benefits extending to various tasks (i.e., discrimination tasks). Then, in the second section, I will follow a complementary approach: I will seek to induce attentional shifts in order to evaluate their role in different cognitive tasks. In the fourth and fifth chapters this will be pursued exploiting sensory stimulations: visual optokinetic stimulation and galvanic vestibular stimulation, respectively. In the fourth chapter I will show that spatial attention is highly involved in numerical cognition, this relationship being bidirectional. Specifically, I will show that optokinetic stimulation modulates the occurrence of procedural errors during mental arithmetics, and that calculation itself affects oculomotor behaviour in turn. In the fifth chapter I will examine the effects of galvanic vestibular stimulation, a particularly promising technique for the rehabilitation of lateralized attention disorders, on spatial representations. I will discuss critically a recent account for unilateral spatial neglect, suggesting that vestibular stimulations or disorders might indeed affect the metric representation of space, but not necessarily resulting in spatial unawareness. Finally, in the sixth chapter I will describe an attentional capture phenomenon by intrinsically rewarding distracters. I will seek, in particular, to predict the degree of attentional capture from resting-state functional magnetic resonance imaging data and the related brain connectivity pattern; I will report preliminary data focused on the importance of the cingulate-opercular network, and discuss the results through a parallel with clinical populations characterized by behavioural addictions.
In questo lavoro presenterò una serie di ricerche che possono sembrare piuttosto eterogenee per quesiti sperimentali e approcci metodologici, ma sono tuttavia legate da un filo conduttore comune: i costrutti di ragionamento e attenzione spaziale. Affronterò in particolare aspetti legati alla valutazione delle asimmetrie attenzionali, nell'individuo sano come nel paziente con disturbi neurologici, il loro ruolo in vari aspetti della cognizione umana, e i loro substrati neurali, guidato dalla convinzione che l’attenzione spaziale giochi un ruolo importante in svariati processi mentali non necessariamente limitati alla percezione. Quanto segue è stato dunque organizzato in due sezioni distinte. Nella prima mi soffermerò sulla valutazione delle asimmetrie visuospaziali, iniziando dalla descrizione di un nuovo paradigma particolarmente adatto a questo scopo. Nel primo capitolo descriverò gli effetti del doppio compito e del carico attenzionale su un test di monitoraggio spaziale; il risultato principale mostra un netto peggioramento nella prestazione al compito di detezione spaziale in funzione del carico di memoria introdotto. Nel secondo capitolo applicherò lo stesso paradigma ad una popolazione clinica contraddistinta da lesione cerebrale dell’emisfero sinistro. Nonostante una valutazione neuropsicologica standard non evidenziasse alcun deficit lateralizzato dell’attenzione, mostrerò che sfruttare un compito accessorio può portare ad una spiccata maggiore sensibilità dei test diagnostici, con evidenti ricadute benefiche sull'iter clinico e terapeutico dei pazienti. Infine, nel terzo capitolo suggerirò, tramite dati preliminari, che asimmetrie attenzionali possono essere individuate, nell'individuo sano, anche lungo l’asse sagittale; argomenterò, in particolare, che attorno allo spazio peripersonale sembrano essere generalmente concentrate più risorse attentive, e che i benefici conseguenti si estendono a compiti di varia natura (ad esempio compiti di discriminazione). Passerò dunque alla seconda sezione, in cui, seguendo una logica inversa, indurrò degli spostamenti nel focus attentivo in modo da valutarne il ruolo in compiti di varia natura. Nei capitoli quarto e quinto sfrutterò delle stimolazioni sensoriali: la stimolazione visiva optocinetica e la stimolazione galvanico vestibolare, rispettivamente. Nel quarto capitolo mostrerò che l’attenzione spaziale è coinvolta nella cognizione numerica, con cui intrattiene rapporti bidirezionali. Nello specifico mostrerò da un lato che la stimolazione optocinetica può modulare l’occorrenza di errori procedurali nel calcolo mentale, dall'altro che il calcolo stesso ha degli effetti sull'attenzione spaziale e in particolare sul comportamento oculomotorio. Nel quinto capitolo esaminerò gli effetti della stimolazione galvanica vestibolare, una tecnica particolarmente promettente per la riabilitazione dei disturbi attentivi lateralizzati, sulle rappresentazioni mentali dello spazio. Discuterò in modo critico un recente modello della negligenza spaziale unilaterale, suggerendo che stimolazioni e disturbi vestibolari possano sì avere ripercussioni sulle rappresentazioni metriche dello spazio, ma senza comportare necessariamente inattenzione per lo spazio stesso. Infine, nel sesto capitolo descriverò gli effetti di cattura dell’attenzione visuospaziale che stimoli distrattori intrinsecamente motivanti possono esercitare nell'adulto sano. Cercherò, in particolare, di predire l’entità di questa cattura attenzionale partendo da immagini di risonanza magnetica funzionale a riposo: riporterò dati preliminari focalizzati sull'importanza del circuito cingolo-opercolare, effettuando un parallelismo con popolazioni cliniche caratterizzate da comportamenti di dipendenza.

Cette thèse est réalisée dans le cadre du projet BPI DeepTech, en collaboration avec la société Fair&Smart, veillant principalement à la protection des données personnelles conformément au Règlement Général sur la Protection des Données (RGPD). Dans ce contexte, nous avons proposé un modèle neuronal profond pour l'extraction d'informations dans les documents administratifs semi-structurés (DSSs). En raison du manque de données d'entraînement publiques, nous avons proposé un générateur artificiel de DSSs qui peut générer plusieurs classes de documents avec une large variation de contenu et de mise en page. Les documents sont générés à l'aide de variables aléatoires permettant de gérer le contenu et la mise en page en respectant des contraintes visant à garantir leur proximité avec des documents réels. Des métriques ont été introduites pour évaluer la diversité des DSSs générés en termes de contenu et de mise en page. Les résultats de l'évaluation ont montré que les jeux de données générés pour trois types de DSSs (fiches de paie, tickets de caisse et factures) présentent un degré élevé de diversité, ce qui permet d'éviter le sur-apprentissage lors de l'entraînement des systèmes d'extraction d'informations. En s'appuyant sur le format spécifique des DSSs, constitué de paires de mots (mots-clés, informations) situés dans des voisinages proches spatialement, le document est modélisé sous forme de graphe où les nœuds représentent les mots et les arcs, les relations de voisinage. Le graphe est incorporé dans un réseau d'attention à graphe (GAT) multi-couches (Multi-GAT). Celui-ci applique le mécanisme d'attention multi-têtes permettant d'apprendre l'importance des voisins de chaque mot pour mieux le classer. Une première version de ce modèle a été utilisée en mode supervisé et a obtenu un score F1 de 96 % sur deux jeux de données de factures et de fiches de paie générées, et de 89 % sur un ensemble de tickets de caisse réels (SROIE). Nous avons ensuite enrichi le Multi-GAT avec un plongement multimodal de l'information au niveau des mots (avec des composantes textuelle, visuelle et positionnelle), et l'avons associé à un auto-encodeur variationnel à graphe (VGAE). Ce modèle fonctionne en mode semi-supervisé, capable d'apprendre à partir des données annotées et non annotées simultanément. Pour optimiser au mieux la classification des nœuds du graphe, nous avons proposé un semi-VGAE dont l'encodeur partage ses premières couches avec le classifieur Multi-GAT. Cette optimisation est encore renforcée par la proposition d'une fonction de perte VGAE gérée par la perte de classification. En utilisant une petite base de données non annotées, nous avons pu améliorer de plus de 3 % le score F1 obtenu sur un ensemble de factures générées. Destiné à fonctionner dans un environnement protégé, nous avons adapté l'architecture du modèle pour son chiffrement homomorphe. Nous avons étudié une méthode de réduction de la dimensionnalité du modèle Multi-GAT. Ensuite, nous avons proposé une approche d'approximation polynomiale des fonctions non-linéaires dans le modèle. Pour réduire la dimension du modèle, nous avons proposé une méthode de fusion de caractéristiques multimodales qui nécessite peu de paramètres supplémentaires et qui réduit les dimensions du modèle tout en améliorant ses performances. Pour l'adaptation au chiffrement, nous avons étudié des approximations polynomiales de degrés faibles aux fonctions non-linéaires avec une utilisation des techniques de distillation de connaissance et de fine tuning pour mieux adapter le modèle aux nouvelles approximations. Nous avons pu minimiser la perte lors de l'approximation d'environ 3 % pour deux jeux de données de factures ainsi qu'un jeu de données de fiches de paie et de 5 % pour SROIE
This thesis is being carried out as part of the BPI DeepTech project, in collaboration with the company Fair&Smart, primarily looking after the protection of personal data in accordance with the General Data Protection Regulation (RGPD). In this context, we have proposed a deep neural model for extracting information in semi-structured administrative documents (SSDs). Due to the lack of public training datasets, we have proposed an artificial generator of SSDs that can generate several classes of documents with a wide variation in content and layout. Documents are generated using random variables to manage content and layout, while respecting constraints aimed at ensuring their similarity to real documents. Metrics were introduced to evaluate the content and layout diversity of the generated SSDs. The results of the evaluation have shown that the generated datasets for three SSD types (payslips, receipts and invoices) present a high diversity level, thus avoiding overfitting when training the information extraction systems. Based on the specific format of SSDs, consisting specifically of word pairs (keywords-information) located in spatially close neighborhoods, the document is modeled as a graph where nodes represent words and edges, neighborhood connections. The graph is fed into a multi-layer graph attention network (Multi-GAT). The latter applies the multi-head attention mechanism to learn the importance of each word's neighbors in order to better classify it. A first version of this model was used in supervised mode and obtained an F1 score of 96% on two generated invoice and payslip datasets, and 89% on a real receipt dataset (SROIE). We then enriched the multi-GAT with multimodal embedding of word-level information (textual, visual and positional), and combined it with a variational graph auto-encoder (VGAE). This model operates in semi-supervised mode, being able to learn on both labeled and unlabeled data simultaneously. To further optimize the graph node classification, we have proposed a semi-VGAE whose encoder shares its first layers with the multi-GAT classifier. This is also reinforced by the proposal of a VGAE loss function managed by the classification loss. Using a small unlabeled dataset, we were able to improve the F1 score obtained on a generated invoice dataset by over 3%. Intended to operate in a protected environment, we have adapted the architecture of the model to suit its homomorphic encryption. We studied a method of dimensionality reduction of the Multi-GAT model. We then proposed a polynomial approximation approach for the non-linear functions in the model. To reduce the dimensionality of the model, we proposed a multimodal feature fusion method that requires few additional parameters and reduces the dimensions of the model while improving its performance. For the encryption adaptation, we studied low-degree polynomial approximations of nonlinear functions, using knowledge distillation and fine-tuning techniques to better adapt the model to the new approximations. We were able to minimize the approximation loss by around 3% on two invoice datasets as well as one payslip dataset and by 5% on SROIE

Personalized recommendation systems use the data generated by user-item interactions (for example, in the form of ratings) to predict different users interests in available items and recommend a set of items or products to the users. The sparsity of data, cold start, and scalability are some of the important challenges faced by the developers of recommendation systems. These problems are alleviated by using external information, which can be in the form of a social network or a heterogeneous information network, or cross-domain knowledge. This thesis develops novel neural network models for designing personalized recommendation systems using the available external information. The first part of the thesis studies the top-N item recommendation setting where the external information is available in the form of a social network or heterogeneous information network. Unlike a simple recommendation setting, capturing complex relationships amongst entities (users, items, and connected objects) becomes essential when a social and heterogeneous information network is available. In a social network, all socially connected users do not have equal influence on each other. Further, estimating the quantum of influence among entities in a user-item interaction network is important when only implicit ratings are available. We address these challenges by proposing a novel neural network model, SoRecGAT, which employs a multi-head and multi-layer graph attention mechanism. The attention mechanism helps the model learn the influence of entities on each other more accurately. Further, we exploit heterogeneous information networks (HIN) to gather multiple views for the items. A novel neural network model -- GAMMA (Graph and Multi-view Memory Attention mechanism) is proposed to extract relevant information from HINs. The proposed model is an end-to-end model which eliminates the need for learning a similarity matrix offline using some manually selected meta-paths before optimizing the desired objective function. In the second part of the thesis, we focus on top-N bundle recommendation and list continuation setting. Bundle recommendation is the task of recommending a group of products instead of individual products to users. We study two interesting challenges -- (1) how to personalize and recommend existing bundles to users and (2) how to generate personalized novel bundles targeting specific users. We propose GRAM-SMOT -- a graph attention-based framework that considers higher-order relationships among the users, items, and bundles and the relative influence of items present in the bundles. For efficiently learning the embeddings of the entities, we define a loss function based on the metric-learning approach. A strategy that leverages submodular optimization ideas is used to generate novel bundles. We also study the problem of top-N personalized list continuation where the task is to curate the next items to user-generated lists (ordered sequence of items) in a personalized way by using the sequential information of the items in the list. The main challenge in this task is understanding the ternary relationships among the users, items, and lists. We propose HyperTeNet -- a self-attention hypergraph and Transformer-based neural network architecture for the personalized list continuation task. Here, graph convolutions are used to learn the multi-hop relationship among entities of the same type. A self-attention-based hypergraph neural network is proposed to learn the ternary relationships among the interacting entities via hyperlink prediction in a 3-uniform hypergraph. Further, the entity embeddings are shared with a Transformer-based architecture and are learned through an alternating optimization procedure. The final part of the thesis focuses on the personalized rating prediction setting where external information is available in the form of cross-domain knowledge. We propose an end-to-end neural network model, NeuCDCF, that provides a way to alleviate data sparsity problems by exploiting the information from related domains. NeuCDCF is based on a wide and deep framework and learns the representations jointly using matrix factorization and deep neural networks. We study the challenges involved in handling diversity between domains and learning complex non-linear relationships among entities within and across domains. We conduct extensive experiments in each of these settings using several real-world datasets and demonstrate the efficacy of the proposed models.

Cette thèse par article comprend quatre articles qui contribuent au domaine de l'apprentissage profond, en particulier à l'accélération de l’apprentissage par le biais de réseaux à faible précision et à l'application de réseaux de neurones profonds au traitement du langage naturel. Dans le premier article, nous étudions un schéma d’entraînement de réseau de neurones qui élimine la plupart des multiplications en virgule flottante. Cette approche consiste à binariser ou à ternariser les poids dans la propagation en avant et à quantifier les états cachés dans la propagation arrière, ce qui convertit les multiplications en changements de signe et en décalages binaires. Les résultats expérimentaux sur des jeux de données de petite à moyenne taille montrent que cette approche produit des performances encore meilleures que l’approche standard de descente de gradient stochastique, ouvrant la voie à un entraînement des réseaux de neurones rapide et efficace au niveau du matériel. Dans le deuxième article, nous avons proposé un mécanisme structuré d’auto-attention d’enchâssement de phrases qui extrait des représentations interprétables de phrases sous forme matricielle. Nous démontrons des améliorations dans 3 tâches différentes: le profilage de l'auteur, la classification des sentiments et l'implication textuelle. Les résultats expérimentaux montrent que notre modèle génère un gain en performance significatif par rapport aux autres méthodes d’enchâssement de phrases dans les 3 tâches. Dans le troisième article, nous proposons un modèle hiérarchique avec graphe de calcul dynamique, pour les données séquentielles, qui apprend à construire un arbre lors de la lecture de la séquence. Le modèle apprend à créer des connexions de saut adaptatives, ce qui facilitent l'apprentissage des dépendances à long terme en construisant des cellules récurrentes de manière récursive. L’entraînement du réseau peut être fait soit par entraînement supervisée en donnant des structures d’arbres dorés, soit par apprentissage par renforcement. Nous proposons des expériences préliminaires dans 3 tâches différentes: une nouvelle tâche d'évaluation de l'expression mathématique (MEE), une tâche bien connue de la logique propositionnelle et des tâches de modélisation du langage. Les résultats expérimentaux montrent le potentiel de l'approche proposée. Dans le quatrième article, nous proposons une nouvelle méthode d’analyse par circonscription utilisant les réseaux de neurones. Le modèle prédit la structure de l'arbre d'analyse en prédisant un scalaire à valeur réelle, soit la distance syntaxique, pour chaque position de division dans la phrase d'entrée. L'ordre des valeurs relatives de ces distances syntaxiques détermine ensuite la structure de l'arbre d'analyse en spécifiant l'ordre dans lequel les points de division seront sélectionnés, en partitionnant l'entrée de manière récursive et descendante. L’approche proposée obtient une performance compétitive sur le jeu de données Penn Treebank et réalise l’état de l’art sur le jeu de données Chinese Treebank.
This thesis by article consists of four articles which contribute to the field of deep learning, specifically in the acceleration of training through low-precision networks, and the application of deep neural networks on natural language processing. In the first article, we investigate a neural network training scheme that eliminates most of the floating-point multiplications. This approach consists of binarizing or ternarizing the weights in the forward propagation and quantizing the hidden states in the backward propagation, which converts multiplications to sign changes and binary shifts. Experimental results on datasets from small to medium size show that this approach result in even better performance than standard stochastic gradient descent training, paving the way to fast, hardware-friendly training of neural networks. In the second article, we proposed a structured self-attentive sentence embedding that extracts interpretable sentence representations in matrix form. We demonstrate improvements on 3 different tasks: author profiling, sentiment classification and textual entailment. Experimental results show that our model yields a significant performance gain compared to other sentence embedding methods in all of the 3 tasks. In the third article, we propose a hierarchical model with dynamical computation graph for sequential data that learns to construct a tree while reading the sequence. The model learns to create adaptive skip-connections that ease the learning of long-term dependencies through constructing recurrent cells in a recursive manner. The training of the network can either be supervised training by giving golden tree structures, or through reinforcement learning. We provide preliminary experiments in 3 different tasks: a novel Math Expression Evaluation (MEE) task, a well-known propositional logic task, and language modelling tasks. Experimental results show the potential of the proposed approach. In the fourth article, we propose a novel constituency parsing method with neural networks. The model predicts the parse tree structure by predicting a real valued scalar, named syntactic distance, for each split position in the input sentence. The order of the relative values of these syntactic distances then determine the parse tree structure by specifying the order in which the split points will be selected, recursively partitioning the input, in a top-down fashion. Our proposed approach was demonstrated with competitive performance on Penn Treebank dataset, and the state-of-the-art performance on Chinese Treebank dataset.

碩士
國立交通大學
資訊管理研究所
108
With the rise of financial technology (FinTech) in recent years, the financial industry seek to make their services more efficient through technology, one of the important topic in Fintech is how to conduct analysis in big data and establish prediction model based on artificial intelligence. In this case, we hope to find out the rules and implicit correlation between these data through algorithms, and forecast the situation of future market. In fact, we believe that there is a complex correlation pattern between different financial commodities. The changes in commodities may lead to a chain reaction of financial market through the complex network, and we may be able to build a relational model, which can be represented by a graph structure from these commodities and take a glance on the real situation of market. We may be able to learn the correlation with the help of deep neural network. So we will focus on the research of graph neural network and apply it to the financial domain. In this work, we proposes a deep learning model based on graph structure and attention mechanism, which is applied to the study of interaction relationship of financial time-series data. Traditional deep learning model perform well while input data are in the Euclidean space such as images and sequences. However, it is very easy to lose the structural information of graph if we learn the graph structure data with traditional deep learning module. Therefore, it is necessary to design a deep learning model specifically used for processing the graph structure. In this study, we expect to formulate various relationship between financial commodities as and learn the representation of graph through the graph neural networks. Moreover, we can highlight the importance of each commodity through the attention mechanism, and finally forecast the future trend of market with the help of our proposed model.

Cette thèse par article se compose de quatre articles qui contribuent au domaine de l’apprentissage profond, en particulier dans la compréhension et l’apprentissage des ap- proches neuronales des systèmes de dialogue. Le premier article fait un pas vers la compréhension si les architectures de dialogue neuronal couramment utilisées capturent efficacement les informations présentes dans l’historique des conversations. Grâce à une série d’expériences de perturbation sur des ensembles de données de dialogue populaires, nous constatons que les architectures de dialogue neuronal couramment utilisées comme les modèles seq2seq récurrents et basés sur des transformateurs sont rarement sensibles à la plupart des perturbations du contexte d’entrée telles que les énoncés manquants ou réorganisés, les mots mélangés, etc. Le deuxième article propose d’améliorer la qualité de génération de réponse dans les systèmes de dialogue de domaine ouvert en modélisant conjointement les énoncés avec les attributs de dialogue de chaque énoncé. Les attributs de dialogue d’un énoncé se réfèrent à des caractéristiques ou des aspects discrets associés à un énoncé comme les actes de dialogue, le sentiment, l’émotion, l’identité du locuteur, la personnalité du locuteur, etc. Le troisième article présente un moyen simple et économique de collecter des ensembles de données à grande échelle pour modéliser des systèmes de dialogue orientés tâche. Cette approche évite l’exigence d’un schéma d’annotation d’arguments complexes. La version initiale de l’ensemble de données comprend 13 215 dialogues basés sur des tâches comprenant six domaines et environ 8 000 entités nommées uniques, presque 8 fois plus que l’ensemble de données MultiWOZ populaire.
This thesis by article consists of four articles which contribute to the field of deep learning, specifically in understanding and learning neural approaches to dialog systems. The first article takes a step towards understanding if commonly used neural dialog architectures effectively capture the information present in the conversation history. Through a series of perturbation experiments on popular dialog datasets, wefindthatcommonly used neural dialog architectures like recurrent and transformer-based seq2seq models are rarely sensitive to most input context perturbations such as missing or reordering utterances, shuffling words, etc. The second article introduces a simple and cost-effective way to collect large scale datasets for modeling task-oriented dialog systems. This approach avoids the requirement of a com-plex argument annotation schema. The initial release of the dataset includes 13,215 task-based dialogs comprising six domains and around 8k unique named entities, almost 8 times more than the popular MultiWOZ dataset. The third article proposes to improve response generation quality in open domain dialog systems by jointly modeling the utterances with the dialog attributes of each utterance. Dialog attributes of an utterance refer to discrete features or aspects associated with an utterance like dialog-acts, sentiment, emotion, speaker identity, speaker personality, etc. The final article introduces an embedding-free method to compute word representations on-the-fly. This approach significantly reduces the memory footprint which facilitates de-ployment in on-device (memory constraints) devices. Apart from being independent of the vocabulary size, we find this approach to be inherently resilient to common misspellings.