Academic literature on the topic 'DEEP LEARNING MODEL'

Scene recognition is a hot research topic in the field of image recognition. It is necessary that we focus on the research on scene recognition, because it is helpful to the scene understanding topic, and can provide important contextual information for object recognition. The traditional approaches for scene recognition still have a lot of shortcomings. In these years, the deep learning method, which uses convolutional neural network, has got state-of-the-art results in this area. This thesis constructs a model based on multi-layer feature extraction of CNN and transfer learning for scene recognition tasks. Because scene images often contain multiple objects, there may be more useful local semantic information in the convolutional layers of the network, which may be lost in the full connected layers. Therefore, this paper improved the traditional architecture of CNN, adopted the existing improvement which enhanced the convolution layer information, and extracted it using Fisher Vector. Then this thesis introduced the idea of transfer learning, and tried to introduce the knowledge of two different fields, which are scene and object. We combined the output of these two networks to achieve better results. Finally, this thesis implemented the method using Python and PyTorch. This thesis applied the method to two famous scene datasets. the UIUC-Sports and Scene-15 datasets. Compared with traditional CNN AlexNet architecture, we improve the result from 81% to 93% in UIUC-Sports, and from 79% to 91% in Scene- 15. It shows that our method has good performance on scene recognition tasks.

fMRI imaging is considered key on the understanding of the brain and the mind, for this reason has been the subject of tremendous research connecting different disciplines. The intrinsic complexity of this 4-D type of data processing and analysis has been approached with every single computational perspective, lately increasing the trend to include artificial intelligence. One step critical on the fMRI pipeline is image registration. A model of Deep Networks based on Fully Convolutional Neural Networks, spatial transformation neural networks with a self-learning strategy was proposed for the implementation of a Fully deformable model image registration algorithm. Publicly available fMRI datasets with images from real-life subjects were used for training, testing and validating the model. The model performance was measured in comparison with ANTs deformable registration method with good results suggesting that Deep Learning can be used successfully for the development of the field using the basic strategy of studying the brain using the brain-self strategies.

Negli ultimi anni, le tecniche di Deep Learning sono state notevolmente migliorate, permettendo di affrontare con successo numerosi problemi. Il Deep Learning ha un approccio sub-simbolico ai problemi, perciò non si rende necessario descrivere esplicitamente informazioni sulla struttura del problema per fare sì che questo possa essere affrontato con successo; l'idea è quindi di utilizzare reti neurali di Deep Learning per affrontare problemi con vincoli (CSPs), senza dover fare affidamento su conoscenza esplicita riguardo ai vincoli dei problemi. Chiamiamo questo approccio Model Agnostic; esso può rivelarsi molto utile se usato sui CSP, dal momento che è spesso difficile esprimerne tutti i dettagli: potrebbero esistere vincoli, o preferenze, che non sono menzionati esplicitamente, e che sono intuibili solamente dall'analisi di soluzioni precedenti del problema. In questi casi, un modello di Deep Learning in grado di apprendere la struttura del CSP potrebbe avere applicazioni pratiche rilevanti. In particolar modo, in questa tesi si è indagato sul fatto che una Deep Neural Network possa essere capace di risolvere il rompicapo delle 8 regine. Sono state create due diverse reti neurali, una rete Generatore e una rete Discriminatore, che hanno dovuto apprendere differenti caratteristiche del problema. La rete Generatore è stata addestrata per produrre un singolo assegnamento, in modo che questo sia globalmente consistente; la rete Discriminatore è stata invece addestrata a distinguere tra soluzioni ammissibili e non ammissibili, con l'idea che possa essere utilizzata come controllore dell'euristica. Infine, sono state combinate le due reti in un unico modello, chiamato Generative Adversarial Network (GAN), in modo che esse possano scambiarsi conoscenza riguardo al problema, con l'obiettivo di migliorare le prestazioni di entrambe.

With the recent advances in deep learning, the gaze estimation models reached new levels, in terms of predictive accuracy, that could not be achieved with older techniques. Nevertheless, deep learning consists of computationally and memory expensive algorithms that do not allow their integration for embedded systems. This work aims to tackle this problem by boosting the predictive power of small networks using a model compression method called "distillation". Under the concept of distillation, we introduce an additional term to the compressed model’s total loss which is a bounding term between the compressed model (the student) and a powerful one (the teacher). We show that the distillation method introduces to the compressed model something more than noise. That is, the teacher’s inductive bias which helps the student to reach a better optimum due to the adaptive error deduction. Furthermore, we show that the MobileNet family exhibits unstable training phases and we report that the distilled MobileNet25 slightly outperformed the MobileNet50. Moreover, we try newly proposed training schemes to increase the predictive power of small and thin networks and we infer that extremely thin architectures are hard to train. Finally, we propose a new training scheme based on the hintlearning method and we show that this technique helps the thin MobileNets to gain stability and predictive power.<br>Den senaste utvecklingen inom djupinlärning har hjälp till att förbättra precisionen hos gaze estimation-modeller till nivåer som inte tidigare varit möjliga. Dock kräver djupinlärningsmetoder oftast både stora mängder beräkningar och minne som därmed begränsar dess användning i inbyggda system med små minnes- och beräkningsresurser. Det här arbetet syftar till att kringgå detta problem genom att öka prediktiv kraft i små nätverk som kan användas i inbyggda system, med hjälp av en modellkomprimeringsmetod som kallas distillation". Under begreppet destillation introducerar vi ytterligare en term till den komprimerade modellens totala optimeringsfunktion som är en avgränsande term mellan en komprimerad modell och en kraftfull modell. Vi visar att destillationsmetoden inför mer än bara brus i den komprimerade modellen. Det vill säga lärarens induktiva bias som hjälper studenten att nå ett bättre optimum tack vare adaptive error deduction. Utöver detta visar vi att MobileNet-familjen uppvisar instabila träningsfaser och vi rapporterar att den destillerade MobileNet25 överträffade sin lärare MobileNet50 något. Dessutom undersöker vi nyligen föreslagna träningsmetoder för att förbättra prediktionen hos små och tunna nätverk och vi konstaterar att extremt tunna arkitekturer är svåra att träna. Slutligen föreslår vi en ny träningsmetod baserad på hint-learning och visar att denna teknik hjälper de tunna MobileNets att stabiliseras under träning och ökar dess prediktiva effektivitet.

Trusted execution environments (TEEs) are an emerging technology that provides a protected hardware environment for processing and storing sensitive information. By using TEEs, developers can bolster the security of software systems. However, incorporating TEE into existing software systems can be a costly and labor-intensive endeavor. Software maintenance—changing software after its initial release—is known to contribute the majority of the cost in the software development lifecycle. The first step of making use of a TEE requires that developers accurately identify which pieces of code would benefit from being protected in a TEE. For large code bases, this identification process can be quite tedious and time-consuming. To help reduce the software maintenance costs associated with introducing a TEE into existing software, this thesis introduces ML-TEE, a recommendation tool that uses a deep learning model to classify whether an input function handles sensitive information or sensitive code. By applying ML-TEE, developers can reduce the burden of manual code inspection and analysis. ML-TEE's model was trained and tested on functions from GitHub repositories that use Intel SGX and on an imbalanced dataset. The accuracy of the final model used in the recommendation system has an accuracy of 98.86% and an F1 score of 80.00%. In addition, we conducted a pilot study, in which participants were asked to identify functions that needed to be placed inside a TEE in a third-party project. The study found that on average, participants who had access to the recommendation system's output had a 4% higher accuracy and completed the task 21% faster.<br>Master of Science<br>Improving the security of software systems has become critically important. A trusted execution environment (TEE) is an emerging technology that can help secure software that uses or stores confidential information. To make use of this technology, developers need to identify which pieces of code handle confidential information and should thus be placed in a TEE. However, this process is costly and laborious because it requires the developers to understand the code well enough to make the appropriate changes in order to incorporate a TEE. This process can become challenging for large software that contains millions of lines of code. To help reduce the cost incurred in the process of identifying which pieces of code should be placed within a TEE, this thesis presents ML-TEE, a recommendation system that uses a deep learning model to help reduce the number of lines of code a developer needs to inspect. Our results show that the recommendation system achieves high accuracy as well as a good balance between precision and recall. In addition, we conducted a pilot study and found that participants from the intervention group who used the output from the recommendation system managed to achieve a higher average accuracy and perform the assigned task faster than the participants in the control group.

Natural Language Processing is a field of Artificial Intelligence referring to the ability of computers to understand human speech and language, often in a written form, mainly by using Machine Learning and Deep Learning methods to extract patterns. Languages are challenging by definition, because of their differences, their abstractions and their ambiguities; consequently, their processing is often very demanding, in terms of modelling the problem and resources. Retrieving all sentences in a given text is something that can be easily accomplished with just few lines of code, but what about checking whether a given sentence conveys a message with sarcasm or not? This is something difficult for humans too and therefore, it requires complex modelling mechanisms to be addressed. This kind of information, in fact, poses the problem of its encoding and representation in a meaningful way. The majority of research involves finding and understanding all characteristics of text, in order to develop sophisticated models to address tasks such as Machine Translation, Text Summarization and Question Answering. This work will focus on ALBERT, from Google Research, which is one of the recently released state-of-the-art models and investigate its performance on the Question Answering task. In addition, some ideas will be developed and experimented in order to improve model's performance on the Stanford Question Answering Dataset (SQuAD), after exploring breakthrough changes that made training and fine-tuning of huge language models possible.

Many researches have been investigated to provide the computer aided diagnosis (CAD) automated tumor segmentation in various medical images, e.g., magnetic resonance (MR), computed tomography (CT) and positron-emission tomography (PET). The recent advances in automated tumor segmentation have been achieved by supervised deep learning (DL) methods trained on large labelled data to cover tumor variations. However, there is a scarcity in such training data due to the cost of labeling process. Thus, with insufficient training data, supervised DL methods have difficulty in generating effective feature representations for tumor segmentation. This thesis aims to develop an unsupervised DL method to exploit large unlabeled data generated during clinical process. Our assumption is unsupervised anomaly detection (UAD) that, normal data have constrained anatomy and variations, while anomalies, i.e., tumors, usually differ from the normality with high diversity. We demonstrate our method for automated tumor segmentation on two different image modalities. Firstly, given that bilateral symmetry in normal human brains and unsymmetry in brain tumors, we propose a symmetric-driven deep UAD model using GAN model to model the normal symmetric variations thus segmenting tumors by their being unsymmetrical. We evaluated our method on two benchmarked datasets. Our results show that our method outperformed the state-of-the-art unsupervised brain tumor segmentation methods and achieved competitive performance to the supervised segmentation methods. Secondly, we propose a multi-modal deep UAD model for PET-CT tumor segmentation. We model a manifold of normal variations shared across normal CT and PET pairs; this manifold representing the normal pairing that can be used to segment the anomalies. We evaluated our method on two PET-CT datasets and the results show that we outperformed the state-of-the-art unsupervised methods, supervised methods and baseline fusion techniques.

Causality detection in text data is a challenging natural language processing task. This is a trivial task for human beings as they acquire vast background knowledge throughout their lifetime. For example, a human knows from their experience that heavy rain may cause flood or plane accidents may cause death. However, it is challenging to automatically detect such causal relationships in texts due to the availability of limited contextual information and the unstructured nature of texts. The task is even more challenging for social media short texts such as Tweets as often they are informal, short, and grammatically incorrect. Generating hand-crafted linguistic rules is an option but is not always effective to detect causal relationships in text because they are rigid and require grammatically correct sentences. Also, the rules are often domain-specific and not always portable to another domain. Therefore, supervised learning techniques are more appropriate in the above scenario. Traditional machine learning-based model also suffers from the high dimensional features of texts. This is why deep learning-based approaches are becoming increasingly popular for natural language processing tasks such as causality detection. However, deep learning models often require large datasets with high-quality features to perform well. Extracting deeply-learnable causal features and applying them to a carefully designed deep learning model is important. Also, preparing a large human-labeled training dataset is expensive and time-consuming. Even if a large training dataset is available, it is computationally expensive to train a deep learning model due to the complex structure of neural networks. We focus on addressing the following challenges: (i) extracting highquality causal features, (ii) designing an effective deep learning model to learn from the causal features, and (iii) reducing the dependency on large training datasets. Our main goals in this thesis are as follows: (i) we aim to study the different aspects of causality and causal discovery in text in depth. (ii) We aim to develop strategies to model causality in text, (iii) and finally, we aim to develop frameworks to design effective and efficient deep neural network structures to discover causality in texts.<br>Thesis (PhD Doctorate)<br>Doctor of Philosophy (PhD)<br>School of Info & Comm Tech<br>Science, Environment, Engineering and Technology<br>Full Text

Дисертація складається з 84 сторінок, 59 Цифри та 29 джерел у довідковому списку. Проблема: Оскільки світ стає більш безпечним, для забезпечення належної передачі даних між сторонами, що спілкуються, було використано більше протоколів шифрування. Класифікація мережі стала більше клопоту з використанням деяких прийомів, оскільки перевірка зашифрованого трафіку в деяких країнах може бути незаконною. Це заважає інженерам мережі мати можливість класифікувати трафік, щоб відрізняти зашифрований від незашифрованого трафіку. Мета роботи: Ця стаття спрямована на проблему, спричинену попередніми методами, використовуваними в шифрованій мережевій класифікації. Деякі з них обмежені розміром даних та обчислювальною потужністю. У даній роботі використовується рішення алгоритму глибокого навчання для вирішення цієї проблеми. Основні завдання дослідження: 1. Порівняйте попередні традиційні методи та порівняйте їх переваги та недоліки 2. Вивчити попередні супутні роботи у сучасній галузі досліджень. 3. Запропонуйте більш сучасний та ефективний метод та алгоритм для зашифрованої класифікації мережевого трафіку Об'єкт дослідження: Простий алгоритм штучної нейронної мережі для точної та надійної класифікації мережевого трафіку, що не залежить від розміру даних та обчислювальної потужності. Предмет дослідження: На основі даних, зібраних із приватного потоку трафіку у нашому власному інструменті моделювання мережі. За 4 допомогою запропонованого нами методу визначаємо відмінності корисних навантажень мережевого трафіку та класифікуємо мережевий трафік. Це допомогло відокремити або класифікувати зашифровані від незашифрованого трафіку. Методи дослідження: Експериментальний метод. Ми провели наш експеримент із моделюванням мережі та збиранням трафіку різних незашифрованих протоколів та зашифрованих протоколів. Використовуючи мову програмування python та бібліотеку Keras, ми розробили згорнуту нейронну мережу, яка змогла прийняти корисне навантаження зібраного трафіку, навчити модель та класифікувати трафік у нашому тестовому наборі з високою точністю без вимоги високої обчислювальної потужності.<br>This dissertation consists of 84 pages, 59 Figures and 29 sources in the reference list. Problem: As the world becomes more security conscious, more encryption protocols have been employed in ensuring suecure data transmission between communicating parties. Network classification has become more of a hassle with the use of some techniques as inspecting encrypted traffic can pose to be illegal in some countries. This has hindered network engineers to be able to classify traffic to differentiate encrypted from unencrypted traffic. Purpose of work: This paper aims at the problem caused by previous techniques used in encrypted network classification. Some of which are limited to data size and computational power. This paper employs the use of deep learning algorithm to solve this problem. The main tasks of the research: 1. Compare previous traditional techniques and compare their advantages and disadvantages 2. Study previous related works in the current field of research. 3. Propose a more modern and efficient method and algorithm for encrypted network traffic classification The object of research: Simple artificial neural network algorithm for accurate and reliable network traffic classification that is independent of data size and computational power. The subject of research: Based on data collected from private traffic flow in our own network simulation tool. We use our proposed method to identify the differences in network traffic payloads and classify network traffic. It helped to separate or classify encrypted from unencrypted traffic. 6 Research methods: Experimental method. We have carried out our experiment with network simulation and gathering traffic of different unencrypted protocols and encrypted protocols. Using python programming language and the Keras library we developed a convolutional neural network that was able to take in the payload of the traffic gathered, train the model and classify the traffic in our test set with high accuracy without the requirement of high computational power.

Chaos theory is a branch of modern mathematics concerning the non-linear dynamic systems that are highly sensitive to their initial states. It has extensive real-world applications, such as weather forecasting and stock market prediction. The Lorenz system, defined by three ordinary differential equations (ODEs), is one of the simplest and most popular chaotic models. Historically research has focused on understanding the Lorenz system's mathematical characteristics and dynamical evolution including the inherent chaotic features it possesses. In this thesis, we take a data-driven approach and propose the task of predicting future states of the chaotic system from limited observations. We explore two directions, answering two distinct fundamental questions of the system based on how informed we are about the underlying model. When we know the data is generated by the Lorenz System with unknown parameters, our task becomes parameter estimation (a white-box problem), or the ``inverse'' problem. When we know nothing about the underlying model (a black-box problem), our task becomes sequence prediction. We propose two algorithms for the white-box problem: Markov-Chain-Monte-Carlo (MCMC) and a Multi-Layer-Perceptron (MLP). Specially, we propose to use the Metropolis-Hastings (MH) algorithm with an additional random walk to avoid the sampler being trapped into local energy wells. The MH algorithm achieves moderate success in predicting the $\rho$ value from the data, but fails at the other two parameters. Our simple MLP model is able to attain high accuracy in terms of the $l_2$ distance between the prediction and ground truth for $\rho$ as well, but also fails to converge satisfactorily for the remaining parameters. We use a Recurrent Neural Network (RNN) to tackle the black-box problem. We implement and experiment with several RNN architectures including Elman RNN, LSTM, and GRU and demonstrate the relative strengths and weaknesses of each of these methods. Our results demonstrate the promising role of machine learning and modern statistical data science methods in the study of chaotic dynamic systems. The code for all of our experiments can be found on \url{https://github.com/Yajing-Zhao/}