Dissertations / Theses on the topic 'Neural networks architecture'

The development of models of dynamical systems behaviour is a fundamental activity in science and engineering disciplines. This thesis examines the problem of modelling a class of dynamical systems using neural networks. Existing research reveals that neural network models have been developed for lumped parameter dynamical systems; that is, systems where the variables of interest vary only over the timedomain. However, there are no adequate neural network models for distributed parameter dynamical systems; that is, systems where the variables of interest vary over some other domain, e.g. the spatial domain, in addition to the time domaln. The main goal of this research is to develop a neural network architecture for modelling distributed dynamical systems where one has limited and incomplete knowledge about the underlying behaviour of the system. The result of this research is a generic neural network architecture - the Interacting Neural Network (INN) architecture - that is capable of modelling a wide range of distributed dynamical systems. The fundamental problem associated with distributed systems which the INN architecture addresses is that of scaling. The scaling problem manifests itself when the complexity of a model increases in a manner which is unmanageable as the problem size increases. The INN architecture solves the scaling problem by using the philosophy of interacting subsystems which is a general methodology for managing complexity. The underlying principle of this methodology is to view the system as a combination of many small subsystems and to focus the modelling effort at the subsystem level rather than at the system level. The resulting models are relatively simpler, but when allowed to interact, the complex behaviour of the original system can be retrieved. The capabilities of the INN architecture are investigated by comparing its performance with other architectures on two distributed systems. First, investigations are carried out in modelling non-linear heat flow which serves as a case study to expound the capabilities of the INN architecture. Secondly, the architecture is applied to an aquifer problem to illustrate its capabilities on modelling practical problems. It is shown that the INN architecture captures the underlying behaviour of both systems, and more significantly, that the trained network can generalise spatially, wherein the same trained network can be applied to different instances of a given system. The spatial generalisation capabilities of the INN architecture is a unique and powerful result, which when used appropriately can significantly extend the usefulness of neural network models. Finally, two major factors that affect the generalisation ability of the INN architecture are investigated: (i) the effect of changing the geometry of a given system and (ii) the effect of the amount of training data available. New relationships are deduced for both factors.

Many biological experiments require a protein sequence to be translated to the nucleic acid sequence that codes for it or require an investigator to possess a means to “backtranslate” a protein to its amino acid sequence. However, the degenerate nature of the genetic code greatly frustrates this process through ambiguities in the wobble bases. One possible solution to this dilemma is to predict codon usage frequencies for a target organism through use of an Artificial Neural Network. Consequently, a Neural Network was trained on amino and nucleic acid sequences to determine the network’s capacity in accurate predictions for a twenty amino acid window. Moreover, 10 different network architectures were surveyed to ascertain which one yields optimum (least error) results when trained on the same nucleic acid sequences. The winning architecture was examined using two new training sets that have been partitioned into those with high bias and those with low bias for mRNA secondary structure. The more negative the bias, the more secondary structure it will have, whereas less negative bias will display less secondary structure. Testing of these two training sets revealed that the neural network was able to distinguish between the two sets; i.e., the training set with greater secondary structure learned the patterns in less training cycles and produced a lower error when compared to the training set with less secondary structure given the same network architecture. Ultimately, this work might be beneficial as a computation tool for backtranslation in degenerate PCR cloning and in identifying the unknown coding regions in genes.

A number of researchers have investigated the impact of network architecture on the performance of artificial neural networks. Particular attention has been paid to the impact on the performance of the multi-layer perceptron of architectural issues, and the use of various strategies to attain an optimal network structure. However, there are still perceived limitations with the multi-layer perceptron and networks that employ a different architecture to the multi-layer perceptron have gained in popularity in recent years, particularly, networks that implement a more localised solution, where the solution in one area of the problem space does not impact, or has a minimal impact, on other areas of the space. In this study, we discuss the major architectural issues affecting the performance of a multi-layer perceptron, before moving on to examine in detail the performance of a new localised network, namely the bumptree. The work presented here examines the impact on the performance of artificial neural networks of employing alternative networks to the long established multi-layer perceptron. In particular, networks that impose a solution where the impact of each parameter in the final network architecture has a localised impact on the problem space being modelled are examined. The alternatives examined are the radial basis function and bumptree neural networks, and the impact of architectural issues on the performance of these networks is examined. Particular attention is paid to the bumptree, with new techniques for both developing the bumptree structure and employing this structure to classify patterns being examined.

In this dissertation, a new artificial neural network (ANN) architecture called fuzzy adaptive recurrent counterpropagation neural network (FARCNN) is presented. FARCNNs can be directly synthesized from a set of training data, making system behavioral learning extremely fast. FARCNNs can be applied directly and effectively to model both static and dynamic system behavior based on observed input/output behavioral patterns alone without need of knowing anything about the internal structure of the system under study. The FARCNN architecture is derived from the methodology of fuzzy inductive reasoning and a basic form of counterpropagation neural networks (CNNs) for efficient implementation of finite state machines. Analog signals are converted to fuzzy signals by use of a new type of fuzzy A/D converter, thereby keeping the size of the Kohonen layer of the CNN manageably small. Fuzzy inferencing is accomplished by an application-independent feedforward network trained by means of backpropagation. Global feedback is used to represent full system dynamics. The FARCNN architecture combines the advantages of the quantitative approach (neural network) with that of the qualitative approach (fuzzy logic) as an efficient autonomous system modeling methodology. It also makes the simulation of mixed quantitative and qualitative models more feasible. In simulation experiments, we shall show that FARCNNs can be applied directly and easily to different types of systems, including static continuous nonlinear systems, discrete sequential systems, and as part of large dynamic continuous nonlinear control systems, embedding the FARCNN into much larger industry-sized quantitative models, even permitting a feedback structure to be placed around the FARCNN.

This thesis searches for the optimal neural architecture by minimizing a proxy of validation loss. Existing neural architecture search (NAS) methods used to discover the optimal neural architecture that best fits the validation examples given the up-to-date network weights. However, back propagation with a number of validation examples could be time consuming, especially when it needs to be repeated many times in NAS. Though these intermediate validation results are invaluable, they would be wasted if we cannot use them to predict the future from the past. In this thesis, we propose to approximate the validation loss landscape by learning a mapping from neural architectures to their corresponding validate losses. The optimal neural architecture thus can be easily identified as the minimum of this proxy validation loss landscape. A novel sampling strategy is further developed for an efficient approximation of the loss landscape. Theoretical analysis indicates that the validation loss estimator learned with our sampling strategy can reach a lower error rate and a lower label complexity compared with a uniform sampling. Experimental results on benchmarks demonstrate that the architecture searched by the proposed algorithm can achieve a satisfactory accuracy with less time cost.

Reverse-engineering the brain is one of the US National Academy of Engineering’s “Grand Challenges.” The structure of the brain can be examined at many different levels, spanning many disciplines from low-level biology through psychology and computer science. This thesis focusses on real-time computation of large neural networks using the Izhikevich spiking neuron model. Neural computation has been described as “embarrassingly parallel” as each neuron can be thought of as an independent system, with behaviour described by a mathematical model. However, the real challenge lies in modelling neural communication. While the connectivity of neurons has some parallels with that of electrical systems, its high fan-out results in massive data processing and communication requirements when modelling neural communication, particularly for real-time computations. It is shown that memory bandwidth is the most significant constraint to the scale of real-time neural computation, followed by communication bandwidth, which leads to a decision to implement a neural computation system on a platform based on a network of Field Programmable Gate Arrays (FPGAs), using commercial off- the-shelf components with some custom supporting infrastructure. This brings implementation challenges, particularly lack of on-chip memory, but also many advantages, particularly high-speed transceivers. An algorithm to model neural communication that makes efficient use of memory and communication resources is developed and then used to implement a neural computation system on the multi- FPGA platform. Finding suitable benchmark neural networks for a massively parallel neural computation system proves to be a challenge. A synthetic benchmark that has biologically-plausible fan-out, spike frequency and spike volume is proposed and used to evaluate the system. It is shown to be capable of computing the activity of a network of 256k Izhikevich spiking neurons with a fan-out of 1k in real-time using a network of 4 FPGA boards. This compares favourably with previous work, with the added advantage of scalability to larger neural networks using more FPGAs. It is concluded that communication must be considered as a first-class design constraint when implementing massively parallel neural computation systems.

Neural architecture search is a popular method for automating architecture design. Bayesian optimization is a widely used approach for hyper-parameter optimization and can estimate a function with limited samples. However, Bayesian optimization methods are not preferred for architecture search as it expects vector inputs while graphs are high dimensional data. This thesis presents a Bayesian approach with Gaussian priors that use graph kernels specifically targeted to work in the higherdimensional graph space. We implemented three different graph kernels and show that on the NAS-Bench-101 dataset, an untrained graph convolutional network kernel outperforms previous methods significantly in terms of the best network found and the number of samples required to find it. We follow the AutoML guidelines to make this work reproducible.<br>Neural arkitektur sökning är en populär metod för att automatisera arkitektur design. Bayesian-optimering är ett vanligt tillvägagångssätt för optimering av hyperparameter och kan uppskatta en funktion med begränsade prover. Bayesianska optimeringsmetoder är dock inte att föredra för arkitektonisk sökning eftersom vektoringångar förväntas medan grafer är högdimensionella data. Denna avhandling presenterar ett Bayesiansk tillvägagångssätt med gaussiska prior som använder grafkärnor som är särskilt fokuserade på att arbeta i det högre dimensionella grafutrymmet. Vi implementerade tre olika grafkärnor och visar att det på NASBench- 101-data, till och med en otränad Grafkonvolutionsnätverk-kärna, överträffar tidigare metoder när det gäller det bästa nätverket som hittats och antalet prover som krävs för att hitta det. Vi följer AutoML-riktlinjerna för att göra detta arbete reproducerbart.

As computers begin to pervade aspects of our everyday lives, so the problem of communication from man-to-machine becomes increasingly evident. In recent years, there has been a concerted interest in speech recognition offering a user to communicate freely with a machine. However, this deceptively simple means for exchanging information is in fact extremely complex. A single utterance can contain a wealth of varied information concerning the speaker's gender, age, dialect and mood. Numerous subtle differences such as intonation, rhythm and stress further add to the complexity, increasing the variability between inter- and intra-speaker utterances. These differences pose an enormous problem, especially for a multi-user system since it is impractical to train for every variation of every utterance from every speaker. Consequently adaptation is of great importance, allowing a system with limited knowledge to dynamically adapt towards a new speakers characteristics. A new modified artificial neural network (ANN) was proposed incorporating One-Class-OneNetwork (OCON) subnet architectures connected via a common front-end adaptation layer. Using vowel phonemes from the TIMIT speech database, the adaptation was concentrated on neurons within the front-end layer, resulting in only information common to all classes, primarily speaker characteristics, being adapted. In addition, this prevented new utterances from interfering with phoneme unique information in the corresponding OCON subnets. Hence a more efficient adaptation procedure was created which, after adaptation towards a single class, also aided in the recognition of the remaining classes within the network. Compared with a conventional multi-layer perceptron network, results for inter- and intraspeaker adaptation showed an equally marked improvement for the recognition of adapted phonemes during both full neuron and front-layer neuron adaptation within the new modified architecture. When testing the effects of adaptation on the remaining unadapted vowel phonemes, the modified architecture (allowing only the neurons in the front-end layer to adapt) yielded better results than the modified architecture allowing full neuron adaptation. These results highlighted the storing of speaker information, common to all classes, in the front-end layer allowing efficient inter- and intra-speaker dynamic adaptation.

В статті розглянуті основи роботи з нейронними мережами, особливу увагу приділено моделі мережі під назвою «перцептрон», запровадженої Френком Розенблаттом. До того ж було розкрито тему найпоширеніших мов програмування, що дозволяють втілити нейронні мережі у життя, шляхом створення програмного забезпечення, пов`язаного з ними.<br>The paper covers the basic principles of Neural Networks’ work. Special attention is paid to Frank Rosenblatt’s model of the network called “perceptron”. In addition, the article touches upon the main programming languages used to write software for Neural Networks.

Modern electronic systems, especially sensor and imaging systems, are beginning to incorporate their own neural network subsystems. In order for these neural systems to learn in real-time they must be implemented using VLSI technology, with as much of the learning processes incorporated on-chip as is possible. The majority of current VLSI implementations literally implement a series of neural processing cells, which can be connected together in an arbitrary fashion. Many do not perform the entire neural learning process on-chip, instead relying on other external systems to carry out part of the computation requirements of the algorithm. The work presented here utilises two dimensional instruction systolic arrays in an attempt to define a general neural architecture which is closer to the biological basis of neural networks - it is the synapses themselves, rather than the neurons, that have dedicated processing units. A unified architecture is described which can be programmed at the microcode level in order to facilitate the processing of multiple neural network types. An essential part of neural network processing is the neuron activation function, which can range from a sequential algorithm to a discrete mathematical expression. The architecture presented can easily carry out the sequential functions, and introduces a fast method of mathematical approximation for the more complex functions. This can be evaluated on-chip, thus implementing the entire neural process within a single system. VHDL circuit descriptions for the chip have been generated, and the systolic processing algorithms and associated microcode instruction set for three different neural paradigms have been designed. A software simulator of the architecture has been written, giving results for several common applications in the field.

Shunting inhibition is a powerful computational mechanism that plays an important role in sensory neural information processing systems. It has been extensively used to model some important visual and cognitive functions. It equips neurons with a gain control mechanism that allows them to operate as adaptive non-linear filters. Shunting Inhibitory Artificial Neural Networks (SIANNs) are biologically inspired networks where the basic synaptic computations are based on shunting inhibition. SIANNs were designed to solve difficult machine learning problems by exploiting the inherent non-linearity mediated by shunting inhibition. The aim was to develop powerful, trainable networks, with non-linear decision surfaces, for classification and non-linear regression tasks. This work enhances and extends the original SIANN architecture to a more general form called the Generalised Feedforward Neural Network (GFNN) architecture, which contains as subsets both SIANN and the conventional Multilayer Perceptron (MLP) architectures. The original SIANN structure has the number of shunting neurons in the hidden layers equal to the number of inputs, due to the neuron model that is used having a single direct excitatory input. This was found to be too restrictive, often resulting in inadequately small or inordinately large network structures.

Model building is a fundamental scientific tool. Increasingly there is interest in building neurally-implemented models of cognitive processes with the intention of modelling brains. However, simulation of such models can be prohibitively expensive in both the time and energy required. For example, Spaun - "the world's first functional brain model", comprising 2.5 million neurons - required 2.5 hours of computation for every second of simulation on a large compute cluster. SpiNNaker is a massively parallel, low power architecture specifically designed for the simulation of large neural models in biological real time. Ideally, SpiNNaker could be used to facilitate rapid simulation of models such as Spaun. However the Neural Engineering Framework (NEF), with which Spaun is built, maps poorly to the architecture - to the extent that models such as Spaun would consume vast portions of SpiNNaker machines and still not run as fast as biology. This thesis investigates whether real time simulation of Spaun on SpiNNaker is at all possible. Three techniques which facilitate such a simulation are presented. The first reduces the memory, compute and network loads consumed by the NEF. Consequently, it is demonstrated that only a twentieth of the cores are required to simulate a core component of the Spaun network than would otherwise have been needed. The second technique uses a small number of additional cores to significantly reduce the network traffic required to simulated this core component. As a result simulation in real time is shown to be feasible. The final technique is a novel logic minimisation algorithm which reduces the size of the routing tables which are used to direct information around the SpiNNaker machine. This last technique is necessary to allow the routing of models of the scale and complexity of Spaun. Together these provide the ability to simulate the Spaun model in biological real time - representing a speed-up of 9000 times over previously reported results - with room for much larger models on full-scale SpiNNaker machines.

Human infertility is considered a serious disease of the the reproductive system that affects more than 10% of couples worldwide,and more than 30% of reported cases are related to men. The crucial step in evaluating male in fertility is a semen analysis, highly dependent on sperm morphology. However,this analysis is done at the laboratory manually and depends mainly on the doctor’s experience. Besides,it is laborious, and there is also a high degree of interlaboratory variability in the results. This article proposes applying a specialized convolutional neural network architecture (U-Net),which focuses on the segmentation of sperm cells in micrographs to overcome these problems.The results showed high scores for the model segmentation metrics such as precisión (93%), IoU score (86%),and DICE score of 93%. Moreover,we can conclude that U-net architecture turned out to be a good option to carry out the segmentation of sperm cells.

The civil applications of Unmanned Aerial Vehicle (UAV) technology are constantly on a rise and the safety rules for the operation of UAVs in populated areas are being drafted. The UAV technology is an active area of academic research due to the challenges related to aerodynamics, tight power and payload budgets, multi-sensor information fusion, reactive real-time path planning, perception and communication bandwidth requirements. Autonomous navigation is a complex problem due to the challenges of algorithmic complexity and their real-time implementation. The challenges like long-term GPS errors/outage/jamming and exponential error growth in inertial sensors increase the complexity of autonomous navigation to an extent that high level of redundancy is mandatory in the design of navigation systems. Typical UAV systems use multi-sensor (GPS + INS +Vision) data fusion coupled with responsive sensors, innovative navigation algorithms, computationally capable onboard computers and reactive electromechanical systems to accomplish the navigational needs of safe operations in urban environments. Machine learning is a very promising technology and has broad applicability in the many real-life problems: ranging from hand-held & wearable computers to intelligent cars and homes. It can be efficiently used in autonomous navigation of UAVs. This work presents a novel absolute position estimation solution that leverages Radial Basis Function (RBF) classifier for robust aerial image registration. The proposed solution covers the entire spectrum of the problem involving algorithm design, hardware architecture and real-time hardware implementation. The system relies on single passive imaging source for acquisition of aerial images. The sensed image is geometrically transformed to bring it in a common view point as the reference satellite image. The orthorectified aerial image is then learned by the RBF network and full search is performed in the Region of Interest (ROI) of the reference satellite image. The real-time implementation of computationally intensive algorithm is accomplished by designing a customized wide data path in Field Programmable Gate Array (FPGA). The proposed architecture offers a reliable drift-free position estimation solution by conglomerating information from the inertial sensors and geo-registration of the aerial images over a geodetically aligned satellite reference image. We compare the robustness of our proposed matching algorithm with the standard normalized area correlation techniques and present limitations and False Acceptance Rates (FAR) of the two algorithms. This analysis has been performed on a set of real aerial and satellite imagery, acquired under different lightening and weather conditions. This is then followed by a discussion on real-time FPGA based architecture and power analysis. We conclude by presenting future directions of the work. Keywords: Inertial Measurement Units, Vision based Navigation, Real-time implementation, FPGA, Neural Network

Les réseaux de neurones ont connu d'impressionnants développements depuis l'émergence de l'apprentissage profond, vers 2012, et sont désormais l'état de l'art de toute une gamme de tâches automatisées, telles que le traitement automatique du langage naturel, la classification, la prédiction, etc. Néanmoins, dans un contexte où la recherche se focalise sur l'optimisation d'un unique indicateur de performance -- typiquement, le taux d'exactitude --, il apparaît que les performances tendent à croître de façon fiable, voire prévisible, en fonction de la taille du jeu de données d'entraînement, du nombre de paramètres, et de la quantité de calculs réalisés pendant l'entraînement. Les progrès réalisés sont-ils alors plus le fait des recherches menées dans le domaine des réseaux de neurones, ou celui de l'écosystème logiciel et matériel sur lequel il s'appuie ? Afin de répondre à cette question, nous avons créé une nouvelle figure de mérite illustrant les choix architecturaux faits entre capacités et complexité. Nous avons choisi pour estimer la complexité d'utiliser la consommation énergétique lors de l'inférience, de sorte à représenter l'adéquation entre l'algorithme et l'architecture. Nous avons établi une façon de mesurer cette consommation énergétique, confirmé sa pertinence, et établi un classement de réseaux de neurones de l'état de l'art selon cette méthodologie. Nous avons ensuite exploré comment différents paramètres d'exécution influencent notre score, et comment le rafiner en allant de l'avant, en insistant sur le besoin de "fonction objectif" adaptées au cas d'usage. Nous finissons en établissant diverses façons de poursuivre le travail entamé durant cette thèse<br>Neural networks have seen impressive developments since the emergence of deep learning, around 2012, and are now the state of the art for diverse tasks, such as natural language processing, classification, prediction, autonomous systems etc. However, as the research tends to focus around the optimization of a single performance metric -- typically accuracy --, it appears that performances tend to scale reliably and even predictably with the size of the training dataset, the neural network's complexity and the total amount of training compute. In this context, we ask how much of the recent progress in the field of neural networks can be attributed to progress made in compute, software support, and hardware optimization. To answer this question, we created a new figure of merit illustrating tradeoffs between the complexity and capability of a network. We used the measured energy consumption per inference as an estimator of complexity and a way of representing the adequation between the algorithm and the architecture. We established a way of measuring this energy consumption, verified its relevance, and benchmarked networks from the state of the art according to this methodology. We then explored how different execution parameters influence our score, and how to further refine it, insisting on the need for diverse objective functions reflecting different usecases in the field of neural networks. We end by acknowledging the social and environmental responsibility of the neural network field, and lay out the envisioned continuation of our work

The size and speed of computer networks continue to expand at a rapid pace, as do the corresponding errors, failures, and faults inherent within such extensive networks. This thesis introduces a novel approach to interface Border Gateway Protocol (BGP) computer networks with neural networks to learn the precursor connectivity patterns that emerge prior to a node failure. Details of the design and construction of a framework that utilizes neural networks to learn and monitor BGP connection states as a means of detecting and predicting BGP peer node failure are presented. Moreover, this framework is used to monitor a BGP network and a suite of tests are conducted to establish that this neural network approach as a viable strategy for predicting BGP peer node failure. For all performed experiments both of the proposed neural network architectures succeed in memorizing and utilizing the network connectivity patterns. Lastly, a discussion of this framework's generic design is presented to acknowledge how other types of networks and alternate machine learning techniques can be accommodated with relative ease.

Unmanned Aerial Vehicles (UAVs) are capable of placing sensors at unique vantage points without endangering a pilot. Therefore, they are well suited to perform target tracking missions. However, performing the mission can be burdensome for the operator. To track a target, the operator must estimate the position of the target from the incoming video stream, update the orientation of the camera, and move the vehicle to an appropriate vantage point. The purpose of the research in this thesis is to provide a target tracking system that performs these tasks automatically in real-time. The first task, which receives the majority of the attention, is estimating the position of the target within the incoming video stream. Because of the inherent clutter in the imagery, the resulting probability distributions are typically non-Gaussian and multi-modal. Therefore, classical state estimation techniques, such as the Kalman filter and its variants are unacceptable solutions. The particle filter has become a popular alternative since it is able to approximate the multi-modal distributions using a set of samples, and it is used as part of this research. To improve the performance of the filter and manage the inherently large computational burden a neural network is used to estimate the performance of the particle filter. The filter parameters are then changed in response to the performance. Once the position of the target is estimated in the frame, it is projected on the ground using the camera orientation and vehicle attitude and input into a linear predictor. The output of the predictor is used to update the orientation of the camera and vehicle waypoints. Through offline, simulation, and flight testing, the approach is shown to provide a powerful visual tracking system for use onboard the GTMax unmanned research helicopter.

Cette dernière décennie a donné lieu à la réémergence des méthodes d'apprentissage machine basées sur les réseaux de neurones formels sous le nom d'apprentissage profond. Bien que ces méthodes aient permis des avancées majeures dans le domaine de l'apprentissage machine, plusieurs obstacles à la possibilité d'industrialiser ces méthodes persistent, notamment la nécessité de collecter et d'étiqueter une très grande quantité de données ainsi que la puissance de calcul nécessaire pour effectuer l'apprentissage et l'inférence avec ce type de réseau neuronal. Dans cette thèse, nous proposons d'étudier l'adéquation entre des algorithmes d'inférence et d'apprentissage issus des réseaux de neurones biologiques pour des architectures matérielles massivement parallèles. Nous montrons avec trois contributions que de telles adéquations permettent d'accélérer drastiquement les temps de calculs inhérents au réseaux de neurones. Dans notre premier axe, nous réalisons l'étude d'adéquation du moteur BCVision de Brainchip SAS pour les plate-formes GPU. Nous proposons également l'introduction d'une architecture hiérarchique basée sur des cellules complexes. Nous montrons que l'adéquation pour GPU accélère les traitements par un facteur sept, tandis que l'architecture hiérarchique atteint un facteur mille. La deuxième contribution présente trois algorithmes de propagation de décharges neuronales adaptés aux architectures parallèles. Nous réalisons une étude complète des modèles computationels de ces algorithmes, permettant de sélectionner ou de concevoir un système matériel adapté aux paramètres du réseau souhaité. Dans notre troisième axe nous présentons une méthode pour appliquer la règle Spike-Timing-Dependent-Plasticity à des données images afin d'apprendre de manière non-supervisée des représentations visuelles. Nous montrons que notre approche permet l'apprentissage d'une hiérarchie de représentations pertinente pour des problématiques de classification d'images, tout en nécessitant dix fois moins de données que les autres approches de la littérature<br>The last decade has seen the re-emergence of machine learning methods based on formal neural networks under the name of deep learning. Although these methods have enabled a major breakthrough in machine learning, several obstacles to the possibility of industrializing these methods persist, notably the need to collect and label a very large amount of data as well as the computing power necessary to perform learning and inference with this type of neural network. In this thesis, we propose to study the adequacy between inference and learning algorithms derived from biological neural networks and massively parallel hardware architectures. We show with three contribution that such adequacy drastically accelerates computation times inherent to neural networks. In our first axis, we study the adequacy of the BCVision software engine developed by Brainchip SAS for GPU platforms. We also propose the introduction of a coarse-to-fine architecture based on complex cells. We show that GPU portage accelerates processing by a factor of seven, while the coarse-to-fine architecture reaches a factor of one thousand. The second contribution presents three algorithms for spike propagation adapted to parallel architectures. We study exhaustively the computational models of these algorithms, allowing the selection or design of the hardware system adapted to the parameters of the desired network. In our third axis we present a method to apply the Spike-Timing-Dependent-Plasticity rule to image data in order to learn visual representations in an unsupervised manner. We show that our approach allows the effective learning a hierarchy of representations relevant to image classification issues, while requiring ten times less data than other approaches in the literature

Neurocomputers offer a massively parallel computing paradigm by mimicking the human brain. Their efficient use in statistical information processing has been proposed to overcome critical bottlenecks with traditional computing schemes for applications such as image and speech processing, and associative memory. In neural networks information is generally represented by phase (e.g., oscillatory neural networks) or amplitude (e.g., cellular neural networks). Phase-based neurocomputing is constructed as a network of coupled oscillatory neurons that are connected via programmable phase elements. Representing each neuron circuit with one oscillatory device and implementing programmable phases among neighboring neurons, however, are not clearly feasible from circuits perspective if not impossible. In contrast to nascent oscillatory neurocomputing circuits, mature amplitude-based neural networks offer more efficient circuit solutions using simpler resistive networks where information is carried via voltage- and current-mode signals. Yet, such circuits have not been efficiently realized by CMOS alone due to the needs for an efficient summing mechanism for weighted neural signals and a digitally-controlled weighting element for representing couplings among artificial neurons. Large power consumption and high circuit complexity of such CMOS-based implementations have precluded adoption of amplitude-based neurocomputing circuits as well, and have led researchers to explore the use of emerging technologies for such circuits. Although they provide intriguing properties, previously proposed neurocomputing components based on emerging technologies have not offered a complete and practical solution to efficiently construct an entire system. In this thesis we explore the generalized problem of co-optimization of technology and architecture for such systems, and develop a recipe for device requirements and target capabilities. We describe four plausible technologies, each of which could potentially enable the implementation of an efficient and fully-functional neurocomputing system. We first investigate fully-digital neural network architectures that have been tried before using CMOS technology in which many large-size logic gates such as D flip-flops and look-up tables are required. Using a newly-proposed all-magnetic non-volatile logic family, mLogic, we demonstrate the efficacy of digitizing the oscillators and phase relationships for an oscillatory neural network by exploiting the inherent storage as well as enabling an all-digital cellular neural network hardware with simplified programmability. We perform system-level comparisons of mLogic and 32nm CMOS for both networks consisting of 60 neurons. Although digital implementations based on mLogic offer improvements over CMOS in terms of power and area, analog neurocomputing architectures seem to be more compatible with the greatest portion of emerging technologies and devices. For this purpose in this dissertation we explore several emerging technologies with unique device configurations and features such as mCell devices, ovenized aluminum nitride resonators, and tunable multi-gate graphene devices to efficiently enable two key components required for such analog networks – that is, summing function and weighting with compact D/A (digital-to-analog) conversion capability. We demonstrate novel ways to implement these functions and elaborate on our building blocks for artificial neurons and synapses using each technology. We verify the functionality of each proposed implementation using various image processing applications based on compact circuit simulation models for such post-CMOS devices. Finally, we design a proof-of-concept neurocomputing circuitry containing 20 neurons using 65nm CMOS technology that is based on the primitives that we define for our analog neurocomputing scheme. This allows us to fully recognize the inefficiencies of an all-CMOS implementation for such specific applications. We share our experimental results that are in agreement with circuit simulations for the same image processing applications based on proposed architectures using emerging technologies. Power and area comparisons demonstrate significant improvements for analog neurocomputing circuits when implemented using beyond- CMOS technologies, thereby promising huge opportunities for future energy-efficient computing.

In this thesis, an architecture for the adaptive mode transition control of unmanned aerial vehicles (UAV) is presented. The proposed architecture consists of three levels: the highest level is occupied by mission planning routines where information about way points the vehicle must follow is processed. The middle level uses a trajectory generation component to coordinate the task execution and provides set points for low-level stabilizing controllers. The adaptive mode transitioning control algorithm resides at the lowest level of the hierarchy consisting of a mode transitioning controller and the accompanying adaptation mechanism. The mode transition controller is composed of a mode transition manager, a set of local controllers, a set of active control models, a set point filter, a state filter, an automatic trimming mechanism and a dynamic compensation filter. Local controllers operate in local modes and active control models operate in transitions between two local modes. The mode transition manager determines the actual mode of operation of the vehicle based on a set of mode membership functions and activates a local controller or an active control model accordingly. The adaptation mechanism uses an indirect adaptive control methodology to adapt the active control models. For this purpose, a set of plant models based on fuzzy neural networks is trained based on input/output information from the vehicle and used to compute sensitivity matrices providing the linearized models required by the adaptation algorithms. The effectiveness of the approach is verified through software-in-the-loop simulations, hardware-in-the-loop simulations and flight testing.

Process modelling is one of the extensively time consuming tasks within the fieldof process management. This thesis represents an alternative approach minimizing the effort for process modelling based on an intelligent, self-learning system.The current situation of process management in practise is the starting point forthe definition of the requirements which an appropriate information system architecture has to fulfil to solve the issues of contemporary process modelling. The current situation is described by a combination of theoretical considerations aboutcontemporary process management and a survey estimating the efforts needed tomodel and update process descriptions and diagrams out of several, very different documents. The identified requirements represent the groundwork for the derivation of manda-tory features and the architectural composition of the basic components of an intelligent software system and its integration in a process manager’s work. Moreover the effects of the integration of the defined intelligent software system on the social-technical interactions within organizations are sketched. Because text analysis and the transformation of texts into process diagrams crucially influences efficiency and effectivity in process management appropriate, established technologies are comparatively evaluated and discussed regarding theirability to reduce these efforts. According to the identified advantages and limitations of artificial neural networks within the qualitative comparison of the dif-ferent technologies a new concept was developed to fit the needs more holisticthan the existing concepts. The new concept consists of aspects of different established technologies, biological findings and philosophical thoughts about the impossibility of a physical representation of human mind. It results in a conceptof three-dimensional, spatiotemporal flexible, artificial neural networks, which change themselves permanently. A final qualitative comparison figures out thedifferences between the characteristics of existing concepts of artificial neural networks and determines the improvements and benefits the new concept of neural networks achieves.

With recent advances in artificial intelligence, new methods are being developed that provide faster, and more consistent predictions for data in complex environments. In the field of landscape assessment, where an array of physical variables effect environmental perception, natural resource managers need tools to assist them in isolating the significant predictors critical for the protection and management of these resources. Recent studies that have utilized neural networks to assist in developing predictive models of scenic beauty that have typically utilized linear regression techniques have found limited success. The goal of this research is to compare NN's with linear regression models to determine their efficiency predictive capability for assessing near view scenic beauty in the Cedar City District of the Dixie National forest (DNF). Results of this study strongly conclude that neural networks are consistently better predictors of near view scenic beauty in spruce/fir dominated forests than hierarchical linear regression models.

Dans le contexte actuel, l’Intelligence Artificielle (IA) est largement répandue et s’applique à de nombreux domaines tels que les transports, la médecine et les véhicules autonomes. Parmi les algorithmes d'IA, on retrouve principalement les réseaux de neurones, qui peuvent être répartis en deux familles : d'une part, les Réseaux de Neurones Impulsionnels (SNNs) qui sont issus du domaine des neurosciences ; d'autre part, les Réseaux de Neurones Analogiques (ANNs) qui sont issus du domaine de l'apprentissage machine. Les ANNs connaissent un succès inédit grâce à des résultats inégalés dans de nombreux secteurs tels que la classification d'images et la reconnaissance d'objets. Cependant, leur déploiement nécessite des capacités de calcul considérables et ne conviennent pas à des systèmes très contraints. Afin de pallier ces limites, de nombreux chercheurs s'intéressent à un calcul bio-inspiré, qui serait la parfaite alternative aux calculateurs conventionnels basés sur l'architecture de Von Neumann. Ce paradigme répond aux exigences de performance de calcul, mais pas aux exigences d'efficacité énergétique. Il faut donc concevoir des circuits matériels neuromorphiques adaptés aux calculs parallèles et distribués. Dans ce contexte, nous avons établi un certain nombre de critères en termes de précision et de coût matériel pour différencier les SNNs et ANNs. Dans le cas de topologies simples, nous avons montré que les SNNs sont plus efficaces en termes de coût matériel que les ANNs, et ce, avec des précisions de prédiction quasiment similaires. Ainsi, dans ce travail, notre objectif est de concevoir une architecture neuromorphique basée sur les SNNs. Dans cette perspective, nous avons mis en place un flot de conception composé de trois niveaux, qui permet la réalisation d’une architecture neuromorphique dédiée et adaptée aux applications d’IA embarquée.Dans un contexte d'efficacité énergétique, nous avons réalisé une étude approfondie sur divers paradigmes de codage neuronal utilisés avec les SNNs. Par ailleurs, nous avons proposé de nouvelles versions dérivées du codage fréquentiel, visant à se rapprocher de l'activité produite avec le codage temporel, qui se caractérise par un nombre réduit d'impulsions (spikes) se propageant dans le SNN. En faisant cela, nous sommes en mesure de réduire le nombre de spikes, ce qui se traduit par un SNN avec moins d'événements à traiter, et ainsi, réduire la consommation énergétique sous-jacente. Pour cela, deux techniques nouvelles ont été proposées : "First Spike", qui se caractérise par l'utilisation d’un seul spike au maximum par donnée ; "Spike Select", qui permet de réguler et de minimiser l'activité globale du SNN.Dans la partie d’exploration RTL, nous avons comparé de manière quantitative un certain nombre d’architectures de SNN avec différents niveaux de parallélisme et multiplexage de calculs. En effet, le codage "Spike Select" engendre une régulation de la distribution des spikes, avec la majorité générée dans la première couche et peu d'entre eux propagés dans les couches profondes. Nous avons constaté que cette distribution bénéficie d’une architecture hybride comportant une première couche parallèle et les autres multiplexées. Par conséquent, la combinaison du "Spike Select" et de l'architecture hybride serait une solution efficace, avec un compromis efficace entre coût matériel, consommation et latence.Enfin, en se basant sur les choix architecturaux et neuronaux issus de l'exploration précédente, nous avons élaboré une architecture évènementielle dédiée aux SNNs mais suffisamment programmable pour supporter différents types et tailles de réseaux de neurones. L'architecture supporte les couches les plus utilisées : convolution, pooling et entièrement connectées. En utilisant cette architecture, nous serons bientôt en mesure de comparer les ANNs et les SNNs sur des applications réalistes et enfin conclure sur l'utilisation des SNNs pour l'IA embarquée<br>Nowadays, Artificial Intelligence (AI) is a widespread concept applied to many fields such as transportation, medicine and autonomous vehicles. The main AI algorithms are artificial neural networks, which can be divided into two families: Spiking Neural Networks (SNNs), which are bio-inspired models resulting from neuroscience, and Analog Neural Networks (ANNs), which result from machine learning. The ANNs are experiencing unprecedented success in research and industrial fields, due to their recent successes in many application contexts such as image classification and object recognition. However, they require considerable computational capacity for their deployment which is not adequate to very constrained systems such as 'embedded systems'. To overcome these limitations, many researchers are interested in brain-inspired computing, which would be the perfect alternative to conventional computers based on the Von Neumann architecture (CPU/GPU). This paradigm meets computing performance but not energy efficiency requirements. Hence, it is necessary to design neuromorphic hardware circuits adaptable to parallel and distributed computing. In this context, we have set criteria in terms of accuracy and hardware implementation cost to differentiate the two neural families (SNNs and ANNs). In the case of simple network topologies, we conducted a study that has shown that the spiking models have significant gains in terms of hardware cost when compared to the analog networks, with almost similar prediction accuracies. Therefore, the objective of this thesis is to design a generic neuromorphic architecture that is based on spiking neural networks. To this end, we have set up a three-level design flow for exploring and implementing neuromorphic architectures.In an energy efficiency context, a thorough exploration of different neural coding paradigms for neural data representation in SNNs has been carried out. Moreover, new derivative versions of rate-based coding have been proposed that aim to get closer to the activity produced by temporal coding, which is characterized by a reduced number of spikes propagating in the network. In this way, the number of spikes can be reduced so that the number of events to be processed in the SNNs gets smaller. The aim in doing this approach is to reduce the hardware architecture's energy consumption. The proposed coding approaches are: First Spike, which is characterized using at most one single spike to present an input data, and Spike Select, which allows to regulate and minimize the overall spiking activity in the SNN.In the RTL design exploration, we quantitatively compared three SNN architectural models having different levels of computing parallelism and multiplexing. Using Spike Select coding results in a distribution regulation of the spiking data, with most of them generated within the first layer and few of them propagate into the deep layers. Such distribution benefits from a so-called 'hybrid architecture' that includes a fully-parallel part for the first layer and multiplexed parts to the other layers. Therefore, combining the Spike Select and the Hybrid Architecture would be an effective solution for embedded AI applications, with an efficient hardware and latency trade-off.Finally, based on the architectural and neural choices resulting from the previous exploration, we have designed a final event-based architecture dedicated to SNNs supporting different neural network types and sizes. The architecture supports the most used layers: convolutional, pooling and fully-connected. Using this architecture, we will be able to compare analog and spiking neural networks on realistic applications and to finally conclude about the use of SNNs for Embedded Artificial Intelligence

Computer vision techniques are becoming increasingly popular. For example, face recognition is used to help police find criminals, vehicle detection is used to prevent drivers from serious traffic accidents, and written word recognition is used to convert written words into printed words. With the rapid development of vehicle detection given the use of deep learning techniques, there are still concerns about the performance of state-of-the-art vehicle detection techniques. For example, state-of-the-art vehicle detectors are restricted by the large variation of scales. People working on vehicle detection are developing techniques to solve this problem. This thesis proposes an advanced vehicle detection model, adopting one of the classical neural networks, which are the residual neural network and the region proposal network. The model utilizes the residual neural network as a feature extractor and the region proposal network to detect the potential objects' information.<br>Master of Science<br>Computer vision techniques are becoming increasingly popular. For example, face recognition is used to help police find criminals, vehicle detection is used to prevent drivers from serious traffic accidents, and written word recognition is used to convert written words into printed words. With the rapid development of vehicle detection given the use of deep learning techniques, there are still concerns about the performance of state-of-the art vehicle detection techniques. For example, state-of-the-art vehicle detectors are restricted by the large variation of scales. People working on vehicle detection are developing techniques to solve this problem. This thesis proposes an advanced vehicle detection model, utilizing deep learning techniques to detect the potential objects’ information.

SpiNNaker is a neuromorphic hardware devised to simulate SNNs effectively.This paper examines how the performance of an SNN simulated on SpiNNaker is affected by its ratio of number of neurons to number of connections between neurons, as the SNN grows in size. The scope of this paper is limited to the SpiNNaker system SpiNN-3. The performance was measured by running Vogels-Abbott benchmark networks, varying the neuron count and connection probability, and noting which configurations the hardware was able to handle. In total, 3; 498 different configurations were examined. The discussion suggests an improvement to the method that is not dependent upon an underlying assumption to be true. Regardless of whether this assumption is true or not, it is concluded there is a non-proportional relationship between the number of neurons and the connection probability. As the connection probability decreases, the number of neurons that can be simulated increases at a faster rate. Secondly, it is found that SpiNN-3 can simulate up to 12; 000 neurons without issues at a relatively high connection probability compared to the sparsely connected human brain. Thus it is concluded that SpiNN-3 lives up to its promise of being able to simulate ~10; 000 neurons. The scaling properties of SNNs on SpiNNaker are thereby determined to be promising. SpiNNaker2 is the planned sequel to the SpiNNaker version that has been investigated. The results of this study suggest, and most importantly, do not contradict, the possibility of a machine such as SpiNNaker2. It is the authors’ perception that SpiNNaker2 is a good candidate for leading the way into a sustainable future with brain-sized SNN-simulations. Several suggestions for future research are made.<br>SpiNNaker är en neuromorfisk hårdvara utformad för att simulera pulserande neuronnät effektivt. Den här rapporten undersöker hur prestandan för ett pulserande neuronnät simulerat på SpiNNaker påverkas av dess förhållande mellan antal neuroner och antal anslutningar mellan neuroner, när det pulserande neuronnätet växer i storlek. Omfånget av denna rapport är begränsat till SpiNNaker-systemet SpiNN-3. Prestandan mättes genom att köra Vogels- Abbott benchmark-nätverk, där neuronantalet och anslutningssannolikheten varierade och det noterades vilka konfigurationer hårdvaran kunde hantera. Totalt undersöktes 3 498 olika konfigurationer. Diskussionen föreslår en förbättring av metoden som inte är beroende av att ett underliggande antagande är sant. Oavsett om detta antagande är sant eller inte, dras slutsatsen att det finns ett icke-proportionellt förhållande mellan antalet neuroner och anslutningssannolikheten. När anslutningssannolikheten minskar ökar antalet neuroner som kan simuleras snabbare. För det andra har det visat sig att SpiNN- 3 kan simulera upp till 12 000 neuroner utan problem med en relativt hög anslutningssannolikhet jämfört med den glest anslutna mänskliga hjärnans. Således dras slutsatsen att SpiNN-3 lever upp till sitt löfte om att kunna simulera ~10 000 neuroner. Skalningsegenskaperna för pulserande neuronnät på SpiNNaker fastställs därmed lovande. SpiNNaker2 är den planerade uppföljaren till den SpiNNakerversion som har undersökts. Resultaten av denna studie antyder, och viktigast av allt, motsäger inte möjligheten till en maskin som SpiNNaker2. Det är författarnas uppfattning att SpiNNaker2 är en bra kandidat för att leda vägen in i en hållbar framtid med pulserande neuronnätssimuleringar i hjärnstorlek. Flera förslag för framtida forskning görs.<br><p>ISBN 978-91-7873-587-7</p>

Speech recognition systems have been implemented using a wide range of signal processing techniques including neuromorphic/biological inspired and Digital Signal Processing techniques. Neuromorphic/biologically inspired techniques, such as silicon cochlea models, are based on fairly simple yet highly parallel computation and/or computational units. While the area of digital signal processing (DSP) is based on block transforms and statistical or error minimization methods. Essential to each of these techniques is the first stage of extracting meaningful information from the speech signal, which is known as feature extraction. This can be done using biologically inspired techniques such as silicon cochlea models, or techniques beginning with a model of speech production and then trying to separate the the vocal tract response from an excitation signal. Even within each of these approaches, there are multiple techniques including cepstrum filtering, which sits under the class of Homomorphic signal processing, or techniques using FFT based predictive approaches. The underlying reality is there are multiple techniques that have attacked the problem in speech recognition but the problem is still far from being solved. The techniques that have shown to have the best recognition rates involve Cepstrum Coefficients for the feature extraction and Hidden-Markov Models to perform the pattern recognition. The presented research develops an analog system based on programmable analog array technology that can perform the initial stages of auditory feature extraction and recognition before passing information to a digital signal processor. The goal being a low power system that can be fully contained on one or more integrated circuit chips. Results show that it is possible to realize advanced filtering techniques such as Cepstrum Filtering and Vector Quantization in analog circuitry. Prior to this work, previous applications of analog signal processing have focused on vision, cochlea models, anti-aliasing filters and other single component uses. Furthermore, classic designs have looked heavily at utilizing op-amps as a basic core building block for these designs. This research also shows a novel design for a Hidden Markov Model (HMM) decoder utilizing circuits that take advantage of the inherent properties of subthreshold transistors and floating-gate technology to create low-power computational blocks.

The high cost of processor fabrication plants and approaching physical limits have started a new wave research in alternative computing paradigms. As an alternative to the top-down manufactured silicon-based computers, research in computing using natural and physical system directly has recently gained a great deal of interest. A branch of this research promotes the idea that any physical system with sufficiently complex dynamics is able to perform computation. The power of networks in representing complex interactions between many parts make them a suitable choice for modeling physical systems. Many studies used networks with a homogeneous structure to describe the computational circuits. However physical systems are inherently heterogeneous. We aim to study the effect of heterogeneity in the dynamics of physical systems that pertains to information processing. Two particularly well-studied network models that represent information processing in a wide range of physical systems are Random Boolean Networks (RBN), that are used to model gene interactions, and Liquid State Machines (LSM), that are used to model brain-like networks. In this thesis, we study the effects of function heterogeneity, in-degree heterogeneity, and interconnect irregularity on the dynamics and the performance of RBN and LSM. First, we introduce the model parameters to characterize the heterogeneity of components in RBN and LSM networks. We then quantify the effects of heterogeneity on the network dynamics. For the three heterogeneity aspects that we studied, we found that the effect of heterogeneity on RBN and LSM are very different. We find that in LSM the in-degree heterogeneity decreases the chaoticity in the network, whereas it increases chaoticity in RBN. For interconnect irregularity, heterogeneity decreases the chaoticity in LSM while its effects on RBN the dynamics depends on the connectivity. For {K} < 2, heterogeneity in the interconnect will increase the chaoticity in the dynamics and for {K} > 2 it decreases the chaoticity. We find that function heterogeneity has virtually no effect on the LSM dynamics. In RBN however, function heterogeneity actually makes the dynamics predictable as a function of connectivity and heterogeneity in the network structure. We hypothesize that node heterogeneity in RBN may help signal processing because of the variety of signal decomposition by different nodes.

Thesis (M.S.)--State University of New York at Binghamton, Thomas J. Watson School of Engineering and Applied Science, Department of Electrical and Computer Engineering, 2009.<br>Includes bibliographical references.

Self-adaptive systems have limited time to adjust their configurations whenever their adaptation goals, i.e., quality requirements, are violated due to some runtime uncertainties. Within the available time, they need to analyze their adaptation space, i.e., a set of configurations, to find the best adaptation option, i.e., configuration, that can achieve their adaptation goals. Existing formal analysis approaches find the best adaptation option by analyzing the entire adaptation space. However, exhaustive analysis requires time and resources and is therefore only efficient when the adaptation space is small. The size of the adaptation space is often in hundreds or thousands, which makes formal analysis approaches inefficient in large-scale self-adaptive systems. In this thesis, we tackle this problem by presenting an online learning approach that enables formal analysis approaches to analyze large adaptation spaces efficiently. The approach integrates with the standard feedback loop and reduces the adaptation space to a subset of adaptation options that are relevant to the current runtime uncertainties. The subset is then analyzed by the formal analysis approaches, which allows them to complete the analysis faster and efficiently within the available time. We evaluate our approach on two different instances of an Internet of Things application. The evaluation shows that our approach dramatically reduces the adaptation space and analysis time without compromising the adaptation goals.

L’objectif de la thèse est d’étudier la gestion de bout en bout des architectures à la SDN, et comment nos briques OSS (Operation Support System) doivent évoluer: cela implique d’étudier les processus métier associés, leurs implémentations ainsi que l’outillage nécessaire. Les objectifs de la thèse sont donc de répondre aux verrous suivants:1. Identifier les changements impliqués par l’émergence de ces réseaux programmables sur les architectures de gestions en termes d’exigences ou "requirements". L’étude peut être focalisée sur un type de réseau, mobile par exemple. 2. Identifier l’évolution à apporter aux interfaces de gestions actuelles: quelles alternatives aux FCAPS (fault, configuration, accounting, performance, and security) ? Quels changements à apporter aux couches de gestions allant du gestionnaire d’équipement ou "Element Management System" jusqu’au OSS ?<br>The main goal of the PhD activities is to define and develop architecture and mechanisms to ensure consistency and continuity of the operations and behaviors in mixed physical/virtual environments, characterized by a high level of dynamicity, elasticity and heterogeneity by applying a cognitive approach to the architecture where applicable. The target is then to avoid the "build it first, manage it later" paradigm. The research questions targeted by the PhD are the following: 1. Identify the changes on Network Operation Support Systems implementation when using SDN as a design approach for future networks. The study could be restricted to mobile networks for example, or sub-part of it (CORE networks, RAN, data centers, etc); 2.Identify the needed evolution at the management interfaces level: a. Shall we need alternative to the well-known FCAPS and do we still need the element management system? b. What will change to provision an SDN based service? c. How to ensure resiliency of SDN based networks?

Dentro do cenário de projeto de circuitos elétricos orientados para o processamento de redes neurais artificiais, este trabalho se concentra no estudo da implementação de Mapas Auto-organizáveis (SOM, do inglês, Self-Organizing Maps) em chips FPGA. A pesquisa aqui realizada busca, fundamentalmente, responder à seguinte pergunta: como devem ser projetadas as arquiteturas computacionais de cada etapa de processamento do SOM para serem adequadamente executadas em FPGA? De forma mais detalhada, o trabalho investiga as possibilidades que diferentes circuitos de computação do SOM oferecem em relação à velocidade de processamento, ao consumo de recursos do FPGA e à consistência com o formalismo teórico que fundamenta esse modelo de rede neural. Tal objetivo de pesquisa é motivado por possibilitar o desenvolvimento de sistemas de processamento neural que exibam as características positivas típicas de implementações diretas em hardware, como o processamento embarcado e a aceleração computacional. CONTRIBUIÇÕES PRINCIPAIS No decorrer da investigação de tais questões, o presente trabalho gerou contribuições com diferentes graus de impacto. A contribuição mais essencial do ponto de vista de estruturação do restante da pesquisa é a fundamentação teórica das propriedades de computação do SOM em hardware. Tal fundamentação é importante pois permitiu a construção dos alicerces necessários para o estudo das diferentes arquiteturas de circuitos exploradas neste trabalho, de forma que estas permanecessem consistentes com as premissas teóricas que certificam o modelo de computação neural estudado. Outra contribuição avaliada como de grande impacto, e que se consolida como um objeto gerado pela pesquisa, é a proposta de um circuito processador para SOM em FPGA que possui o estado-da-arte em velocidade de computação, medido em CUPS (Connections Updated Per Second). Tal processador permite atingir 52,67 GCUPS, durante a fase de treinamento do SOM, um ganho de aproximadamente 100% em relação aos trabalhos publicados na literatura. A aceleração possibilitada pela exploração de processamentos paralelos em FPGA, desenvolvida neste trabalho, é de três a quatro ordens de grandeza em relação a execuções em software do SOM com a mesma configuração. A última contribuição considerada como de grande impacto é a caracterização da execução do SOM em FPGA. Tal avaliação se faz necessária porque os processos de computação dos modelos neurais em hardware, embora semelhantes, não são necessariamente idênticos aos mesmos processos executados em software. Desta forma, a contribuição deste ponto de pesquisa pode ser entendida como a análise do impacto das mudanças implementadas na computação do SOM em FPGA em relação à execução tradicional do algoritmo, feita pela avaliação dos resultados produzidos pela rede neural por medidas de erros topográficos e de quantização. Este trabalho também gerou contribuições consideradas como de médio impacto, que podem ser divididas em dois grupos: aplicações práticas e aportes teóricos. A primeira contribuição de origem prática é a investigação de trabalhos publicados na literatura envolvendo SOM cujas aplicações podem ser viabilizadas por implementações em hardware. Os trabalhos localizados nesse levantamento foram organizados em diferentes categorias, conforme a área de pesquisa - como, por exemplo, Indústria, Robótica e Medicina - e, em geral, eles utilizam o SOM em aplicações que possuem requisitos de velocidade computacional ou embarque do processamento, portanto, a continuidade de seus desenvolvimentos é beneficiada pela execução direta em hardware. As outras duas contribuições de médio impacto de origem prática são as aplicações que serviram como plataforma de teste dos circuitos desenvolvidos para a implementação do SOM. A primeira aplicação pertence à área de telecomunicações e objetiva a identificação de símbolos transmitidos por 16-QAM ou 64-QAM. Estas duas técnicas de modulação são empregadas em diversas aplicações com requisitos de mobilidade - como telefonia celular, TV digital em dispositivos portáteis e Wi-Fi - e o SOM é utilizado para identificar sinais QAM recepcionados com ruídos e distorções. Esta aplicação gerou a publicação de um artigo na revista da Springer, Neural Computing and Applications: Sousa; Pires e Del-Moral-Hernandez (2017). A segunda aplicação pertence à área de processamento de imagem e visa reconhecer ações humanas capturadas por câmeras de vídeo. O processamento autônomo de imagens executado por chips FPGA junto às câmeras de vídeo pode ser empregado em diferentes utilizações, como, por exemplo, sistemas de vigilância automática ou assistência remota em locais públicos. Esta segunda aplicação também é caracterizada por demandar arquiteturas computacionais de alto desempenho. Todas as contribuições teóricas deste trabalho avaliadas como de médio impacto estão relacionadas ao estudo das características de arquiteturas de hardware para computação do modelo SOM. A primeira destas é a proposta de uma função de vizinhança do SOM baseada em FPGA. O objetivo de tal proposta é desenvolver uma expressão computacional para ser executada no chip que constitua uma alternativa eficiente tanto à função gaussiana, tradicionalmente empregada no processo de treinamento do SOM, quanto à função retangular, utilizada de forma rudimentar nas primeiras pesquisas publicadas sobre a implementação do SOM em FPGA. A segunda destas contribuições é a descrição detalhada dos componentes básicos e dos blocos computacionais utilizados nas diferentes etapas de execução do SOM em FPGA. A apresentação dos detalhes da arquitetura de processamento, incluindo seus circuitos internos e a função computada por cada um de seus blocos, permite que trabalhos futuros utilizem os desenvolvimentos realizados nesta pesquisa. Esta descrição detalhada e funcional foi aceita para publicação no IEEE World Congress on Computational Intelligence (WCCI 2018): Sousa et al. (2018). A terceira contribuição teórica de médio impacto é a elaboração de um modelo distribuído de execução do SOM em FPGA sem o uso de uma unidade central de controle. Tal modelo permite a execução das fases de aprendizado e operação da rede neural em hardware de forma distribuída, a qual alcança um comportamento global de auto-organização dos neurônios apenas pela troca local de dados entre elementos de processamento vizinhos. A descrição do modelo distribuído, em conjunto com sua caracterização, está publicada em um artigo no International Joint Conference on Neural Networks do IEEE (IJCNN 2017): Sousa e Del-Moral-Hernandez (2017a). A última contribuição deste grupo de aporte teórico é a comparação entre diferentes modelos de execução do SOM em FPGA. A comparação tem a função de avaliar e contrastar três diferentes possibilidades de implementação do SOM: o modelo distribuído, o modelo centralizado e o modelo híbrido. Os testes realizados e os resultados obtidos estão publicados em um trabalho no International Symposium on Circuits and Systems do IEEE (ISCAS 2017): Sousa e Del-Moral-Hernandez (2017b). Finalmente, apresentam-se a seguir as contribuições avaliadas como de menor impacto, em comparação com as contribuições já descritas, ou ainda incipientes (e que possibilitam continuidades da pesquisa em trabalhos futuros), sendo relacionadas a seguir como contribuições complementares: * Pesquisa de literatura científica sobre o estado-da-arte da área da Engenharia de Sistemas Neurais Artificiais. * Identificação de grupos internacionais de pesquisa de execução do SOM em hardware, os quais foram reconhecidos por publicarem regularmente seus estudos sobre diferentes tipos de implementações e categorias de circuitos computacionais. * Enumeração das justificativas e motivações mais frequentes na literatura para o processamento de sistemas neurais de computação em hardware. * Comparação e contraste das características de microprocessadores, GPUs, FPGAs e ASICs (tais como, custo médio do componente, paralelismo computacional oferecido e consumo típico de energia) para contextualização do tipo de aplicações que a escolha pela pesquisa com o dispositivo FPGA possibilita. * Levantamento das propriedades de computação do SOM em hardware mais frequentemente utilizadas nas pesquisas publicadas na literatura, tais como, quantidade de bits usados nos cálculos, tipo de representação de dados e arquitetura típica dos circuitos de execução das diferentes etapas de processamento do SOM. * Comparação do consumo de área do FPGA e da velocidade de processamento entre a execução da função de vizinhança tradicional gaussiana e a função de vizinhança proposta neste trabalho (com resultados obtidos de aproximadamente 4 vezes menos área do chip e 5 vezes mais velocidade de operação). * Caracterização do aumento dos recursos consumidos no chip e da velocidade de operação do sistema, em relação à implementação do SOM com diferentes complexidades (quantidade de estágios decrescentes do fator de aprendizado e da abertura da função de vizinhança) e comparação destas propriedades da arquitetura proposta em relação aos valores publicados na literatura. * Proposta de uma nova métrica para caracterização do erro topográfico na configuração final do SOM após o treinamento.<br>In the context of design electrical circuits for processing artificial neural networks, this work focuses on the study of Self-Organizing Maps (SOM) executed on FPGA chips. The work attempts to answer the following question: how should the computational architecture be designed to efficiently implement in FPGA each one of the SOM processing steps? More specifically, this thesis investigates the distinct possibilities that different SOM computing architectures offer, regarding the processing speed, the consumption of FPGA resources and the consistency to the theory that underlies this neural network model. The motivation of the present work is enabling the development of neural processing systems that exhibit the positive features typically associate to hardware implementations, such as, embedded processing and computational acceleration. MAIN CONTRIBUITIONS In the course of the investigation, the present work generated contributions with different degrees of impact. The most essential contribution from the point of view of structuring the research process is the theoretical basis of the hardware-oriented SOM properties. This is important because it allowed the construction of the foundations for the study of different circuit architectures, so that the developments remained consistent with the theory that underpins the neural computing model. Another major contribution is the proposal of a processor circuit for implementing SOM in FPGA, which is the state-of-the-art in computational speed measured in CUPS (Connections Updated Per Second). This processor allows achieving 52.67 GCUPS, during the training phase of the SOM, which means a gain of 100%, approximately, in relation to other published works. The acceleration enabled by the FPGA parallel processing developed in this work reaches three to four orders of magnitude compared with software implementations of the SOM with the same configuration. The highlights made in the text indicate pieces of writing that synthesize the idea presented. The last main contribution of the work is the characterization of the FPGA-based SOM. This evaluation is important because, although similar, the computing processes of neural models in hardware are not necessarily identical to the same processes implemented in software. Hence, this contribution can be described as the analysis of the impact of the implemented changes, regarding the FPGA-based SOM compared to traditional algorithms. The comparison was performed evaluating the measures of topographic and quantization errors for the outputs produced by both implementations. This work also generated medium impact contributions, which can be divided into two groups: empirical and theoretical. The first empirical contribution is the survey of SOM applications which can be made possible by hardware implementations. The papers presented in this survey are classified according to their research area - such as Industry, Robotics and Medicine - and, in general, they use SOM in applications that require computational speed or embedded processing. Therefore, the continuity of their developments is benefited by direct hardware implementations of the neural network. The other two empirical contributions are the applications employed for testing the circuits developed. The first application is related to the reception of telecommunications signals and aims to identify 16-QAM and 64-QAM symbols. These two modulation techniques are used in a variety of applications with mobility requirements, such as cell phones, digital TV on portable devices and Wi-Fi. The SOM is used to identify QAM distorted signals received with noise. This research work was published in the Springer Journal on Neural Computing and Applications: Sousa; Pires e Del-Moral-Hernandez (2017). The second is an image processing application and it aims to recognize human actions captured by video cameras. Autonomous image processing performed by FPGA chips inside video cameras can be used in different scenarios, such as automatic surveillance systems or remote assistance in public areas. This second application is also characterized by demanding high performance from the computing architectures. All the theoretical contributions with medium impact are related to the study of the properties of hardware circuits for implementing the SOM model. The first of these is the proposal of an FPGA-based neighborhood function. The aim of the proposal is to develop a computational function to be implemented on chip that enables an efficient alternative to both: the Gaussian function (traditionally employed in the SOM training process) and the rectangular function (used rudimentary in the first published works on hardware-based SOMs). The second of those contributions is the detailed description of the basic components and blocks used to compute the different steps of the SOM algorithm in hardware. The description of the processing architecture includes its internal circuits and computed functions, allowing the future works to use the architecture proposed. This detailed and functional description was accepted for publication in the IEEE World Congress on Computational Intelligence (WCCI 2018): Sousa et al. (2018). The development of an FPGA distributed implementation model for the SOM composes the third of those contributions. Such a model allows an execution of the neural network learning and operational phases without the use of a central control unit. The proposal achieves a global self-organizing behavior only by using local data exchanges among the neighboring processing elements. The description and characterization of the distributed model are published in a paper in the IEEE International Joint Conference on Neural Networks (IJCNN 2017): Sousa e Del-Moral-Hernandez (2017a). The last contribution of this group is the comparison between different FPGA architectures for implementing the SOM. This comparison has the function of evaluating and contrasting three different SOM architectures: the distributed model, the centralized model and the hybrid model. The tests performed and the results obtained are published in an article in the IEEE International Symposium on Circuits and Systems (ISCAS 2017): Sousa e Del-Moral-Hernandez (2017b). Finally, the contributions assessed as having a minor impact, compared to contributions already described, or still incipient (and which allow the continuity of the research in possible future works), are presented as complementary contributions: * Research in the scientific literature on the state-of-the-art works in the field of Artificial Neural Systems Engineering. * Identification of the international research groups on hardware-based SOM, which were recognized for regularly publishing their studies on different types of implementations and categories of computational circuits. * Enumeration of the justifications and motivations often mentioned in works on hardware developments of neural computing systems. * Comparison and contrast of the characteristics of microprocessors, GPUs, FPGAs and ASICs (such as, average cost, parallelism and typical power consumption) to contextualize the type of applications enabled by the choice of FPGA as the target device. * Survey of literature for the most commonly hardware properties used for computing the SOM, such as the number of bits used in the calculations, the type of data representation and the typical architectures of the FPGA circuits. * Comparison of the FPGA resources consumption and processing speed between the execution of the traditional Gaussian neighborhood function and the proposed alternative neighborhood function (with obtained results of approximately 4 times less chip area and 5 times more computational speed). * Characterization of the increase in chip resources consumptions and the decrease in system speeds, according to the implementations of the SOM with different complexities (such as, the number of stages in learning factor and the width of the neighborhood function). Comparison of these properties between the proposed architecture and the works published in the literature. * Proposal of a new metric for the characterization of the topographic error in the final configuration of the SOM after the training phase.

L’objectif de la thèse est d’étudier la gestion de bout en bout des architectures à la SDN, et comment nos briques OSS (Operation Support System) doivent évoluer: cela implique d’étudier les processus métier associés, leurs implémentations ainsi que l’outillage nécessaire. Les objectifs de la thèse sont donc de répondre aux verrous suivants:1. Identifier les changements impliqués par l’émergence de ces réseaux programmables sur les architectures de gestions en termes d’exigences ou "requirements". L’étude peut être focalisée sur un type de réseau, mobile par exemple. 2. Identifier l’évolution à apporter aux interfaces de gestions actuelles: quelles alternatives aux FCAPS (fault, configuration, accounting, performance, and security) ? Quels changements à apporter aux couches de gestions allant du gestionnaire d’équipement ou "Element Management System" jusqu’au OSS ?<br>The main goal of the PhD activities is to define and develop architecture and mechanisms to ensure consistency and continuity of the operations and behaviors in mixed physical/virtual environments, characterized by a high level of dynamicity, elasticity and heterogeneity by applying a cognitive approach to the architecture where applicable. The target is then to avoid the "build it first, manage it later" paradigm. The research questions targeted by the PhD are the following: 1. Identify the changes on Network Operation Support Systems implementation when using SDN as a design approach for future networks. The study could be restricted to mobile networks for example, or sub-part of it (CORE networks, RAN, data centers, etc); 2.Identify the needed evolution at the management interfaces level: a. Shall we need alternative to the well-known FCAPS and do we still need the element management system? b. What will change to provision an SDN based service? c. How to ensure resiliency of SDN based networks?

Розроблено інтелектуальну систему прогнозу курсу криптовалют методом рекурентних нейромереж, спроектовано та реалізовано програмне забезпечення з підтримкою можливостей проведення операцій над криптовалютами у вигляді веб-сервісів з мікросервісною архітектурою для підвищення їх ефективності.

Depuis son introduction en 1982, la carte auto-organisatrice de Kohonen (Self-Organizing Map : SOM) a prouvé ses capacités de classification et visualisation des données multidimensionnelles dans différents domaines d’application. Les implémentations matérielles de la carte SOM, en exploitant le taux de parallélisme élevé de l’algorithme de Kohonen, permettent d’augmenter les performances de ce modèle neuronal souvent au détriment de la flexibilité. D’autre part, la flexibilité est offerte par les implémentations logicielles qui quant à elles ne sont pas adaptées pour les applications temps réel à cause de leurs performances temporelles limitées. Dans cette thèse nous avons proposé une architecture matérielle distribuée, adaptable, flexible et extensible de la carte SOM à base de NoC dédiée pour une implantation matérielle sur FPGA. A base de cette approche, nous avons également proposé une architecture matérielle innovante d’une carte SOM à structure croissante au cours de la phase d’apprentissage<br>Since its introduction in 1982, Kohonen’s Self-Organizing Map (SOM) showed its ability to classify and visualize multidimensional data in various application fields. Hardware implementations of SOM, by exploiting the inherent parallelism of the Kohonen algorithm, allow to increase the overall performances of this neuronal network, often at the expense of the flexibility. On the other hand, the flexibility is offered by software implementations which on their side are not suited for real-time applications due to the limited time performances. In this thesis we proposed a distributed, adaptable, flexible and scalable hardware architecture of SOM based on Network-on-Chip (NoC) designed for FPGA implementation. Moreover, based on this approach we also proposed a novel hardware architecture of a growing SOM able to evolve its own structure during the learning phase

Submitted by Silvana Teresinha Dornelles Studzinski (sstudzinski) on 2016-06-29T14:42:16Z No. of bitstreams: 1 Pablo Eduardo Pereira de Araujo Cottens_.pdf: 1315690 bytes, checksum: 78ac4ce471c2b51e826c7523a01711bd (MD5)<br>Made available in DSpace on 2016-06-29T14:42:16Z (GMT). No. of bitstreams: 1 Pablo Eduardo Pereira de Araujo Cottens_.pdf: 1315690 bytes, checksum: 78ac4ce471c2b51e826c7523a01711bd (MD5) Previous issue date: 2016-03-07<br>Nenhuma<br>Normalmente Redes Neurais Artificiais (RNAs) necessitam estações de trabalho para o seu processamento, por causa da complexidade do sistema. Este tipo de arquitetura de processamento requer que instrumentos de campo estejam localizados na vizinhança da estação de trabalho, caso exista a necessidade de processamento em tempo real, ou que o dispositivo de campo possua como única tarefa a de coleta de dados para processamento futuro. Este projeto visa criar uma arquitetura em lógica programável para um neurônio genérico, no qual as RNAs podem fazer uso da natureza paralela de FPGAs para executar a aplicação de forma rápida. Este trabalho mostra que a utilização de lógica programável para a implementação de RNAs de baixa resolução de bits é viável e as redes neurais, devido à natureza paralelizável, se beneficiam pela implementação em hardware, podendo obter resultados de forma muito rápida.<br>Currently, modern Artificial Neural Networks (ANN), according to their complexity, require a workstation for processing all their input data. This type of processing architecture requires that the field device is located somewhere in the vicintity of a workstation, in case real-time processing is required, or that the field device at hand will have the sole task of collecting data for future processing, when field data is required. This project creates a generic neuron architecture in programmabl logic, where Artifical Neural Networks can use the parallel nature of FPGAs to execute applications in a fast manner, albeit not using the same resolution for its otputs. This work shows that the utilization of programmable logic for the implementation of low bit resolution ANNs is not only viable, but the neural network, due to its parallel nature, benefits greatly from the hardware implementation, giving fast and accurate results.