Dissertations / Theses on the topic 'Neural networks'

Thesis (Ph. D.)--University of Oregon, 2006.<br>Typescript. Includes vita and abstract. Includes bibliographical references (leaves 122- 131). Also available for download via the World Wide Web; free to University of Oregon users.

The goal of this thesis is to explore some variations of neural networks. The thesis is mainly split into two parts: a variation of the shaping functions in neural networks and a variation of learning rules in neural networks. In the first part, we mainly investigate polynomial perceptrons - a perceptron with a polynomial shaping function instead of a linear one. We prove the polynomial perceptron convergence theorem and illustrate the notion by showing that a higher order perceptron can learn the XOR function through empirical experiments with implementation. In the second part, we propose three models (SMLP, SA, SA2) for stream learning and anomaly detection in streams. The main technique allowing these models to perform at a level comparable to the state-of-the-art algorithms in stream learning is the learning rule used. We employ mini-batch gradient descent algorithm and stochastic gradient descent algorithm to speed up the models. In addition, the use of parallel processing with multi-threads makes the proposed methods highly efficient in dealing with streaming data. Our analysis shows that all models have linear runtime and constant memory requirement. We also demonstrate empirically that the proposed methods feature high detection rate, low false alarm rate, and fast response. The paper on the first two models (SMLP, SA) is published in the 29th Canadian AI Conference and won the best paper award. The invited journal paper on the third model (SA2) for Computational Intelligence is under peer review.

Artificial Neural Networks have been widely probed by worldwide researchers to cope with the problems such as function approximation and data simulation. This thesis deals with Feed-forward Neural Networks (FNN's) with a new neuron activation function called Neuron-adaptive Activation Function (NAF), and Feed-forward Higher Order Neural Networks (HONN's) with this new neuron activation function. We have designed a new neural network model, the Neuron-Adaptive Neural Network (NANN), and mathematically proved that one NANN can approximate any piecewise continuous function to any desired accuracy. In the neural network literature only Zhang proved the universal approximation ability of FNN Group to any piecewise continuous function. Next, we have developed the approximation properties of Neuron Adaptive Higher Order Neural Networks (NAHONN's), a combination of HONN's and NAF, to any continuous function, functional and operator. Finally, we have created a software program called MASFinance which runs on the Solaris system for the approximation of continuous or discontinuous functions, and for the simulation of any continuous or discontinuous data (especially financial data). Our work distinguishes itself from previous work in the following ways: we use a new neuron-adaptive activation function, while the neuron activation functions in most existing work are all fixed and can't be tuned to adapt to different approximation problems; we only use on NANN to approximate any piecewise continuous function, while a neural network group must be utilised in previous research; we combine HONN's with NAF and investigate its approximation properties to any continuous function, functional, and operator; we present a new software program, MASFinance, for function approximation and data simulation. Experiments running MASFinance indicate that the proposed NANN's present several advantages over traditional neuron-fixed networks (such as greatly reduced network size, faster learning, and lessened simulation errors), and that the suggested NANN's can effectively approximate piecewise continuous functions better than neural networks groups. Experiments also indicate that NANN's are especially suitable for data simulation<br>Doctor of Philosophy (PhD)

Thesis: M. Eng., Massachusetts Institute of Technology, Department of Electrical Engineering and Computer Science, 2018.<br>This electronic version was submitted by the student author. The certified thesis is available in the Institute Archives and Special Collections.<br>Cataloged from student-submitted PDF version of thesis.<br>Includes bibliographical references (pages 65-68).<br>Bus schedules are unreliable, leaving passengers waiting and increasing commute times. This problem can be solved by modeling the traffic network, and delivering predicted arrival times to passengers. Research attempts to model traffic networks use historical, statistical and learning based models, with learning based models achieving the best results. This research compares several neural network architectures trained on historical data from Boston buses. Three models are trained: multilayer perceptron, convolutional neural network and recurrent neural network. Recurrent neural networks show the best performance when compared to feed forward models. This indicates that neural time series models are effective at modeling bus networks. The large amount of data available for training bus network models and the effectiveness of large neural networks at modeling this data show that great progress can be made in improving commutes for passengers.<br>by Cooper Stokes Sloan.<br>M. Eng.

To create neural networks that work, one needs to specify a structure and the interconnection weights between each pair of connected computing elements. The structure of a network can be selected by the designer depending on the application, although the selection of interconnection weights is a much larger problem. Algorithms have been developed to alter the weights slightly in order to produce the desired results. Learning algorithms such as Hebb's rule, the Delta rule and error propagation have been used, with success, to learn the appropriate weights. The major objection to this class of algorithms is that one cannot specify what is not desired in the network in addition to what is desired. An alternate method to learning the correct interconnection weights is to evolve a network in an environment that rewards "good” behavior and punishes "bad" behavior, This technique allows interesting networks to appear which otherwise may not be discovered by other methods of learning. In order to teach a network the correct weights, this approach simply needs a direction where an acceptable solution can be obtained rather than a complete answer to the problem.<br>Master of Science

Les xarxes neuronals cel·lulars altrament anomenades CNNs, són un tipus de sistema dinàmic que relaciona diferents elements que s'anomenen neurones via unes plantilles de paràmetres. Aquest sistema queda completament determinat coneixent quines són les entrades a la xarxa, les sortides i els paràmetres o pesos. En aquest treball fem un estudi exhaustiu sobre aquest tipus de xarxa en el cas més senzill on només hi intervenen dues neurones. Tot i la simplicitat del sistema, veurem que pot tenir una dinàmica molt rica. <br/><br/>Primer de tot, revisem l'estabilitat d'aquest sistema des de dos punts de vista diferents. Usant la teoria de Lyapunov, trobem el rang de paràmetres en el que hem de treballar per aconseguir la convergència de la xarxa cap a un punt fix. Aquest mètode ens obre les portes per abordar els diferents tipus de problemes que es poden resoldre usant una xarxa neuronal cel·lular de dues neurones. D'altra banda, el comportament dinàmic de la CNN està determinat per la funció lineal a trossos que defineix les sortides del sistema. Això ens permet estudiar els diferents sistemes que apareixen en cada una de les regions on el sistema és lineal, aconseguint un estudi complet de l'estabilitat de la xarxa en funció de les posicions locals dels diferents punts d'equilibri del sistema. D'aquí obtenim bàsicament dos tipus de convergència, cap a un punt fix o bé cap a un cicle límit. Aquests resultats ens permeten organitzar aquest estudi bàsicament en aquests dos tipus de convergència. Entendre el sistema d'equacions diferencials que defineixen la CNN en dimensió 1 usant només dues neurones, ens permet trobar les dificultats intrínseques de les xarxes neuronals cel·lulars així com els possibles usos que els hi podem donar. A més, ens donarà les claus per a poder entendre el cas general. <br/><br/>Un dels primers problemes que abordem és la dependència de les sortides del sistema respecte les condicions inicials. La funció de Lyapunov que usem en l'estudi de l'estabilitat es pot veure com una quàdrica si la pensem com a funció de les sortides. La posició i la geometria d'aquesta forma quadràtica ens permeten trobar condicions sobre els paràmetres que descriuen el sistema dinàmic. Treballant en aquestes regions aconseguim abolir el problema de la dependència. A partir d'aquí ja comencem a estudiar les diferents aplicacions de les CNN treballant en un rang de paràmetres on el sistema convergeix a un punt fix. Una primera aplicació la trobem usant aquest tipus de xarxa per a reproduir distribucions de probabilitat tipus Bernoulli usant altre cop la funció de Lyapunov emprada en l'estudi de l'estabilitat. Una altra aplicació apareix quan ens centrem a treballar dins del quadrat unitat. En aquest cas, el sistema és capaç de reproduir funcions lineals. <br/><br/>L'existència de la funció de Lyapunov permet també de construir unes gràfiques que depenen dels paràmetres de la CNN que ens indiquen la relació que hi ha entre les entrades de la CNN i les sortides. Aquestes gràfiques ens donen un algoritme per a dissenyar plantilles de paràmetres reproduint aquestes relacions. També ens obren la porta a un nou problema: com composar diferents plantilles per aconseguir una determinada relació entrada¬sortida. Tot aquest estudi ens porta a pensar en buscar una relació funcional entre les entrades externes a la xarxa i les sortides. Com que les possibles sortides és un conjunt discret d'elements gràcies a la funció lineal a trossos, la correspondència entrada¬sortida es pot pensar com un problema de classificació on cada una de les classes està definida per les diferent possibles sortides. Pensant¬ho d'aquesta manera, estudiem quins problemes de classificació es poden resoldre usant una CNN de dues neurones i trobem quina relació hi ha entre els paràmetres de la CNN, les entrades i les sortides. Això ens permet trobar un mètode per a dissenyar plantilles per a cada problema concret de classificació. A més, els resultats obtinguts d'aquest estudi ens porten cap al problema de reproduir funcions Booleanes usant CNNs i ens mostren alguns dels límits que tenen les xarxes neuronals cel·lulars tot intentant reproduir el capçal de la màquina universal de Turing descoberta per Marvin Minsky l'any 1962. <br/><br/>A partir d'aquí comencem a estudiar la xarxa neuronal cel·lular quan convergeix cap a un cicle límit. Basat en un exemple particular extret del llibre de L.O Chua, estudiem primer com trobar cicles límit en el cas que els paràmetres de la CNN que connecten les diferents neurones siguin antisimètrics. D'aquesta manera trobem en quin rang de paràmetres hem de treballar per assegurar que l'estat final de la xarxa sigui una corba tancada. A més ens dona la base per poder abordar el problema en el cas general. El comportament periòdic d'aquestes corbes ens incita primer a calcular aquest període per cada cicle i després a pensar en possibles aplicacions com ara usar les CNNs per a generar senyals de rellotge. <br/><br/>Finalment, un cop estudiats els diferents tipus de comportament dinàmics i les seves possibles aplicacions, fem un estudi comparatiu de la xarxa neuronal cel·lular quan la sortida està definida per la funció lineal a trossos i quan està definida per la tangent hiperbòlica ja que moltes vegades en la literatura s'usa l'una en comptes de l'altra aprofitant la seva diferenciabilitat. Aquest estudi ens indica que no sempre es pot usar la tangent hiperbòlica en comptes de la funció lineal a trossos ja que la convergència del sistema és diferent en un segons com es defineixin les sortides de la CNN.<br>Les redes neuronales celulares o CNNs, son un tipo de sistema dinámico que relaciona diferentes elementos llamados neuronas a partir de unas plantillas de parámetros. Este sistema queda completamente determinado conociendo las entradas de la red, las salidas y los parámetros o pesos. En este trabajo hacemos un estudio exhaustivo de estos tipos de red en el caso más sencillo donde sólo intervienen dos neuronas. Este es un sistema muy sencillo que puede llegar a tener una dinámica muy rica. <br/><br/>Primero, revisamos la estabilidad de este sistema desde dos puntos de vista diferentes. Usando la teoría de Lyapunov, encontramos el rango de parámetros en el que hemos de trabajar para conseguir que la red converja hacia un punto fijo. Este método nos abre las puertas parar poder abordar los diferentes tipos de problemas que se pueden resolver usando una red neuronal celular de dos neuronas. Por otro lado, el comportamiento dinámico de la CNN está determinado por la función lineal a tramos que define las salidas del sistema. Esto nos permite estudiar los diferentes sistemas que aparecen en cada una de las regiones donde el sistema es lineal, consiguiendo un estudio completo de la estabilidad de la red en función de las posiciones locales de los diferentes puntos de equilibrio del sistema. Obtenemos básicamente dos tipos de convergencia, hacia a un punto fijo o hacia un ciclo límite. Estos resultados nos permiten organizar este estudio básicamente en estos dos tipos de convergencia. Entender el sistema de ecuaciones diferenciales que definen la CNN en dimensión 1 usando solamente dos neuronas, nos permite encontrar las dificultades intrínsecas de las redes neuronales celulares así como sus posibles usos. Además, nos va a dar los puntos clave para poder entender el caso general. Uno de los primeros problemas que abordamos es la dependencia de las salidas del sistema respecto de las condiciones iniciales. La función de Lyapunov que usamos en el estudio de la estabilidad es una cuadrica si la pensamos como función de las salidas. La posición y la geometría de esta forma cuadrática nos permiten encontrar condiciones sobre los parámetros que describen el sistema dinámico. Trabajando en estas regiones logramos resolver el problema de la dependencia. A partir de aquí ya podemos empezar a estudiar las diferentes aplicaciones de las CNNs trabajando en un rango de parámetros donde el sistema converge a un punto fijo. Una primera aplicación la encontramos usando este tipo de red para reproducir distribuciones de probabilidad tipo Bernoulli usando otra vez la función de Lyapunov usada en el estudio de la estabilidad. Otra aplicación aparece cuando nos centramos en trabajar dentro del cuadrado unidad. En este caso, el sistema es capaz de reproducir funciones lineales. <br/><br/>La existencia de la función de Lyapuno v permite también construir unas graficas que dependen de los parámetros de la CNN que nos indican la relación que hay entre las entradas de la CNN y las salidas. Estas graficas nos dan un algoritmo para diseñar plantillas de parámetros reproduciendo estas relaciones. También nos abren la puerta hacia un nuevo problema: como componer diferentes plantillas para conseguir una determinada relación entrada¬salida. Todo este estudio nos lleva a pensar en buscar una relación funcional entre las entradas externas a la red y las salidas. Teniendo en cuenta que las posibles salidas es un conjunto discreto de elementos gracias a la función lineal a tramos, la correspondencia entrada¬salida se puede pensar como un problema de clasificación donde cada una de las clases está definida por las diferentes posibles salidas. Pensándolo de esta forma, estudiamos qué problemas de clasificación se pueden resolver usando una CNN de dos neuronas y encontramos la relación que hay entre los parámetros de la CNN, las entradas y las salidas. Esto nos permite encontrar un método de diseño de plantillas para cada problema concreto de clasificación. Además, los resultados obtenidos en este estudio nos conducen hacia el problema de reproducir funciones Booleanas usando CNNs y nos muestran algunos de los límites que tienen las redes neuronales celulares al intentar reproducir el cabezal (la cabeza) de la máquina universal de Turing descubierta por Marvin Minsky el año 1962. <br/><br/>A partir de aquí empezamos a estudiar la red neuronal celular cuando ésta converge hacia un ciclo límite. Basándonos en un ejemplo particular sacado del libro de L.O Chua, estudiamos primero como encontrar ciclos límite en el caso que los parámetros de la CNN que conectan las diferentes neuronas sean anti¬simétricos. De esta forma encontramos el rango de parámetros en el cuál hemos de trabajar para asegurar que el estado final de la red sea una curva cerrada. Además nos da la base para poder abordar el problema en el caso general. El comportamiento periódico de estas curvas incita primero a calcular su periodo para cada ciclo y luego a pensar en posibles aplicaciones como por ejemplo usar las CNNs para generar señales de reloj. <br/><br/>Finalmente, estudiados ya los diferentes tipos de comportamiento dinámico y sus posibles aplicaciones, hacemos un estudio comparativo de la red neuronal celular cuando la salida está definida por la función lineal a trozos y cuando está definida por la tangente hiperbólica ya que muchas veces en la literatura se usa una en vez de la otra intentado aprovechar su diferenciabilidad. Este estudio nos indica que no siempre se puede intercambiar dichas funciones ya que la convergencia del sistema es distinta según como se definan las salidas de la CNN.<br>In this dissertation we review the two neuron cellular neural network stability using the Lyapunov theory, and using the different local dynamic behavior derived from the piecewise linear function use. We study then a geometrical way to understand the system dynamics. The Lyapunov stability, gives us the key point to tackle the different convergence problems that can be studied when the CNN system converges to a fixed¬point. The geometric stability shed light on the convergence to limit cycles. This work is basically organized based on these two convergence classes. <br/><br/>We try to make an exhaustive study about Cellular Neural Networks in order to find the intrinsic difficulties, and the possible uses of a CNN. Understanding the CNN system in a lower dimension, give us some of the main keys in order to understand the general case. That's why we will focus our study in the one dimensional CNN case with only two neurons. <br/><br/>From the results obtained using the Lyapunov function, we propose some methods to avoid the dependence on initial conditions problem. Its intrinsic characteristics as a quadratic form of the output values gives us the key points to find parameters where the final outputs do not depend on initial conditions. At this point, we are able to study different CNN applications for parameter range where the system converges to a fixed¬point. We start by using CNNs to reproduce Bernoulli probability distributions, based on the Lyapunov function geometry. Secondly, we reproduce linear functions while working inside the unit square. <br/><br/>The existence of the Lyapunov function allows us to construct a map, called convergence map, depending on the CNN parameters, which relates the CNN inputs with the final outputs. This map gives us a recipe to design templates performing some desired input¬output associations. The results obtained drive us into the template composition problem. We study the way different templates can be applied in sequence. From the results obtained in the template design problem, we may think on finding a functional relation between the external inputs and the final outputs. Because the set of final states is discrete, thanks to the piecewise linear function, this correspondence can be thought as a classification problem. Each one of the different classes is defined by the different final states which, will depend on the CNN parameters. <br/><br/>Next, we study which classifications problems can be solved by a two neuron CNN, and relate them with weight parameters. In this case, we also find a recipe to design templates performing these classification problems. The results obtained allow us to tackle the problem to realize Boolean functions using CNNs, and show us some CNN limits trying to reproduce the header of a universal Turing machine. <br/><br/>Based on a particular limit cycle example extracted from Chua's book, we start this study with anti symmetric connections between cells. The results obtained can be generalized for CNNs with opposite sign parameters. We have seen in the stability study that limit cycles have the possibility to exist for this parameter range. Periodic behavior of these curves is computed in a particular case. The limit cycle period can be expressed as a function of the CNN parameters, and can be used to generate clock signals. <br/><br/>Finally, we compare the CNN dynamic behavior using different output functions, hyperbolic tangent and piecewise linear function. Many times in the literature, hyperbolic tangent is used instead of piecewise linear function because of its differentiability along the plane. Nevertheless, in some particular regions in the parameter space, they exhibit a different number of equilibrium points. Then, for theoretical results, hyperbolic tangent should not be used instead of piecewise linear function.

Affective computing has gained significant attention from researchers in the last decade due to the wide variety of applications that can benefit from this technology. Often, researchers describe affect using emotional dimensions such as arousal and valence. Valence refers to the spectrum of negative to positive emotions while arousal determines the level of excitement. Describing emotions through continuous dimensions (e.g. valence and arousal) allows us to encode subtle and complex affects as opposed to discrete emotions, such as the basic six emotions: happy, anger, fear, disgust, sad and neutral. Recognizing spontaneous and subtle emotions remains a challenging problem for computers. In our work, we employ two modalities of information: video and audio. Hence, we extract visual and audio features using deep neural network models. Given that emotions are time-dependent, we apply the Temporal Convolutional Neural Network (TCN) to model the variations in emotions. Additionally, we investigate an alternative model that combines a Convolutional Neural Network (CNN) and a Recurrent Neural Network (RNN). Given our inability to fit the latter deep model into the main memory, we divide the RNN into smaller segments and propose a scheme to back-propagate gradients across all segments. We configure the hyperparameters of all models using Gaussian processes to obtain a fair comparison between the proposed models. Our results show that TCN outperforms RNN for the recognition of the arousal and valence emotional dimensions. Therefore, we propose the adoption of TCN for emotion detection problems as a baseline method for future work. Our experimental results show that TCN outperforms all RNN based models yielding a concordance correlation coefficient of 0.7895 (vs. 0.7544) on valence and 0.8207 (vs. 0.7357) on arousal on the validation dataset of SEWA dataset for emotion prediction.

Orientadores: José Antônio Brum, Marcus Aloizio Martinez de Aguiar<br>Dissertação (mestrado) - Universidade Estadual de Campinas, Instituto de Física Gleb Wataghin<br>Made available in DSpace on 2018-08-29T15:40:55Z (GMT). No. of bitstreams: 1 Reis_ElohimFonsecados_M.pdf: 2277988 bytes, checksum: 08f2c3b84a391217d575c0f425159fca (MD5) Previous issue date: 2015<br>Resumo: Este trabalho é dividido em duas partes. Na primeira parte, uma rede de correlação é construída baseada em um modelo de Ising em diferentes temperaturas, crítica, subcrítica e supercrítica, usando um algorítimo de Metropolis Monte-Carlo com dinâmica de \textit{single-spin-flip}. Este modelo teórico é comparado com uma rede do cérebro construída a partir de correlações das séries temporais do sinal BOLD de fMRI de regiões do cérebro. Medidas de rede, como coeficiente de aglomeração, mínimo caminho médio e distribuição de grau são analisadas. As mesmas medidas de rede são calculadas para a rede obtida pelas correlações das séries temporais dos spins no modelo de Ising. Os resultados da rede cerebral são melhor explicados pelo modelo teórico na temperatura crítica, sugerindo aspectos de criticalidade na dinâmica cerebral. Na segunda parte, é estudada a dinâmica temporal da atividade de um população neural, ou seja, a atividade de células ganglionares da retina gravadas em uma matriz de multi-eletrodos. Vários estudos têm focado em descrever a atividade de redes neurais usando modelos de Ising com desordem, não dando atenção à estrutura dinâmica. Tratando o tempo como uma dimensão extra do sistema, a dinâmica temporal da atividade da população neural é modelada. O princípio de máxima entropia é usado para construir um modelo de Ising com interação entre pares das atividades de diferentes neurônios em tempos diferentes. O ajuste do modelo é feito com uma combinação de amostragem de Monte-Carlo e método do gradiente descendente. O sistema é caracterizado pelos parâmetros aprendidos, questões como balanço detalhado e reversibilidade temporal são analisadas e variáveis termodinâmicas, como o calor específico, podem ser calculadas para estudar aspectos de criticalidade<br>Abstract: This work is divided in two parts. In the first part, a correlation network is build based on an Ising model at different temperatures, critical, subcritical and supercritical, using a Metropolis Monte-Carlo algorithm with single-spin-flip dynamics. This theoretical model is compared with a brain network built from the correlations of BOLD fMRI temporal series of brain regions activity. Network measures, such as clustering coefficient, average shortest path length and degree distributions are analysed. The same network measures are calculated to the network obtained from the time series correlations of the spins in the Ising model. The results from the brain network are better explained by the theoretical model at the critical temperature, suggesting critical aspects in the brain dynamics. In the second part, the temporal dynamics of the activity of a neuron population, that is, the activity of retinal ganglion cells recorded in a multi-electrode array was studied. Many studies have focused on describing the activity of neural networks using disordered Ising models, with no regard to the dynamic nature. Treating time as an extra dimension of the system, the temporal dynamics of the activity of the neuron population is modeled. The maximum entropy principle approach is used to build an Ising model with pairwise interactions between the activities of different neurons at different times. Model fitting is performed by a combination of Metropolis Monte Carlo sampling with gradient descent methods. The system is characterized by the learned parameters, questions like detailed balance and time reversibility are analysed and thermodynamic variables, such as specific heat, can be calculated to study critical aspects<br>Mestrado<br>Física<br>Mestre em Física<br>2013/25361-6<br>FAPESP

Neural networks have frequently been found to give accurate solutions to hard classification problems. However neural networks do not make explained classifications because the class boundaries are implicitly defined by the network weights, and these weights do not lend themselves to simple analysis. Explanation is desirable because it gives problem insight both to the designer and to the user of the classifier. Many methods have been suggested for explaining the classification given by a neural network, but they all suffer from one or more of the following disadvantages: a lack of equivalence between the network and the explanation; the absence of a probability framework required to express the uncertainty present in the data; a restriction to problems with binary or coarsely discretised features; reliance on axis-aligned rules, which are intrinsically poor at describing the boundaries generated by neural networks. The structure of the solution presented in this thesis rests on the following steps: Train a standard neural network to estimate the class conditional probabilities. Bayes’ rule then defines the optimal class boundaries. Obtain an explicit representation of these class boundaries using a piece-wise linearisation technique. Note that the class boundaries are otherwise only implicitly defined by the network weights. Obtain a safe but possibly partial description of this explicit representation using rules based upon the city-block distance to a prototype pattern. The methods required to achieve the last two represent novel work which seeks to explain the answers given by a proven neural network solution to the classification problem.

CIVINS<br>In many applications, ranging from character recognition to signal detection to automatic target identification, the problem of signal classification is of interest. Often, for example, a signal is known to belong to one of a family of sets C sub 1..., C sub n and the goal is to classify the signal according to the set to which it belongs. The main purpose of this thesis is to show that under certain conditions placed on the sets, the theory of uniform approximation can be applied to solve this problem. Specifically, if we assume that sets C sub j are compact subsets of a normed linear space, several approaches using the Stone-Weierstrass theorem give us a specific structure for classification. This structure is a single hidden layer feedforward neural network. We then discuss the functions which comprise the elements of this neural network and give an example of an application

Neuromorphic Engineering (NE) has led to the development of biologically-inspired computer architectures whose long-term goal is to approach the performance of the human brain in terms of energy efficiency and cognitive capabilities. Although there are a number of neuromorphic platforms available for large-scale Spiking Neural Network (SNN) simulations, the problem of programming these brain-like machines to be competent in cognitive applications still remains unsolved. On the other hand, Deep Learning has emerged in Artificial Neural Network (ANN) research to dominate state-of-the-art solutions for cognitive tasks. Thus the main research problem emerges of understanding how to operate and train biologically-plausible SNNs to close the gap in cognitive capabilities between SNNs and ANNs. SNNs can be trained by first training an equivalent ANN and then transferring the tuned weights to the SNN. This method is called ‘off-line’ training, since it does not take place on an SNN directly, but rather on an ANN instead. However, previous work on such off-line training methods has struggled in terms of poor modelling accuracy of the spiking neurons and high computational complexity. In this thesis we propose a simple and novel activation function, Noisy Softplus (NSP), to closely model the response firing activity of biologically-plausible spiking neurons, and introduce a generalised off-line training method using the Parametric Activation Function (PAF) to map the abstract numerical values of the ANN to concrete physical units, such as current and firing rate in the SNN. Based on this generalised training method and its fine tuning, we achieve the state-of-the-art accuracy on the MNIST classification task using spiking neurons, 99.07%, on a deep spiking convolutional neural network (ConvNet). We then take a step forward to ‘on-line’ training methods, where Deep Learning modules are trained purely on SNNs in an event-driven manner. Existing work has failed to provide SNNs with recognition accuracy equivalent to ANNs due to the lack of mathematical analysis. Thus we propose a formalised Spike-based Rate Multiplication (SRM) method which transforms the product of firing rates to the number of coincident spikes of a pair of rate-coded spike trains. Moreover, these coincident spikes can be captured by the Spike-Time-Dependent Plasticity (STDP) rule to update the weights between the neurons in an on-line, event-based, and biologically-plausible manner. Furthermore, we put forward solutions to reduce correlations between spike trains; thereby addressing the result of performance drop in on-line SNN training. The promising results of spiking Autoencoders (AEs) and Restricted Boltzmann Machines (SRBMs) exhibit equivalent, sometimes even superior, classification and reconstruction capabilities compared to their non-spiking counterparts. To provide meaningful comparisons between these proposed SNN models and other existing methods within this rapidly advancing field of NE, we propose a large dataset of spike-based visual stimuli and a corresponding evaluation methodology to estimate the overall performance of SNN models and their hardware implementations.

This thesis introduces the concept of an encoder-decoder neural network and develops architectures for the construction of such networks. Encoder-decoder neural networks are probabilistic conditional generative models of high-dimensional structured items such as natural language utterances and natural images. Encoder-decoder neural networks estimate a probability distribution over structured items belonging to a target set conditioned on structured items belonging to a source set. The distribution over structured items is factorized into a product of tractable conditional distributions over individual elements that compose the items. The networks estimate these conditional factors explicitly. We develop encoder-decoder neural networks for core tasks in natural language processing and natural image and video modelling. In Part I, we tackle the problem of sentence modelling and develop deep convolutional encoders to classify sentences; we extend these encoders to models of discourse. In Part II, we go beyond encoders to study the longstanding problem of translating from one human language to another. We lay the foundations of neural machine translation, a novel approach that views the entire translation process as a single encoder-decoder neural network. We propose a beam search procedure to search over the outputs of the decoder to produce a likely translation in the target language. Besides known recurrent decoders, we also propose a decoder architecture based solely on convolutional layers. Since the publication of these new foundations for machine translation in 2013, encoder-decoder translation models have been richly developed and have displaced traditional translation systems both in academic research and in large-scale industrial deployment. In services such as Google Translate these models process in the order of a billion translation queries a day. In Part III, we shift from the linguistic domain to the visual one to study distributions over natural images and videos. We describe two- and three- dimensional recurrent and convolutional decoder architectures and address the longstanding problem of learning a tractable distribution over high-dimensional natural images and videos, where the likely samples from the distribution are visually coherent. The empirical validation of encoder-decoder neural networks as state-of- the-art models of tasks ranging from machine translation to video prediction has a two-fold significance. On the one hand, it validates the notions of assigning probabilities to sentences or images and of learning a distribution over a natural language or a domain of natural images; it shows that a probabilistic principle of compositionality, whereby a high- dimensional item is composed from individual elements at the encoder side and whereby a corresponding item is decomposed into conditional factors over individual elements at the decoder side, is a general method for modelling cognition involving high-dimensional items; and it suggests that the relations between the elements are best learnt in an end-to-end fashion as non-linear functions in distributed space. On the other hand, the empirical success of the networks on the tasks characterizes the underlying cognitive processes themselves: a cognitive process as complex as translating from one language to another that takes a human a few seconds to perform correctly can be accurately modelled via a learnt non-linear deterministic function of distributed vectors in high-dimensional space.

We extend the in-situ Hebbian-linked SOMs network by Miikkulainen to come up with two unsupervised neural networks that learn the mapping between the individual modes of a multimodal dataset. The first network, the single-pass Hebbian linked SOMs network, extends the in-situ Hebbian-linked SOMs network by enabling the Hebbian link weights to be computed through one- shot learning. The second network, a modified counter propagation network, extends the unsupervised learning of crossmodal mappings by making it possible for only one self-organising map to implement the crossmodal mapping. The two proposed networks each have a smaller computation time and achieve lower crossmodal mean squared errors than the in-situ Hebbian- linked SOMs network when assessed on two bimodal datasets, an audio-acoustic speech utterance dataset and a phonological-semantics child utterance dataset. Of the three network architectures, the modified counterpropagation network achieves the highest percentage of correct classifications comparable to that of the LVQ-2 algorithm by Kohonen and the neural network for category learning by de Sa and Ballard in classification tasks using the audio-acoustic speech utterance dataset. To facilitate multimodal processing of temporal data, we propose a Temporal Hypermap neural network architecture that learns and recalls multiple temporal patterns in an unsupervised manner. The Temporal Hypermap introduces flexibility in the recall of temporal patterns - a stored temporal pattern can be retrieved by prompting the network with the temporal pattern's identity vector, whilst the incorporation of short term memory allows the recall of a temporal pattern, starting from the pattern item specified by contextual information up to the last item in the pattern sequence. Finally, we extend the connectionist modelling of child language acquisition in two important respects. First, we introduce the concept of multimodal representation of speech utterances at the one-word and two-word stage. This allows us to model child language at the one-word utterance stage with a single modified counterpropagation network, which is an improvement on previous models in which multiple networks are required to simulate the different aspects of speech at the one-word utterance stage. Secondly, we present, for the time, a connectionist model of the transition of child language from the one-word utterance stage to the two-word utterance stage. We achieve this using a gated multi-net comprising a modified counterpropagation network and a Temporal Hypermap.

Includes abstract.<br>Includes bibliographical references (leaves 46-49).<br>In this research, the salience of an entity refers to its state or quality of standing out, or receiving increased attention, relative to neighboring entities. By neighbouring entities we refer to both spatial (i.e. similar visual objects) and temporal (i.e. related concepts). In this research we model the effect of non-local connections using an ANN, creating a salience-affected neural network (SANN). We adapt an ANN to embody the capacity to respond to an input salience signal and to produce a reverse salience signal during testing. The input salience signal applied during training to each node has the effect of varying the node’s thresholds, depending on the activation level of the node. Each node produces a nodal reverse salience signal during testing (a measure of the threshold bias for the individual node). The reverse salience signal is defined as the summation of the nodal reverse salience signals observed at each node.

Significant breakthroughs in the last decade in the Machine Learning (ML) field have ushered in a new era of Artificial Intelligence (AI). Particularly, recent advances in Deep Learning (DL) have enabled to develop a new breed of modeling and optimization tools with a plethora of applications in different fields like natural language processing, or computer vision. In this context, the Knowledge-Defined Networking (KDN) paradigm highlights the lack of adoption of AI techniques in computer networks and – as a result – proposes a novel architecture that relies on Software-Defined Networking (SDN) and modern network analytics techniques to facilitate the deployment of ML-based solutions for efficient network operation. This dissertation aims to be a step forward in the realization of Knowledge-Defined Networks. In particular, we focus on the application of AI techniques to control and optimize networks more efficiently and automatically. To this end, we identify two components within the KDN context whose development may be crucial to achieve self-operating networks in the future: (i) the automatic control module, and (ii) the network analytics platform. The first part of this thesis is devoted to the construction of efficient automatic control modules. First, we explore the application of Deep Reinforcement Learning (DRL) algorithms to optimize the routing configuration in networks. DRL has recently demonstrated an outstanding capability to solve efficiently decision-making problems in other fields. However, first DRL-based attempts to optimize routing in networks have failed to achieve good results, often under-performing traditional heuristics. In contrast to previous DRL-based solutions, we propose a more elaborate network representation that facilitates DRL agents to learn efficient routing strategies. Our evaluation results show that DRL agents using the proposed representation achieve better performance and learn faster how to route traffic in an Optical Transport Network (OTN) use case. Second, we lay the foundations on the use of Graph Neural Networks (GNN) to build ML-based network optimization tools. GNNs are a newly proposed family of DL models specifically tailored to operate and generalize over graphs of variable size and structure. In this thesis, we posit that GNNs are well suited to model the relationships between different network elements inherently represented as graphs (e.g., topology, routing). Particularly, we use a custom GNN architecture to build a routing optimization solution that – unlike previous ML-based proposals – is able to generalize well to topologies, routing configurations, and traffic never seen during the training phase. The second part of this thesis investigates the design of practical and efficient network analytics solutions in the KDN context. Network analytics tools are crucial to provide the control plane with a rich and timely view of the network state. However this is not a trivial task considering that all this information turns typically into big data in real-world networks. In this context, we analyze the main aspects that should be considered when measuring and classifying traffic in SDN (e.g., scalability, accuracy, cost). As a result, we propose a practical solution that produces flow-level measurement reports similar to those of NetFlow/IPFIX in traditional networks. The proposed system relies only on native features of OpenFlow – currently among the most established standards in SDN – and incorporates mechanisms to maintain efficiently flow-level statistics in commodity switches and report them asynchronously to the control plane. Additionally, a system that combines ML and Deep Packet Inspection (DPI) identifies the applications that generate each traffic flow.<br>La evolución del campo del Aprendizaje Maquina (ML) en la última década ha dado lugar a una nueva era de la Inteligencia Artificial (AI). En concreto, algunos avances en el campo del Aprendizaje Profundo (DL) han permitido desarrollar nuevas herramientas de modelado y optimización con múltiples aplicaciones en campos como el procesado de lenguaje natural, o la visión artificial. En este contexto, el paradigma de Redes Definidas por Conocimiento (KDN) destaca la falta de adopción de técnicas de AI en redes y, como resultado, propone una nueva arquitectura basada en Redes Definidas por Software (SDN) y en técnicas modernas de análisis de red para facilitar el despliegue de soluciones basadas en ML. Esta tesis pretende representar un avance en la realización de redes basadas en KDN. En particular, investiga la aplicación de técnicas de AI para operar las redes de forma más eficiente y automática. Para ello, identificamos dos componentes en el contexto de KDN cuyo desarrollo puede resultar esencial para conseguir redes operadas autónomamente en el futuro: (i) el módulo de control automático y (ii) la plataforma de análisis de red. La primera parte de esta tesis aborda la construcción del módulo de control automático. En primer lugar, se explora el uso de algoritmos de Aprendizaje Profundo por Refuerzo (DRL) para optimizar el encaminamiento de tráfico en redes. DRL ha demostrado una capacidad sobresaliente para resolver problemas de toma de decisiones en otros campos. Sin embargo, los primeros trabajos que han aplicado DRL a la optimización del encaminamiento en redes no han conseguido rendimientos satisfactorios. Frente a dichas soluciones previas, proponemos una representación más elaborada de la red que facilita a los agentes DRL aprender estrategias de encaminamiento eficientes. Nuestra evaluación muestra que cuando los agentes DRL utilizan la representación propuesta logran mayor rendimiento y aprenden más rápido cómo encaminar el tráfico en un caso práctico en Redes de Transporte Ópticas (OTN). En segundo lugar, se presentan las bases sobre la utilización de Redes Neuronales de Grafos (GNN) para construir herramientas de optimización de red. Las GNN constituyen una nueva familia de modelos de DL específicamente diseñados para operar y generalizar sobre grafos de tamaño y estructura variables. Esta tesis destaca la idoneidad de las GNN para modelar las relaciones entre diferentes elementos de red que se representan intrínsecamente como grafos (p. ej., topología, encaminamiento). En particular, utilizamos una arquitectura GNN específicamente diseñada para optimizar el encaminamiento de tráfico que, a diferencia de las propuestas anteriores basadas en ML, es capaz de generalizar correctamente sobre topologías, configuraciones de encaminamiento y tráfico nunca vistos durante el entrenamiento La segunda parte de esta tesis investiga el diseño de herramientas de análisis de red eficientes en el contexto de KDN. El análisis de red resulta esencial para proporcionar al plano de control una visión completa y actualizada del estado de la red. No obstante, esto no es una tarea trivial considerando que esta información representa una cantidad masiva de datos en despliegues de red reales. Esta parte de la tesis analiza los principales aspectos a considerar a la hora de medir y clasificar el tráfico en SDN (p. ej., escalabilidad, exactitud, coste). Como resultado, se propone una solución práctica que genera informes de medidas de tráfico a nivel de flujo similares a los de NetFlow/IPFIX en redes tradicionales. El sistema propuesto utiliza sólo funciones soportadas por OpenFlow, actualmente uno de los estándares más consolidados en SDN, y permite mantener de forma eficiente estadísticas de tráfico en conmutadores con características básicas y enviarlas de forma asíncrona hacia el plano de control. Asimismo, un sistema que combina ML e Inspección Profunda de Paquetes (DPI) identifica las aplicaciones que generan cada flujo de tráfico.

Performance of neural network based restoration of optical networks is evaluated and a few possible improvements are proposed. Neural network based restoration is simulated with optical link capacities assigned by a new method. Two new improvement methods are developed to reduce the neural network size and the restoration time of severed optical connections. Cycle based restoration is suggested, which reduces the neural network structure by restoring the severed connections for each optical node, iteratively. Additionally, to reduce the restoration time, the necessary waiting time before the neuron outputs fire for the propagation delay over the network is computed and embedded in the control structure of the neural network. The improvement methods are evaluated by simulations in terms of restorability, restoration time, network redundancy and average length of restoration paths for different failure cases and different security requirements.

The growing usage of computer networks is requiring improvements in network technologies and management techniques so users will receive high quality service. As more individuals transmit data through a computer network, the quality of service received by the users begins to degrade. A major aspect of computer networks that is vital to quality of service is data routing. A more effective method for routing data through a computer network can assist with the new problems being encountered with today's growing networks. Effective routing algorithms use various techniques to determine the most appropriate route for transmitting data. Determining the best route through a wide area network (WAN), requires the routing algorithm to obtain information concerning all of the nodes, links, and devices present on the network. The most relevant routing information involves various measures that are often obtained in an imprecise or inaccurate manner, thus suggesting that fuzzy reasoning is a natural method to employ in an improved routing scheme. The neural network is deemed as a suitable accompaniment because it maintains the ability to learn in dynamic situations. Once the neural network is initially designed, any alterations in the computer routing environment can easily be learned by this adaptive artificial intelligence method. The capability to learn and adapt is essential in today's rapidly growing and changing computer networks. These techniques, fuzzy reasoning and neural networks, when combined together provide a very effective routing algorithm for computer networks. Computer simulation is employed to prove the new fuzzy routing algorithm outperforms the Shortest Path First (SPF) algorithm in most computer network situations. The benefits increase as the computer network migrates from a stable network to a more variable one. The advantages of applying this fuzzy routing algorithm are apparent when considering the dynamic nature of modern computer networks.<br>Ph. D.

Networks of spiking neurons can be used not only for brain modeling but also to solve graph problems. With the use of a computationally efficient Izhikevich neuron model combined with plasticity rules, the networks possess self-organizing characteristics. Two different time-based synaptic plasticity rules are used to adjust weights among nodes in a graph resulting in solutions to graph prob- lems such as finding the shortest path and clustering.

Thesis (Ph.D.)--University of Nebraska-Lincoln, 2005.<br>Title from title screen (sites viewed on Oct. 18, 2005). PDF text: 100 p. : col. ill. Includes bibliographical references (p. 95-100 of dissertation).

Le Reti Neurali, e in particolare le Reti Neurali Convoluzionali, hanno recentemente dimostrato risultati straordinari in vari campi. Purtroppo, comunque, non vi è ancora una chiara comprensione del perchè queste architetture funzionino così bene e soprattutto è difficile spiegare il comportamento nel caso di fallimenti. Questa mancanza di chiarezza è quello che separa questi modelli dall’essere applicati in scenari concreti e critici della vita reale, come la sanità o le auto a guida autonoma. Per questa ragione, durante gli ultimi anni sono stati portati avanti diversi studi in modo tale da creare metodi che siano capaci di spiegare al meglio cosa sta succedendo dentro una rete neurale oppure dove la rete sta guardando per predire in un certo modo. Proprio queste tecniche sono il centro di questa tesi e il ponte tra i due casi di studio che sono presentati sotto. Lo scopo di questo lavoro è quindi duplice: per prima cosa, usare questi metodi per analizzare e quindi capire come migliorare applicazioni basate su reti neurali convoluzionali e in secondo luogo, per investigare la capacità di generalizzazione di queste architetture, sempre grazie a questi metodi.

Evolutionary computation has been around ever since the late 50s. This thesis aims at elaborate on genetic algorithms, a subset of evolutionary computation, with particular regard to the field of neuroevolution, which is the application of GAs to the generation of functioning neural networks. The most widely adopted techniques are thereby explained and contrasted. The experimentation chapter finally shows an implementation of a genetic algorithm, inspired by existing algorithms, with the objective of optimizing a novel kind of artificial neural network.

Deep learning is the most effective and used approach to artificial intelligence, and yet it is far from being properly understood. The understanding of it is the way to go to further improve its effectiveness and in the best case to gain some understanding of the "natural" intelligence. We attempt a step in this direction with the aim of physics. We describe a convolutional neural network for image classification (trained on CIFAR-10) within the descriptive framework of Thermodynamics. In particular we define and study the temperature of each component of the network. Our results provides a new point of view on deep learning models, which may be a starting point towards a better understanding of artificial intelligence.

Deep Convolutional Neural Networks and "deep learning" in general stand at the cutting edge on a range of applications, from image based recognition and classification to natural language processing, speech and speaker recognition and reinforcement learning. Very deep models however are often large, complex and computationally expensive to train and evaluate. Deep learning models are thus seldom deployed natively in environments where computational resources are scarce or expensive. To address this problem we turn our attention towards a range of techniques that we collectively refer to as "model compression" where a lighter student model is trained to approximate the output produced by the model we wish to compress. To this end, the output from the original model is used to craft the training labels of the smaller student model. This work contains some experiments on CIFAR-10 and demonstrates how to use the aforementioned techniques to compress a people counting model whose precision, recall and F1-score are improved by as much as 14% against our baseline.

Stock trading is increasingly done pseudo-automatically orfully automatically, using algorithms which make day-todayor even moment-to-moment decisions.This report investigates the possibility of creating a virtualstock trader, using a method used in Artificial Intelligence,called Neural Networks, to make intelligent decisions onwhen to buy and sell stocks on the stock market.We found that it might be possible to earn money overa longer period of time, although the profit is less than theaverage stock index. However, the method also performedwell in situations where the stock index is going down.<br>Aktiehandel genomförs till allt större grad automatiskt ellerhalvautomatiskt, med algoritmer som fattar beslut pådaglig basis eller över ännu kortare tidsintervall.Denna rapport undersöker möjligheten att göra en virtuellaktiehandlare med hjälp av en metod inom artificiellintelligens kallad neurala nätverk, och fatta intelligenta beslutom när aktier på aktiemarknaden ska köpas eller säljas.Vi fann att det är möjligt att tjäna pengar över en längretidsperiod, men vinsten vår algoritm gör över den behandladetidsperioden är mindre än börsindex ökning. Däremotvisar vår algoritm positiva resultat även under sjunkandebörsindex.

Barcodes are ubiquituous in modern society and they have had industrial application for decades. However, for noisy images modern methods can underperform. Poor lighting conditions, occlusions and low resolution can be problematic in decoding. This thesis aims to solve this problem by using neural networks, which have enjoyed great success in many computer vision competitions the last years. We investigate how three different networks perform on data sets with noisy images. The first network is a single classifier, the second network is an ensemble classifier and the third is based on a pre-trained feature extractor. For comparison, we also test two baseline methods that are used in industry today. We generate training data using software and modify it to ensure proper generalization. Testing data is created by photographing barcodes in different settings, creating six image classes - normal, dark, white, rotated, occluded and wrinkled. The proposed single classifier and ensemble classifier outperform the baseline as well as the pre-trained feature extractor by a large margin. The thesis work was performed at SICK IVP, a machine vision company in Linköping in 2017.

Neural networks need to be able to guarantee their intrinsic generalisation abilities if they are to be used reliably. Mitchell's concept and version spaces technique is able to guarantee generalisation in the symbolic concept-learning environment in which it is implemented. Generalisation, according to Mitchell, is guaranteed when there is no alternative concept that is consistent with all the examples presented so far, except the current concept, given the bias of the user. A form of bidirectional convergence is used by Mitchell to recognise when the no-alternative situation has been reached. Mitchell's technique has problems of search and storage feasibility in its symbolic environment. This thesis aims to show that by evolving the technique further in a neural environment, these problems can be overcome. Firstly, the biasing factors which affect the kind of concept that can be learned are explored in a neural network context. Secondly, approaches for abstracting the underlying features of the symbolic technique that enable recognition of the no-alternative situation are discussed. The discussion generates neural techniques for guaranteeing generalisation and culminates in a neural technique which is able to recognise when the best fit neural weight state has been found for a given set of data and topology.

Data mining is about the search for relationships and global patterns in large databases that are increasing in size. Data mining is beneficial for anyone who has a huge amount of data, for example, customer and business data, transaction, marketing, financial, manufacturing and web data etc. The results of data mining are also referred to as knowledge in the form of rules, regularities and constraints. Rule mining is one of the popular data mining methods since rules provide concise statements of potentially important information that is easily understood by end users and also actionable patterns. At present rule mining has received a good deal of attention and enthusiasm from data mining researchers since rule mining is capable of solving many data mining problems such as classification, association, customer profiling, summarization, segmentation and many others. This thesis makes several contributions by proposing rule mining methods using genetic algorithms and neural networks. The thesis first proposes rule mining methods using a genetic algorithm. These methods are based on an integrated framework but capable of mining three major classes of rules. Moreover, the rule mining processes in these methods are controlled by tuning of two data mining measures such as support and confidence. The thesis shows how to build data mining predictive models using the resultant rules of the proposed methods. Another key contribution of the thesis is the proposal of rule mining methods using supervised neural networks. The thesis mathematically analyses the Widrow-Hoff learning algorithm of a single-layered neural network, which results in a foundation for rule mining algorithms using single-layered neural networks. Three rule mining algorithms using single-layered neural networks are proposed for the three major classes of rules on the basis of the proposed theorems. The thesis also looks at the problem of rule mining where user guidance is absent. The thesis proposes a guided rule mining system to overcome this problem. The thesis extends this work further by comparing the performance of the algorithm used in the proposed guided rule mining system with Apriori data mining algorithm. Finally, the thesis studies the Kohonen self-organization map as an unsupervised neural network for rule mining algorithms. Two approaches are adopted based on the way of self-organization maps applied in rule mining models. In the first approach, self-organization map is used for clustering, which provides class information to the rule mining process. In the second approach, automated rule mining takes the place of trained neurons as it grows in a hierarchical structure.

The basis of this thesis is the presumption that while neural networks are useful structures that can be used to model complex, highly non-linear systems, current methods of training the neural networks are inadequate in some problem domains. Genetic algorithms have been used to optimise both the weights and architectures of neural networks, but these approaches do not treat the neural network in a sensible manner. In this thesis, I define the basis of computation within a neural network as a single neuron and its associated input connections. Sets of these neurons, stored in a matrix representation, comprise the building blocks that are transferred during one or more epochs of a genetic algorithm. I develop the concept of a Neural Building Block and two new genetic algorithms are created that utilise this concept. The first genetic algorithm utilises the micro neural building block (micro-NBB); a unit consisting of one or more neurons and their input connections. The micro-NBB is a unit that is transmitted through the process of crossover and hence requires the introduction of a new crossover operator. However the micro NBB can not be stored as a reusable component and must exist only as the product of the crossover operator. The macro neural building block (macro-NBB) is utilised in the second genetic algorithm, and encapsulates the idea that fit neural networks contain fit sub-networks, that need to be preserved across multiple epochs. A macro-NBB is a micro-NBB that exists across multiple epochs. Macro-NBBs must exist across multiple epochs, and this necessitates the use of a genetic store, and a new operator to introduce macro-NBBs back into the population at random intervals. Once the theoretical presentation is completed the newly developed genetic algorithms are used to evolve weights for a variety of architectures of neural networks to demonstrate the feasibility of the approach. Comparison of the new genetic algorithm with other approaches is very favourable on two problems: a multiplexer problem and a robot control problem.