Academic literature on the topic 'LSTM ALGORITHM'

LSTM neural networks have been used for speech recognition, image recognition and other artificial intelligence applications for many years. Most applications perform the LSTM algorithm and the required calculations on cloud computers. Off-line solutions include the use of FPGAs and GPUs but the most promising solutions include ASIC accelerators designed for this purpose only. This report presents an ASIC design capable of performing the multiple iterations of the LSTM algorithm on a unidirectional and without peepholes neural network architecture. The proposed design provides arithmetic level parallelism options as blocks are instantiated based on parameters. The internal structure of the design implements pipelined, parallel or serial solutions depending on which is optimal in every case. The implications concerning these decisions are discussed in detail in the report. The design process is described in detail and the evaluation of the design is also presented to measure accuracy and error of the design output.This thesis work resulted in a complete synthesizable ASIC design implementing an LSTM layer, a Fully Connected layer and a Softmax layer which can perform classification of data based on trained weight matrices and bias vectors. The design primarily uses 16-bit fixed point format with 5 integer and 11 fractional bits but increased precision representations are used in some blocks to reduce error output. Additionally, a verification environment has also been designed and is capable of performing simulations, evaluating the design output by comparing it with results produced from performing the same operations with 64-bit floating point precision on a SystemVerilog testbench and measuring the encountered error. The results concerning the accuracy and the design output error margin are presented in this thesis report. The design went through Logic and Physical synthesis and successfully resulted in a functional netlist for every tested configuration. Timing, area and power measurements on the generated netlists of various configurations of the design show consistency and are reported in this report.<br>LSTM neurala nätverk har använts för taligenkänning, bildigenkänning och andra artificiella intelligensapplikationer i många år. De flesta applikationer utför LSTM-algoritmen och de nödvändiga beräkningarna i digitala moln. Offline lösningar inkluderar användningen av FPGA och GPU men de mest lovande lösningarna inkluderar ASIC-acceleratorer utformade för endast dettaändamål. Denna rapport presenterar en ASIC-design som kan utföra multipla iterationer av LSTM-algoritmen på en enkelriktad neural nätverksarkitetur utan peepholes. Den föreslagna designed ger aritmetrisk nivå-parallellismalternativ som block som är instansierat baserat på parametrar. Designens inre konstruktion implementerar pipelinerade, parallella, eller seriella lösningar beroende på vilket anternativ som är optimalt till alla fall. Konsekvenserna för dessa beslut diskuteras i detalj i rapporten. Designprocessen beskrivs i detalj och utvärderingen av designen presenteras också för att mäta noggrannheten och felmarginal i designutgången. Resultatet av arbetet från denna rapport är en fullständig syntetiserbar ASIC design som har implementerat ett LSTM-lager, ett fullständigt anslutet lager och ett Softmax-lager som kan utföra klassificering av data baserat på tränade viktmatriser och biasvektorer. Designen använder huvudsakligen 16bitars fast flytpunktsformat med 5 heltal och 11 fraktions bitar men ökade precisionsrepresentationer används i vissa block för att minska felmarginal. Till detta har även en verifieringsmiljö utformats som kan utföra simuleringar, utvärdera designresultatet genom att jämföra det med resultatet som produceras från att utföra samma operationer med 64-bitars flytpunktsprecision på en SystemVerilog testbänk och mäta uppstådda felmarginal. Resultaten avseende noggrannheten och designutgångens felmarginal presenteras i denna rapport.Designen gick genom Logisk och Fysisk syntes och framgångsrikt resulterade i en funktionell nätlista för varje testad konfiguration. Timing, area och effektmätningar på den genererade nätlistorna av olika konfigurationer av designen visar konsistens och rapporteras i denna rapport.

The success of Internet of Things solutions allowed the establishment of new applications such as smart hydroponic agriculture. One typical problem in such an application is the rapid degradation of the deployed sensors. Traditionally, this problem is resolved by frequent manual maintenance, which is considered to be ineffective and may harm the crops in the long run. The main purpose of this thesis was to propose a machine learning approach for automating the detection of sensor fault drifts. In addition, the solution’s operability was investigated in a cloud computing environment in terms of the response time. This thesis proposes a detection algorithm that utilizes RNN in predicting sensor drifts from time-series data streams. The detection algorithm was later named; Predictive Sliding Detection Window (PSDW) and consisted of both forecasting and classification models. Three different RNN algorithms, i.e., LSTM, CNN-LSTM, and GRU, were designed to predict sensor drifts using forecasting and classification techniques. The algorithms were compared against each other in terms of relevant accuracy metrics for forecasting and classification. The operability of the solution was investigated by developing a web server that hosted the PSDW algorithm on an AWS computing instance. The resulting forecasting and classification algorithms were able to make reasonably accurate predictions for this particular scenario. More specifically, the forecasting algorithms acquired relatively low RMSE values as ~0.6, while the classification algorithms obtained an average F1-score and accuracy of ~80% but with a high standard deviation. However, the response time was ~5700% slower during the simulation of the HTTP requests. The obtained results suggest the need for future investigations to improve the accuracy of the models and experiment with other computing paradigms for more reliable deployments.

This thesis deals with the issue of automatic measurement of the duration of QRS complexes in ECG signals. Special emphasis is then placed on the possibility of automatic detection of QRS complexes while exciting cardiac tissue with a pacemaker. The content of this work is divided into four logical units, where the first part deals with the heart as an organ. It describes the origin and spread of excitement in the heart, its possible pathologies and their manifestations in ECG recording, it also deals with pacing and measuring ECG recording during simultaneous pacing. The second part of the thesis contains a brief introduction to the topic of machine and deep learning. The third part of the thesis contains a search of current approaches using methods based on deep learning to solve the detection of QRSd. The fourth part deals with the design and implementation of its own model of deep learning, able to detect the beginnings and ends of QRS complexes from ECG recordings. It describes the data preprocessing implemented in the MATLAB programming environment. The actual implementation of the model was performed in the Python using the PyTorch and NumPy moduls.

A cheap and powerful solution to lower the electricity usage and making the residents more energy aware in a home is to simply make the residents aware of what appliances that are consuming electricity. Meaning the residents can then take decisions to turn them off in order to save energy. Non-intrusive load monitoring (NILM) is a cost-effective solution to identify different appliances based on their unique load signatures by only measuring the energy consumption at a single sensing point. In this thesis, a low-cost hardware platform is developed with the help of an Arduino to collect consumption signatures in real time, with the help of a single CT-sensor. Three different algorithms and one recurrent neural network are implemented with Python to find out which of them is the most suited for this kind of work. The tested algorithms are k-Nearest Neighbors, Random Forest and Decision Tree Classifier and the recurrent neural network is Long short-term memory.

Automatically detecting anomalies in text is of great interest for surveillance entities as vast amounts of data can be analysed to find suspicious activity. In this thesis, three distinct machine learning algorithms are evaluated as a chat message classifier is being implemented for the purpose of market surveillance. Naive Bayes and Support Vector Machine belong to the statistical class of machine learning algorithms being evaluated in this thesis and both require feature selection, a side objective of the thesis is thus to find a suitable feature selection technique to ensure mentioned algorithms achieve high performance. Long Short-Term Memory network is the deep learning algorithm being evaluated in the thesis, rather than depend on feature selection, the deep neural network will be evaluated as it is trained using word embeddings. Each of the algorithms achieved high performance but the findings ofthe thesis suggest Naive Bayes algorithm in conjunction with a feature counting feature selection technique is the most suitable choice for this particular learning problem.<br>Att automatiskt kunna upptäcka anomalier i text har stora implikationer för företag och myndigheter som övervakar olika sorters kommunikation. I detta examensarbete utvärderas tre olika maskininlärningsalgoritmer för chattmeddelandeklassifikation i ett marknadsövervakningsystem. Naive Bayes och Support Vector Machine tillhör båda den statistiska klassen av maskininlärningsalgoritmer som utvärderas i studien och bådar kräver selektion av vilka särdrag i texten som ska användas i algoritmen. Ett sekundärt mål med studien är således att hitta en passande selektionsteknik för att de statistiska algoritmerna ska prestera så bra som möjligt. Long Short-Term Memory Network är djupinlärningsalgoritmen som utvärderas i studien. Istället för att använda en selektionsteknik kommer djupinlärningsalgoritmen nyttja ordvektorer för att representera text. Resultaten visar att alla utvärderade algoritmer kan nå hög prestanda för ändamålet, i synnerhet Naive Bayes tillsammans med termfrekvensselektion.

Time series prediction is one of the main areas of statistics and machine learning. In 2018 the two new algorithms higher order hidden Markov model and temporal convolutional network were proposed and emerged as challengers to the more traditional recurrent neural network and long-short term memory network as well as the autoregressive integrated moving average (ARIMA). In this study most major algorithms together with recent innovations for time series forecasting is trained and evaluated on two datasets from the theme park industry with the aim of predicting future number of visitors. To develop models, Python libraries Keras and Statsmodels were used. Results from this thesis show that the neural network models are slightly better than ARIMA and the hidden Markov model, and that the temporal convolutional network do not perform significantly better than the recurrent or long-short term memory networks although having the lowest prediction error on one of the datasets. Interestingly, the Markov model performed worse than all neural network models even when using no independent variables.

Actualmente las operaciones de mantenimiento preventivo de los parques eólicos se soportan sobre técnicas de Machine Learning para reducir los costes de las paradas no planificadas. Por eso se necesita una predicción de fallos con cierta anticipación que funcione sobre los datos de SCADA. Estos datos necesitan ser procesados en distintas etapas descritas en esta tesis, con resultados publicados en cada una de ellas. En una primera fase se limpian los valores extremos (Outliers), indicando cómo deben ser tratados para no eliminar la información sobre los fallos. En una segunda, las distintas variables son seleccionadas por diversos métodos de selección de características (Feature Selection). En la misma fase, se compara el uso de variables transformadas mediante Autoencoders. En una tercera se construye el modelo, mediante métodos supervisados y no supervisados, obteniendo resultados destacables con Self Organizing Maps (SOM) y con técnicas de Deep Learning incluyendo redes ANN y LSTM multicapa.<br>Nowadays, the preventive maintenance operations of wind farms are supported by Machine Learning techniques to reduce the costs of unplanned downtime. That is why an early fault prediction that works with SCADA data is required. These data need to be processed at different stages described in this thesis, with results published in each of them. In a first phase, the extreme values (Outliers) are cleaned, indicating how they should address in order not to eliminate the information about the faults. In a second step, the different variables are selected by different Feature Selection methods. At the same step, the use of variables transformed by Autoencoders is also compared. In a third, the model is constructed using Supervised and Unsupervised methods, obtaining outstanding results with Self Organizing Maps (SOM) and Deep Learning techniques including ANN and LSTM multi-layer networks.<br>Actualment les operacions de manteniment preventiu dels parcs eòlics se suporten sobre tècniques de Machine Learning per a reduir els costos de les parades no planificades. Per això es necessita una predicció de fallades amb certa anticipació que funcioni sobre les dades de SCADA. Aquestes dades necessiten ser processades en diferents etapes descrites a aquesta tesi, amb resultats publicats en cadascuna d'elles. En una primera fase es netegen els valors extrems (Outliers), indicant com han de ser tractats per no eliminar la informació sobre les fallades. En una segona, les diferents variables són seleccionades per diversos mètodes de selecció de característiques (Feature Selection). En la mateixa fase, es compara l'ús de variables transformades mitjançant Autoencoders. En una tercera es construeix el model, mitjançant mètodes supervisats i no supervisats, obtenint resultats destacables amb Self Organizing Maps (SOM) i amb tècniques de Deep Learning incloent xarxes ANN i LSTM multicapa.

Predicting the behavior of financial markets is largely an unsolved problem. The problem hasbeen approached with many different methods ranging from binary logic, statisticalcalculations and genetic algorithms. In this thesis, the problem is approached with a machinelearning method, namely the Long Short-Term Memory (LSTM) variant of Recurrent NeuralNetworks (RNNs). Recurrent neural networks are artificial neural networks (ANNs)—amachine learning algorithm mimicking the neural processing of the mammalian nervoussystem—specifically designed for time series sequences. The thesis investigates the capabilityof the LSTM in modeling financial market behavior as well as compare it to the traditionalRNN, evaluating their performances using various measures.<br>Prediktion av den finansiella marknadens beteende är i stort ett olöst problem. Problemet hartagits an på flera sätt med olika metoder så som binär logik, statistiska uträkningar ochgenetiska algoritmer. I den här uppsatsen kommer problemet undersökas medmaskininlärning, mer specifikt Long Short-Term Memory (LSTM), en variant av rekurrentaneurala nätverk (RNN). Rekurrenta neurala nätverk är en typ av artificiellt neuralt nätverk(ANN), en maskininlärningsalgoritm som ska efterlikna de neurala processerna hos däggdjursnervsystem, specifikt utformat för tidsserier. I uppsatsen undersöks kapaciteten hos ett LSTMatt modellera finansmarknadens beteenden och jämförs den mot ett traditionellt RNN, merspecifikt mäts deras effektivitet på olika vis.

The demand for robots to work in environments together with humans is growing. This calls for new requirements on robots systems, such as the need to be perceived as responsive and accurate in human interactions. This thesis explores the possibility of using AI methods to predict the movement of a human and evaluating if that information can assist a robot with human interactions. The AI methods that were used is a Long Short Term Memory(LSTM) network and an artificial neural network(ANN). Both networks were trained on data from a motion capture dataset and on four different prediction times: 1/2, 1/4, 1/8 and a 1/16 second. The evaluation was performed directly on the dataset to determine the prediction error. The neural networks were also evaluated on a robotic arm in a simulated environment, to show if the prediction methods would be suitable for a real-life system. Both methods show promising results when comparing the prediction error. From the simulated system, it could be concluded that with the LSTM prediction the robotic arm would generally precede the actual position. The results indicate that the methods described in this thesis report could be used as a stepping stone for a human-robot interactive system.