Dissertations / Theses on the topic 'Network anomally'

This semester project presents an analysis of network traffic and detection of anomalies in network traffic by several various means. In the first part of the paper there is an explanation of the methods aiming at denial of service. Then in the second part an implementation of protection by means of selected solutions is presented. The intent is to compare these means which are supposed to detect cyber attacks aiming at denial of service. Another intent is to choose the best solutions from the categories of open-source and commercial solutions. The target of the master thesis was to work out a comparison between actual solutions for detection of DoS and DDoS attacks.

Les systèmes SCADA sont de plus en plus ciblés par les cyberattaques en raison de nombreuses vulnérabilités dans le matériel, les logiciels, les protocoles et la pile de communication. Ces systèmes utilisent aujourd'hui du matériel, des logiciels, des systèmes d'exploitation et des protocoles standard. De plus, les systèmes SCADA qui étaient auparavant isolés sont désormais interconnectés aux réseaux d'entreprise et à Internet, élargissant ainsi la surface d'attaque. Dans cette thèse, nous utilisons une approche deep learning pour proposer un réseau de neurones profonds hybride efficace pour la détection d'anomalies dans les systèmes SCADA. Les principales caractéristiques des données SCADA sont apprises de manière automatique et non supervisée, puis transmises à un classificateur supervisé afin de déterminer si ces données sont normales ou anormales, c'est-à-dire s'il y a une cyber-attaque ou non. Par la suite, en réponse au défi dû au temps d’entraînement élevé des modèles deep learning, nous avons proposé une approche distribuée de notre système de détection d'anomalies afin de réduire le temps d’entraînement de notre modèle<br>SCADA systems are more and more targeted by cyber-attacks because of many vulnerabilities inhardware, software, protocols and the communication stack. Those systems nowadays use standard hardware, software, operating systems and protocols. Furthermore, SCADA systems which used to be air-gaped are now interconnected to corporate networks and to the Internet, widening the attack surface.In this thesis, we are using a deep learning approach to propose an efficient hybrid deep neural network for anomaly detection in SCADA systems. The salient features of SCADA data are automatically and unsupervisingly learnt, and then fed to a supervised classifier in order to dertermine if those data are normal or abnormal, i.e if there is a cyber-attack or not. Afterwards, as a response to the challenge caused by high training time of deep learning models, we proposed a distributed approach of our anomaly detection system in order lo lessen the training time of our model

Abstract Industrial control system (ICS) networks used to be isolated environments, typically separated by physical air gaps from the wider area networks. This situation has been changing and the change has brought with it new cybersecurity issues. The process has also exacerbated existing problems that were previously less exposed due to the systems’ relative isolation. This process of increasing connectivity between devices, systems and persons can be seen as part of a paradigm shift called the Internet of Things (IoT). This change is progressing and the industry actors need to take it into account when working to improve the cybersecurity of ICS environments and thus their reliability. Ensuring that proper security processes and mechanisms are being implemented and enforced on the ICS network level is an important part of the general security posture of any given industrial actor. Network security and the detection of intrusions and anomalies in the context of ICS networks are the main high-level research foci of this thesis. These issues are investigated through work on machine learning (ML) based anomaly detection (AD). Potentially suitable features, approaches and algorithms for implementing a network anomaly detection system for use in ICS environments are investigated. After investigating the challenges, different approaches and methods, a proof-ofconcept (PoC) was implemented. The PoC implementation is built on top of the Bro network security monitoring framework (Bro) for testing the selected approach and tools. In the PoC, a Self-Organizing Map (SOM) algorithm is implemented using Bro scripting language to demonstrate the feasibility of using Bro as a base system. The implemented approach also represents a minimal case of event-driven machine learning anomaly detection (EMLAD) concept conceived during the research. The contributions of this thesis are as follows: a set of potential features for use in machine learning anomaly detection, proof of the feasibility of the machine learning approach in ICS network setting, a concept for event-driven machine learning anomaly detection, a design and initial implementation of user configurable and extendable machine learning anomaly detection framework for ICS networks<br>Tiivistelmä Kehittyneet yhteiskunnat käyttävät teollisuuslaitoksissaan ja infrastruktuuriensa operoinnissa monimuotoisia automaatiojärjestelmiä. Näiden automaatiojärjestelmien tieto- ja kyberturvallisuuden tila on hyvin vaihtelevaa. Laitokset ja niiden hyödyntämät järjestelmät voivat edustaa usean eri aikakauden tekniikkaa ja sisältää useiden eri aikakauden heikkouksia ja haavoittuvaisuuksia. Järjestelmät olivat aiemmin suhteellisen eristyksissä muista tietoverkoista kuin omista kommunikaatioväylistään. Tämä automaatiojärjestelmien eristyneisyyden heikkeneminen on luonut uuden joukon uhkia paljastamalla niiden kommunikaatiorajapintoja ympäröivälle maailmalle. Nämä verkkoympäristöt ovat kuitenkin edelleen verrattaen eristyneitä ja tätä ominaisuutta voidaan hyödyntää niiden valvonnassa. Tässä työssä esitetään tutkimustuloksia näiden verkkojen turvallisuuden valvomisesta erityisesti poikkeamien havainnoinnilla käyttäen hyväksi koneoppimismenetelmiä. Alkuvaiheen haasteiden ja erityispiirteiden tutkimuksen jälkeen työssä käytetään itsejärjestyvien karttojen (Self-Organizing Map, SOM) algoritmia esimerkkiratkaisun toteutuksessa uuden konseptin havainnollistamiseksi. Tämä uusi konsepti on tapahtumapohjainen koneoppiva poikkeamien havainnointi (Event-Driven Machine Learning Anomaly Detection, EMLAD). Työn kontribuutiot ovat seuraavat, kaikki teollisuusautomaatioverkkojen kontekstissa: ehdotus yhdeksi anomalioiden havainnoinnissa käytettävien ominaisuuksien ryhmäksi, koneoppivan poikkeamien havainnoinnin käyttökelpoisuuden toteaminen, laajennettava ja joustava esimerkkitoteutus uudesta EMLAD-konseptista toteutettuna Bro NSM työkalun ohjelmointikielellä

La détection d'anomalies est une tâche critique de l'administration des réseaux. L'apparition continue de nouvelles anomalies et la nature changeante du trafic réseau compliquent de fait la détection d'anomalies. Les méthodes existantes de détection d'anomalies s'appuient sur une connaissance préalable du trafic : soit via des signatures créées à partir d'anomalies connues, soit via un profil de normalité. Ces deux approches sont limitées : la première ne peut détecter les nouvelles anomalies et la seconde requiert une constante mise à jour de son profil de normalité. Ces deux aspects limitent de façon importante l'efficacité des méthodes de détection existantes.Nous présentons une approche non-supervisée qui permet de détecter et caractériser les anomalies réseaux de façon autonome. Notre approche utilise des techniques de partitionnement afin d'identifier les flux anormaux. Nous proposons également plusieurs techniques qui permettent de traiter les anomalies extraites pour faciliter la tâche des opérateurs. Nous évaluons les performances de notre système sur des traces de trafic réel issues de la base de trace MAWI. Les résultats obtenus mettent en évidence la possibilité de mettre en place des systèmes de détection d'anomalies autonomes et fonctionnant sans connaissance préalable<br>Anomaly detection has become a vital component of any network in today’s Internet. Ranging from non-malicious unexpected events such as flash-crowds and failures, to network attacks such as denials-of-service and network scans, network traffic anomalies can have serious detrimental effects on the performance and integrity of the network. The continuous arising of new anomalies and attacks create a continuous challenge to cope with events that put the network integrity at risk. Moreover, the inner polymorphic nature of traffic caused, among other things, by a highly changing protocol landscape, complicates anomaly detection system's task. In fact, most network anomaly detection systems proposed so far employ knowledge-dependent techniques, using either misuse detection signature-based detection methods or anomaly detection relying on supervised-learning techniques. However, both approaches present major limitations: the former fails to detect and characterize unknown anomalies (letting the network unprotected for long periods) and the latter requires training over labeled normal traffic, which is a difficult and expensive stage that need to be updated on a regular basis to follow network traffic evolution. Such limitations impose a serious bottleneck to the previously presented problem.We introduce an unsupervised approach to detect and characterize network anomalies, without relying on signatures, statistical training, or labeled traffic, which represents a significant step towards the autonomy of networks. Unsupervised detection is accomplished by means of robust data-clustering techniques, combining Sub-Space clustering with Evidence Accumulation or Inter-Clustering Results Association, to blindly identify anomalies in traffic flows. Correlating the results of several unsupervised detections is also performed to improve detection robustness. The correlation results are further used along other anomaly characteristics to build an anomaly hierarchy in terms of dangerousness. Characterization is then achieved by building efficient filtering rules to describe a detected anomaly. The detection and characterization performances and sensitivities to parameters are evaluated over a substantial subset of the MAWI repository which contains real network traffic traces.Our work shows that unsupervised learning techniques allow anomaly detection systems to isolate anomalous traffic without any previous knowledge. We think that this contribution constitutes a great step towards autonomous network anomaly detection.This PhD thesis has been funded through the ECODE project by the European Commission under the Framework Programme 7. The goal of this project is to develop, implement, and validate experimentally a cognitive routing system that meet the challenges experienced by the Internet in terms of manageability and security, availability and accountability, as well as routing system scalability and quality. The concerned use case inside the ECODE project is network anomaly

Critical infrastructure provides us with the most important parts of modern society, electricity, water and transport. To increase efficiency and to meet new demands from the customer remote monitoring and control of the systems is necessary. This opens new ways for an attacker to reach the Supervisory Control And Data Acquisition (SCADA) systems that control and monitors the physical processes involved. This also increases the need for security features specially designed for these settings. Anomaly-based detection is a technique suitable for the more deterministic SCADA systems. This thesis uses a combination of two techniques to detect anomalies. The first technique is an automatic whitelist that learns the behavior of the network flows. The second technique utilizes the differences in arrival times of the network packets. A prototype anomaly detector has been developed in Bro. To analyze the IEC 60870-5-104 protocol a new parser for Bro was also developed. The resulting anomaly detector was able to achieve a high detection rate for three of the four different types of attacks evaluated. The studied methods of detection are promising when used in a highly deterministic setting, such as a SCADA system.

Network data (also referred to as relational data, social network data, real graph data) has become ubiquitous, and understanding patterns in this data has become an important research problem. We investigate how interactions in social networks are formed and how these interactions facilitate diffusion, model these behaviors, and apply these findings to real-world problems. We examined graphs of size up to 16 million nodes, across many domains from academic citation networks, to campaign contributions and actor-movie networks. We also performed several case studies in online social networks such as blogs and message board communities. Our major contributions are the following: (a) We discover several surprising patterns in network topology and interactions, such as Popularity Decay power law (in-links to a blog post decay with a power law with -1:5 exponent) and the oscillating size of connected components; (b) We propose generators such as the Butterfly generator that reproduce both established and new properties found in real networks; (c) several case studies, including a proposed method of detecting misstatements in accounting data, where using network effects gave a significant boost in detection accuracy.

Anomaly detection in dynamic communication networks has many important security applications. These networks can be extremely large and so detecting any changes in their structure can be computationally challenging; hence, computationally fast, parallelisable methods for monitoring the network are paramount. For this reason the methods presented here use independent node and edge based models to detect locally anomalous substructures within communication networks. As a first stage, the aim is to detect changes in the data streams arising from node or edge communications. Throughout the thesis simple, conjugate Bayesian models for counting processes are used to model these data streams. A second stage of analysis can then be performed on a much reduced subset of the network comprising nodes and edges which have been identified as potentially anomalous in the first stage. The first method assumes communications in a network arise from an inhomogeneous Poisson process with piecewise constant intensity. Anomaly detection is then treated as a changepoint problem on the intensities. The changepoint model is extended to incorporate seasonal behaviour inherent in communication networks. This seasonal behaviour is also viewed as a changepoint problem acting on a piecewise constant Poisson process. In a static time frame, inference is made on this extended model via a Gibbs sampling strategy. In a sequential time frame, where the data arrive as a stream, a novel, fast Sequential Monte Carlo (SMC) algorithm is introduced to sample from the sequence of posterior distributions of the changepoints over time. A second method is considered for monitoring communications in a large scale computer network. The usage patterns in these types of networks are very bursty in nature and don't fit a Poisson process model. For tractable inference, discrete time models are considered, where the data are aggregated into discrete time periods and probability models are fitted to the communication counts. In a sequential analysis, anomalous behaviour is then identified from outlying behaviour with respect to the fitted predictive probability models. Seasonality is again incorporated into the model and is treated as a changepoint model on the transition probabilities of a discrete time Markov process. Second stage analytics are then developed which combine anomalous edges to identify anomalous substructures in the network.

In this degree project, we study the anomaly detection problem in log files of computer networks. In particular, we try to find an efficient way to detect anomalies in our data, which consist of different logging messages from different systems in CERN’s network for the LHC-b experiment. The contributions of the thesis are double: 1) The thesis serves as a survey on how we can detect threats, and errors in systems that are logging a huge amount of messages in the databases of a computer network. 2) Scientists in the LHC-b experiment make use of the Elasticsearch, which is an open source search engine and logging platform with great reputation, providing log monitoring, as well as data stream processing. Moreover, the Elasticsearch provides a machine learning feature that automatically models the behavior of the data, learning trends, and periodicity to identify anomalies. Alternatively to the Elasticsearch machine learning feature, we build, test and evaluate some machine learning models that can be used for the same purpose from the scientists of the experiment. We further provide results that our models generalize well to unseen log messages in the database.<br>I detta examensarbete studerar vi problemet med att upptäcka avvikelser i loggfiler från ett datanätverk. Specifikt försöker vi hitta ett effektivt sätt att upptäcka avvikelser i datan, som består av olika loggningsmeddelanden från olika system i CERNs nätverk för LHC-b-experimentet. Avhandlingens dubbla bidrag är: 1)Avhandlingen kan anses som en undersökning om hur vi kan upptäcka hot och fel i system som loggar en enorm mängd meddelanden i databaser från ett datanätverk. 2) Forskare i LHC-bexperimentet använder sig av Elasticsearch, som är en sökmotor och loggningsplattform med öppen källkod och ett avsevärt rykte, som tillhandahåller loggövervakning och automatisk datahantering. Dessutom är Elasticsearch försedd med en maskinlärningsfunktion som automatiskt modellerar beteenden med hjälp av data, trender och periodicitet för att identifiera avvikelser. Vi bygger, testar och utvärderar ett fåtal maskininlärningsmodeller som ett alternativt till Elasticsearch maskininlärningsfunktion. Forskarna i experimentet kan använda maskininlärningsmodellerna till samma ändamål som Elasticsearch maskininlärningsfunktion. Vi presenterar också resultat som visar att våra modeller generaliserar väl för osedda loggmeddelanden i databasen.

Available security standards for RFID networks (e.g. ISO/IEC 29167) are designed to secure individual tag-reader sessions and do not protect against active attacks that could also compromise the system as a whole (e.g. tag cloning or replay attacks). Proper traffic characterization models of the communication within an RFID network can lead to better understanding of operation under “normal” system state conditions and can consequently help identify security breaches not addressed by current standards. This study of RFID traffic characterization considers two piecewise-constant data smoothing techniques, namely Bayesian blocks and Knuth’s algorithms, over time-tagged events and compares them in the context of rate-based anomaly detection. This was accomplished using data from experimental RFID readings and comparing (1) the event counts versus time if using the smoothed curves versus empirical histograms of the raw data and (2) the threshold-dependent alert-rates based on inter-arrival times obtained if using the smoothed curves versus that of the raw data itself. Results indicate that both algorithms adequately model RFID traffic in which inter-event time statistics are stationary but that Bayesian blocks become superior for traffic in which such statistics experience abrupt changes.

With the ever increasing deployment and usage of gigabit networks, traditional network anomaly detection based intrusion detection systems have not scaled accordingly. Most, if not all, systems deployed assume the availability of complete and clean data for the purpose of intrusion detection. We contend that this assumption is not valid. Factors like noise in the audit data, mobility of the nodes, and the large amount of data generated by the network make it difficult to build a normal traffic profile of the network for the purpose of anomaly detection. From this perspective, the leitmotif of the research effort described in this dissertation is the design of a novel intrusion detection system that has the capability to detect intrusions with high accuracy even when complete audit data is not available. In this dissertation, we take a holistic approach to anomaly detection to address the threats posed by network based denial-of-service attacks by proposing improvements in every step of the intrusion detection process. At the data collection phase, we have implemented an adaptive sampling scheme that intelligently samples incoming network data to reduce the volume of traffic sampled, while maintaining the intrinsic characteristics of the network traffic. A Bloom filters based fast flow aggregation scheme is employed at the data pre-processing stage to further reduce the response time of the anomaly detection scheme. Lastly, this dissertation also proposes an expectation-maximization algorithm based anomaly detection scheme that uses the sampled audit data to detect intrusions in the incoming network traffic.<br>Ph. D.

The dissertation investigated the creation of an anomaly detection approach to identify anomalies in the SGW elements of a LTE network. Unsupervised techniques were compared and used to identify and remove anomalies in the training data set. This “cleaned” data set was then used to train an autoencoder in an semi-supervised approach. The resultant autoencoder was able to indentify normal observations. A subsequent data set was then analysed by the autoencoder. The resultant reconstruction errors were then compared to the ground truth events to investigate the effectiveness of the autoencoder’s anomaly detection capability.

Grâce à l'évolution des outils d'automatisation et d'intelligence artificielle, les réseauxmobiles sont devenus de plus en plus dépendants de la machine. De nos jours, une grandepartie des tâches de gestion de réseaux est exécutée d'une façon autonome, sans interventionhumaine. Dans cette thèse, nous avons focalisé sur l'utilisation des techniques d'analyse dedonnées dans le but d'automatiser et de consolider le processus de résolution de défaillancesdans les réseaux. Pour ce faire, nous avons défini deux objectifs principaux : la détectiond'anomalies et le diagnostic des causes racines de ces anomalies. Le premier objectif consiste àdétecter automatiquement les anomalies dans les réseaux sans faire appel aux connaissancesdes experts. Pour atteindre cet objectif, nous avons proposé un algorithme, Watchmen AnomalyDetection (WAD), basé sur le concept de la reconnaissance de formes (pattern recognition). Cetalgorithme apprend le modèle du trafic réseau à partir de séries temporelles périodiques etdétecte des distorsions par rapport à ce modèle dans le flux de nouvelles données. Le secondobjectif a pour objet la détermination des causes racines des problèmes réseau sans aucuneconnaissance préalable sur l'architecture du réseau et des différents services. Pour ceci, nousavons conçu un algorithme, Automatic Root Cause Diagnosis (ARCD), qui permet de localiser lessources d'inefficacité dans le réseau. ARCD est composé de deux processus indépendants :l'identification des contributeurs majeurs à l'inefficacité globale du réseau et la détection desincompatibilités. WAD et ARCD ont fait preuve d'efficacité. Cependant, il est possible d'améliorerces algorithmes sur plusieurs aspects<br>With the evolution of automation and artificial intelligence tools, mobile networks havebecome more and more machine reliant. Today, a large part of their management tasks runs inan autonomous way, without human intervention. In this thesis, we have focused on takingadvantage of the data analysis tools to automate the troubleshooting task and carry it to a deeperlevel. To do so, we have defined two main objectives: anomaly detection and root causediagnosis. The first objective is about detecting issues in the network automatically withoutincluding expert knowledge. To meet this objective, we have proposed an algorithm, WatchmenAnomaly Detection (WAD), based on pattern recognition. It learns patterns from periodic timeseries and detect distortions in the flow of new data. The second objective aims at identifying theroot cause of issues without any prior knowledge about the network topology and services. Toaddress this question, we have designed an algorithm, Automatic Root Cause Diagnosis (ARCD)that identifies the roots of network issues. ARCD is composed of two independent threads: MajorContributor identification and Incompatibility detection. WAD and ARCD have been proven to beeffective. However, many improvements of these algorithms are possible

Thesis (Ph.D.)--Boston University<br>PLEASE NOTE: Boston University Libraries did not receive an Authorization To Manage form for this thesis or dissertation. It is therefore not openly accessible, though it may be available by request. If you are the author or principal advisor of this work and would like to request open access for it, please contact us at open-help@bu.edu. Thank you.<br>This dissertation makes fundamental contributions to statistical methods relating to the detection of anomalies in the context of computer network traffic monitoring. In particular, it contributes basic statistical tools for socially-based network anomaly characterization and detection, it extends a popular detection methodology to high-dimensional contexts, and it demonstrates that standard flow sampling can interact with inherent network topology in ways unexpected. In the first contribution of my research, I define anomalous intrusion in terms of locations in social space, rather than in physical space. I develop statistical detectors based on simple graph-based summaries of the network, with a focus on detecting anti-social behaviors. This research suggests that certain values of local graphical measurements, like clustering coefficients and betweenness centrality, are associated with the malicious antisocial behaviors in the types of network representations of IP flow measurements used in this work. This motivates me to propose a simple, efficient and robust anomaly detection technique. I evaluate this methodology on different network representations and using different social summaries. In the second contribution of my research, I extend the use of the PCA subspace method to high-dimensional spaces. Specifically, I show that, under appropriate conditions,with high probability the magnitude of the residuals of a standard PCA subspace analysis of randomly projected data behaves comparably to that of the residuals of a similar PCA analysis of the original data. My results indicate the feasibility of applying subspacebased anomaly detection algorithms to Gaussian random projection data. This concept is illustrated in the context of computer network traffic anomaly detection for the purpose of detecting volume anomalies. The impact of sampling on so-called Peer-to-Peer (P2P) network analysis is the focus of the third contribution of my research. In this research I use a combination of probability calculations and simulation techniques to characterize the extent to which standard packet sampling in the Internet can adversely affect the topology of stylized versions of Bittorrent download networks reconstructed from measurements of network flows. The results indicate that a certain stratification observed in these networks impacts the reconstructed topology in ways decidedly different from typical networks which have no stratification.<br>2031-01-01

Ces dernières années, le piratage est devenu une industrie à part entière, augmentant le nombre et la diversité des cyberattaques. Les menaces qui pèsent sur les réseaux informatiques vont des logiciels malveillants aux attaques par déni de service, en passant par le phishing et l'ingénierie sociale. Un plan de cybersécurité efficace ne peut plus reposer uniquement sur des antivirus et des pare-feux pour contrer ces menaces : il doit inclure plusieurs niveaux de défense. Les systèmes de détection d'intrusion (IDS) réseaux sont un moyen complémentaire de renforcer la sécurité, avec la possibilité de surveiller les paquets de la couche 2 (liaison) à la couche 7 (application) du modèle OSI. Les techniques de détection d'intrusion sont traditionnellement divisées en deux catégories : la détection par signatures et la détection par anomalies. La plupart des IDS utilisés aujourd'hui reposent sur la détection par signatures ; ils ne peuvent cependant détecter que des attaques connues. Les IDS utilisant la détection par anomalies sont capables de détecter des attaques inconnues, mais sont malheureusement moins précis, ce qui génère un grand nombre de fausses alertes. Dans ce contexte, la création d'IDS précis par anomalies est d'un intérêt majeur pour pouvoir identifier des attaques encore inconnues.Dans cette thèse, les modèles d'apprentissage automatique sont étudiés pour créer des IDS qui peuvent être déployés dans de véritables réseaux informatiques. Tout d'abord, une méthode d'optimisation en trois étapes est proposée pour améliorer la qualité de la détection : 1/ augmentation des données pour rééquilibrer les jeux de données, 2/ optimisation des paramètres pour améliorer les performances du modèle et 3/ apprentissage ensembliste pour combiner les résultats des meilleurs modèles. Les flux détectés comme des attaques peuvent être analysés pour générer des signatures afin d'alimenter les bases de données d'IDS basées par signatures. Toutefois, cette méthode présente l'inconvénient d'exiger des jeux de données étiquetés, qui sont rarement disponibles dans des situations réelles. L'apprentissage par transfert est donc étudié afin d'entraîner des modèles d'apprentissage automatique sur de grands ensembles de données étiquetés, puis de les affiner sur le trafic normal du réseau à surveiller. Cette méthode présente également des défauts puisque les modèles apprennent à partir d'attaques déjà connues, et n'effectuent donc pas réellement de détection d'anomalies. C'est pourquoi une nouvelle solution basée sur l'apprentissage non supervisé est proposée. Elle utilise l'analyse de l'en-tête des protocoles réseau pour modéliser le comportement normal du trafic. Les anomalies détectées sont ensuite regroupées en attaques ou ignorées lorsqu'elles sont isolées. Enfin, la détection la congestion réseau est étudiée. Le taux d'utilisation de la bande passante entre les différents liens est prédit afin de corriger les problèmes avant qu'ils ne se produisent<br>In recent years, hacking has become an industry unto itself, increasing the number and diversity of cyber attacks. Threats on computer networks range from malware to denial of service attacks, phishing and social engineering. An effective cyber security plan can no longer rely solely on antiviruses and firewalls to counter these threats: it must include several layers of defence. Network-based Intrusion Detection Systems (IDSs) are a complementary means of enhancing security, with the ability to monitor packets from OSI layer 2 (Data link) to layer 7 (Application). Intrusion detection techniques are traditionally divided into two categories: signatured-based (or misuse) detection and anomaly detection. Most IDSs in use today rely on signature-based detection; however, they can only detect known attacks. IDSs using anomaly detection are able to detect unknown attacks, but are unfortunately less accurate, which generates a large number of false alarms. In this context, the creation of precise anomaly-based IDS is of great value in order to be able to identify attacks that are still unknown.In this thesis, machine learning models are studied to create IDSs that can be deployed in real computer networks. Firstly, a three-step optimization method is proposed to improve the quality of detection: 1/ data augmentation to rebalance the dataset, 2/ parameters optimization to improve the model performance and 3/ ensemble learning to combine the results of the best models. Flows detected as attacks can be analyzed to generate signatures to feed signature-based IDS databases. However, this method has the disadvantage of requiring labelled datasets, which are rarely available in real-life situations. Transfer learning is therefore studied in order to train machine learning models on large labeled datasets, then finetune them on benign traffic of the network to be monitored. This method also has flaws since the models learn from already known attacks, and therefore do not actually perform anomaly detection. Thus, a new solution based on unsupervised learning is proposed. It uses network protocol header analysis to model normal traffic behavior. Anomalies detected are then aggregated into attacks or ignored when isolated. Finally, the detection of network congestion is studied. The bandwidth utilization between different links is predicted in order to correct issues before they occur

As social networks become more prevalent, there is significant interest in studying these network data, the focus often being on detecting anomalous events. This area of research is referred to as social network surveillance or social network change detection. While there are a variety of proposed methods suitable for different monitoring situations, two important issues have yet to be completely addressed in network surveillance literature. First, performance assessments using simulated data to evaluate the statistical performance of a particular method. Second, the study of aggregated data in social network surveillance. The research presented tackle these issues in two parts, evaluation of a popular anomaly detection method and investigation of the effects of different aggregation levels on network anomaly detection.<br>Ph. D.

This thesis presents Bayesian approach for a contextual network anomaly detection. Network anomaly detection is important in a computer system performance monitoring perspective. Detecting a contextual anomaly is much harder since we need to take the context into account in order to explain whether it is normal or abnormal. The main idea of this thesis is to find contextual attributes from a set of indicators, then to estimate the resource loads through the Bayesian model. The proposed algorithm offers three advantages. Firstly, the model can estimate resource loads with automatically selected indicators and its credible intervals. Secondly, both point and collective contextual anomalies can be captured by the posterior predictive distribution. Lastly, the structural interpretation of the model gives us a way to find similar nodes. This thesis employs real data from Radio Network Controller (RNC) to validate the effectiveness in detecting contextual anomalies.

Information systems and their services (referred to as cyberspace) are ubiquitous and touch all aspects of our life. With the exponential growth in cyberspace activities, the number and complexity of cyber-attacks have increased significantly due to an increase in the number of applications with vulnerabilities and the number of attackers. Consequently, it becomes extremely critical to develop efficient network Intrusion Detection Systems (IDS) that can mitigate and protect cyberspace resources and services against cyber-attacks. On the other hand, since each network system and application has its own specification as defined in its protocol, it is hard to develop a single IDS which works properly for all network protocols. The keener approach is to design customized detection engines for each protocol and then aggregate the reports from these engines to define the final security state of the system. In this dissertation, we developed a general methodology based on data mining, statistical analysis and protocol semantics to perform anomaly behavior analysis and detection for network-centric systems and their protocols. In our approach, we develop runtime models of protocol's state transitions during a time interval ΔΤ. We consider any n consecutive messages in a session during the time interval ΔΤ as an n-transition pattern called n-gram. By applying statistical analysis over these n-gram patterns we can accurately model the normal behavior of any protocol. Then we use the amount of the deviation from this normal model to quantify the anomaly score of the protocol activities. If this anomaly score is higher than a well-defined threshold the system marks that activity as a malicious activity. To validate our methodology, we have applied it to two different protocols: DNS (Domain Name System) at the application layer and the IEEE 802.11(WiFi) at the data link layer, where we have achieved good detection results (>95%) with low detection errors (<0.1%).

Thesis: S.M. in Engineering and Management, Massachusetts Institute of Technology, System Design and Management Program, 2018.<br>Cataloged from PDF version of thesis.<br>Includes bibliographical references (pages 84-86).<br>Cybersecurity is a growing research area with direct commercial impact to organizations and companies in every industry. With all other technological advancements in the Internet of Things (IoT), mobile devices, cloud computing, 5G network, and artificial intelligence, the need for cybersecurity is more critical than ever before. These technologies drive the need for tighter cybersecurity implementations, while at the same time act as enablers to provide more advanced security solutions. This paper will discuss a framework that can predict cybersecurity risk by identifying normal network behavior and detect network traffic anomalies. Our research focuses on the analysis of the historical network traffic data to identify network usage trends and security vulnerabilities. Specifically, this thesis will focus on multiple components of the data analytics platform. It explores the big data platform architecture, and data ingestion, analysis, and engineering processes. The experiments were conducted utilizing various time series algorithms (Seasonal ETS, Seasonal ARIMA, TBATS, Double-Seasonal Holt-Winters, and Ensemble methods) and Long Short-Term Memory Recurrent Neural Network algorithm. Upon creating the baselines and forecasting network traffic trends, the anomaly detection algorithm was implemented using specific thresholds to detect network traffic trends that show significant variation from the baseline. Lastly, the network traffic data was analyzed and forecasted in various dimensions: total volume, source vs. destination volume, protocol, port, machine, geography, and network structure and pattern. The experiments were conducted with multiple approaches to get more insights into the network patterns and traffic trends to detect anomalies.<br>by Lwin P. Moe.<br>S.M. in Engineering and Management

Anomaly detection has become a crucial part of the protection of information and integrity. Due to the increase of cyber threats the demand for anomaly detection has grown for companies. Anomaly detection on time series data aims to detect unexpected behavior on the system. Anomalies often occur online, and companies need to be able to protect themselves from these intrusions. Multiple machine learning algorithms have been used and researched to solve the problem with anomaly detection and it is ongoing research to find the most optimal algorithms. Therefore, this study investigates algorithms such as K-means, Mean Shift and DBSCAN algorithm could be a solution for the problem. The study also investigates if combining the algorithms will improve the result. The results that the study reveals that the combinations of the algorithms perform slightly worse than the individual algorithms regarding speed and accuracy to detect anomalies. The algorithms without combinations did perform well during this study, they have slight differences between each other, and the results show the DBSCAN algorithm has slightly better total detection compared to the other algorithms and has slower execution time. The conclusion for this study reveals that the Mean Shift algorithm had the fastest execution time and the DBSCAN algorithm had the highest accuracy. The study also reveals most of the combinations between the algorithms did not improve during the fusion. However, the DBSCAN + Mean Shift fusion did improve the accuracy, and the K-means + Mean Shift fusion did improve the execution time.

In this thesis, the use of unsupervised and semi-supervised machine learning techniques was analyzed as potential tools for anomaly detection in the sensor network that the electrical system in a Scania truck is comprised of. The experimentation was designed to analyse the need for both point and contextual anomaly detection in this setting. For the point anomaly detection the method of Isolation Forest was experimented with and for contextual anomaly detection two different recurrent neural network architectures using Long Short Term Memory units was relied on. One model was simply a many to one regression model trained to predict a certain signal, while the other was an encoder-decoder network trained to reconstruct a sequence. Both models were trained in an semi-supervised manner, i.e. on data that only depicts normal behaviour, which theoretically should lead to a performance drop on abnormal sequences resulting in higher error terms. In both setting the parameters of a Gaussian distribution were estimated using these error terms which allowed for a convenient way of defining a threshold which would decide if the observation would be flagged as anomalous or not. Additional experimentation's using an exponential weighted moving average over a number of past observations to filter the signal was also conducted. The models performance on this particular task was very different but the regression model showed a lot of promise especially when combined with a filtering preprocessing step to reduce the noise in the data. However the model selection will always be governed by the nature the particular task at hand so the other methods might perform better in other settings.<br>I den här avhandlingen var användningen av oövervakad och halv-övervakad maskininlärning analyserad som ett möjligt verktyg för att upptäcka avvikelser av anomali i det sensornätverk som elektriska systemet en Scanialastbil består av. Experimentet var konstruerat för att analysera behovet av både punkt och kontextuella avvikelser av anomali i denna miljö. För punktavvikelse av anomali var metoden Isolation Forest experimenterad med och för kontextuella avvikelser av anomali användes två arkitekturer av återkommande neurala nätverk. En av modellerna var helt enkelt många-till-en regressionmodell tränad för att förutspå ett visst märke, medan den andre var ett kodare-avkodare nätverk tränat för att rekonstruera en sekvens.Båda modellerna blev tränade på ett halv-övervakat sätt, d.v.s. på data som endast visar normalt beteende, som teoretiskt skulle leda till minskad prestanda på onormala sekvenser som ger ökat antal feltermer. I båda fallen blev parametrarna av en Gaussisk distribution estimerade på grund av dessa feltermer som tillåter ett bekvämt sätt att definera en tröskel som skulle bestämma om iakttagelsen skulle bli flaggad som en anomali eller inte. Ytterligare experiment var genomförda med exponentiellt viktad glidande medelvärde över ett visst antal av tidigare iakttagelser för att filtera märket. Modellernas prestanda på denna uppgift var välidt olika men regressionmodellen lovade mycket, särskilt kombinerad med ett filterat förbehandlingssteg för att minska bruset it datan. Ändå kommer modelldelen alltid styras av uppgiftens natur så att andra metoder skulle kunna ge bättre prestanda i andra miljöer.

The frequent and large-scale network attacks have led to an increased need for developing techniques for analyzing network traffic. If efficient analysis tools were available, it could become possible to detect the attacks, anomalies and to appropriately take action to contain the attacks before they have had time to propagate across the network. In this dissertation, we suggest a technique for traffic anomaly detection based on analyzing the correlation of destination IP addresses and distribution of image-based signal in postmortem and real-time, by passively monitoring packet headers of traffic. This address correlation data are transformed using discrete wavelet transform for effective detection of anomalies through statistical analysis. Results from trace-driven evaluation suggest that the proposed approach could provide an effective means of detecting anomalies close to the source. We present a multidimensional indicator using the correlation of port numbers as a means of detecting anomalies. We also present a network measurement approach that can simultaneously detect, identify and visualize attacks and anomalous traffic in real-time. We propose to represent samples of network packet header data as frames or images. With such a formulation, a series of samples can be seen as a sequence of frames or video. Thisenables techniques from image processing and video compression such as DCT to be applied to the packet header data to reveal interesting properties of traffic. We show that ??scene change analysis?? can reveal sudden changes in traffic behavior or anomalies. We show that ??motion prediction?? techniques can be employed to understand the patterns of some of the attacks. We show that it may be feasible to represent multiple pieces of data as different colors of an image enabling a uniform treatment of multidimensional packet header data. Measurement-based techniques for analyzing network traffic treat traffic volume and traffic header data as signals or images in order to make the analysis feasible. In this dissertation, we propose an approach based on the classical Neyman-Pearson Test employed in signal detection theory to evaluate these different strategies. We use both of analytical models and trace-driven experiments for comparing the performance of different strategies. Our evaluations on real traces reveal differences in the effectiveness of different traffic header data as potential signals for traffic analysis in terms of their detection rates and false alarm rates. Our results show that address distributions and number of flows are better signals than traffic volume for anomaly detection. Our results also show that sometimes statistical techniques can be more effective than the NP-test when the attack patterns change over time.

Aquesta tesi proposa una plataforma de detecció d’intrusions per a revelar atacs a les xarxes de sensors sense fils (WSN, per les sigles en anglès) de les ciutats intel·ligents (smart cities). La plataforma està dissenyada tenint en compte les necessitats dels administradors de la ciutat intel·ligent, els quals necessiten accés a una arquitectura centralitzada que pugui gestionar alarmes de seguretat en un sistema altament heterogeni i distribuït. En aquesta tesi s’identifiquen els diversos passos necessaris des de la recollida de dades fins a l’execució de les tècniques de detecció d’intrusions i s’avalua que el procés sigui escalable i capaç de gestionar dades típiques de ciutats intel·ligents. A més, es comparen diversos algorismes de detecció d’anomalies i s’observa que els mètodes de vectors de suport d’una mateixa classe (one-class support vector machines) resulten la tècnica multivariant més adequada per a descobrir atacs tenint en compte les necessitats d’aquest context. Finalment, es proposa un esquema per a ajudar els administradors a identificar els tipus d’atacs rebuts a partir de les alarmes disparades.<br>Esta tesis propone una plataforma de detección de intrusiones para revelar ataques en las redes de sensores inalámbricas (WSN, por las siglas en inglés) de las ciudades inteligentes (smart cities). La plataforma está diseñada teniendo en cuenta la necesidad de los administradores de la ciudad inteligente, los cuales necesitan acceso a una arquitectura centralizada que pueda gestionar alarmas de seguridad en un sistema altamente heterogéneo y distribuido. En esta tesis se identifican los varios pasos necesarios desde la recolección de datos hasta la ejecución de las técnicas de detección de intrusiones y se evalúa que el proceso sea escalable y capaz de gestionar datos típicos de ciudades inteligentes. Además, se comparan varios algoritmos de detección de anomalías y se observa que las máquinas de vectores de soporte de una misma clase (one-class support vector machines) resultan la técnica multivariante más adecuada para descubrir ataques teniendo en cuenta las necesidades de este contexto. Finalmente, se propone un esquema para ayudar a los administradores a identificar los tipos de ataques recibidos a partir de las alarmas disparadas.<br>This thesis proposes an intrusion detection platform which reveals attacks in smart city wireless sensor networks (WSN). The platform is designed taking into account the needs of smart city administrators, who need access to a centralized architecture that can manage security alarms in a highly heterogeneous and distributed system. In this thesis, we identify the various necessary steps from gathering WSN data to running the detection techniques and we evaluate whether the procedure is scalable and capable of handling typical smart city data. Moreover, we compare several anomaly detection algorithms and we observe that one-class support vector machines constitute the most suitable multivariate technique to reveal attacks, taking into account the requirements in this context. Finally, we propose a classification schema to assist administrators in identifying the types of attacks compromising their networks.

Thesis (Ph. D.)--University of Alabama at Birmingham, 2007.<br>Additional advisors: Elliot J. Lefkowitz, Kevin D. Reilly, Robert Thacker, Chengcui Zhang. Description based on contents viewed Feb. 7, 2008; title from title screen. Includes bibliographical references (p. 125-132).

This work presents an investigation of tailoring Network Representation Learning (NRL) for an application in the Financial Industry. NRL approaches are data-driven models that learn how to encode graph structures into low-dimensional vector spaces, which can be further exploited by downstream Machine Learning applications. They can potentially bring a lot of benefits in the Financial Industry since they extract in an automatic way features that can provide useful input regarding graph structures, called embeddings. Financial transactions can be represented as a network, and through NRL, it is possible to extract embeddings that reflect the intrinsic inter-connected nature of economic relationships. Such embeddings can be used for several purposes, among which Anomaly Detection to fight financial crime.This work provides a qualitative analysis over state-of-the-art NRL models, which identifies Graph Convolutional Network (ConvGNN) as the most suitable category of approaches for Financial Industry but with a certain need for further improvement. Financial Industry poses additional challenges when modelling a NRL solution. Despite the need of having a scalable solution to handle real-world graph with considerable dimensions, it is necessary to take into consideration several characteristics: transactions graphs are inherently dynamic since every day new transactions are executed and nodes can be heterogeneous. Besides, everything is further complicated by the need to have updated information in (near) real-time due to the sensitivity of the application domain. For these reasons, GraphSAGE has been considered as a base for the experiments, which is an inductive ConvGNN model. Two variants of GraphSAGE are presented: a dynamic variant whose weights evolve accordingly with the input sequence of graph snapshots, and a variant specifically meant to handle bipartite graphs. These variants have been evaluated by applying them to real-world data and leveraging the generated embeddings to perform Anomaly Detection. The experiments demonstrate that leveraging these variants leads toimagecomparable results with other state-of-the-art approaches, but having the advantage of being suitable to handle real-world financial data sets.<br>Detta arbete presenterar en undersökning av tillämpningar av Network Representation Learning (NRL) inom den finansiella industrin. Metoder inom NRL möjliggör datadriven kondensering av grafstrukturer till lågdimensionella och lätthanterliga vektorer.Dessa vektorer kan sedan användas i andra maskininlärningsuppgifter. Närmare bestämt, kan metoder inom NRL underlätta hantering av och informantionsutvinning ur beräkningsintensiva och storskaliga grafer inom den finansiella sektorn, till exempel avvikelsehantering bland finansiella transaktioner. Arbetet med data av denna typ försvåras av det faktum att transaktionsgrafer är dynamiska och i konstant förändring. Utöver detta kan noderna, dvs transaktionspunkterna, vara vitt skilda eller med andra ord härstamma från olika fördelningar.I detta arbete har Graph Convolutional Network (ConvGNN) ansetts till den mest lämpliga lösningen för nämnda tillämpningar riktade mot upptäckt av avvikelser i transaktioner. GraphSAGE har använts som utgångspunkt för experimenten i två olika varianter: en dynamisk version där vikterna uppdateras allteftersom nya transaktionssekvenser matas in, och en variant avsedd särskilt för bipartita (tvådelade) grafer. Dessa varianter har utvärderats genom användning av faktiska datamängder med avvikelsehantering som slutmål.

Network monitoring has always been a topic of foremost importance for both network operators and researchers for multiple reasons ranging from anomaly detection to tra c classi cation or capacity planning. Nowadays, as networks become more and more complex, tra c increases and security threats reproduce, achieving a deeper understanding of what is happening in the network has become an essential necessity. In particular, due to the considerable growth of cybercrime, research on the eld of anomaly detection has drawn signi cant attention in recent years and tons of proposals have been made. All the same, when it comes to deploying solutions in real environments, some of them fail to meet some crucial requirements. Taking this into account, this thesis focuses on lling this gap between the research and the non-research world. Prior to the start of this work, we identify several problems. First, there is a clear lack of detailed and updated information on the most common anomalies and their characteristics. Second, unawareness of sampled data is still common although the performance of anomaly detection algorithms is severely a ected. Third, operators currently need to invest many work-hours to manually inspect and also classify detected anomalies to act accordingly and take the appropriate mitigation measures. This is further exacerbated due to the high number of false positives and false negatives and because anomaly detection systems are often perceived as extremely complex black boxes. Analysing an issue is essential to fully comprehend the problem space and to be able to tackle it properly. Accordingly, the rst block of this thesis seeks to obtain detailed and updated real-world information on the most frequent anomalies occurring in backbone networks. It rst reports on the performance of di erent commercial systems for anomaly detection and analyses the types of network nomalies detected. Afterwards, it focuses on further investigating the characteristics of the anomalies found in a backbone network using one of the tools for more than half a year. Among other results, this block con rms the need of applying sampling in an operational environment as well as the unacceptably high number of false positives and false negatives still reported by current commercial tools. On the whole, the presence of ampling in large networks for monitoring purposes has become almost mandatory and, therefore, all anomaly detection algorithms that do not take that into account might report incorrect results. In the second block of this thesis, the dramatic impact of sampling on the performance of well-known anomaly detection techniques is analysed and con rmed. However, we show that the results change signi cantly depending on the sampling technique used and also on the common metric selected to perform the comparison. In particular, we show that, Packet Sampling outperforms Flow Sampling unlike previously reported. Furthermore, we observe that Selective Sampling (SES), a sampling technique that focuses on small ows, obtains much better results than traditional sampling techniques for scan detection. Consequently, we propose Online Selective Sampling, a sampling technique that obtains the same good performance for scan detection than SES but works on a per-packet basis instead of keeping all ows in memory. We validate and evaluate our proposal and show that it can operate online and uses much less resources than SES. Although the literature is plenty of techniques for detecting anomalous events, research on anomaly classi cation and extraction (e.g., to further investigate what happened or to share evidence with third parties involved) is rather marginal. This makes it harder for network operators to analise reported anomalies because they depend solely on their experience to do the job. Furthermore, this task is an extremely time-consuming and error-prone process. The third block of this thesis targets this issue and brings it together with the knowledge acquired in the previous blocks. In particular, it presents a system for automatic anomaly detection, extraction and classi cation with high accuracy and very low false positives. We deploy the system in an operational environment and show its usefulness in practice. The fourth and last block of this thesis presents a generalisation of our system that focuses on analysing all the tra c, not only network anomalies. This new system seeks to further help network operators by summarising the most signi cant tra c patterns in their network. In particular, we generalise our system to deal with big network tra c data. In particular, it deals with src/dst IPs, src/dst ports, protocol, src/dst Autonomous Systems, layer 7 application and src/dst geolocation. We rst deploy a prototype in the European backbone network of G EANT and show that it can process large amounts of data quickly and build highly informative and compact reports that are very useful to help comprehending what is happening in the network. Second, we deploy it in a completely di erent scenario and show how it can also be successfully used in a real-world use case where we analyse the behaviour of highly distributed devices related with a critical infrastructure sector.<br>La monitoritzaci o de xarxa sempre ha estat un tema de gran import ancia per operadors de xarxa i investigadors per m ultiples raons que van des de la detecci o d'anomalies fins a la classi caci o d'aplicacions. Avui en dia, a mesura que les xarxes es tornen m es i m es complexes, augmenta el tr ansit de dades i les amenaces de seguretat segueixen creixent, aconseguir una comprensi o m es profunda del que passa a la xarxa s'ha convertit en una necessitat essencial. Concretament, degut al considerable increment del ciberactivisme, la investigaci o en el camp de la detecci o d'anomalies ha crescut i en els darrers anys s'han fet moltes i diverses propostes. Tot i aix o, quan s'intenten desplegar aquestes solucions en entorns reals, algunes d'elles no compleixen alguns requisits fonamentals. Tenint aix o en compte, aquesta tesi se centra a omplir aquest buit entre la recerca i el m on real. Abans d'iniciar aquest treball es van identi car diversos problemes. En primer lloc, hi ha una clara manca d'informaci o detallada i actualitzada sobre les anomalies m es comuns i les seves caracter stiques. En segona inst ancia, no tenir en compte la possibilitat de treballar amb nom es part de les dades (mostreig de tr ansit) continua sent bastant est es tot i el sever efecte en el rendiment dels algorismes de detecci o d'anomalies. En tercer lloc, els operadors de xarxa actualment han d'invertir moltes hores de feina per classi car i inspeccionar manualment les anomalies detectades per actuar en conseqüencia i prendre les mesures apropiades de mitigaci o. Aquesta situaci o es veu agreujada per l'alt nombre de falsos positius i falsos negatius i perqu e els sistemes de detecci o d'anomalies s on sovint percebuts com caixes negres extremadament complexes. Analitzar un tema es essencial per comprendre plenament l'espai del problema i per poder-hi fer front de forma adequada. Per tant, el primer bloc d'aquesta tesi pret en proporcionar informaci o detallada i actualitzada del m on real sobre les anomalies m es freqüents en una xarxa troncal. Primer es comparen tres eines comercials per a la detecci o d'anomalies i se n'estudien els seus punts forts i febles, aix com els tipus d'anomalies de xarxa detectats. Posteriorment, s'investiguen les caracter stiques de les anomalies que es troben en la mateixa xarxa troncal utilitzant una de les eines durant m es de mig any. Entre d'altres resultats, aquest bloc con rma la necessitat de l'aplicaci o de mostreig de tr ansit en un entorn operacional, aix com el nombre inacceptablement elevat de falsos positius i falsos negatius en eines comercials actuals. En general, el mostreig de tr ansit de dades de xarxa ( es a dir, treballar nom es amb una part de les dades) en grans xarxes troncals s'ha convertit en gaireb e obligatori i, per tant, tots els algorismes de detecci o d'anomalies que no ho tenen en compte poden veure seriosament afectats els seus resultats. El segon bloc d'aquesta tesi analitza i confi rma el dram atic impacte de mostreig en el rendiment de t ecniques de detecci o d'anomalies plenament acceptades a l'estat de l'art. No obstant, es mostra que els resultats canvien signi cativament depenent de la t ecnica de mostreig utilitzada i tamb e en funci o de la m etrica usada per a fer la comparativa. Contr ariament als resultats reportats en estudis previs, es mostra que Packet Sampling supera Flow Sampling. A m es, a m es, s'observa que Selective Sampling (SES), una t ecnica de mostreig que se centra en mostrejar fluxes petits, obt e resultats molt millors per a la detecci o d'escanejos que no pas les t ecniques tradicionals de mostreig. En conseqü encia, proposem Online Selective Sampling, una t ecnica de mostreig que obt e el mateix bon rendiment per a la detecci o d'escanejos que SES, per o treballa paquet per paquet enlloc de mantenir tots els fluxes a mem oria. Despr es de validar i evaluar la nostra proposta, demostrem que es capa c de treballar online i utilitza molts menys recursos que SES. Tot i la gran quantitat de tècniques proposades a la literatura per a la detecci o d'esdeveniments an omals, la investigaci o per a la seva posterior classi caci o i extracci o (p.ex., per investigar m es a fons el que va passar o per compartir l'evid encia amb tercers involucrats) es m es aviat marginal. Aix o fa que sigui m es dif cil per als operadors de xarxa analalitzar les anomalies reportades, ja que depenen unicament de la seva experi encia per fer la feina. A m es a m es, aquesta tasca es un proc es extremadament lent i propens a errors. El tercer bloc d'aquesta tesi se centra en aquest tema tenint tamb e en compte els coneixements adquirits en els blocs anteriors. Concretament, presentem un sistema per a la detecci o extracci o i classi caci o autom atica d'anomalies amb una alta precisi o i molt pocs falsos positius. Adicionalment, despleguem el sistema en un entorn operatiu i demostrem la seva utilitat pr actica. El quart i ultim bloc d'aquesta tesi presenta una generalitzaci o del nostre sistema que se centra en l'an alisi de tot el tr ansit, no nom es en les anomalies. Aquest nou sistema pret en ajudar m es als operadors ja que resumeix els patrons de tr ansit m es importants de la seva xarxa. En particular, es generalitza el sistema per fer front al "big data" (una gran quantitat de dades). En particular, el sistema tracta IPs origen i dest i, ports origen i destí , protocol, Sistemes Aut onoms origen i dest , aplicaci o que ha generat el tr ansit i fi nalment, dades de geolocalitzaci o (tamb e per origen i dest ). Primer, despleguem un prototip a la xarxa europea per a la recerca i la investigaci o (G EANT) i demostrem que el sistema pot processar grans quantitats de dades r apidament aix com crear informes altament informatius i compactes que s on de gran utilitat per ajudar a comprendre el que est a succeint a la xarxa. En segon lloc, despleguem la nostra eina en un escenari completament diferent i mostrem com tamb e pot ser utilitzat amb exit en un cas d' us en el m on real en el qual s'analitza el comportament de dispositius altament distribuïts.

Per far fronte a esigenze computazionali elevate, necessarie per la risoluzione di problemi complessi, la scienza e le industrie fanno spesso uso di sistemi di calcolo di enormi dimensioni e potenza. I sistemi HPC (High Performance Computing) sono identificabili come un insieme di tanti computer cooperanti e connessi tra loro, chiamati singolarmente “nodi”. I costi da sostenere per l’acquisto o la costruzione di questi sistemi ammontano a svariate decine di milioni di euro. Per questo motivo viene spesso affittata la potenza di calcolo di questi sistemi in modalità on-demand, grazie alla tecnologia del Cloud Computing. In queste circostanze risulta necessario garantire quella che viene definita “qualità del servizio”(QoS) , la quale comprende la disponibilità del sistema (availability). Questo significa che un cluster HPC non deve mai (o quasi) essere inaccessibile a chi ha pagato per usufruirne. L’oggetto di questa tesi di laurea magistrale nasce da questa necessità e si propone di trovare una soluzione costruendo un modello, basato sull’utilizzo di tecniche facenti riferimento all’Intelligenza Artificiale (ed in particolare al Machine Learning), che sia in grado di prevedere in anticipo il sorgere di nuove anomalie nei nodi del sistema, di modo che un amministratore possa intervenire tempestivamente, evitando cosı̀ che questo debba essere reso inaccessibile all’utenza. Il sistema HPC su cui è stato testato il modello è di proprietà di CINECA, un consorzio universitario formato da 67 università italiane e 13 istituzioni, prende il nome di MARCONI e, grazie alla sua potenza di calcolo di 20 Pflops/s, si posiziona al 19esimo posto su scala mondiale in termini di prestazioni.

<p>Critical networks require defence in depth incorporating many different security technologies including intrusion detection. One important intrusion detection approach is called anomaly detection where normal (good) behaviour of users of the protected system is modelled, often using machine learning or data mining techniques. During detection new data is matched against the normality model, and deviations are marked as anomalies. Since no knowledge of attacks is needed to train the normality model, anomaly detection may detect previously unknown attacks.</p><p>In this thesis we present ADWICE (Anomaly Detection With fast Incremental Clustering) and evaluate it in IP networks. ADWICE has the following properties:</p><p>(i) Adaptation - Rather than making use of extensive periodic retraining sessions on stored off-line data to handle changes, ADWICE is fully incremental making very flexible on-line training of the model possible without destroying what is already learnt. When subsets of the model are not useful anymore, those clusters can be forgotten.</p><p>(ii) Performance - ADWICE is linear in the number of input data thereby heavily reducing training time compared to alternative clustering algorithms. Training time as well as detection time is further reduced by the use of an integrated search-index.</p><p>(iii) Scalability - Rather than keeping all data in memory, only compact cluster summaries are used. The linear time complexity also improves scalability of training.</p><p>We have implemented ADWICE and integrated the algorithm in a software agent. The agent is a part of the Safeguard agent architecture, developed to perform network monitoring, intrusion detection and correlation as well as recovery. We have also applied ADWICE to publicly available network data to compare our approach to related works with similar approaches. The evaluation resulted in a high detection rate at reasonable false positives rate.</p><br>Report code: LiU-Tek-Lic-2006:12.

L'utilisation des réseaux de capteurs sans fil (WSN) ne cesse d'augmenter au point de couvrir divers domaines et applications. Cette tendance est supportée par les avancements techniques achevés dans la conception des capteurs, qui ont permis de réduire le coût ainsi que la taille de ces composants. Toutefois, il reste plusieurs défis qui font face au déploiement et au bon fonctionnement de ce type de réseaux et qui parviennent principalement de la limitation des ressources de capteurs ainsi de l'imperfection des données collectées. Dans cette thèse, on adresse le problème de collecte de données et de détection d'anomalies dans les réseaux de capteurs. Nous visons à assurer ces deux fonctionnalités tout en économisant l'utilisation des ressources de capteurs et en prolongeant la durée de vie de réseaux. Tout au long de ce travail, nous présentons plusieurs solutions qui permettent une collecte efficace de données de capteurs ainsi que une bonne détection des éventuelles anomalies. Dans notre première contribution, nous décrivons une solution basée sur la technique Compressive Sensing (CS) qui permet d'équilibrer le trafic transmis par les nœuds dans le réseau. Notre approche diffère des solutions existantes par la prise en compte de la corrélation temporelle ainsi que spatiale dans le processus de décompression des données. De plus, nous proposons une nouvelle formulation pour détecter les anomalies. Les simulations réalisées sur des données réelles prouvent l'efficacité de notre approche en termes de reconstruction de données et de détection d'anomalies par rapport aux approches existantes. Pour mieux optimiser l'utilisation des ressources de WSNs, nous proposons dans une deuxième contribution une solution de collecte de données et de détection d'anomalies basée sur la technique Matrix Completion (MC) qui consiste à transmettre un sous ensemble aléatoire de données de capteurs. Nous développons un algorithme qui estime les mesures manquantes en se basant sur plusieurs propriétés des données. L'algorithme développé permet également de dissimuler les anomalies de la structure normale des données. Cette solution est améliorée davantage dans notre troisième contribution, où nous proposons une formulation différente du problème de collecte de données et de détection d'anomalies. Nous reformulons les connaissances a priori sur les données cibles par des contraintes convexes. Ainsi, les paramètres impliqués dans l'algorithme développé sont liés a certaines propriétés physiques du phénomène observé et sont faciles à ajuster. Nos deux approches montrent de bonnes performances en les simulant sur des données réelles. Enfin, nous proposons dans la dernière contribution une nouvelle technique de collecte de données qui consiste à envoyer que les positions les plus importantes dans la représentation parcimonieuse des données uniquement. Nous considérons dans cette approche le bruit qui peut s'additionner aux données reçues par le nœud collecteur. Cette solution permet aussi de détecter les pics dans les mesures prélevées. En outre, nous validons l'efficacité de notre solution par une analyse théorique corroborée par des simulations sur des données réelles<br>The use of Wireless Sensor Networks (WSN)s is steadily increasing to cover various applications and domains. This trend is supported by the technical advancements in sensor manufacturing process which allow a considerable reduction in the cost and size of these components. However, there are several challenges facing the deployment and the good functioning of this type of networks. Indeed, WSN's applications have to deal with the limited energy, memory and processing capacities of sensor nodes as well as the imperfection of the probed data. This dissertation addresses the problem of collecting data and detecting anomalies in WSNs. The aforementioned functionality needs to be achieved while ensuring a reliable data quality at the collector node, a good anomaly detection accuracy, a low false alarm rate as well as an efficient energy consumption solution. Throughout this work, we provide different solutions that allow to meet these requirements. Foremost, we propose a Compressive Sensing (CS) based solution that allows to equilibrate the traffic carried by nodes regardless their distance from the sink. This solution promotes a larger lifespan of the WSN since it balances the energy consumption between sensor nodes. Our approach differs from existing CS-based solutions by taking into account the sparsity of sensory representation in the temporal domain in addition to the spatial dimension. Moreover, we propose a new formulation to detect aberrant readings. The simulations carried on real datasets prove the efficiency of our approach in terms of data recovering and anomaly detection compared to existing solutions. Aiming to further optimize the use of WSN resources, we propose in our second contribution a Matrix Completion (MC) based data gathering and anomaly detection solution where an arbitrary subset of nodes contributes at the data gathering process at each operating period. To fill the missing values, we mainly relay on the low rank structure of sensory data as well as the sparsity of readings in some transform domain. The developed algorithm also allows to dissemble anomalies from the normal data structure. This solution is enhanced in our third contribution where we propose a constrained formulation of the data gathering and anomalies detection problem. We reformulate the textit{a prior} knowledge about the target data as hard convex constraints. Thus, the involved parameters into the developed algorithm become easy to adjust since they are related to some physical properties of the treated data. Both MC based approaches are tested on real datasets and demonstrate good capabilities in terms of data reconstruction quality and anomaly detection performance. Finally, we propose in the last contribution a position based compressive data gathering scheme where nodes cooperate to compute and transmit only the relevant positions of their sensory sparse representation. This technique provide an efficient tool to deal with the noisy nature of WSN environment as well as detecting spikes in the sensory data. Furthermore, we validate the efficiency of our solution by a theoretical analysis and corroborate it by a simulation evaluation

The project's objective is to detect network anomalies happening in a telecommunication network due to hardware malfunction or software defects after a vast upgrade on the network's system over a specific area, such as a city. The network's system generates statistical data at a 15-minute interval for different locations in the area of interest. For every interval, all statistical data generated over an area are aggregated and converted to images. In this way, an image represents a snapshot of the network for a specific interval, where statistical data are represented as points having different density values. To that problem, this project makes use of Generative Adversarial Networks (GANs), which learn a manifold of the normal network pattern. Additionally, mapping from new unseen images to the learned manifold results in an anomaly score used to detect anomalies. The anomaly score is a combination of the reconstruction error and the learned feature representation. Two models for detecting anomalies are used in this project, AnoGAN and f-AnoGAN. Furthermore, f-AnoGAN uses a state-of-the-art approach called Wasstestein GAN with gradient penalty, which improves the initial implementation of GANs. Both quantitative and qualitative evaluation measurements are used to assess GANs models, where F1 Score and Wasserstein loss are used for the quantitative evaluation and linear interpolation in the hidden space for qualitative evaluation. Moreover, to set a threshold, a prediction model used to predict the expected behaviour of the network for a specific interval. Then, the predicted behaviour is used over the anomaly detection model to define a threshold automatically. Our experiments were implemented successfully for both prediction and anomaly detection models. We additionally tested known abnormal behaviours which were detected and visualised. However, more research has to be done over the evaluation of GANs, as there is no universal approach to evaluate them.

Supervisory Control and Data Acquisition (SCADA) systems control and monitor critical infrastructure in society, such as electricity transmission and distribution systems. Modern SCADA systems are increasingly adopting open architectures, protocols, and standards and being connected to the Internet to enable remote control. A boost in sophisticated attacks against SCADA systems makes SCADA security a pressing issue. An Intrusion Detection System (IDS) is a security countermeasure that monitors a network and tracks unauthenticated activities inside the network. Most commercial IDSs used in general IT systems are signature-based, by which an IDS compares the system behaviors with known attack patterns. Unfortunately, recent attacks against SCADA systems exploit zero-day vulnerabilities in SCADA devices which are undetectable by signature-based IDSs. This thesis aims to enhance SCADA system monitoring by anomaly detection that models normal behaviors and finds deviations from the model. With anomaly detection, zero-day attacks are possible to detect. We focus on modeling the timing attributes of SCADA traffic for two reasons: (1) the timing regularity fits the automation nature of SCADA systems, and (2) the timing information (i.e., arrival time) of a packet is captured and sent by a network driver where an IDS is located. Hence, it’s less prone to intentional manipulation by an attacker, compared to the payload of a packet. This thesis first categorises SCADA traffic into two groups, request-response and spontaneous traffic, and studies data collected in three different protocol formats (Modbus, Siemens S7, and IEC-60870-5-104). The request-response traffic is generated by a polling mechanism. For this type of traffic, we model the inter-arrival times for each command and response pair with a statistical approach. Results presented in this thesis show that request-response traffic exists in several SCADA traffic sets collected from systems with different sizes and settings. The proposed statistical approach for request-response traffic can detect attacks having subtle changes in timing, such as a single packet insertion and TCP prediction for two of the three SCADA protocols studied. The spontaneous traffic is generated by remote terminal units when they see significant changes in measurement values. For this type of traffic, we first use a pattern mining approach to find the timing characteristics of the data. Then, we model the suggested attributes with machine learning approaches and run it on traffic collected in a real power facility. We test our anomaly detection model with two types of attacks. One causes persistent anomalies and another only causes intermittent ones. Our anomaly detector exhibits a 100% detection rate with at most 0.5% false positive rate for the attacks with persistent anomalies. For the attacks with intermittent anomalies, we find our approach effective when (1) the anomalies last for a longer period (over 1 hour), or (2) the original traffic has relatively low volume.

The recent proliferation of sensors in inhospitable environments such as disaster or battle zones has not been matched by in situ data processing capabilities due to a lack of computing infrastructure in the field. We envision a solution based on small, low-altitude unmanned aerial vehicles (UAVs) that can deploy elastically-scalable computing infrastructure anywhere, at any time. This airborne compute cloud—essentially, micro-data centers hosted on UAVs—would communicate with terrestrial assets over a bandwidth-constrained wireless network with variable, unpredictable link qualities. Achieving high performance over this ground-to-air mobile radio channel thus requires making full and efficient use of every single transmission opportunity. To this end, this dissertation presents two system primitives that improve throughput and reduce network overhead by using recent distributed coding methods to exploit natural properties of the airborne environment (i.e., antenna beam diversity and anomaly sparsity). We first built and deployed an UAV wireless networking testbed and used it to characterize the ground-to-UAV wireless channel. Our flight experiments revealed that antenna beam diversity from using multiple SISO radios boosts reception range and aggregate throughput. This observation led us to develop our first primitive: ground-to-UAV bulk data transport. We designed and implemented FlowCode, a reliable link layer for uplink data transport that uses network coding to harness antenna beam diversity gains. Via flight experiments, we show that FlowCode can boost reception range and TCP throughput as much as 4.5-fold. Our second primitive permits low-overhead cloud status monitoring. We designed CloudSense, a network switch that compresses cloud status streams in-network via compressive sensing. CloudSense is particularly useful for anomaly detection tasks requiring global relative comparisons (e.g., MapReduce straggler detection) and can achieve up to 16.3-fold compression as well as early detection of the worst anomalies. Our efforts have also shed light on the close relationship between network coding and compressive sensing. Thus, we offer FlowCode and CloudSense not only as first steps toward the airborne compute cloud, but also as exemplars of two classes of applications—approximation intolerant and tolerant—to which network coding and compressive sensing should be judiciously and selectively applied.<br>Engineering and Applied Sciences

Anomaly detection in time series is a broad field with many application areas, and has been researched for many years. In recent years the need for monitoring and DevOps has increased, partly due to the increased usage of microservice infrastructures. Applying time series anomaly detection to the metrics emitted by these microservices can yield new insights into the system health and could enable detecting anomalous conditions before they are escalated into a full incident. This thesis investigates how two proposed anomaly detectors, one based on the RPCA algorithm and the other on the HTM neural network, perform on metrics emitted by a microservice infrastructure, with the goal of enhancing the infrastructure monitoring. The detectors are evaluated against a random sample of metrics from a digital rights management company’s microservice infrastructure, as well as the open source NAB dataset. It is illustrated that both algorithms are able to detect every known incident in the company metrics tested. Their ability to detect anomalies is shown to be dependent on the defined threshold value for what qualifies as an outlier. The RPCA Detector proved to be better at detecting anomalies on the company microservice metrics, however the HTM detector performed better on the NAB dataset. Findings also highlight the difficulty of manually annotating anomalies even with domain knowledge. An issue found to be true for both the dataset created for this project, and the NAB dataset. The thesis concludes that the proposed detectors possess different abilities, both having their respective trade-offs. Although they are similar in detection accuracy and false positive rates, each has different inert abilities to perform tasks such as continuous monitoring or ease of deployment in an existing monitoring setup.<br>Anomalitetsdetektering i tidsserier är ett brett område med många användningsområden och har undersökts under många år. De senaste åren har behovet av övervakning och DevOps ökat, delvis på grund av ökad användning av microservice-infrastrukturer. Att tillämpa tidsserieanomalitetsdetektering på de mätvärden som emitteras av dessa microservices kan ge nya insikter i systemhälsan och kan möjliggöra detektering av avvikande förhållanden innan de eskaleras till en fullständig incident. Denna avhandling undersöker hur två föreslagna anomalitetsdetektorer, en baserad på RPCA-algoritmen och den andra på HTM neurala nätverk, presterar på mätvärden som emitteras av en microservice-infrastruktur, med målet att förbättra infrastrukturövervakningen. Detektorerna utvärderas mot ett slumpmässigt urval av mätvärden från en microservice-infrastruktur på en digital underhållningstjänst, och från det öppet tillgängliga NAB-dataset. Det illustreras att båda algoritmerna kunde upptäcka alla kända incidenter i de testade underhållningstjänst-mätvärdena. Deras förmåga att upptäcka avvikelser visar sig vara beroende av det definierade tröskelvärdet för vad som kvalificeras som en anomali. RPCA-detektorn visade sig bättre på att upptäcka anomalier i underhållningstjänstens mätvärden, men HTM-detektorn presterade bättre på NAB-datasetet. Fynden markerar också svårigheten med att manuellt annotera avvikelser, även med domänkunskaper. Ett problem som visat sig vara sant för datasetet skapat för detta projekt och NAB-datasetet. Avhandlingen slutleder att de föreslagna detektorerna har olikaförmågor, vilka båda har sina respektive avvägningar. De har liknande detekteringsnoggrannhet, men har olika inerta förmågor för att utföra uppgifter som kontinuerlig övervakning, eller enkelhet att installera i en befintlig övervakningsinstallation.

Thesis (Ph. D.)--State University of New York at Binghamton, Thomas J. Watson School of Engineering and Applied Science, Department of Electrical and Computer Engineering, 2009.<br>Includes bibliographical references.