Academic literature on the topic 'Concept Drift Detection'
Create a spot-on reference in APA, MLA, Chicago, Harvard, and other styles
Consult the lists of relevant articles, books, theses, conference reports, and other scholarly sources on the topic 'Concept Drift Detection.'
Next to every source in the list of references, there is an 'Add to bibliography' button. Press on it, and we will generate automatically the bibliographic reference to the chosen work in the citation style you need: APA, MLA, Harvard, Chicago, Vancouver, etc.
You can also download the full text of the academic publication as pdf and read online its abstract whenever available in the metadata.
Journal articles on the topic "Concept Drift Detection"
1
Zhu, Jiaqi, Shaofeng Cai, Fang Deng, Beng Chin Ooi, and Wenqiao Zhang. "METER: A Dynamic Concept Adaptation Framework for Online Anomaly Detection." Proceedings of the VLDB Endowment 17, no. 4 (2023): 794–807. http://dx.doi.org/10.14778/3636218.3636233.
Full textAbstract:
Real-time analytics and decision-making require online anomaly detection (OAD) to handle drifts in data streams efficiently and effectively. Unfortunately, existing approaches are often constrained by their limited detection capacity and slow adaptation to evolving data streams, inhibiting their efficacy and efficiency in handling concept drift , which is a major challenge in evolving data streams. In this paper, we introduce METER, a novel dynamic concept adaptation framework that introduces a new paradigm for OAD. METER addresses concept drift by first training a base detection model on historical data to capture recurring central concepts , and then learning to dynamically adapt to new concepts in data streams upon detecting concept drift. Particularly, METER employs a novel dynamic concept adaptation technique that leverages a hypernetwork to dynamically generate the parameter shift of the base detection model, providing a more effective and efficient solution than conventional retraining or fine-tuning approaches. Further, METER incorporates a lightweight drift detection controller, underpinned by evidential deep learning, to support robust and interpretable concept drift detection. We conduct an extensive experimental evaluation, and the results show that METER significantly outperforms existing OAD approaches in various application scenarios.
2
Sakurai, Guilherme Yukio, Jessica Fernandes Lopes, Bruno Bogaz Zarpelão, and Sylvio Barbon Junior. "Benchmarking Change Detector Algorithms from Different Concept Drift Perspectives." Future Internet 15, no. 5 (2023): 169. http://dx.doi.org/10.3390/fi15050169.
Full textAbstract:
The stream mining paradigm has become increasingly popular due to the vast number of algorithms and methodologies it provides to address the current challenges of Internet of Things (IoT) and modern machine learning systems. Change detection algorithms, which focus on identifying drifts in the data distribution during the operation of a machine learning solution, are a crucial aspect of this paradigm. However, selecting the best change detection method for different types of concept drift can be challenging. This work aimed to provide a benchmark for four drift detection algorithms (EDDM, DDM, HDDMW, and HDDMA) for abrupt, gradual, and incremental drift types. To shed light on the capacity and possible trade-offs involved in selecting a concept drift algorithm, we compare their detection capability, detection time, and detection delay. The experiments were carried out using synthetic datasets, where various attributes, such as stream size, the amount of drifts, and drift duration can be controlled and manipulated on our generator of synthetic stream. Our results show that HDDMW provides the best trade-off among all performance indicators, demonstrating superior consistency in detecting abrupt drifts, but has suboptimal time consumption and a limited ability to detect incremental drifts. However, it outperforms other algorithms in detection delay for both abrupt and gradual drifts with an efficient detection performance and detection time performance.
3
Toor, Affan Ahmed, Muhammad Usman, Farah Younas, Alvis Cheuk M. Fong, Sajid Ali Khan, and Simon Fong. "Mining Massive E-Health Data Streams for IoMT Enabled Healthcare Systems." Sensors 20, no. 7 (2020): 2131. http://dx.doi.org/10.3390/s20072131.
Full textAbstract:
With the increasing popularity of the Internet-of-Medical-Things (IoMT) and smart devices, huge volumes of data streams have been generated. This study aims to address the concept drift, which is a major challenge in the processing of voluminous data streams. Concept drift refers to overtime change in data distribution. It may occur in the medical domain, for example the medical sensors measuring for general healthcare or rehabilitation, which may switch their roles for ICU emergency operations when required. Detecting concept drifts becomes trickier when the class distributions in data are skewed, which is often true for medical sensors e-health data. Reactive Drift Detection Method (RDDM) is an efficient method for detecting long concepts. However, RDDM has a high error rate, and it does not handle class imbalance. We propose an Enhanced Reactive Drift Detection Method (ERDDM), which systematically generates strategies to handle concept drift with class imbalance in data streams. We conducted experiments to compare ERDDM with three contemporary techniques in terms of prediction error, drift detection delay, latency, and ability to handle data imbalance. The experimentation was done in Massive Online Analysis (MOA) on 48 synthetic datasets customized to possess the capabilities of data streams. ERDDM can handle abrupt and gradual drifts and performs better than all benchmarks in almost all experiments.
4
Kumar, Sanjeev, Ravendra Singh, Mohammad Zubair Khan, and Abdulfattah Noorwali. "Design of adaptive ensemble classifier for online sentiment analysis and opinion mining." PeerJ Computer Science 7 (August 5, 2021): e660. http://dx.doi.org/10.7717/peerj-cs.660.
Full textAbstract:
DataStream mining is a challenging task for researchers because of the change in data distribution during classification, known as concept drift. Drift detection algorithms emphasize detecting the drift. The drift detection algorithm needs to be very sensitive to change in data distribution for detecting the maximum number of drifts in the data stream. But highly sensitive drift detectors lead to higher false-positive drift detections. This paper proposed a Drift Detection-based Adaptive Ensemble classifier for sentiment analysis and opinion mining, which uses these false-positive drift detections to benefit and minimize the negative impact of false-positive drift detection signals. The proposed method creates and adds a new classifier to the ensemble whenever a drift happens. A weighting mechanism is implemented, which provides weights to each classifier in the ensemble. The weight of the classifier decides the contribution of each classifier in the final classification results. The experiments are performed using different classification algorithms, and results are evaluated on the accuracy, precision, recall, and F1-measures. The proposed method is also compared with these state-of-the-art methods, OzaBaggingADWINClassifier, Accuracy Weighted Ensemble, Additive Expert Ensemble, Streaming Random Patches, and Adaptive Random Forest Classifier. The results show that the proposed method handles both true positive and false positive drifts efficiently.
5
M, Thangam, Bhuvaneswari A, and Sangeetha J. "A Framework to Detect and Classify Time-based Concept Drift." Indian Journal of Science and Technology 16, no. 48 (2023): 4631–37. https://doi.org/10.17485/IJST/v16i48.583.
Full textAbstract:
Abstract <strong>Objectives:</strong> To design a framework that performs time series decomposition to detect and classify the types of concept drift in a data stream. The aim of this research is to increase the classification accuracy in the detection and classification of drifts. <strong>Methods:</strong> The proposed method is validated using the Beijing PM2.5 dataset available in the UCI Machine Learning Repository. This dataset has 13 attributes and experiments were performed with the existing drift detection framework algorithms such as EFCDD, Meta-ADD, CIDD, and comparisons were performed with the proposed TBD framework. The outcome of this research is aggregated with Classification accuracy. An effective algorithm selection framework is presented that detects and classifies time-based concept drift existing in the data. The temporal aspects of the data are decomposed to determine the algorithm to be applied to detect and classify the types of drifts. Depending on the decomposed levels, three varied algorithms have been applied and used for the effective detection and classification of time-based drifts. <strong>Findings:</strong> The performance of the proposed method is validated using the classification accuracy and compared with the existing drift detection framework algorithms. The proposed framework achieves maximum classification accuracy of 95.24% than all the other existing methods. <strong>Novelty:</strong> A novel framework has been proposed with better classification accuracy for the detection and classification of time-based concept drift. <strong>Keywords</strong>: Feature Selection, Concept Drift, Multiple Drift Detection, Time­series decomposition, Classification Accuracy
6
Dries, Anton, and Ulrich Rückert. "Adaptive concept drift detection." Statistical Analysis and Data Mining: The ASA Data Science Journal 2, no. 5-6 (2009): 311–27. http://dx.doi.org/10.1002/sam.10054.
Full text
7
Lu, Pengqian, Jie Lu, Anjin Liu, and Guangquan Zhang. "Early Concept Drift Detection via Prediction Uncertainty." Proceedings of the AAAI Conference on Artificial Intelligence 39, no. 18 (2025): 19124–32. https://doi.org/10.1609/aaai.v39i18.34105.
Full textAbstract:
Concept drift, characterized by unpredictable changes in data distribution over time, poses significant challenges to machine learning models in streaming data scenarios. Although error rate-based concept drift detectors are widely used, they often fail to identify drift in the early stages when the data distribution changes but error rates remain constant. This paper introduces the Prediction Uncertainty Index (PU-index), derived from the prediction uncertainty of the classifier, as a superior alternative to the error rate for drift detection. Our theoretical analysis demonstrates that: (1) The PU-index can detect drift even when error rates remain stable. (2) Any change in the error rate will lead to a corresponding change in the PU-index. These properties make the PU-index a more sensitive and robust indicator for drift detection compared to existing methods. We also propose a PU-index-based Drift Detector (PUDD) that employs a novel Adaptive PU-index Bucketing algorithm for detecting drift. Empirical evaluations on both synthetic and real-world datasets demonstrate PUDD’s efficacy in detecting drift in structured and image data.
8
Palli, Abdul Sattar, Jafreezal Jaafar, Heitor Murilo Gomes, Manzoor Ahmed Hashmani, and Abdul Rehman Gilal. "An Experimental Analysis of Drift Detection Methods on Multi-Class Imbalanced Data Streams." Applied Sciences 12, no. 22 (2022): 11688. http://dx.doi.org/10.3390/app122211688.
Full textAbstract:
The performance of machine learning models diminishes while predicting the Remaining Useful Life (RUL) of the equipment or fault prediction due to the issue of concept drift. This issue is aggravated when the problem setting comprises multi-class imbalanced data. The existing drift detection methods are designed to detect certain drifts in specific scenarios. For example, the drift detector designed for binary class data may not produce satisfactory results for applications that generate multi-class data. Similarly, the drift detection method designed for the detection of sudden drift may struggle with detecting incremental drift. Therefore, in this experimental investigation, we seek to investigate the performance of the existing drift detection methods on multi-class imbalanced data streams with different drift types. For this reason, this study simulated the streams with various forms of concept drift and the multi-class imbalance problem to test the existing drift detection methods. The findings of current study will aid in the selection of drift detection methods for use in developing solutions for real-time industrial applications that encounter similar issues. The results revealed that among the compared methods, DDM produced the best average F1 score. The results also indicate that the multi-class imbalance causes the false alarm rate to increase for most of the drift detection methods.
9
Hu, Hanqing, and Mehmed Kantardzic. "Heuristic ensemble for unsupervised detection of multiple types of concept drift in data stream classification." Intelligent Decision Technologies 15, no. 4 (2022): 609–22. http://dx.doi.org/10.3233/idt-210115.
Full textAbstract:
Real-world data stream classification often deals with multiple types of concept drift, categorized by change characteristics such as speed, distribution, and severity. When labels are unavailable, traditional concept drift detection algorithms, used in stream classification frameworks, are often focused on only one type of concept drift. To overcome the limitations of traditional detection algorithms, this study proposed a Heuristic Ensemble Framework for Drift Detection (HEFDD). HEFDD aims to detect all types of concept drift by employing an ensemble of selected concept drift detection algorithms, each capable of detecting at least one type of concept drift. Experimental results show HEFDD provides significant improvement based on the z-score test when comparing detection accuracy with state-of-the-art individual algorithms. At the same time, HEFDD is able to reduce false alarms generated by individual concept drift detection algorithms.
10
Sobolewski, Piotr. "Concept Drift Detection and Model Selection with Simulated Recurrence and Ensembles of Statistical Detectors." JUCS - Journal of Universal Computer Science 19, no. (4) (2013): 462–83. https://doi.org/10.3217/jucs-019-04-0462.
Full textAbstract:
The paper presents a concept drift detection method for unsupervised learning which takes into consideration the prior knowledge to select the most appropriate classification model. The prior knowledge carries information about the data distribution patterns that reflect different concepts, which may occur in the data stream. The presented method serves as a temporary solution for a classification system after a virtual concept drift and also provides additional information about the concept data distribution for adapting the classification model. Presented detector uses a developed method called simulated recurrence and detector ensembles based on statistical tests. Evaluation is performed on benchmark datasets.
Dissertations / Theses on the topic "Concept Drift Detection"
1
Ostovar, Alireza. "Business process drift: Detection and characterization." Thesis, Queensland University of Technology, 2019. https://eprints.qut.edu.au/127157/1/Alireza_Ostovar_Thesis.pdf.
Full textAbstract:
This research contributes a set of techniques for the early detection and characterization of process drifts, i.e. statistically significant changes in the behavior of business operations, as recorded in transactional data. Early detection and subsequent characterization of process drifts allows organizations to take prompt remedial actions and avoid potential repercussions resulting from unplanned changes in the behavior of their operations.
2
ESCOVEDO, TATIANA. "NEUROEVOLUTIVE LEARNING AND CONCEPT DRIFT DETECTION IN NON-STATIONARY ENVIRONMENTS." PONTIFÍCIA UNIVERSIDADE CATÓLICA DO RIO DE JANEIRO, 2015. http://www.maxwell.vrac.puc-rio.br/Busca_etds.php?strSecao=resultado&nrSeq=26748@1.
Full textAbstract:
PONTIFÍCIA UNIVERSIDADE CATÓLICA DO RIO DE JANEIRO<br>COORDENAÇÃO DE APERFEIÇOAMENTO DO PESSOAL DE ENSINO SUPERIOR<br>PROGRAMA DE EXCELENCIA ACADEMICA<br>Os conceitos do mundo real muitas vezes não são estáveis: eles
mudam com o tempo. Assim como os conceitos, a distribuição de dados
também pode se alterar. Este problema de mudança de conceitos ou
distribuição de dados é conhecido como concept drift e é um desafio para um
modelo na tarefa de aprender a partir de dados. Este trabalho apresenta um
novo modelo neuroevolutivo com inspiração quântica, baseado em um comitê
de redes neurais do tipo Multi-Layer Perceptron (MLP), para a aprendizagem
em ambientes não estacionários, denominado NEVE (Neuro-EVolutionary
Ensemble). Também apresenta um novo mecanismo de detecção de concept
drift, denominado DetectA (Detect Abrupt) com a capacidade de detectar
mudanças tanto de forma proativa quanto de forma reativa. O algoritmo
evolutivo com inspiração quântica binário-real AEIQ-BR é utilizado no NEVE
para gerar automaticamente novos classificadores para o comitê, determinando
a topologia mais adequada para a nova rede, selecionando as variáveis de
entrada mais apropriadas e determinando todos os pesos da rede neural MLP.
O algoritmo AEIQ-R determina os pesos de votação de cada rede neural
membro do comitê, sendo possível utilizar votação por combinação linear,
votação majoritária ponderada e simples. São implementadas quatro diferentes
abordagens do NEVE, que se diferem uma da outra pela forma de detectar e
tratar os drifts ocorridos. O trabalho também apresenta resultados de
experimentos realizados com o método DetectA e com o modelo NEVE em
bases de dados reais e artificiais. Os resultados mostram que o detector se
mostrou robusto e eficiente para bases de dados de alta dimensionalidade,
blocos de tamanho intermediário, bases de dados com qualquer proporção de
drift e com qualquer balanceamento de classes e que, em geral, os melhores
resultados obtidos foram usando algum tipo de detecção. Comparando a
acurácia do NEVE com outros modelos consolidados da literatura, verifica-se
que o NEVE teve acurácia superior na maioria dos casos. Isto reforça que a
abordagem por comitê neuroevolutivo é uma escolha robusta para situações
em que as bases de dados estão sujeitas a mudanças repentinas de
comportamento.<br>Real world concepts are often not stable: they change with time. Just as
the concepts, data distribution may change as well. This problem of change in
concepts or distribution of data is known as concept drift and is a challenge for
a model in the task of learning from data. This work presents a new
neuroevolutive model with quantum inspiration called NEVE (Neuro-
EVolutionary Ensemble), based on an ensemble of Multi-Layer Perceptron
(MLP) neural networks for learning in non-stationary environments. It also
presents a new concept drift detection mechanism, called DetectA (DETECT
Abrupt) with the ability to detect changes both proactively as reactively. The
evolutionary algorithm with binary-real quantum inspiration AEIQ-BR is used in
NEVE to automatically generate new classifiers for the ensemble, determining
the most appropriate topology for the new network and by selecting the most
appropriate input variables and determining all the weights of the neural
network. The AEIQ-R algorithm determines the voting weight of each neural
network ensemble member, and you can use voting by linear combination and
voting by weighted or simple majority. Four different approaches of NEVE are
implemented and they differ from one another by the way of detecting and
treating occurring drifts. The work also presents results of experiments
conducted with the DetectA method and with the NEVE model in real and
artificial databases. The results show that the detector has proved efficient and
suitable for data bases with high-dimensionality, intermediate sized blocks, any
proportion of drifts and with any class balancing. Comparing the accuracy of
NEVE with other consolidated models in the literature, it appears that NEVE
had higher accuracy in most cases. This reinforces that the neuroevolution
ensemble approach is a robust choice to situations in which the databases are
subject to sudden changes in behavior.
3
Roded, Keren. "The concept of drift and operationalization of its detection in simulated data." Thesis, University of British Columbia, 2017. http://hdl.handle.net/2429/63135.
Full textAbstract:
In this paper, the phenomenon of changes in item characteristics over time (often referred to as drift) is discussed from several theoretical perspectives, and a new procedure for the detection of Item Parameter Drift (IPD) is proposed. An initial evaluation of the utility of the proposed procedure is conducted using simulated data modeled by the 2-Parameter Logistic (2PL) Item Response Theory (IRT) model. In addition to the proposed procedure, an IPD analysis of the simulated data is conducted using two known methods: Kim, Cohen, and Park's (1995) extension of Lord's (1980) Chi-square test of Differential Item Functioning (DIF) to multiple groups, and logistic regression. The results indicate high agreement and accuracy in the detection of true IPD using the two known methods, but poor performance of the proposed procedure. Possible explanations of the findings and future directions are discussed.<br>Education, Faculty of<br>Educational and Counselling Psychology, and Special Education (ECPS), Department of<br>Graduate
4
D'Ettorre, Sarah. "Fine-Grained, Unsupervised, Context-based Change Detection and Adaptation for Evolving Categorical Data." Thesis, Université d'Ottawa / University of Ottawa, 2016. http://hdl.handle.net/10393/35518.
Full textAbstract:
Concept drift detection, the identfication of changes in data distributions in streams,
is critical to understanding the mechanics of data generating processes and ensuring that data models remain representative through time [2]. Many change detection methods utilize statistical techniques that take numerical data as input. However, many applications produce data streams containing categorical attributes. In this context, numerical statistical methods are unavailable, and different approaches are required. Common solutions use error monitoring, assuming that
fluctuations in the error measures of a learning system correspond to concept drift [4]. There has been very little research, though, on context-based concept drift detection in categorical streams. This approach observes changes in the actual data distribution and is less popular due to the challenges associated with categorical data analysis. However, context-based change detection is arguably more informative as it is data-driven, and more widely applicable in that it can function in an unsupervised setting [4]. This study offers a contribution to this gap in the research by proposing a novel context-based change detection and adaptation algorithm for categorical data, namely Fine-Grained Change Detection in Categorical Data Streams (FG-CDCStream). This unsupervised method exploits elements of ensemble learning, a technique whereby decisions are made according to the majority vote of a set of models representing different random subspaces of the data [5]. These ideas are applied to a set of concept drift detector objects and merged with concepts from a recent, state-of-the-art, context-based change detection algorithm, the so-called Change Detection in Categorical Data Streams (CDCStream) [4]. FG-CDCStream is proposed as an extension of the batch-based CDCStream, providing instance-by-instance analysis and improving its change detection capabilities especially in data streams containing abrupt changes or a combination of abrupt and gradual changes. FG-CDCStream also enhances the adaptation strategy of CDCStream producing more representative post-change models.
5
Pesaranghader, Ali. "A Reservoir of Adaptive Algorithms for Online Learning from Evolving Data Streams." Thesis, Université d'Ottawa / University of Ottawa, 2018. http://hdl.handle.net/10393/38190.
Full textAbstract:
Continuous change and development are essential aspects of evolving environments and applications, including, but not limited to, smart cities, military, medicine, nuclear reactors, self-driving cars, aviation, and aerospace. That is, the fundamental characteristics of such environments may evolve, and so cause dangerous consequences, e.g., putting people lives at stake, if no reaction is adopted. Therefore, learning systems need to apply intelligent algorithms to monitor evolvement in their environments and update themselves effectively. Further, we may experience fluctuations regarding the performance of learning algorithms due to the nature of incoming data as it continuously evolves. That is, the current efficient learning approach may become deprecated after a change in data or environment. Hence, the question 'how to have an efficient learning algorithm over time against evolving data?' has to be addressed. In this thesis, we have made two contributions to settle the challenges described above.
In the machine learning literature, the phenomenon of (distributional) change in data is known as concept drift. Concept drift may shift decision boundaries, and cause a decline in accuracy. Learning algorithms, indeed, have to detect concept drift in evolving data streams and replace their predictive models accordingly. To address this challenge, adaptive learners have been devised which may utilize drift detection methods to locate the drift points in dynamic and changing data streams. A drift detection method able to discover the drift points quickly, with the lowest false positive and false negative rates, is preferred. False positive refers to incorrectly alarming for concept drift, and false negative refers to not alarming for concept drift. In this thesis, we introduce three algorithms, called as the Fast Hoeffding Drift Detection Method (FHDDM), the Stacking Fast Hoeffding Drift Detection Method (FHDDMS), and the McDiarmid Drift Detection Methods (MDDMs), for detecting drift points with the minimum delay, false positive, and false negative rates. FHDDM is a sliding window-based algorithm and applies Hoeffding’s inequality (Hoeffding, 1963) to detect concept drift. FHDDM slides its window over the prediction results, which are either 1 (for a correct prediction) or 0 (for a wrong prediction). Meanwhile, it compares the mean of elements inside the window with the maximum mean observed so far; subsequently, a significant difference between the two means, upper-bounded by the Hoeffding inequality, indicates the occurrence of concept drift. The FHDDMS extends the FHDDM algorithm by sliding multiple windows over its entries for a better drift detection regarding the detection delay and false negative rate. In contrast to FHDDM/S, the MDDM variants assign weights to their entries, i.e., higher weights are associated with the most recent entries in the sliding window, for faster detection of concept drift. The rationale is that recent examples reflect the ongoing situation adequately. Then, by putting higher weights on the latest entries, we may detect concept drift quickly. An MDDM algorithm bounds the difference between the weighted mean of elements in the sliding window and the maximum weighted mean seen so far, using McDiarmid’s inequality (McDiarmid, 1989). Eventually, it alarms for concept drift once a significant difference is experienced. We experimentally show that FHDDM/S and MDDMs outperform the state-of-the-art by representing promising results in terms of the adaptation and classification measures.
Due to the evolving nature of data streams, the performance of an adaptive learner, which is defined by the classification, adaptation, and resource consumption measures, may fluctuate over time. In fact, a learning algorithm, in the form of a (classifier, detector) pair, may present a significant performance before a concept drift point, but not after. We define this problem by the question 'how can we ensure that an efficient classifier-detector pair is present at any time in an evolving environment?' To answer this, we have developed the Tornado framework which runs various kinds of learning algorithms simultaneously against evolving data streams. Each algorithm incrementally and independently trains a predictive model and updates the statistics of its drift detector. Meanwhile, our framework monitors the (classifier, detector) pairs, and recommends the efficient one, concerning the classification, adaptation, and resource consumption performance, to the user. We further define the holistic CAR measure that integrates the classification, adaptation, and resource consumption measures for evaluating the performance of adaptive learning algorithms. Our experiments confirm that the most efficient algorithm may differ over time because of the developing and evolving nature of data streams.
6
Henke, Márcia. "Deteção de Spam baseada na evolução das características com presença de Concept Drift." Universidade Federal do Amazonas, 2015. http://tede.ufam.edu.br/handle/tede/4708.
Full textAbstract:
Submitted by Geyciane Santos (geyciane_thamires@hotmail.com) on 2015-11-12T20:17:58Z
No. of bitstreams: 1
Tese - Márcia Henke.pdf: 2984974 bytes, checksum: a103355c1a7895956d40d4fa9422347a (MD5)<br>Approved for entry into archive by Divisão de Documentação/BC Biblioteca Central (ddbc@ufam.edu.br) on 2015-11-16T18:36:36Z (GMT) No. of bitstreams: 1
Tese - Márcia Henke.pdf: 2984974 bytes, checksum: a103355c1a7895956d40d4fa9422347a (MD5)<br>Approved for entry into archive by Divisão de Documentação/BC Biblioteca Central (ddbc@ufam.edu.br) on 2015-11-16T18:43:03Z (GMT) No. of bitstreams: 1
Tese - Márcia Henke.pdf: 2984974 bytes, checksum: a103355c1a7895956d40d4fa9422347a (MD5)<br>Made available in DSpace on 2015-11-16T18:43:03Z (GMT). No. of bitstreams: 1
Tese - Márcia Henke.pdf: 2984974 bytes, checksum: a103355c1a7895956d40d4fa9422347a (MD5)
Previous issue date: 2015-03-30<br>CAPES - Coordenação de Aperfeiçoamento de Pessoal de Nível Superior<br>Electronic messages (emails) are still considered the most significant tools in business
and personal applications due to their low cost and easy access. However, e-mails have
become a major problem owing to the high amount of junk mail, named spam, which fill
the e-mail boxes of users. Among the many problems caused by spam messages, we may
highlight the fact that it is currently the main vector for the spread of malicious activities
such as viruses, worms, trojans, phishing, botnets, among others. Such activities allow
the attacker to have illegal access to penetrating data, trade secrets or to invade the privacy of the sufferers to get some advantage. Several approaches have been proposed to prevent sending unsolicited e-mail messages, such as filters implemented in e-mail servers, spam message classification mechanisms for users to define when particular issue or author is a source of spread of spam and even filters implemented in network electronics. In general, e-mail filter approaches are based on analysis of message content to determine whether or not a message is spam. A major problem with this approach is spam detection in the presence of concept drift. The literature defines concept drift as changes occurring in the concept of data over time, as the change in the features that describe an attack or occurrence of new features. Numerous Intrusion Detection Systems (IDS) use machine learning techniques to monitor the classification error rate in order to detect change. However, when detection occurs, some damage has been caused to the system, a fact that requires updating the classification process and the system operator intervention. To overcome the problems mentioned above, this work proposes a new changing detection method, named Method oriented to the Analysis of the Development of Attacks Characteristics (MECA). The proposed method consists of three steps: 1) classification model training; 2) concept drift detection; and 3) transfer learning. The first step generates classification models as it is commonly conducted in machine learning. The second step introduces two new strategies to avoid concept drift: HFS (Historical-based Features Selection) that analyzes the evolution of the features based on over time historical; and SFS (Similarity-based Features Selection) that analyzes the evolution of the features from the level of similarity obtained between the features vectors of the source and target domains. Finally, the third step focuses on the following questions: what, how and when to transfer acquired knowledge. The answer to the first question is provided by the concept drift detection strategies that identify the new features and store them to be
transferred. To answer the second question, the feature representation transfer approach
is employed. Finally, the transfer of new knowledge is executed as soon as changes that
compromise the classification task performance are identified. The proposed method was developed and validated using two public databases, being one of the datasets built along this thesis. The results of the experiments shown that it is possible to infer a threshold to detect changes in order to ensure the classification model is updated through knowledge transfer. In addition, MECA architecture is able to perform the classification task, as well as the concept drift detection, as two parallel and independent tasks. Finally, MECA uses SVM machine learning algorithm (Support Vector Machines), which is less adherent to the training samples. The results obtained with MECA showed that it is possible to detect changes through feature evolution monitoring before a significant degradation in classification models is achieved.<br>As mensagens eletrônicas (e-mails) ainda são consideradas as ferramentas de maior prestígio no meio empresarial e pessoal, pois apresentam baixo custo e facilidade de acesso. Por outro lado, os e-mails tornaram-se um grande problema devido à elevada quantidade de mensagens não desejadas, denominadas spam, que lotam as caixas de emails dos usuários. Dentre os diversos problemas causados pelas mensagens spam, destaca-se o fato de ser atualmente o principal vetor de propagação de atividades
maliciosas como vírus, worms, cavalos de Tróia, phishing, botnets, dentre outros. Tais atividades permitem ao atacante acesso indevido a dados sigilosos, segredos de negócios ou mesmo invadir a privacidade das vítimas para obter alguma vantagem. Diversas abordagens, comerciais e acadêmicas, têm sido propostas para impedir o envio de mensagens de e-mails indesejados como filtros implementados nos servidores
de e-mail, mecanismos de classificação de mensagens de spam para que os usuários definam quando determinado assunto ou autor é fonte de propagação de spam e até mesmo filtros implementados em componentes eletrônicos de rede. Em geral, as abordagens de filtros de e-mail são baseadas na análise do conteúdo das mensagens para determinar se tal mensagem é ou não um spam. Um dos maiores problemas com essa abordagem é a deteção de spam na presença de concept drift. A literatura conceitua concept drift como mudanças que ocorrem no conceito dos dados ao longo do tempo como a alteração das características que descrevem um ataque ou ocorrência de novas características. Muitos Sistemas de Deteção de Intrusão (IDS) usam técnicas de aprendizagem de máquina para monitorar a taxa de erro de
classificação no intuito de detetar mudança. Entretanto, quando a deteção ocorre, algum dano já foi causado ao sistema, fato que requer atualização do processo de classificação e a intervenção do operador do sistema. Com o objetivo de minimizar os problemas mencionados acima, esta tese propõe um método de deteção de mudança, denominado Método orientado à Análise da Evolução das Características de Ataques (MECA). O método proposto é composto por três etapas: 1) treino do modelo de classificação; 2) deteção de mudança; e 3) transferência do aprendizado. A primeira etapa emprega modelos de classificação comumente adotados em qualquer método que utiliza aprendizagem de máquina. A segunda etapa apresenta duas novas estratégias para contornar concept drift: HFS (Historical-based Features Selection) que analisa a evolução das características com base no histórico ao longo do tempo; e SFS (Similarity based Features Selection) que observa a evolução das características a partir do nível de similaridade obtido entre os vetores de características dos domínios fonte e alvo. Por fim, a terceira etapa concentra seu objetivo nas seguintes questões: o que, como e quando transferir conhecimento adquirido. A resposta à primeira questão é fornecida pelas estratégias de deteção de mudança, que identificam as novas características e as armazenam para que sejam transferidas. Para responder a segunda questão, a abordagem de transferência de representação de características é adotada. Finalmente, a transferência do novo conhecimento é realizada tão logo mudanças que comprometam o desempenho da tarefa de classificação sejam identificadas. O método MECA foi desenvolvido e validado usando duas bases de dados
públicas, sendo que uma das bases foi construída ao longo desta tese. Os resultados dos experimentos indicaram que é possível inferir um limiar para detetar mudanças a fim de garantir o modelo de classificação sempre atualizado por meio da transferência de conhecimento. Além disso, um diferencial apresentado no método MECA é a possibilidade de executar a tarefa de classificação em paralelo com a deteção de mudança, sendo as duas tarefas independentes. Por fim, o MECA utiliza o algoritmo de aprendizagem de máquina SVM (Support Vector Machines), que é menos aderente às amostras de treinamento. Os resultados obtidos com o MECA mostraram que é possível detetar mudanças por meio da evolução das características antes de ocorrer uma degradação significativa no modelo de classificação utilizado.
7
SANTOS, Silas Garrido Teixeira de Carvalho. "Avaliação criteriosa dos algoritmos de detecção de concept drifts." Universidade Federal de Pernambuco, 2015. https://repositorio.ufpe.br/handle/123456789/17310.
Full textAbstract:
Submitted by Fabio Sobreira Campos da Costa (fabio.sobreira@ufpe.br) on 2016-07-11T12:33:28Z
No. of bitstreams: 2
license_rdf: 1232 bytes, checksum: 66e71c371cc565284e70f40736c94386 (MD5)
silas-dissertacao-versao-final-2016.pdf: 1708159 bytes, checksum: 6c0efc5f2f0b27c79306418c9de516f1 (MD5)<br>Made available in DSpace on 2016-07-11T12:33:28Z (GMT). No. of bitstreams: 2
license_rdf: 1232 bytes, checksum: 66e71c371cc565284e70f40736c94386 (MD5)
silas-dissertacao-versao-final-2016.pdf: 1708159 bytes, checksum: 6c0efc5f2f0b27c79306418c9de516f1 (MD5)
Previous issue date: 2015-02-27<br>FACEPE<br>A extração de conhecimento em ambientes com fluxo contínuo de dados é uma atividade que
vem crescendo progressivamente. Diversas são as situações que necessitam desse mecanismo,
como o monitoramento do histórico de compras de clientes; a detecção de presença por meio
de sensores; ou o monitoramento da temperatura da água. Desta maneira, os algoritmos
utilizados para esse fim devem ser atualizados constantemente, buscando adaptar-se às
novas instâncias e levando em consideração as restrições computacionais. Quando se
trabalha em ambientes com fluxo contínuo de dados, em geral não é recomendável supor
que sua distribuição permanecerá estacionária. Diversas mudanças podem ocorrer ao longo
do tempo, desencadeando uma situação geralmente conhecida como mudança de conceito
(concept drift). Neste trabalho foi realizado um estudo comparativo entre alguns dos
principais métodos de detecção de mudanças: ADWIN, DDM, DOF, ECDD, EDDM, PL e
STEPD. Para execução dos experimentos foram utilizadas bases artificiais – simulando
mudanças abruptas, graduais rápidas, e graduais lentas – e também bases com problemas
reais. Os resultados foram analisados baseando-se na precisão, tempo de execução, uso
de memória, tempo médio de detecção das mudanças, e quantidade de falsos positivos e
negativos. Já os parâmetros dos métodos foram definidos utilizando uma versão adaptada
de um algoritmo genético. De acordo com os resultados do teste de Friedman juntamente
com Nemenyi, em termos de precisão, DDM se mostrou o método mais eficiente com as
bases utilizadas, sendo estatisticamente superior ao DOF e ECDD. Já EDDM foi o método
mais rápido e também o mais econômico no uso da memória, sendo superior ao DOF,
ECDD, PL e STEPD, em ambos os casos. Conclui-se então que métodos mais sensíveis
às detecções de mudanças, e consequentemente mais propensos a alarmes falsos, obtêm
melhores resultados quando comparados a métodos menos sensíveis e menos suscetíveis a
alarmes falsos.<br>Knowledge extraction from data streams is an activity that has been progressively receiving
an increased demand. Examples of such applications include monitoring purchase history
of customers, movement data from sensors, or water temperatures. Thus, algorithms used
for this purpose must be constantly updated, trying to adapt to new instances and taking
into account computational constraints. When working in environments with a continuous
flow of data, there is no guarantee that the distribution of the data will remain stationary.
On the contrary, several changes may occur over time, triggering situations commonly
known as concept drift. In this work we present a comparative study of some of the main
drift detection methods: ADWIN, DDM, DOF, ECDD, EDDM, PL and STEPD. For
the execution of the experiments, artificial datasets were used – simulating abrupt, fast
gradual, and slow gradual changes – and also datasets with real problems. The results
were analyzed based on the accuracy, runtime, memory usage, average time to change
detection, and number of false positives and negatives. The parameters of methods were
defined using an adapted version of a genetic algorithm. According to the Friedman test
with Nemenyi results, in terms of accuracy, DDM was the most efficient method with
the datasets used, and statistically superior to DOF and ECDD. EDDM was the fastest
method and also the most economical in memory usage, being statistically superior to
DOF, ECDD, PL and STEPD, in both cases. It was concluded that more sensitive change
detection methods, and therefore more prone to false alarms, achieve better results when
compared to less sensitive and less susceptible to false alarms methods.
8
Dal, Pozzolo Andrea. "Adaptive Machine Learning for Credit Card Fraud Detection." Doctoral thesis, Universite Libre de Bruxelles, 2015. http://hdl.handle.net/2013/ULB-DIPOT:oai:dipot.ulb.ac.be:2013/221654.
Full textAbstract:
Billions of dollars of loss are caused every year by fraudulent credit card transactions. The design of efficient fraud detection algorithms is key for reducing these losses, and more and more algorithms rely on advanced machine learning techniques to assist fraud investigators. The design of fraud detection algorithms is however particularly challenging due to the non-stationary distribution of the data, the highly unbalanced classes distributions and the availability of few transactions labeled by fraud investigators. At the same time public data are scarcely available for confidentiality issues, leaving unanswered many questions about what is the best strategy. In this thesis we aim to provide some answers by focusing on crucial issues such as: i) why and how undersampling is useful in the presence of class imbalance (i.e. frauds are a small percentage of the transactions), ii) how to deal with unbalanced and evolving data streams (non-stationarity due to fraud evolution and change of spending behavior), iii) how to assess performances in a way which is relevant for detection and iv) how to use feedbacks provided by investigators on the fraud alerts generated. Finally, we design and assess a prototype of a Fraud Detection System able to meet real-world working conditions and that is able to integrate investigators’ feedback to generate accurate alerts.<br>Doctorat en Sciences<br>info:eu-repo/semantics/nonPublished
9
Dong, Yue. "Higher Order Neural Networks and Neural Networks for Stream Learning." Thesis, Université d'Ottawa / University of Ottawa, 2017. http://hdl.handle.net/10393/35731.
Full textAbstract:
The goal of this thesis is to explore some variations of neural networks. The thesis is mainly split into two parts: a variation of the shaping functions in neural networks and a variation of learning rules in neural networks.
In the first part, we mainly investigate polynomial perceptrons - a perceptron with a polynomial shaping function instead of a linear one. We prove the polynomial perceptron convergence theorem and illustrate the notion by showing that a higher order perceptron can learn the XOR function through empirical experiments with implementation. In the second part, we propose three models (SMLP, SA, SA2) for stream learning and anomaly detection in streams. The main technique allowing these models to perform at a level comparable to the state-of-the-art algorithms in stream learning is the learning rule used. We employ mini-batch gradient descent algorithm and stochastic gradient descent algorithm to speed up the models. In addition, the use of parallel processing with multi-threads makes the proposed methods highly efficient in dealing with streaming data. Our analysis shows that all models have linear runtime and constant memory requirement. We also demonstrate empirically that the proposed methods feature high detection rate, low false alarm rate, and fast response.
The paper on the first two models (SMLP, SA) is published in the 29th Canadian AI Conference and won the best paper award. The invited journal paper on the third model (SA2) for Computational Intelligence is under peer review.
10
Togbe, Maurras Ulbricht. "Détection distribuée d'anomalies dans les flux de données." Electronic Thesis or Diss., Sorbonne université, 2022. http://www.theses.fr/2022SORUS400.
Full textAbstract:
La détection d'anomalies est une problématique importante dans de nombreux domaines d'application comme la santé, le transport, l'industrie etc. Il s'agit d'un sujet d'actualité qui tente de répondre à la demande toujours croissante dans différents domaines tels que la détection d'intrusion, de fraude, etc. Dans cette thèse, après un état de l'art général complet, la méthode non supervisé Isolation Forest (IForest) a été étudiée en profondeur en présentant ses limites qui n'ont pas été abordées dans la littérature. Notre nouvelle version de IForest appelée Majority Voting IForest permet d'améliorer son temps d'exécution. Nos méthodes ADWIN-based IForest ASD et NDKSWIN-based IForest ASD permettent la détection d'anomalies dans les flux de données avec une meilleure gestion du concept drift. Enfin, la détection distribuée d'anomalies en utilisant IForest a été étudiée et évaluée. Toutes nos propositions ont été validées avec des expérimentations sur différents jeux de données<br>Anomaly detection is an important issue in many application areas such as healthcare, transportation, industry etc. It is a current topic that tries to meet the ever increasing demand in different areas such as intrusion detection, fraud detection, etc. In this thesis, after a general complet state of the art, the unsupervised method Isolation Forest (IForest) has been studied in depth by presenting its limitations that have not been addressed in the literature. Our new version of IForest called Majority Voting IForest improves its execution time. Our ADWIN-based IForest ASD and NDKSWIN-based IForest ASD methods allow the detection of anomalies in data stream with a better management of the drift concept. Finally, distributed anomaly detection using IForest has been studied and evaluated. All our proposals have been validated with experiments on different datasets
Book chapters on the topic "Concept Drift Detection"
1
Putatunda, Sayan. "Concept Drift Detection in Data Streams." In Practical Machine Learning for Streaming Data with Python. Apress, 2021. http://dx.doi.org/10.1007/978-1-4842-6867-4_2.
Full text
2
Ogasawara, Eduardo, Rebecca Salles, Fabio Porto, and Esther Pacitti. "Change Points and Concept Drift Detection." In Synthesis Lectures on Data Management. Springer Nature Switzerland, 2025. https://doi.org/10.1007/978-3-031-75941-3_4.
Full text
3
Zenisek, Jan, Gabriel Kronberger, Josef Wolfartsberger, Norbert Wild, and Michael Affenzeller. "Concept Drift Detection with Variable Interaction Networks." In Computer Aided Systems Theory – EUROCAST 2019. Springer International Publishing, 2020. http://dx.doi.org/10.1007/978-3-030-45093-9_36.
Full text
4
Liu, Anjin, Guangquan Zhang, and Jie Lu. "Concept Drift Detection Based on Anomaly Analysis." In Neural Information Processing. Springer International Publishing, 2014. http://dx.doi.org/10.1007/978-3-319-12637-1_33.
Full text
5
Yu, Shujian, and Zubin Abraham. "Concept Drift Detection with Hierarchical Hypothesis Testing." In Proceedings of the 2017 SIAM International Conference on Data Mining. Society for Industrial and Applied Mathematics, 2017. http://dx.doi.org/10.1137/1.9781611974973.86.
Full text
6
Abirami, M. G., and Gilad Gressel. "Concept Drift Detection Using Minimum Prediction Deviation." In Advances in Intelligent Systems and Computing. Springer Singapore, 2021. http://dx.doi.org/10.1007/978-981-16-1249-7_24.
Full text
7
Menon, Aditya Gopal, and Gilad Gressel. "Concept Drift Detection in Phishing Using Autoencoders." In Communications in Computer and Information Science. Springer Singapore, 2021. http://dx.doi.org/10.1007/978-981-16-0419-5_17.
Full text
8
Sobolewski, Piotr, and Michał Woźniak. "Enhancing Concept Drift Detection with Simulated Recurrence." In Advances in Intelligent Systems and Computing. Springer Berlin Heidelberg, 2013. http://dx.doi.org/10.1007/978-3-642-32518-2_15.
Full text
9
Takano, Taisei, and Hisashi Koga. "Fast Concept Drift Detection Exploiting Product Quantization." In Lecture Notes in Computer Science. Springer Nature Switzerland, 2024. http://dx.doi.org/10.1007/978-3-031-68312-1_20.
Full text
10
Friedrich, Björn. "Unsupervised Statistical Concept Drift Detection for Behaviour Abnormality Detection." In Empowering Independent Living using the ICF. Springer Fachmedien Wiesbaden, 2024. http://dx.doi.org/10.1007/978-3-658-44688-8_5.
Full textConference papers on the topic "Concept Drift Detection"
1
Li, Jin, Kleanthis Malialis, Stelios G. Vrachimis, and Marios M. Polycarpou. "Online Detection of Water Contamination Under Concept Drift." In 2025 IEEE Symposia on Computational Intelligence for Energy, Transport and Environmental Sustainability (CIETES Companion). IEEE, 2025. https://doi.org/10.1109/cietescompanion65203.2025.11003354.
Full text
2
Ilic, Aleksandra Stojnev, and Dragan Stojanovic. "Visual Analytics of Streaming Data in Concept Drift Detection." In 2024 11th International Conference on Electrical, Electronic and Computing Engineering (IcETRAN). IEEE, 2024. http://dx.doi.org/10.1109/icetran62308.2024.10645138.
Full text
3
Yu, Jiangbin, Han Liu, Qing Guo, Xia Chen, and Meng Xue. "A Concept Drift Detection Algorithm for Power Data Stream." In 2024 6th International Conference on Energy Systems and Electrical Power (ICESEP). IEEE, 2024. http://dx.doi.org/10.1109/icesep62218.2024.10651984.
Full text
4
Li, Mengyuan. "An Online Anomaly Detection Algorithm with Adaptive Concept Drift." In 2024 6th International Conference on Communications, Information System and Computer Engineering (CISCE). IEEE, 2024. http://dx.doi.org/10.1109/cisce62493.2024.10653010.
Full text
5
Rahimli, Leyla, Feras M. Awaysheh, Sawsan Al Zubi, and Sadi Alawadi. "Federated Learning Drift Detection: An Empirical Study on the Impact of Concept and Data Drift." In 2024 2nd International Conference on Federated Learning Technologies and Applications (FLTA). IEEE, 2024. https://doi.org/10.1109/flta63145.2024.10839814.
Full text
6
Chen, Yijie, and Wei Guo. "Concept drift data stream regression model based on adaptive drift detection and incremental broad learning." In International Conference on Computer Graphics, Artificial Intelligence, and Data Processing (ICCAID 2024), edited by Xin Xu and Azlan bin Mohd Zain. SPIE, 2025. https://doi.org/10.1117/12.3061638.
Full text
7
Yang, Liusha, Zhongwen Peng, and Haosheng Yu. "Unsupervised Concept Drift Detection and Adaptation Based on Random Forest." In 2025 8th International Conference on Advanced Algorithms and Control Engineering (ICAACE). IEEE, 2025. https://doi.org/10.1109/icaace65325.2025.11019964.
Full text
8
Yu, Xiangyu, Carlos Natalino, Paolo Monti, et al. "Enhancing Operational Security of Human-to-Machine Applications through Concept Drift Detection." In Optical Fiber Communication Conference. Optica Publishing Group, 2025. https://doi.org/10.1364/ofc.2025.w3j.5.
Full textAbstract:
We propose a novel concept drift detection framework to detect and mitigate malicious traffic for human-to-machine applications in fiber-wireless access networks, reducing uplink latency by up to 98% and enhancing its operational security.
9
Dries, Anton, and Ulrich Rückert. "Adaptive Concept Drift Detection." In Proceedings of the 2009 SIAM International Conference on Data Mining. Society for Industrial and Applied Mathematics, 2009. http://dx.doi.org/10.1137/1.9781611972795.21.
Full text
10
Pedro Mauad Nogueira, João, and Gilberto Reynoso Meza. "OPTIMIZED ENSEMBLED CONCEPT DRIFT DETECTION." In ANAIS DO LVI SIMPóSIO BRASILEIRO DE PESQUISA OPERACIONAL. Galoa, 2024. https://doi.org/10.59254/sbpo-2024-193497.
Full text