Academic literature on the topic 'Identical and Independent Distributed (IID)'
Create a spot-on reference in APA, MLA, Chicago, Harvard, and other styles
Consult the lists of relevant articles, books, theses, conference reports, and other scholarly sources on the topic 'Identical and Independent Distributed (IID).'
Next to every source in the list of references, there is an 'Add to bibliography' button. Press on it, and we will generate automatically the bibliographic reference to the chosen work in the citation style you need: APA, MLA, Harvard, Chicago, Vancouver, etc.
You can also download the full text of the academic publication as pdf and read online its abstract whenever available in the metadata.
Journal articles on the topic "Identical and Independent Distributed (IID)"
1
Wu, Jikun, JiaHao Yu, and YuJun Zheng. "Research on Federated Learning Algorithms in Non-Independent Identically Distributed Scenarios." Highlights in Science, Engineering and Technology 85 (March 13, 2024): 104–12. http://dx.doi.org/10.54097/7newsv97.
Full textAbstract:
Federal learning is distributed learning and is mainly training locally by using multiple distributed devices. After receiving a local parameter, a server performs aggregation and performs multiple iterations until convergence to a final stable model. However, in actual application, due to different preferences of clients and differences in local data of different clients, data in federal learning may be not independently and identically distributed. (Non-Independent Identically Distribution). The main research work of this article is as follows: 1)Analyze and summarize the methods and techniques for solving the non-IID data problem in past experiments.2) Perform in-depth research on the basic methods of federal learning on non-IID data, such as FedAvg and FedProx.3) By using the FedAvg algorithm, using the CIFAR-10 data set, the simulation method is used to simulate the number of types contained in each client, and the distribution of the data set divided according to the distribution of Dirichlet to simulate the non-independent identical distribution of data. The detailed data analysis is made on the influence of the data on the accuracy and loss of model training.
2
Collins, Megan. "Distribution and Properties of the Critical Values of Random Polynomials With Non-Independent and Non-Identically Distributed Roots." PUMP Journal of Undergraduate Research 3 (November 6, 2020): 244–76. http://dx.doi.org/10.46787/pump.v3i0.2282.
Full textAbstract:
This paper considers the pairing between the distribution of the roots and the distribution of the critical values of random polynomials. The primary model of random polynomial considered in this paper consists of monic polynomials of degree n with a single complex variable z where the roots of the random polynomial are complex valued random variables that are chosen from two independent sequences of iid, complex valued random variables. The distribution of the random variables from each of the two sequences are different, producing roots of the random polynomial which have non-identical distributions. Furthermore, both the iid, complex valued random variables from one of the sequences of random variables and the complex conjugates of those random variables are roots of the random polynomial. Hence, this model of monic random polynomials of degree n has roots that are random variables which are not independent, due to the dependence based on complex conjugates, and the non-identical distributions which arise from the use of the two independent sequences. This paper also describes the relationship between the roots and critical values of monic random polynomials of degree n where the roots are chosen to be random variables and their complex conjugates where the random variables are from a sequence of iid complex valued random variables.
3
Aggarwal, Meenakshi, Vikas Khullar, Nitin Goyal, Abdullah Alammari, Marwan Ali Albahar, and Aman Singh. "Lightweight Federated Learning for Rice Leaf Disease Classification Using Non Independent and Identically Distributed Images." Sustainability 15, no. 16 (2023): 12149. http://dx.doi.org/10.3390/su151612149.
Full textAbstract:
Rice (Oryza sativa L.) is a vital food source all over the world, contributing 15% of the protein and 21% of the energy intake per person in Asia, where most rice is produced and consumed. However, bacterial, fungal, and other microbial diseases that have a negative effect on the health of plants and crop yield are a major problem for rice farmers. It is challenging to diagnose these diseases manually, especially in areas with a shortage of crop protection experts. Automating disease identification and providing readily available decision-support tools are essential for enabling effective rice leaf protection measures and minimising rice crop losses. Although there are numerous classification systems for the diagnosis of rice leaf disease, no reliable, secure method has been identified that meets these needs. This paper proposes a lightweight federated deep learning architecture while maintaining data privacy constraints for rice leaf disease classification. The distributed client–server design of this framework protects the data privacy of all clients, and by using independent and identically distributed (IID) and non-IID data, the validity of the federated deep learning models was examined. To validate the framework’s efficacy, the researchers conducted experiments in a variety of settings, including conventional learning, federated learning via a single client, as well as federated learning via multiple clients. The study began by extracting features from various pre-trained models, ultimately selecting EfficientNetB3 with an impressive 99% accuracy as the baseline model. Subsequently, experimental results were conducted using the federated learning (FL) approach with both IID and non-IID datasets. The FL approach, along with a dense neural network trained and evaluated on an IID dataset, achieved outstanding training and evaluated accuracies of 99% with minimal losses of 0.006 and 0.03, respectively. Similarly, on a non-IID dataset, the FL approach maintained a high training accuracy of 99% with a loss of 0.04 and an evaluation accuracy of 95% with a loss of 0.08. These results indicate that the FL approach performs nearly as well as the base model, EfficientNetB3, highlighting its effectiveness in handling both IID and non-IID data. It was found that federated deep learning models with multiple clients outperformed conventional pre-trained models. The unique characteristics of the proposed framework, such as its data privacy for edge devices with limited resources, set it apart from the existing classification schemes for rice leaf diseases. The framework is the best alternative solution for the early classification of rice leaf disease because of these additional features.
4
Alotaibi, Basmah, Fakhri Alam Khan, and Sajjad Mahmood. "Communication Efficiency and Non-Independent and Identically Distributed Data Challenge in Federated Learning: A Systematic Mapping Study." Applied Sciences 14, no. 7 (2024): 2720. http://dx.doi.org/10.3390/app14072720.
Full textAbstract:
Federated learning has emerged as a promising approach for collaborative model training across distributed devices. Federated learning faces challenges such as Non-Independent and Identically Distributed (non-IID) data and communication challenges. This study aims to provide in-depth knowledge in the federated learning environment by identifying the most used techniques for overcoming non-IID data challenges and techniques that provide communication-efficient solutions in federated learning. The study highlights the most used non-IID data types, learning models, and datasets in federated learning. A systematic mapping study was performed using six digital libraries, and 193 studies were identified and analyzed after the inclusion and exclusion criteria were applied. We identified that enhancing the aggregation method and clustering are the most widely used techniques for non-IID data problems (used in 18% and 16% of the selected studies), and a quantization technique was the most common technique in studies that provide communication-efficient solutions in federated learning (used in 27% and 15% of the selected studies). Additionally, our work shows that label distribution skew is the most used case to simulate a non-IID environment, specifically, the quantity label imbalance. The supervised learning model CNN model is the most commonly used learning model, and the image datasets MNIST and Cifar-10 are the most widely used datasets when evaluating the proposed approaches. Furthermore, we believe the research community needs to consider the client’s limited resources and the importance of their updates when addressing non-IID and communication challenges to prevent the loss of valuable and unique information. The outcome of this systematic study will benefit federated learning users, researchers, and providers.
5
Zhu, Feng, Jiangshan Hao, Zhong Chen, Yanchao Zhao, Bing Chen, and Xiaoyang Tan. "STAFL: Staleness-Tolerant Asynchronous Federated Learning on Non-iid Dataset." Electronics 11, no. 3 (2022): 314. http://dx.doi.org/10.3390/electronics11030314.
Full textAbstract:
With the development of the Internet of Things, edge computing applications are paying more and more attention to privacy and real-time. Federated learning, a promising machine learning method that can protect user privacy, has begun to be widely studied. However, traditional synchronous federated learning methods are easily affected by stragglers, and non-independent and identically distributed data sets will also reduce the convergence speed. In this paper, we propose an asynchronous federated learning method, STAFL, where users can upload their updates at any time and the server will immediately aggregate the updates and return the latest global model. Secondly, STAFL will judge the user’s data distribution according to the user’s update and dynamically change the aggregation parameters according to the user’s network weight and staleness to minimize the impact of non-independent and identically distributed data sets on asynchronous updates. The experimental results show that our method performs better on non-independent and identically distributed data sets than existing methods.
6
Tayyeh, Huda Kadhim, and Ahmed Sabah Ahmed AL-Jumaili. "Balancing Privacy and Performance: A Differential Privacy Approach in Federated Learning." Computers 13, no. 11 (2024): 277. http://dx.doi.org/10.3390/computers13110277.
Full textAbstract:
Federated learning (FL), a decentralized approach to machine learning, facilitates model training across multiple devices, ensuring data privacy. However, achieving a delicate privacy preservation–model convergence balance remains a major problem. Understanding how different hyperparameters affect this balance is crucial for optimizing FL systems. This article examines the impact of various hyperparameters, like the privacy budget (ϵ), clipping norm (C), and the number of randomly chosen clients (K) per communication round. Through a comprehensive set of experiments, we compare training scenarios under both independent and identically distributed (IID) and non-independent and identically distributed (Non-IID) data settings. Our findings reveal that the combination of ϵ and C significantly influences the global noise variance, affecting the model’s performance in both IID and Non-IID scenarios. Stricter privacy conditions lead to fluctuating non-converging loss behavior, particularly in Non-IID settings. We consider the number of clients (K) and its impact on the loss fluctuations and the convergence improvement, particularly under strict privacy measures. Thus, Non-IID settings are more responsive to stricter privacy regulations; yet, with a higher client interaction volume, they also can offer better convergence. Collectively, knowledge of the privacy-preserving approach in FL has been extended and useful suggestions towards an ideal privacy–convergence balance were achieved.
7
LYONS, RUSSELL. "Factors of IID on Trees." Combinatorics, Probability and Computing 26, no. 2 (2016): 285–300. http://dx.doi.org/10.1017/s096354831600033x.
Full textAbstract:
Classical ergodic theory for integer-group actions uses entropy as a complete invariant for isomorphism of IID (independent, identically distributed) processes (a.k.a. product measures). This theory holds for amenable groups as well. Despite recent spectacular progress of Bowen, the situation for non-amenable groups, including free groups, is still largely mysterious. We present some illustrative results and open questions on free groups, which are particularly interesting in combinatorics, statistical physics and probability. Our results include bounds on minimum and maximum bisection for random cubic graphs that improve on all past bounds.
8
Gao, Huiguo, Mengyuan Lee, Guanding Yu, and Zhaolin Zhou. "A Graph Neural Network Based Decentralized Learning Scheme." Sensors 22, no. 3 (2022): 1030. http://dx.doi.org/10.3390/s22031030.
Full textAbstract:
As an emerging paradigm considering data privacy and transmission efficiency, decentralized learning aims to acquire a global model using the training data distributed over many user devices. It is a challenging problem since link loss, partial device participation, and non-independent and identically distributed (non-iid) data distribution would all deteriorate the performance of decentralized learning algorithms. Existing work may restrict to linear models or show poor performance over non-iid data. Therefore, in this paper, we propose a decentralized learning scheme based on distributed parallel stochastic gradient descent (DPSGD) and graph neural network (GNN) to deal with the above challenges. Specifically, each user device participating in the learning task utilizes local training data to compute local stochastic gradients and updates its own local model. Then, each device utilizes the GNN model and exchanges the model parameters with its neighbors to reach the average of resultant global models. The iteration repeats until the algorithm converges. Extensive simulation results over both iid and non-iid data validate the algorithm’s convergence to near optimal results and robustness to both link loss and partial device participation.
9
Zhang, You, Jin Wang, Liang-Chih Yu, Dan Xu, and Xuejie Zhang. "Multi-Attribute Multi-Grained Adaptation of Pre-Trained Language Models for Text Understanding from Bayesian Perspective." Proceedings of the AAAI Conference on Artificial Intelligence 39, no. 24 (2025): 25967–75. https://doi.org/10.1609/aaai.v39i24.34791.
Full textAbstract:
Current neural networks often employ multi-domain-learning or attribute-injecting mechanisms to incorporate non-independent and identically distributed (non-IID) information for text understanding tasks by capturing individual characteristics and the relationships among samples. However, the extent of the impact of non-IID information and how these methods affect pre-trained language models (PLMs) remains unclear. This study revisits the assumption that non-IID information enhances PLMs to achieve performance improvements from a Bayesian perspective, which unearths and integrates non-IID and IID features. Furthermore, we proposed a multi-attribute multi-grained framework for PLM adaptations (M2A), which combines multi-attribute and multi-grained views to mitigate uncertainty in a lightweight manner. We evaluate M2A through prevalent text-understanding datasets and demonstrate its superior performance, mainly when data are implicitly non-IID, and PLMs scale larger.
10
Liu, Ying, Zhiqiang Wang, Shufang Pang, and Lei Ju. "Distributed Malicious Traffic Detection." Electronics 13, no. 23 (2024): 4720. http://dx.doi.org/10.3390/electronics13234720.
Full textAbstract:
With the wide deployment of edge devices, distributed network traffic data are rapidly increasing. Traditional detection methods for malicious traffic rely on centralized training, in which a single server is often used to aggregate private traffic data from edge devices, so as to extract and identify features. However, these methods face difficult data collection, heavy computational complexity, and high privacy risks. To address these issues, this paper proposes a federated learning-based distributed malicious traffic detection framework, FL-CNN-Traffic. In this framework, edge devices utilize a convolutional neural network (CNN) to process local detection, data collection, feature extraction, and training. A server aggregates model updates from edge devices using four federated learning algorithms (FedAvg, FedProx, Scaffold, and FedNova) to build a global model. This framework allows multiple devices to collaboratively train a model without sharing private traffic data, addressing the “Data Silo” problem while ensuring privacy. Evaluations on the USTC-TFC2016 dataset show that for independent and identically distributed (IID) data, this framework can reach or exceed the performance of centralized deep learning methods. For Non-IID data, this framework outperforms other neural networks based on federated learning, with accuracy improvements ranging from 2.59% to 4.73%.
Dissertations / Theses on the topic "Identical and Independent Distributed (IID)"
1
Hsu, Shih-Yi, and 許世易. "The Rate of Complete Convergence for 2m Independent and Identical Distributed Random Variables." Thesis, 1999. http://ndltd.ncl.edu.tw/handle/17501214365158068459.
Full textBook chapters on the topic "Identical and Independent Distributed (IID)"
1
Jacobs, Konrad. "Independent Identically Distributed (IID) Random Variables." In Discrete Stochastics. Birkhäuser Basel, 1992. http://dx.doi.org/10.1007/978-3-0348-8645-1_4.
Full text
2
Griffith, Daniel A., and Larry J. Layne. "Important Modeling Assumptions." In A Casebook For Spatial Statistical Data Analysis. Oxford University PressNew York, NY, 1999. http://dx.doi.org/10.1093/oso/9780195109580.003.0002.
Full textAbstract:
Abstract Popular conventional statistical assumptions invoked to guard against misleading or incorrect conclusions (from other than random error) include one positing independent and identically distributed (iid) values, one positing normality, and one positing linearity. Statistical inference can be either sampling design based or model based (de Gruijter and ter Braak 1990; Hahn and Meeker 1993). Sampling design-based inference exploits central limit theorems and considers the only marked source of variation to result from the sampling process employed. Given a large enough sample, then, a central limit theorem guarantees iid and normality; some researchers questionably argue that a sufficiently large sample will circumvent many nonlinearity complications, too. Model-based inference exploits the notion of random variables, treating a set of georeferenced values as just one realization of an underlying spatial process constituting a conceptual population. It requires the critical postulate that the process under study is statistically identical to that from which data have been collected. Attribute value locations need not be selected at random; rather, inference is based upon some specified model, with the soundness of inferences being dictated by the validity of model assumptions or the robustness of model results.
3
Feng, Chao, Alberto Huertas Celdrán, Janosch Baltensperger, et al. "Sentinel: An Aggregation Function to Secure Decentralized Federated Learning." In Frontiers in Artificial Intelligence and Applications. IOS Press, 2024. http://dx.doi.org/10.3233/faia240686.
Full textAbstract:
Decentralized Federated Learning (DFL) emerges as an innovative paradigm to train collaborative models, addressing the single point of failure limitation. However, the security and trustworthiness of FL and DFL are compromised by poisoning attacks, negatively impacting its performance. Existing defense mechanisms have been designed for centralized FL and they do not adequately exploit the particularities of DFL. Thus, this work introduces Sentinel, a defense strategy to counteract poisoning attacks in DFL. Sentinel leverages the accessibility of local data and defines a three-step aggregation protocol consisting of similarity filtering, bootstrap validation, and normalization to safeguard against malicious model updates. Sentinel has been evaluated with diverse datasets and data distributions. Besides, various poisoning attack types and threat levels have been verified. The results improve the state-of-the-art performance against both untargeted and targeted poisoning attacks when data follows an IID (Independent and Identically Distributed) configuration. Besides, under non-IID configuration, it is analyzed how performance degrades both for Sentinel and other state-of-the-art robust aggregation methods.
4
Kanimozhi, S., R. Deebika, and Ram N. Hajare. "Federated and Transfer Learning for Distributed Anomaly Detection in IoT-Enabled Power Electronics for Industrial Automation." In Power Electronics for IoT-Enabled Smart Grids and Industrial Automation. RADemics Research Institute, 2025. https://doi.org/10.71443/9789349552111-06.
Full textAbstract:
The rapid integration of IIoT in power electronics has transformed industrial automation, enabling real-time monitoring, predictive maintenance, and intelligent decision-making. The distributed nature of IIoT-enabled power electronics introduces significant challenges in anomaly detection, including data heterogeneity, privacy concerns, and computational limitations of edge devices. Traditional centralized learning approaches are inefficient in handling these constraints, necessitating the adoption of decentralized learning paradigms. Federated Learning (FL) has emerged as a transformative approach, enabling collaborative model training across edge devices while preserving data privacy. FL faces challenges such as communication overhead, resource constraints, and performance degradation due to non-independent and identically distributed (non-IID) data. To address these limitations, Transfer Learning (TL) was integrated with FL to enhance model adaptability, enabling efficient knowledge transfer across different industrial environments and reducing dependency on extensive labeled datasets. This book chapter presents a comprehensive study on the integration of FL and TL for distributed anomaly detection in IIoT-enabled power electronics. The research explores optimization techniques for scalable FL deployment, including low-latency model aggregation, edge-to-cloud collaboration, and privacy-preserving secure model aggregation using Secure Multi-Party Computation (SMPC). The role of meta-learning in improving FL model generalization for handling heterogeneous data was analyzed. To address computational inefficiencies, the study examines Federated Knowledge Distillation (FKD) as a lightweight learning approach that minimizes resource consumption while maintaining high anomaly detection accuracy. The findings highlight the advantages of hybrid FL-TL frameworks in enhancing fault diagnosis, reducing communication overhead, and ensuring energy-efficient real-time anomaly detection. The proposed approach strengthens the reliability and security of industrial automation by providing a scalable and adaptive learning framework for power electronics systems. Future research directions include optimizing FL-TL integration for dynamic industrial environments, developing energy-efficient federated architectures, and enhancing privacy-preserving techniques for large-scale IIoT networks.
5
Zhao, Juan, Yuankai Zhang, Ruixuan Li, et al. "XFed: Improving Explainability in Federated Learning by Intersection Over Union Ratio Extended Client Selection." In Frontiers in Artificial Intelligence and Applications. IOS Press, 2023. http://dx.doi.org/10.3233/faia230628.
Full textAbstract:
Federated Learning (FL) allows massive clients to collaboratively train a global model without revealing their private data. Because of the participants’ not independently and identically distributed (non-IID) statistical characteristics, it will cause divergence among the client’s Deep Neural Network model weights and require more communication rounds before training can be converged. Moreover, models trained from non-IID data may also extract biased features and the rationale behind the model is still not fully analyzed and exploited. In this paper, we propose eXplainable-Fed (XFed) which is a novel client selection mechanism that takes both accuracy and explainability into account. Specifically, XFed selects participants in each round based on a small test set’s accuracy via cross-entropy loss and interpretability via XAI-accuracy. XAI-accuracy is calculated by Intersection over Union Ratio between the heat map and the truth mask to evaluate the overall rationale of accuracy. The results of our experiments show that our method has comparable accuracy to state-of-the-art methods specially designed for accuracy while increasing explainability by 14%-35% in terms of rationality.
Conference papers on the topic "Identical and Independent Distributed (IID)"
1
Arafeh, Mohamad, Ahmad Hammoud, Hadi Otrok, Azzam Mourad, Chamseddine Talhi, and Zbigniew Dziong. "Independent and Identically Distributed (IID) Data Assessment in Federated Learning." In GLOBECOM 2022 - 2022 IEEE Global Communications Conference. IEEE, 2022. http://dx.doi.org/10.1109/globecom48099.2022.10001718.
Full text
2
Li, Yang, Liangliang Shi, and Junchi Yan. "IID-GAN: an IID Sampling Perspective for Regularizing Mode Collapse." In Thirty-Second International Joint Conference on Artificial Intelligence {IJCAI-23}. International Joint Conferences on Artificial Intelligence Organization, 2023. http://dx.doi.org/10.24963/ijcai.2023/437.
Full textAbstract:
Despite its success, generative adversarial networks (GANs) still suffer from mode collapse, i.e., the generator can only map latent variables to a partial set of modes in the target distribution. In this paper, we analyze and seek to regularize this issue with an independent and identically distributed (IID) sampling perspective and emphasize that holding the IID property referring to the target distribution for generation can naturally avoid mode collapse. This is based on the basic IID assumption for real data in machine learning. However, though the source samples {z} obey IID, the generations {G(z)} may not necessarily be IID sampling from the target distribution. Based on this observation, considering a necessary condition of IID generation, we propose a new loss to encourage the closeness between the inverse samples of real data and the Gaussian source in the latent space to regularize the generation to be IID from the target distribution. The logic is that the inverse samples from target data should also be IID in the source distribution. Experiments on both synthetic and real-world data show the effectiveness of our model.
3
Sousa, John Lucas R. P. de, Wellington Lobato, Denis Rosário, Eduardo Cerqueira, and Leandro A. Villas. "Entropy-based Client Selection Mechanism for Vehicular Federated Environments." In Workshop em Desempenho de Sistemas Computacionais e de Comunicação. Sociedade Brasileira de Computação - SBC, 2023. http://dx.doi.org/10.5753/wperformance.2023.230700.
Full textAbstract:
Autonomous driving requires machine learning models to be trained at the edge for improved efficiency and reduced communication latency. Federated learning (FL) allows knowledge sharing among all devices, but Not Independent and Identically Distributed (non-IID) scenarios with biased device data distributions can lead to statistical heterogeneity and lower classification accuracy. This paper proposes an entropy-based client selection approach for vehicular federated learning environments that aims to address the challenges posed by non-IID data in vehicular networks. The proposed method is compared to a random selection mechanism in both IID and non-IID scenarios, as well as in a scenario with random client drops. The results show that the entropy-based selection method outperforms the random selection method in all compared metrics, particularly in non-IID scenarios.
4
Liao, Xinting, Weiming Liu, Chaochao Chen, et al. "HyperFed: Hyperbolic Prototypes Exploration with Consistent Aggregation for Non-IID Data in Federated Learning." In Thirty-Second International Joint Conference on Artificial Intelligence {IJCAI-23}. International Joint Conferences on Artificial Intelligence Organization, 2023. http://dx.doi.org/10.24963/ijcai.2023/440.
Full textAbstract:
Federated learning (FL) collaboratively models user data in a decentralized way. However, in the real world, non-identical and independent data distributions (non-IID) among clients hinder the performance of FL due to three issues, i.e., (1) the class statistics shifting, (2) the insufficient hierarchical information utilization, and (3) the inconsistency in aggregating clients. To address the above issues, we propose HyperFed which contains three main modules, i.e., hyperbolic prototype Tammes initialization (HPTI), hyperbolic prototype learning (HPL), and consistent aggregation (CA). Firstly, HPTI in the server constructs uniformly distributed and fixed class prototypes, and shares them with clients to match class statistics, further guiding consistent feature representation for local clients. Secondly, HPL in each client captures the hierarchical information in local data with the supervision of shared class prototypes in the hyperbolic model space. Additionally, CA in the server mitigates the impact of the inconsistent deviations from clients to server. Extensive studies of four datasets prove that HyperFed is effective in enhancing the performance of FL under the non-IID setting.
5
Chandran, Pravin, Raghavendra Bhat, Avinash Chakravarthy, and Srikanth Chandar. "Divide-and-Conquer Federated Learning Under Data Heterogeneity." In International Conference on AI, Machine Learning and Applications (AIMLA 2021). Academy and Industry Research Collaboration Center (AIRCC), 2021. http://dx.doi.org/10.5121/csit.2021.111302.
Full textAbstract:
Federated Learning allows training of data stored in distributed devices without the need for centralizing training-data, thereby maintaining data-privacy. Addressing the ability to handle data heterogeneity (non-identical and independent distribution or non-IID) is a key enabler for the wider deployment of Federated Learning. In this paper, we propose a novel Divide-andConquer training methodology that enables the use of the popular FedAvg aggregation algorithm by over-coming the acknowledged FedAvg limitations in non-IID environments. We propose a novel use of Cosine-distance based Weight Divergence metric to determine the exact point where a Deep Learning network can be divided into class-agnostic initial layers and class-specific deep layers for performing a Divide and Conquer training. We show that the methodology achieves trained-model accuracy at-par with (and in certain cases exceeding) the numbers achieved by state-of-the-art algorithms like FedProx, FedMA, etc. Also, we show that this methodology leads to compute and/or bandwidth optimizations under certain documented conditions.
6
Xu, Yi, Ying Li, Haoyu Luo, Xiaoliang Fan, and Xiao Liu. "FBLG: A Local Graph Based Approach for Handling Dual Skewed Non-IID Data in Federated Learning." In Thirty-Third International Joint Conference on Artificial Intelligence {IJCAI-24}. International Joint Conferences on Artificial Intelligence Organization, 2024. http://dx.doi.org/10.24963/ijcai.2024/585.
Full textAbstract:
In real-world situations, federated learning often needs to process non-IID (non-independent and identically distributed) data with multiple skews, causing inadequate model performance. Existing federated learning methods mainly focus on addressing the problem with a single skew of non-IID, and hence the performance of global models can be degraded when faced with dual skewed non-IID data caused by heterogeneous label distributions and sample sizes among clients. To address the problem with dual skewed non-IID data, in this paper, we propose a federated learning algorithm based on local graph, named FBLG. Specifically, to address the label distribution skew, we firstly construct a local graph based on clients' local losses and Jensen-Shannon (JS) divergence, so that similar clients can be selected for aggregation to ensure a highly consistent global model. Afterwards, to address the sample size skew, we design the objective function to favor clients with more samples as models trained with more samples tend to carry more useful information. Experiments on four datasets with dual skewed non-IID data demonstrate FBLG outperforms nine baseline methods and achieves up to 9% improvement in accuracy. Simultaneously, both theoretical analysis and experiments show FBLG can converge quickly.
7
Guruprasad, Kamalesh Kumar Mandakolathur, Gayatri Sunil Ambulkar, and Geetha Nair. "Federated Learning for Seismic Data Denoising: Privacy-Preserving Paradigm." In International Petroleum Technology Conference. IPTC, 2024. http://dx.doi.org/10.2523/iptc-23888-ms.
Full textAbstract:
Summary Federated Learning (FL) is a framework that empowers multiple clients to develop robust machine learning (ML) algorithms while safeguarding data privacy and security. This paper's primary goal is to investigate the capability of the FL framework in preserving privacy and to assess its efficacy for clients operating within the oil and gas industry. To demonstrate the practicality of this framework, we apply it to seismic denoising use cases incorporating data from clients with IID (independent & and identically distributed) and Non-IID (non-independent and non-identically distributed) or domain-shifted data distributions. The FL setup is implemented using the well-established Flower framework. The experiment involves injecting noise into 3D seismic data and subsequently employing various ML algorithms to eliminate this noise. All experiments were conducted using both IID and Non-IID data, employing both traditional and FL approaches, various tests considering different types of noise, noise factors, number of 2D seismic slices, diverse models, number of clients, and aggregations strategies. We tested different model aggregation strategies, such as FedAvg, FedProx, and Fedcyclic, alongside client selection strategies that consider model divergence, convergence trend similarity, and client weight analysis to improve the aggregation process. We also incorporated batch normalization into the network architecture to reduce data discrepancies among clients. The denoising process was evaluated using metrics like mean-square-error (MSE), signal-to-noise ratio (SNR), and peak signal-to-noise ratio (PSNR). A comparison between conventional methods and FL demonstrated that FL exhibited a reduced error rate, especially when dealing with larger datasets. Furthermore, FL harnessed the power of parallel computing, resulting in a notable 30% increase in processing speed, enhanced resource utilization, and a remarkable 99% reduction in communication costs. To sum it up, this study underscores the potential of FL in the context of seismic denoising, safeguarding data privacy, and enhancing overall performance. We addressed the associated challenges by experimenting with various approaches for client selection and aggregation within a privacy-preserving framework. Notably, among these aggregation strategies, FedCyclic stands out as it offers faster convergence, achieving performance levels comparable to FedAvg and FedProx with fewer training iterations.
8
Errami, Latifa, and El Houcine Bergou. "Tolerating Outliers: Gradient-Based Penalties for Byzantine Robustness and Inclusion." In Thirty-Third International Joint Conference on Artificial Intelligence {IJCAI-24}. International Joint Conferences on Artificial Intelligence Organization, 2024. http://dx.doi.org/10.24963/ijcai.2024/435.
Full textAbstract:
This work investigates the interplay between Robustness and Inclusion in the context of poisoning attacks targeting the convergence of Stochastic Gradient Descent (SGD). While robustness has received significant attention, the standard Byzantine defenses rely on the Independent and Identically Distributed (IID) assumption causing their performance to deteriorate on non-IID data distributions, even without any attack. This is largely due to these defenses being excessively cautious and discarding benign outliers. We introduce a penalty-based aggregation that accounts for the discrepancy between trusted clients and outliers. We propose the use of Linear Scalarization (LS) as an enhancing method to enable current defenses to simultaneously circumvent Byzantine attacks while also granting inclusion of outliers. This empowers existing defenses to not only counteract malicious adversaries effectively but also to incorporate outliers into the learning process. We conduct a theoretical analysis to demonstrate the convergence of our approach. Specifically, we establish the robustness and resilience of our method under standard assumptions. Empirical analysis further validates the viability of the proposed approach. Across mild to strong non-IID data splits, our method consistently either matches or surpasses the performance of current approaches in the literature, under state-of-the-art Byzantine attack scenarios.
9
Wu, Yawen, Zhepeng Wang, Dewen Zeng, Meng Li, Yiyu Shi, and Jingtong Hu. "Decentralized Unsupervised Learning of Visual Representations." In Thirty-First International Joint Conference on Artificial Intelligence {IJCAI-22}. International Joint Conferences on Artificial Intelligence Organization, 2022. http://dx.doi.org/10.24963/ijcai.2022/323.
Full textAbstract:
Collaborative learning enables distributed clients to learn a shared model for prediction while keeping the training data local on each client. However, existing collaborative learning methods require fully-labeled data for training, which is inconvenient or sometimes infeasible to obtain due to the high labeling cost and the requirement of expertise. The lack of labels makes collaborative learning impractical in many realistic settings. Self-supervised learning can address this challenge by learning from unlabeled data. Contrastive learning (CL), a self-supervised learning approach, can effectively learn visual representations from unlabeled image data. However, the distributed data collected on clients are usually not independent and identically distributed (non-IID) among clients, and each client may only have few classes of data, which degrades the performance of CL and learned representations. To tackle this problem, we propose a collaborative contrastive learning framework consisting of two approaches: feature fusion and neighborhood matching, by which a unified feature space among clients is learned for better data representations. Feature fusion provides remote features as accurate contrastive information to each client for better local learning. Neighborhood matching further aligns each client’s local features to the remote features such that well-clustered features among clients can be learned. Extensive experiments show the effectiveness of the proposed framework. It outperforms other methods by 11% on IID data and matches the performance of centralized learning.
10
Souza, Lucas Airam C. de, Miguel Elias M. Campista, and Luís Henrique M. K. Costa. "Federated Learning with Accurate Model Training and Low Communication Cost in Heterogeneous Scenarios." In Anais Estendidos do Simpósio Brasileiro de Redes de Computadores e Sistemas Distribuídos. Sociedade Brasileira de Computação - SBC, 2024. http://dx.doi.org/10.5753/sbrc_estendido.2024.1633.
Full textAbstract:
Federated learning (FL) is a distributed approach to train machine learning models without disclosing private data from participating clients to a central server. Nevertheless, FL performance depends on the data distribution, and the training struggles to converge when clients have distinct data distributions, increasing overall training time and the final model prediction error. This work proposes two strategies to reduce the impact of data heterogeneity in FL scenarios. Firstly, we propose a hierarchical client clustering system to mitigate the convergence obstacles of federated learning in non-Independent and Identically Distributed (IID) scenarios. The results show that our system has a better classification performance than FedAVG, increasing its accuracy by approximately 16% on non-IID scenarios. Furthermore, we improve our first proposal by implementing ATHENA-FL, a federated learning system that shares knowledge among different clusters. The proposed system also uses the one-versus-all model to train one binary detector for each class in the cluster. Thus, clients can compose complex models combining multiple detectors. ATHENA-FL mitigates data heterogeneity by maintaining the clustering step before training to mitigate data heterogeneity. Our results show that ATHENA-FL correctly identifies samples, achieving up to 10.9% higher accuracy than traditional training. Finally, ATHENA-FL achieves lower training communication costs than MobileNet architecture, reducing the number of transmitted bytes between 25% and 97% across evaluated scenarios.