Academic literature on the topic 'Missforest'

The global positioning system (GPS) can provide the daily coordinate time series to help geodesy and geophysical studies. However, due to logistics and malfunctioning, missing values are often “seen” in GPS time series, especially in polar regions. Acquiring a consistent and complete time series is the prerequisite for accurate and reliable statical analysis. Previous imputation studies focused on the temporal relationship of time series, and only a few studies used spatial relationships and/or were based on machine learning methods. In this study, we impute 20 Greenland GPS time series using missForest, which is a new machine learning method for data imputation. The imputation performance of missForest and that of four traditional methods are assessed, and the methods’ impacts on principal component analysis (PCA) are investigated. Results show that missForest can impute more than a 30-day gap, and its imputed time series has the least influence on PCA. When the gap size is 30 days, the mean absolute value of the imputed and true values for missForest is 2.71 mm. The normalized root mean squared error is 0.065, and the distance of the first principal component is 0.013. missForest outperforms the other compared methods. missForest can effectively restore the information of GPS time series and improve the results of related statistical processes, such as PCA analysis.

Targeted proteomics utilizing antibody-based proximity extension assays provides sensitive and highly specific quantifications of plasma protein levels. Multivariate analysis of this data is hampered by frequent missing values (random or left censored), calling for imputation approaches. While appropriate missing-value imputation methods exist, benchmarks of their performance in targeted proteomics data are lacking. Here, we assessed the performance of two methods for imputation of values missing completely at random, the previously top-benchmarked ‘missForest’ and the recently published ‘GSimp’ method. Evaluation was accomplished by comparing imputed with remeasured relative concentrations of 91 inflammation related circulating proteins in 86 samples from a cohort of 645 patients with venous thromboembolism. The median Pearson correlation between imputed and remeasured protein expression values was 69.0% for missForest and 71.6% for GSimp (p = 5.8e-4). Imputation with missForest resulted in stronger reduction of variance compared to GSimp (median relative variance of 25.3% vs. 68.6%, p = 2.4e-16) and undesired larger bias in downstream analyses. Irrespective of the imputation method used, the 91 imputed proteins revealed large variations in imputation accuracy, driven by differences in signal to noise ratio and information overlap between proteins. In summary, GSimp outperformed missForest, while both methods show good overall imputation accuracy with large variations between proteins.

In environmental research, missing data are often a challenge for statistical modeling. This paper addressed some advanced techniques to deal with missing values in a data set measuring air quality using a multiple imputation (MI) approach. MCAR, MAR, and NMAR missing data techniques are applied to the data set. Five missing data levels are considered: 5%, 10%, 20%, 30%, and 40%. The imputation method used in this paper is an iterative imputation method, missForest, which is related to the random forest approach. Air quality data sets were gathered from five monitoring stations in Kuwait, aggregated to a daily basis. Logarithm transformation was carried out for all pollutant data, in order to normalize their distributions and to minimize skewness. We found high levels of missing values for NO2 (18.4%), CO (18.5%), PM10 (57.4%), SO2 (19.0%), and O3 (18.2%) data. Climatological data (i.e., air temperature, relative humidity, wind direction, and wind speed) were used as control variables for better estimation. The results show that the MAR technique had the lowest RMSE and MAE. We conclude that MI using the missForest approach has a high level of accuracy in estimating missing values. MissForest had the lowest imputation error (RMSE and MAE) among the other imputation methods and, thus, can be considered to be appropriate for analyzing air quality data.

The problem of incomplete data and its implications for drawing valid conclusions from statistical analyses is not related to any particular scientific domain, it arises in economics, sociology, education, behavioural sciences or medicine. Almost all standard statistical methods presume that every object has information on every variable to be included in the analysis and the typical approach to missing data is simply to delete them. However, this leads to ineffective and biased analysis results and is not recommended in the literature. The state of the art technique for handling missing data is multiple imputation. In the paper, some selected multiple imputation methods were taken into account. Special attention was paid to using principal components analysis (PCA) as an imputation method. The goal of the study was to assess the quality of PCA‑based imputations as compared to two other multiple imputation techniques: multivariate imputation by chained equations (MICE) and missForest. The comparison was made by artificially simulating different proportions (10–50%) and mechanisms of missing data using 10 complete data sets from the UCI repository of machine learning databases. Then, missing values were imputed with the use of MICE, missForest and the PCA‑based method (MIPCA). The normalised root mean square error (NRMSE) was calculated as a measure of imputation accuracy. On the basis of the conducted analyses, missForest can be recommended as a multiple imputation method providing the lowest rates of imputation errors for all types of missingness. PCA‑based imputation does not perform well in terms of accuracy.

Automatic anomaly detection monitoring plays a vital role in water utilities’ distribution systems to reduce the risk posed by unclean water to consumers. One of the major problems with anomaly detection is imbalanced datasets. Dynamic selection techniques combined with ensemble models have proven to be effective for imbalanced datasets classification tasks. In this paper, water quality anomaly detection is formulated as a classification problem in the presences of class imbalance. To tackle this problem, considering the asymmetry dataset distribution between the majority and minority classes, the performance of sixteen previously proposed single and static ensemble classification methods embedded with resampling strategies are first optimised and compared. After that, six dynamic selection techniques, namely, Modified Class Rank (Rank), Local Class Accuracy (LCA), Overall-Local Accuracy (OLA), K-Nearest Oracles Eliminate (KNORA-E), K-Nearest Oracles Union (KNORA-U) and Meta-Learning for Dynamic Ensemble Selection (META-DES) in combination with homogeneous and heterogeneous ensemble models and three SMOTE-based resampling algorithms (SMOTE, SMOTE+ENN and SMOTE+Tomek Links), and one missing data method (missForest) are proposed and evaluated. A binary real-world drinking-water quality anomaly detection dataset is utilised to evaluate the models. The experimental results obtained reveal all the models benefitting from the combined optimisation of both the classifiers and resampling methods. Considering the three performance measures (balanced accuracy, F-score and G-mean), the result also shows that the dynamic classifier selection (DCS) techniques, in particular, the missForest+SMOTE+RANK and missForest+SMOTE+OLA models based on homogeneous ensemble-bagging with decision tree as the base classifier, exhibited better performances in terms of balanced accuracy and G-mean, while the Bg+mF+SMENN+LCA model based on homogeneous ensemble-bagging with random forest has a better overall F1-measure in comparison to the other models.

We propose a machine learning-based methodology which makes use of ensemble methods with the aims (i) of treating missing data in time series with irregular observation times and detecting anomalies in the observed time behavior; (ii) of defining suitable models of the system dynamics. We applied this methodology to US wholesale electricity price time series that are characterized by missing data, high and stochastic volatility, jumps and pronounced spikes. For missing data, we provide a repair approach based on the missForest algorithm, an imputation algorithm which is completely agnostic about the data distribution. To identify anomalies, i.e., turbulent movements of power prices in which jumps and spikes are observed, we took into account the no-gap reconstructed electricity price time series, and then we detected anomalous regions using the isolation forest algorithm, an anomaly detection method that isolates anomalies instead of profiling normal data points as in the most common techniques. After removing anomalies, the additional gaps will be newly filled by the missForest imputation algorithm. In this way, a complete and clean time series describing the stable dynamics of power prices can be obtained. The decoupling between the stable motion and the turbulent motion allows us to define suitable jump-diffusion models of power prices and to provide an estimation procedure that uses the full information contained in both the stable and the turbulent dynamics.

Data preprocessing is a necessary core in data mining. Preprocessing involves handling missing values, outlier and noise removal, data normalization, etc. The problem with existing methods which handle missing values is that they deal with the whole data ignoring the characteristics of the data (e.g., similarities and differences between cases). This paper focuses on handling the missing values using machine learning methods taking into account the characteristics of the data. The proposed preprocessing method clusters the data, then imputes the missing values in each cluster depending on the data belong to this cluster rather than the whole data. The author performed a comparative study of the proposed method and ten popular imputation methods namely mean, median, mode, KNN, IterativeImputer, IterativeSVD, Softimpute, Mice, Forimp, and Missforest. The experiments were done on four datasets with different number of clusters, sizes, and shapes. The empirical study showed better effectiveness from the point of view of imputation time, Root Mean Square Error (RMSE), Mean Absolute Error (MAE), and coefficient of determination (R2 score) (i.e., the similarity of the original removed value to the imputed one).

Emergency situations in nuclear power plants are accompanied by an automatic reactor shutdown, which gives a big task burden to the plant operators under highly stressful conditions. Diagnosis of the occurred accident is an essential sequence for optimum mitigations; however, it is also a critical source of error because the results of accident identification determine the task flow connected to all subsequent tasks. To support accident identification in nuclear power plants, recurrent neural network (RNN)-based approaches have recently shown outstanding performances. Despite the achievements though, the robustness of RNN models is not promising because wrong inputs have been shown to degrade the performance of RNNs to a greater extent than other methods in some applications. In this research, an accident diagnosis system that is tolerant to sensor faults is developed based on an existing RNN model and tested with anticipated sensor errors. To find the optimum strategy to mitigate sensor error, Missforest, selected from among various imputation methods, and gated recurrent unit with decay (GRUD), developed for multivariate time series imputation based on the RNN model, are compared to examine the extent that they recover the diagnosis accuracies within a given threshold.

Background:Missing data in clinical epidemiological researches violate the intention to treat principle,reduce statistical power and can induce bias if they are related to patient’s response to treatment. In multiple imputation (MI), covariates are included in the imputation equation to predict the values of missing data.Objectives:To find the best approach to estimate and impute the missing values in Kuwait Registry for Rheumatic Diseases (KRRD) patients data.Methods:A number of methods were implemented for dealing with missing data. These includedMultivariate imputation by chained equations(MICE),K-Nearest Neighbors(KNN),Bayesian Principal Component Analysis(BPCA),EM with Bootstrapping(Amelia II),Sequential Random Forest(MissForest) and mean imputation. Choosing the best imputation method wasjudged by the minimum scores ofRoot Mean Square Error(RMSE),Mean Absolute Error(MAE) andKolmogorov–Smirnov D test statistic(KS) between the imputed datapoints and the original datapoints that were subsequently sat to missing.Results:A total of 1,685 rheumatoid arthritis (RA) patients and 10,613 hospital visits were included in the registry. Among them, we found a number of variables that had missing values exceeding 5% of the total values. These included duration of RA (13.0%), smoking history (26.3%), rheumatoid factor (7.93%), anti-citrullinated peptide antibodies (20.5%), anti-nuclear antibodies (20.4%), sicca symptoms (19.2%), family history of a rheumatic disease (28.5%), steroid therapy (5.94%), ESR (5.16%), CRP (22.9%) and SDAI (38.0%), The results showed that among the methods used, MissForest gave the highest level of accuracy to estimate the missing values. It had the least imputation errors for both continuous and categorical variables at each frequency of missingness and it had the smallest prediction differences when the models used imputed laboratory values. In both data sets, MICE had the second least imputation errors and prediction differences, followed by KNN and mean imputation.Conclusion:MissForest is a highly accurate method of imputation for missing data in KRRD and outperforms other common imputation techniques in terms of imputation error and maintenance of predictive ability with imputed values in clinical predictive models. This approach can be used in registries to improve the accuracy of data, including the ones for rheumatoid arthritis patients.References:[1]Junninen, H.; Niska, H.; Tuppurainen, K.; Ruuskanen, J.; Kolehmainen, M. Methods for imputation ofmissing values in air quality data sets.Atmospheric Environment2004,38, 2895–2907.[2]Norazian, M.N.; Shukri, Y.A.; Azam, R.N.; Al Bakri, A.M.M. Estimation of missing values in air pollutiondata using single imputation techniques.ScienceAsia2008,34, 341–345.[3]Plaia, A.; Bondi, A. Single imputation method of missing values in environmental pollution data sets.Atmospheric Environment2006,40, 7316–7330.[4]Kabir, G.; Tesfamariam, S.; Hemsing, J.; Sadiq, R. Handling incomplete and missing data in water networkdatabase using imputation methods.Sustainable and Resilient Infrastructure2019, pp. 1–13.[5]Di Zio, M.; Guarnera, U.; Luzi, O. Imputation through finite Gaussian mixture models.ComputationalStatistics & Data Analysis2007,51, 5305–5316.Disclosure of Interests:None declared

Partiellt bortfall är en vanligt förekommande felkälla vid statistiska undersökningar. Med partiellt bortfall avses avsaknad av vissa variabelvärden för ett observationsobjekt, något som riskerar leda till förlust av statistisk styrka och skeva parameterskattningar. Ett stort antal metoder har utvecklats för att hantera denna problematik, och syftet med denna uppsats är att undersöka vilken effekt några av dessa metoder har på parameterskattningarna i en logistisk regressionsmodell, och huruvida dessa metoder är lämpliga att tillämpa på aktuellt datamaterial. De metoder som inkluderats i denna studie är complete case analysis, MICE och missForest. För ändamålet simuleras partiellt bortfall av olika omfattningar och under olika bortfallsmekanismer i ett verkligt datamaterial som består av 2987 observationer och fem variabler. Metoderna utvärderas sedan med avseende på normalized root mean squared error (NRMSE), samt genom att undersöka hur de regressionskoefficienter som skattats med de imputerade datamaterialen avviker från de regressionskoefficienter som skattats med det kompletta, observerade datamaterialet. missForest resulterar i lägst NRMSE. I den efterföljande logistiska regressionsanalysen resulterar dock MICE i betydligt lägre bias än missForest.
Missing data is a common problem in research and can lead to loss of statistical power and bias in parameter estimates. Numerous methods have been developed for dealing with missing data, and the aim of this thesis is to evaluate how a number of these methods affect the parameter estimates in a logistic regression model, and whether these methods are suitable for the data in question. The methods included in this study are complete case analysis, MICE and missForest. For the purpose of evaluating the methods, missing values in varying proportions and under different missing mechanisms are generated in a real dataset consisting of 2987 observations and five variables. The performance of the methods is assessed by normalized root mean squared error (NRMSE), and by comparing the regression coefficients estimated using the original, true data set with the regression coefficients estimated using imputed data sets. missForest results in the lowest NRMSE. In the subsequent logistic regression analysis, however, MICE results in considerably lower bias than missForest.

Nos dias de hoje existe um grande volume de dados disponíveis e inúmeros algoritmos que permitem analisar estes conjuntos. No entanto, a maioria dos algoritmos necessita que o conjunto de dados seja completo, isto é, não pode possuir valores omissos. Existem então métodos de imputação que permitem fazer o tratamento dos valores omissos. Neste estudo foram comparados três métodos disponíveis no software R, comparando a sua performance em conjuntos de dados na área da saúde disponíveis no UCI Machine Learning Repository, com tipos de variáveis mistas (numéricas e categóricas). Foram gerados valores omissos para cada conjunto, nas percentagens de 10%, 20%, 30%, 40% e 50%, posteriormente sujeitos a métodos de imputação simples e múltipla. Foram analisados depois os erros de imputação para as variáveis numéricas e categóricas, comparando também o tempo que cada método demorou a imputar cada conjunto de dados, e o seu impacto na classificação. Os resultados mostraram que o método mais consistente a imputar conjuntos de dados clínicos é o missForest, apresentando de forma quase constante o menor erro de imputação, mas devido à sua maior complexidade também é o método que leva mais tempo a imputar
Nowadays there is a great volume of available data and countless algorithms that allows us to analyse it. However, most algorithms only work with complete datasets, with no missing values. To solve this problem there are imputation methods that treat the missing data. In this study three methods available in R were used, comparing their performance in imputing medical datasets available at the UCI Machine Learning Repository, with mixed type variables (numeric and categorical). Missing values were generated for each dataset, creating new datasets with 10%, 20%, 30%, 40% and 50% of missing values, and single and multiple imputation methods were applied. The imputation erros were analysed for each type of variable, numeric and categorical, also comparing the imputation time, as well as the impact that each imputation has on classifying each dataset. The results show that the missForest method is the most consistent for clinical datasets, usually presenting the smaller imputation error, but because of its complexity it’s also the method that takes longer to impute the missing values
Mestrado em Matemática e Aplicações