Academic literature on the topic 'Data missingness'

Thesis (Ph. D.)--Ohio State University, 2005.<br>Title from first page of PDF file. Document formatted into pages; contains xiii, 108 p.; also includes graphics. Includes bibliographical references (p. 104-108). Available online via OhioLINK's ETD Center

Longitudinal data are collected for studying changes across time. In social sciences, interest is often in theoretical constructs, such as attitudes, behaviour or abilities, which cannot be directly measured. In that case, multiple related manifest (observed) variables, for example survey questions or items in an ability test, are used as indicators for the constructs, which are themselves treated as latent (unobserved) variables. In this thesis, multivariate longitudinal data is considered where multiple observed variables, measured at each time point, are used as indicators for theoretical constructs (latent variables) of interest. The observed items and the latent variables are linked together via statistical latent variable models. A common problem in longitudinal studies is missing data, where missingness can be classiffed into one of two forms. Dropout occurs when subjects exit the study prematurely, while intermittent missingness takes place when subjects miss one or more occasions but show up on a subsequent wave of the study. Ignoring the missingness mechanism can lead to biased estimates, especially when the missingness is nonrandom. The approach proposed in this thesis uses latent variable models to capture the evolution of a latent phenomenon over time, while incorporating a missingness mechanism to account for possibly nonrandom forms of missingness. Two model specifications are presented, the first of which incorporates dropout only in the missingness mechanism, while the other accounts for both dropout and intermittent missingness allowing them to be informative by being modelled as functions of the latent variables and possibly observed covariates. Models developed in this thesis consider ordinal and binary observed items, because such variables are often met in social surveys, while the underlying latent variables are assumed to be continuous. The proposed models are illustrated by analysing people's perceptions on women's work using three questions from five waves of the British Household Panel Survey.

The purpose of this research is to get a better understanding of how different machine learning algorithms work with different amounts of data corruption. This is important since data corruption is an overbearing issue within data collection and thus, in extension, any work that relies on the collected data. The questions we were looking at were: What feature is the most important? How significant is the correlation of features? What algorithms should be used given the data available? And, How much noise (inaccurate or unhelpful captured data) is acceptable?  The study is structured to introduce AI in healthcare, data missingness, and the machine learning algorithms we used in the study. In the method section, we give a recommended workflow for handling data with machine learning in mind. The results show us that when a dataset is filled with random values, the run-time of algorithms increases since many patterns are lost. Randomly removing values also caused less of a problem than first anticipated since we ran multiple trials, evening out any problems caused by the lost values. Lastly, imputation is a preferred way of handling missing data since it retained many dataset structures. One has to keep in mind if the imputation is done on categories or numerical values. However, there is no easy "best-fit" for any dataset. It is hard to give a concrete answer when choosing a machine learning algorithm that fits any dataset. Nevertheless, since it is easy to simply plug-and-play with many algorithms, we would recommend any user try different ones before deciding which one fits a project the best.

Normatively, protecting civilians from the conduct of hostilities is grounded in the Geneva Conventions and the UN Security Council protection of civilian agenda, both of which celebrate their 70 and 20 year anniversaries in 2019. Previous research focusses heavily on protection of civilians through peacekeeping whereas this research focuses on ‘non-armed’ approaches to enhancing civilian protection in conflict. Prior research and experience reveals a high level of missingness and variation in the level of available data on civilian harm in conflict. Where civilian harm is considered in the peace and conflict literature, it is predominantly from a securitized lens of understanding insurgent recruitment strategies and more recent counter-insurgent strategies aimed at winning ‘hearts and minds’. Through a structured focused comparison of four case studies the correlation between the level of quality, timely and consistent data on civilian harm and affect on the conduct of hostilities will be reviewed and potential confounders identified. Following this the hypothesized causal mechanism will be process traced through the pathway case of Afghanistan. The findings and analysis from both methods identify support for the theory and it’s refinement with important nuances in the factors conducive to quality, timely and consistent data collection on civilian harm in armed conflict.

Nesta tese apresentam-se desenvolvimentos metodológicos para analisar dados com omissão e também estudos delineados para compreender os resultados de tais análises. Escrutinam-se análises de sensibilidade bayesiana e clássica para dados com respostas categorizadas sujeitas a omissão. Mostra-se que as componentes subjetivas de cada abordagem podem influenciar os resultados de maneira não-trivial, independentemente do tamanho da amostra, e que, portanto, as conclusões devem ser cuidadosamente avaliadas. Especificamente, demonstra-se que distribuições \\apriori\\ comumente consideradas como não-informativas ou levemente informativas podem, na verdade, ser bastante informativas para parâmetros inidentificáveis, e que a escolha do modelo sobreparametrizado também tem um papel importante. Quando há omissão em variáveis explicativas, também é necessário propor um modelo marginal para as covariáveis mesmo se houver interesse apenas no modelo condicional. A especificação incorreta do modelo para as covariáveis ou do modelo para o mecanismo de omissão leva a inferências enviesadas para o modelo de interesse. Trabalhos anteriormente publicados têm-se dividido em duas vertentes: ou utilizam distribuições semiparamétricas/não-paramétricas, flexíveis para as covariáveis, e identificam o modelo com a suposição de um mecanismo de omissão não-informativa, ou empregam distribuições paramétricas para as covariáveis e permitem um mecanismo mais geral, de omissão informativa. Neste trabalho analisam-se respostas binárias, combinando um mecanismo de omissão informativa com um modelo não-paramétrico para as covariáveis contínuas, por meio de uma mistura induzida pela distribuição \\apriori\\ de processo de Dirichlet. No caso em que o interesse recai apenas em momentos da distribuição das respostas, propõe-se uma nova análise de sensibilidade sob o enfoque clássico para respostas incompletas que evita suposições distribucionais e utiliza parâmetros de sensibilidade de fácil interpretação. O procedimento tem, em particular, grande apelo na análise de dados contínuos, campo que tradicionalmente emprega suposições de normalidade e/ou utiliza parâmetros de sensibilidade de difícil interpretação. Todas as análises são ilustradas com conjuntos de dados reais.<br>We present methodological developments to conduct analyses with missing data and also studies designed to understand the results of such analyses. We examine Bayesian and classical sensitivity analyses for data with missing categorical responses and show that the subjective components of each approach can influence results in non-trivial ways, irrespectively of the sample size, concluding that they need to be carefully evaluated. Specifically, we show that prior distributions commonly regarded as slightly informative or non-informative may actually be too informative for non-identifiable parameters, and that the choice of over-parameterized models may drastically impact the results. When there is missingness in explanatory variables, we also need to consider a marginal model for the covariates even if the interest lies only on the conditional model. An incorrect specification of either the model for the covariates or of the model for the missingness mechanism leads to biased inferences for the parameters of interest. Previously published works are commonly divided into two streams: either they use semi-/non-parametric flexible distributions for the covariates and identify the model via a non-informative missingness mechanism, or they employ parametric distributions for the covariates and allow a more general informative missingness mechanism. We consider the analysis of binary responses, combining an informative missingness model with a non-parametric model for the continuous covariates via a Dirichlet process mixture. When the interest lies only in moments of the response distribution, we consider a new classical sensitivity analysis for incomplete responses that avoids distributional assumptions and employs easily interpreted sensitivity parameters. The procedure is particularly useful for analyses of missing continuous data, an area where normality is traditionally assumed and/or relies on hard-to-interpret sensitivity parameters. We illustrate all analyses with real data sets.

碩士<br>國立陽明大學<br>公共衛生研究所<br>101<br>Objectives: This dissertation aims to explore the genome-wide pattern of informative missingness among parent genotypes due to various qualities of genotyping. Methods: Genotype, quality score, and pedigree of HapMap data were merged together and genotype scores below 10000, 9000, 8000, and 7000 were assigned to be missing values. Therefore, four sets of trio data with partial missing parental genotypes were implemented by the TIMBD (Guo, 2012), which determines whether parental genotypes are missing informatively or not. SNPs that are significant in the four sets of trio data were studied and 20 of them were matched with RS numbers. Using the NCBI (The National Center for Biotechnology Information) data base, the SNP regional map was used to identify published associations and nearby SNPs in LD with the 20 SNPs. Haploview was used to find linkage disequilibrium information between the 20 SNPs and nearby SNPs. Results: Among the 20 SNPs where parental genotypes are missing informatively due to various genotyping qualities, only one SNP was reported to be associated with obesity and cardiovascular disease. No replications were reported by the 20 SNPs. It is likely that these significant SNPs are false positives.

Trabalho de Projeto do Mestrado Integrado em Engenharia Biomédica apresentado à Faculdade de Ciências e Tecnologia<br>Com a evolução tecnológica, verificou-se um aumento exponencial da quantidade de dados recolhidos e armazenados. Assim, surgiu a necessidade de criar mecanismos automáticos para extrair conhecimento dos referidos dados. Estes mecanismos automáticos, conhecidos por modelos de aprendizagem automática, foram, na sua maioria, desenvolvidos para dados completos, requisito que nem sempre é possível cumprir. Neste contexto, a imputação dos dados (substituição dos valores em falta por estimativas plausíveis) surge como uma possível solução, garantindo a qualidade dos dados para posterior análise.Nos últimos anos, vários estudos têm proposto novas técnicas de imputação, de entre as quais se destaca a utilização de Stacked Denoising Autoencoders. Dada a sua extraordinária capacidade de recuperar dados corrompidos, os Denoising Autoencoders mostram-se promissores na área da imputação de dados, tendo despertado um interesse crescente por parte da comunidade científica.No entanto, sendo um tópico recente, a sua aplicação ainda não se encontra suficientemente bem estudada, apresentando diversos aspetos por explorar; em particular, a sua adequação a diferentes mecanismos de dados em falta (Missing Completely At Random, Missing At Random e Missing Not At Random). Esta tese apresenta um estudo aprofundado da imputação de dados via Stacked Denoising Autoencoders, considerando diferentes mecanismos e percentagens de dados em falta. Em comparação com métodos de imputação do estado da arte, os Stacked Denoising Autoencoders mostraram ser abordagens robustas para a imputação de elevadas percentagens de dados em falta, especialmente quando o mecanismo subjacente à sua geração é Missing Not At Random.<br>The evolution of technology led to an exponential increase in the amount of data being collected and stored, thus creating the need to develop automatic mechanisms to extract knowledge from data. These automatic mechanisms, known as Machine Learning techniques, were mostly designed for complete data, a requirement that is not always fulfilled. In this context, data imputation (replacement of missing values by plausible estimates) arises as a possible solution, ensuring the quality of data for later analysis. Over the years, several studies presented alternative imputation strategies, among which Stacked Denoising Autoencoders stand out. Given their ability to recover corrupted data, Stacked Denoising Autoencoders are promising in the area of data imputation, generating great interest in the scientific community. However, their application is an understudied topic, still presenting challenging aspects for research; namely, their suitability for different missing data mechanisms (Missing Completely At Random, Missing At Random and Missing Not At Random). This thesis presents a thorough study of data imputation via Stacked Denoising Autoencoders, considering different missing data mechanisms and missing rates. In comparison to state-of-the-art imputation methods, Stacked Denoising Autoencoders proved to be robust for imputing high missing rates, especially, when the mechanism underlying their generation is Missing Not At Random.