Academic literature on the topic 'AI annotation'

This paper introduces an AI-driven Android application designed to automate image annotation, addressing the need for efficient labeling across diverse domains. Leveraging deep learning algorithms, the application analyzes images uploaded or captured by users, assigning relevant annotations. Through extensive testing, the application demonstrates high accuracy and efficiency in annotating images, closely matching human annotators' results. The findings underscore the potential of AI-driven annotation tools to revolutionize image dataset creation and utilization, facilitating advancements in machine learning and computer vision applications.

Introduction. Collaborative human-AI annotation is a promising approach for various tasks with large-scale and complex data. Tools and methods to support effective human-AI collaboration for data annotation are an important direction for research. In this paper, we present CHAIRA: a Collaborative Human-AI Risk Annotation tool that enables human and AI agents to collaboratively annotate online incivility. Method. We leveraged Large Language Models (LLMs) to facilitate the interaction between human and AI annotators and examine four different prompting strategies. The developed CHAIRA system combines multiple prompting approaches with human-AI collaboration for online incivility data annotation. Analysis. We evaluated CHAIRA on 457 user comments with ground truth labels based on the inter-rater agreement between human and AI coders. Results. We found that the most collaborative prompt supported a high level of agreement between a human agent and AI, comparable to that of two human coders. While the AI missed some implicit incivility that human coders easily identified, it also spotted politically nuanced incivility that human coders overlooked. Conclusions. Our study reveals the benefits and challenges of using AI agents for incivility annotation and provides design implications and best practices for human-AI collaboration in subjective data annotation.

While artificial intelligence (AI) holds promise for supporting healthcare providers and improving the accuracy of medical diagnoses, a lack of transparency in the composition of datasets exposes AI models to the possibility of unintentional and avoidable mistakes. In particular, public and private image datasets of dermatological conditions rarely include information on skin color. As a start towards increasing transparency, AI researchers have appropriated the use of the Fitzpatrick skin type (FST) from a measure of patient photosensitivity to a measure for estimating skin tone in algorithmic audits of computer vision applications including facial recognition and dermatology diagnosis. In order to understand the variability of estimated FST annotations on images, we compare several FST annotation methods on a diverse set of 460 images of skin conditions from both textbooks and online dermatology atlases. These methods include expert annotation by board-certified dermatologists, algorithmic annotation via the Individual Typology Angle algorithm, which is then converted to estimated FST (ITA-FST), and two crowd-sourced, dynamic consensus protocols for annotating estimated FSTs. We find the inter-rater reliability between three board-certified dermatologists is comparable to the inter-rater reliability between the board-certified dermatologists and either of the crowdsourcing methods. In contrast, we find that the ITA-FST method produces annotations that are significantly less correlated with the experts' annotations than the experts' annotations are correlated with each other. These results demonstrate that algorithms based on ITA-FST are not reliable for annotating large-scale image datasets, but human-centered, crowd-based protocols can reliably add skin type transparency to dermatology datasets. Furthermore, we introduce the concept of dynamic consensus protocols with tunable parameters including expert review that increase the visibility of crowdwork and provide guidance for future crowdsourced annotations of large image datasets.

Aim: As a preliminary stage in the development of an artificial intelligence (AI) algorithm for surgery, this work aimed to study the inter- and intra-individual variability of phase annotations in videos of minimally invasive plate osteosynthesis of distal radius fractures (MIPO). The main hypothesis was that the inter-individual variability was almost perfect if Cohen's kappa coefficient (k) was ≥ 81% overall; the secondary hypothesis was that the intra-individual variability was almost perfect if the F1-score (F1) was ≥ 81%. Methods: The material comprised 9 annotators and three annotated MIPO videos with 5 phases and 4 sub-phases. Each video was presented 3 times to each annotator. The method involved analysing the inter-individual variability of annotations by computing k and F1 from a reference annotator. The intra-individual variability of annotations was analysed by computing F1. Results: Annotation anomalies were noticed: either absences or differences in phase and sub-phase annotations. Regarding the inter-individual variability, an almost perfect agreement between annotators was observed because k ≥ 81% for the three videos. Regarding the intra-individual variability, F1 ≥ 81% for most phases and sub-phases with the nine annotators. Conclusion: The homogeneity of annotations must be as high as possible to develop an AI algorithm in surgery. Therefore, it is necessary to identify the least efficient annotators (measurement of the intra-individual variability) to provide them with individual training and a personalised annotation rhythm. It is also important to optimise the definition of the phases, improve the annotation protocol and choose suitable training videos.

Gene annotation is a critical process in genomics that involves the description of not only the position but also the function of an encoded element of a genome. In general, this provides biological context to sequence data, enabling an advanced level of understanding of genetic information. This is important in areas aligned with genetic engineering, studies of diseases, and evolution. Through ML and DL methodologies, AI enhances functional annotation and gene prediction effectively and accurately. This review focuses on AI in genomic research and assesses its effectiveness compared to traditional annotation tools. Using Escherichia coli as the representative model organism, the study focuses on a systematic approach of gene prediction using web Augustus with functional annotation using DeepGOPlus, an artificial intelligence tool, instead of the conventional BLAST-based annotation using the UniProt database. The study researches the extent of GO term coverage, the specificity of the annotations, and the concordance among these various tools. Artificial intelligence is highly beneficial owing to its speed, scalability, and proficiency in annotating intricate or poorly defined genomic areas. Notable instances include DeepGOPlus, which has demonstrated enhanced coverage by suggesting new terms that were frequently missed by earlier traditional tools. Notwithstanding these, AI tools face challenges such as dependence on high-quality training data, concerns about interpretability, and the need for biological validation to support the predictions. This review emphasizes the transformative impact that artificial intelligence brings to the field of gene annotation by presenting novel applications in many fields, including personalized medicine and synthetic biology, in which traditional methods suffer from severe limitations.

BACKGROUND Missing occult cancer lesions accounts for the most diagnostic errors in retrospective radiology reviews as early cancer can be small or subtle, making the lesions difficult to detect. Second-observer is the most effective technique for reducing these events and can be economically implemented with the advent of artificial intelligence (AI). AIM To achieve appropriate AI model training, a large annotated dataset is necessary to train the AI models. Our goal in this research is to compare two methods for decreasing the annotation time to establish ground truth: Skip-slice annotation and AI-initiated annotation. METHODS We developed a 2D U-Net as an AI second observer for detecting colorectal cancer (CRC) and an ensemble of 5 differently initiated 2D U-Net for ensemble technique. Each model was trained with 51 cases of annotated CRC computed tomography of the abdomen and pelvis, tested with 7 cases, and validated with 20 cases from The Cancer Imaging Archive cases. The sensitivity, false positives per case, and estimated Dice coefficient were obtained for each method of training. We compared the two methods of annotations and the time reduction associated with the technique. The time differences were tested using Friedman’s two-way analysis of variance. RESULTS Sparse annotation significantly reduces the time for annotation particularly skipping 2 slices at a time (P < 0.001). Reduction of up to 2/3 of the annotation does not reduce AI model sensitivity or false positives per case. Although initializing human annotation with AI reduces the annotation time, the reduction is minimal, even when using an ensemble AI to decrease false positives. CONCLUSION Our data support the sparse annotation technique as an efficient technique for reducing the time needed to establish the ground truth.

Accurately estimating the six degree of freedom (6-DoF) pose of objects in images is essential for a variety of applications such as robotics, autonomous driving, and autonomous, AI, and vision-based navigation for unmanned aircraft systems (UAS). Developing such algorithms requires large datasets; however, generating those is tedious as it requires annotating the 6-DoF relative pose of each object of interest present in the image w.r.t. to the camera. Therefore, this work presents a novel approach that automates the data acquisition and annotation process and thus minimizes the annotation effort to the duration of the recording. To maximize the quality of the resulting annotations, we employ an optimization-based approach for determining the extrinsic calibration parameters of the camera. Our approach can handle multiple objects in the scene, automatically providing ground-truth labeling for each object and taking into account occlusion effects between different objects. Moreover, our approach can not only be used to generate data for 6-DoF pose estimation and corresponding 3D-models but can be also extended to automatic dataset generation for object detection, instance segmentation, or volume estimation for any kind of object.

Background/Objectives: In the burgeoning field of medical imaging and Artificial Intelligence (AI), high-quality annotations for training AI-models are crucial. However, there are still only a few large datasets, as segmentation is time-consuming, experts have limited time. This study investigates how the experience of radiologists affects the quality of annotations. Methods: We randomly collected 53 anonymized chest radiographs. Fifteen readers with varying levels of expertise annotated the anatomical structures of different complexity, pneumonic opacities and central venous catheters (CVC) as examples of pathologies and foreign material. The readers were divided into three groups of five. The groups consisted of medical students (MS), junior professionals (JP) with less than five years of working experience and senior professionals (SP) with more than five years of experience. Each annotation was compared to a gold standard consisting of a consensus annotation of three senior board-certified radiologists. We calculated the Dice coefficient (DSC) and Hausdorff distance (HD) to evaluate annotation quality. Inter- and intrareader variability and time dependencies were investigated using Intraclass Correlation Coefficient (ICC) and Ordinary Least Squares (OLS). Results: Senior professionals generally showed better performance, while medical students had higher variability in their annotations. Significant differences were noted, especially for complex structures (DSC Pneumonic Opacities as mean [standard deviation]: MS: 0.516 [0.246]; SP: 0.631 [0.211]). However, it should be noted that overall deviation and intraclass variance was higher for these structures even for seniors, highlighting the inherent limitations of conventional radiography. Experience showed a positive relationship with annotation quality for VCS and lung but was not a significant factor for other structures. Conclusions: Experience level significantly impacts annotation quality. Senior radiologists provided higher-quality annotations for complex structures, while less experienced readers could still annotate simpler structures with satisfying accuracy. We suggest a mixed-expertise approach, enabling the highly experienced to utilize their knowledge most effectively. With the increase in numbers of examinations, radiology will rely on AI support tools in the future. Therefore, economizing the process of data acquisition and AI-training; for example, by integrating less experienced radiologists, will help to meet the coming challenges.

142 Background: Colorectal cancer (CRC) is the second leading cause of cancer-related deaths, and its outcome can be improved with better detection of incidental early CRC on routine CT of the abdomen and pelvis (CTAP). AI-second observer (AI) has the potential as shown in our companion abstract. The bottleneck in training AI is the time required for radiologists to segment the CRC. We compared two techniques for accelerating the segmentation process: 1) Sparse annotation (annotating some of the CT slice containing CRC instead of every slice); 2) Allowing AI to perform initial segmentation followed by human adjustment. Methods: 2D U-Net convolutional neural network (CNN) containing 31 million trainable parameters was trained with 58 CRC CT images from Banner MD Anderson (AZ) and MD Anderson Cancer Center (TX) (51 used for training and 7 for validation) and 59 normal CT scans from Banner MD Anderson Cancer Center. Twenty of the 25 CRC cases from public domain data (The Cancer Genome Atlas) were used to evaluate the performance of the models. The CRC was segmented using ITK-SNAP open-source software (v. 3.8). For the first objective, 3 separate models were trained (fully annotated CRC, every other slice, and every third slice). The AI-annotation on the TCGA dataset was analyzed by the percentage of correct detection of CRC, the number of false positives, and the Dice similarity coefficient (DSC). If parts of the CRC were flagged by AI, then it was considered correct. A detection was considered false positive if the marked lesion did not overlap with CRC; contiguous false positives across different slices of CT image were considered a single false positive. DSC measures the quality of the segmentation by measuring the overlap between the ground-truth and AI detected lesion. For the second objective, the time required to adjust the AI-produced annotation was compared to the time required for annotating the entire CRC without AI assistance. The AI-models were trained using ensemble learning (see our companion abstract for details of the techniques). Results: Our results showed that skipping slices of tumor in training did not alter the accuracy, false positives, or DSC classification of the model. When adjusting the AI-observer segmentation, there was a trend toward decreasing the time required to adjust the annotation compared to full manual segmentation, but the difference was not statistically significant (Table; p=0.121). Conclusions: Our results show that both skipping slices of tumor as well as starting with AI-produced annotation can potentially decrease the effort required to produce high-quality ground truth without compromising the performance of AI. These techniques can help improve the throughput to obtain a large volume of cases to train AI for detecting CRC.[Table: see text]

Objective: Reliable reading and annotation of chest X-ray (CXR) images are essential for both clinical decision-making and AI model development. While most of the literature emphasizes pulmonary findings, this study evaluates the consistency and reliability of annotations for extrapulmonary findings, using a labelling scheme. Methods: Six clinicians with varying experience levels (novice, intermediate, and experienced) annotated 100 CXR images using a diagnostic labelling scheme, in two rounds, separated by a three-week washout period. Annotation consistency was assessed using Randolph’s free-marginal kappa (RK), prevalence- and bias-adjusted kappa (PABAK), proportion positive agreement (PPA), and proportion negative agreement (PNA). Pairwise comparisons and the McNemar’s test were conducted to assess inter-reader and intra-reader agreement. Results: PABAK values indicated high overall grouped labelling agreement (novice: 0.86, intermediate: 0.90, experienced: 0.91). PNA values demonstrated strong agreement on negative findings, while PPA values showed moderate-to-low consistency in positive findings. Significant differences in specific agreement emerged between novice and experienced clinicians for eight labels, but there were no significant variations in RK across experience levels. The McNemar’s test confirmed annotation stability between rounds. Conclusions: This study demonstrates that clinician annotations of extrapulmonary findings in CXR are consistent and reliable across different experience levels using a pre-defined diagnostic labelling scheme. These insights aid in optimizing training strategies for both clinicians and AI models.