Pour voir les autres types de publications sur ce sujet consultez le lien suivant : AI annotation.
Articles de revues sur le sujet « AI annotation »
Créez une référence correcte selon les styles APA, MLA, Chicago, Harvard et plusieurs autres
Consultez les 50 meilleurs articles de revues pour votre recherche sur le sujet « AI annotation ».
À côté de chaque source dans la liste de références il y a un bouton « Ajouter à la bibliographie ». Cliquez sur ce bouton, et nous générerons automatiquement la référence bibliographique pour la source choisie selon votre style de citation préféré : APA, MLA, Harvard, Vancouver, Chicago, etc.
Vous pouvez aussi télécharger le texte intégral de la publication scolaire au format pdf et consulter son résumé en ligne lorsque ces informations sont inclues dans les métadonnées.
Parcourez les articles de revues sur diverses disciplines et organisez correctement votre bibliographie.
1
Sisodiya, Hariom. "AnnotImage: An Image Annotation App." INTERANTIONAL JOURNAL OF SCIENTIFIC RESEARCH IN ENGINEERING AND MANAGEMENT 08, no. 06 (2024): 1–5. http://dx.doi.org/10.55041/ijsrem35581.
Texte intégralRésumé :
This paper introduces an AI-driven Android application designed to automate image annotation, addressing the need for efficient labeling across diverse domains. Leveraging deep learning algorithms, the application analyzes images uploaded or captured by users, assigning relevant annotations. Through extensive testing, the application demonstrates high accuracy and efficiency in annotating images, closely matching human annotators' results. The findings underscore the potential of AI-driven annotation tools to revolutionize image dataset creation and utilization, facilitating advancements in machine learning and computer vision applications.
2
Park, Jinkyung Katie, Rahul Dev Ellezhuthil, Pamela Wisniewski, and Vivek Singh. "Collaborative human-AI risk annotation: co-annotating online incivility with CHAIRA." Information Research an international electronic journal 30, iConf (2025): 992–1008. https://doi.org/10.47989/ir30iconf47146.
Texte intégralRésumé :
Introduction. Collaborative human-AI annotation is a promising approach for various tasks with large-scale and complex data. Tools and methods to support effective human-AI collaboration for data annotation are an important direction for research. In this paper, we present CHAIRA: a Collaborative Human-AI Risk Annotation tool that enables human and AI agents to collaboratively annotate online incivility. Method. We leveraged Large Language Models (LLMs) to facilitate the interaction between human and AI annotators and examine four different prompting strategies. The developed CHAIRA system combines multiple prompting approaches with human-AI collaboration for online incivility data annotation. Analysis. We evaluated CHAIRA on 457 user comments with ground truth labels based on the inter-rater agreement between human and AI coders. Results. We found that the most collaborative prompt supported a high level of agreement between a human agent and AI, comparable to that of two human coders. While the AI missed some implicit incivility that human coders easily identified, it also spotted politically nuanced incivility that human coders overlooked. Conclusions. Our study reveals the benefits and challenges of using AI agents for incivility annotation and provides design implications and best practices for human-AI collaboration in subjective data annotation.
3
Groh, Matthew, Caleb Harris, Roxana Daneshjou, Omar Badri, and Arash Koochek. "Towards Transparency in Dermatology Image Datasets with Skin Tone Annotations by Experts, Crowds, and an Algorithm." Proceedings of the ACM on Human-Computer Interaction 6, CSCW2 (2022): 1–26. http://dx.doi.org/10.1145/3555634.
Texte intégralRésumé :
While artificial intelligence (AI) holds promise for supporting healthcare providers and improving the accuracy of medical diagnoses, a lack of transparency in the composition of datasets exposes AI models to the possibility of unintentional and avoidable mistakes. In particular, public and private image datasets of dermatological conditions rarely include information on skin color. As a start towards increasing transparency, AI researchers have appropriated the use of the Fitzpatrick skin type (FST) from a measure of patient photosensitivity to a measure for estimating skin tone in algorithmic audits of computer vision applications including facial recognition and dermatology diagnosis. In order to understand the variability of estimated FST annotations on images, we compare several FST annotation methods on a diverse set of 460 images of skin conditions from both textbooks and online dermatology atlases. These methods include expert annotation by board-certified dermatologists, algorithmic annotation via the Individual Typology Angle algorithm, which is then converted to estimated FST (ITA-FST), and two crowd-sourced, dynamic consensus protocols for annotating estimated FSTs. We find the inter-rater reliability between three board-certified dermatologists is comparable to the inter-rater reliability between the board-certified dermatologists and either of the crowdsourcing methods. In contrast, we find that the ITA-FST method produces annotations that are significantly less correlated with the experts' annotations than the experts' annotations are correlated with each other. These results demonstrate that algorithms based on ITA-FST are not reliable for annotating large-scale image datasets, but human-centered, crowd-based protocols can reliably add skin type transparency to dermatology datasets. Furthermore, we introduce the concept of dynamic consensus protocols with tunable parameters including expert review that increase the visibility of crowdwork and provide guidance for future crowdsourced annotations of large image datasets.
4
Graëff, Camille, Thomas Lampert, Jean-Paul Mazellier, Nicolas Padoy, Laela El Amiri, and Philippe Liverneaux. "The preliminary stage in developing an artificial intelligence algorithm: a study of the inter- and intra-individual variability of phase annotations in internal fixation of distal radius fracture videos." Artificial Intelligence Surgery 3, no. 3 (2023): 147–59. http://dx.doi.org/10.20517/ais.2023.12.
Texte intégralRésumé :
Aim: As a preliminary stage in the development of an artificial intelligence (AI) algorithm for surgery, this work aimed to study the inter- and intra-individual variability of phase annotations in videos of minimally invasive plate osteosynthesis of distal radius fractures (MIPO). The main hypothesis was that the inter-individual variability was almost perfect if Cohen's kappa coefficient (k) was ≥ 81% overall; the secondary hypothesis was that the intra-individual variability was almost perfect if the F1-score (F1) was ≥ 81%. Methods: The material comprised 9 annotators and three annotated MIPO videos with 5 phases and 4 sub-phases. Each video was presented 3 times to each annotator. The method involved analysing the inter-individual variability of annotations by computing k and F1 from a reference annotator. The intra-individual variability of annotations was analysed by computing F1. Results: Annotation anomalies were noticed: either absences or differences in phase and sub-phase annotations. Regarding the inter-individual variability, an almost perfect agreement between annotators was observed because k ≥ 81% for the three videos. Regarding the intra-individual variability, F1 ≥ 81% for most phases and sub-phases with the nine annotators. Conclusion: The homogeneity of annotations must be as high as possible to develop an AI algorithm in surgery. Therefore, it is necessary to identify the least efficient annotators (measurement of the intra-individual variability) to provide them with individual training and a personalised annotation rhythm. It is also important to optimise the definition of the phases, improve the annotation protocol and choose suitable training videos.
5
Xu, Yixuan, and Jingyi Cui. "Artificial Intelligence in Gene Annotation: Current Applications, Challenges, and Future Prospects." Theoretical and Natural Science 98, no. 1 (2025): 8–15. https://doi.org/10.54254/2753-8818/2025.21464.
Texte intégralRésumé :
Gene annotation is a critical process in genomics that involves the description of not only the position but also the function of an encoded element of a genome. In general, this provides biological context to sequence data, enabling an advanced level of understanding of genetic information. This is important in areas aligned with genetic engineering, studies of diseases, and evolution. Through ML and DL methodologies, AI enhances functional annotation and gene prediction effectively and accurately. This review focuses on AI in genomic research and assesses its effectiveness compared to traditional annotation tools. Using Escherichia coli as the representative model organism, the study focuses on a systematic approach of gene prediction using web Augustus with functional annotation using DeepGOPlus, an artificial intelligence tool, instead of the conventional BLAST-based annotation using the UniProt database. The study researches the extent of GO term coverage, the specificity of the annotations, and the concordance among these various tools. Artificial intelligence is highly beneficial owing to its speed, scalability, and proficiency in annotating intricate or poorly defined genomic areas. Notable instances include DeepGOPlus, which has demonstrated enhanced coverage by suggesting new terms that were frequently missed by earlier traditional tools. Notwithstanding these, AI tools face challenges such as dependence on high-quality training data, concerns about interpretability, and the need for biological validation to support the predictions. This review emphasizes the transformative impact that artificial intelligence brings to the field of gene annotation by presenting novel applications in many fields, including personalized medicine and synthetic biology, in which traditional methods suffer from severe limitations.
6
Grudza, Matthew, Brandon Salinel, Sarah Zeien, et al. "Methods for improving colorectal cancer annotation efficiency for artificial intelligence-observer training." World Journal of Radiology 15, no. 12 (2023): 359–69. http://dx.doi.org/10.4329/wjr.v15.i12.359.
Texte intégralRésumé :
BACKGROUND Missing occult cancer lesions accounts for the most diagnostic errors in retrospective radiology reviews as early cancer can be small or subtle, making the lesions difficult to detect. Second-observer is the most effective technique for reducing these events and can be economically implemented with the advent of artificial intelligence (AI). AIM To achieve appropriate AI model training, a large annotated dataset is necessary to train the AI models. Our goal in this research is to compare two methods for decreasing the annotation time to establish ground truth: Skip-slice annotation and AI-initiated annotation. METHODS We developed a 2D U-Net as an AI second observer for detecting colorectal cancer (CRC) and an ensemble of 5 differently initiated 2D U-Net for ensemble technique. Each model was trained with 51 cases of annotated CRC computed tomography of the abdomen and pelvis, tested with 7 cases, and validated with 20 cases from The Cancer Imaging Archive cases. The sensitivity, false positives per case, and estimated Dice coefficient were obtained for each method of training. We compared the two methods of annotations and the time reduction associated with the technique. The time differences were tested using Friedman’s two-way analysis of variance. RESULTS Sparse annotation significantly reduces the time for annotation particularly skipping 2 slices at a time (P < 0.001). Reduction of up to 2/3 of the annotation does not reduce AI model sensitivity or false positives per case. Although initializing human annotation with AI reduces the annotation time, the reduction is minimal, even when using an ensemble AI to decrease false positives. CONCLUSION Our data support the sparse annotation technique as an efficient technique for reducing the time needed to establish the ground truth.
7
Apud Baca, Javier Gibran, Thomas Jantos, Mario Theuermann, et al. "Automated Data Annotation for 6-DoF AI-Based Navigation Algorithm Development." Journal of Imaging 7, no. 11 (2021): 236. http://dx.doi.org/10.3390/jimaging7110236.
Texte intégralRésumé :
Accurately estimating the six degree of freedom (6-DoF) pose of objects in images is essential for a variety of applications such as robotics, autonomous driving, and autonomous, AI, and vision-based navigation for unmanned aircraft systems (UAS). Developing such algorithms requires large datasets; however, generating those is tedious as it requires annotating the 6-DoF relative pose of each object of interest present in the image w.r.t. to the camera. Therefore, this work presents a novel approach that automates the data acquisition and annotation process and thus minimizes the annotation effort to the duration of the recording. To maximize the quality of the resulting annotations, we employ an optimization-based approach for determining the extrinsic calibration parameters of the camera. Our approach can handle multiple objects in the scene, automatically providing ground-truth labeling for each object and taking into account occlusion effects between different objects. Moreover, our approach can not only be used to generate data for 6-DoF pose estimation and corresponding 3D-models but can be also extended to automatic dataset generation for object detection, instance segmentation, or volume estimation for any kind of object.
8
Multusch, Malte Michel, Lasse Hansen, Mattias Paul Heinrich, et al. "Impact of Radiologist Experience on AI Annotation Quality in Chest Radiographs: A Comparative Analysis." Diagnostics 15, no. 6 (2025): 777. https://doi.org/10.3390/diagnostics15060777.
Texte intégralRésumé :
Background/Objectives: In the burgeoning field of medical imaging and Artificial Intelligence (AI), high-quality annotations for training AI-models are crucial. However, there are still only a few large datasets, as segmentation is time-consuming, experts have limited time. This study investigates how the experience of radiologists affects the quality of annotations. Methods: We randomly collected 53 anonymized chest radiographs. Fifteen readers with varying levels of expertise annotated the anatomical structures of different complexity, pneumonic opacities and central venous catheters (CVC) as examples of pathologies and foreign material. The readers were divided into three groups of five. The groups consisted of medical students (MS), junior professionals (JP) with less than five years of working experience and senior professionals (SP) with more than five years of experience. Each annotation was compared to a gold standard consisting of a consensus annotation of three senior board-certified radiologists. We calculated the Dice coefficient (DSC) and Hausdorff distance (HD) to evaluate annotation quality. Inter- and intrareader variability and time dependencies were investigated using Intraclass Correlation Coefficient (ICC) and Ordinary Least Squares (OLS). Results: Senior professionals generally showed better performance, while medical students had higher variability in their annotations. Significant differences were noted, especially for complex structures (DSC Pneumonic Opacities as mean [standard deviation]: MS: 0.516 [0.246]; SP: 0.631 [0.211]). However, it should be noted that overall deviation and intraclass variance was higher for these structures even for seniors, highlighting the inherent limitations of conventional radiography. Experience showed a positive relationship with annotation quality for VCS and lung but was not a significant factor for other structures. Conclusions: Experience level significantly impacts annotation quality. Senior radiologists provided higher-quality annotations for complex structures, while less experienced readers could still annotate simpler structures with satisfying accuracy. We suggest a mixed-expertise approach, enabling the highly experienced to utilize their knowledge most effectively. With the increase in numbers of examinations, radiology will rely on AI support tools in the future. Therefore, economizing the process of data acquisition and AI-training; for example, by integrating less experienced radiologists, will help to meet the coming challenges.
9
Salinel, Brandon, Matthew Grudza, Sarah Zeien, et al. "Comparison of segmentation methods to improve throughput in annotating AI-observer for detecting colorectal cancer." Journal of Clinical Oncology 40, no. 4_suppl (2022): 142. http://dx.doi.org/10.1200/jco.2022.40.4_suppl.142.
Texte intégralRésumé :
142 Background: Colorectal cancer (CRC) is the second leading cause of cancer-related deaths, and its outcome can be improved with better detection of incidental early CRC on routine CT of the abdomen and pelvis (CTAP). AI-second observer (AI) has the potential as shown in our companion abstract. The bottleneck in training AI is the time required for radiologists to segment the CRC. We compared two techniques for accelerating the segmentation process: 1) Sparse annotation (annotating some of the CT slice containing CRC instead of every slice); 2) Allowing AI to perform initial segmentation followed by human adjustment. Methods: 2D U-Net convolutional neural network (CNN) containing 31 million trainable parameters was trained with 58 CRC CT images from Banner MD Anderson (AZ) and MD Anderson Cancer Center (TX) (51 used for training and 7 for validation) and 59 normal CT scans from Banner MD Anderson Cancer Center. Twenty of the 25 CRC cases from public domain data (The Cancer Genome Atlas) were used to evaluate the performance of the models. The CRC was segmented using ITK-SNAP open-source software (v. 3.8). For the first objective, 3 separate models were trained (fully annotated CRC, every other slice, and every third slice). The AI-annotation on the TCGA dataset was analyzed by the percentage of correct detection of CRC, the number of false positives, and the Dice similarity coefficient (DSC). If parts of the CRC were flagged by AI, then it was considered correct. A detection was considered false positive if the marked lesion did not overlap with CRC; contiguous false positives across different slices of CT image were considered a single false positive. DSC measures the quality of the segmentation by measuring the overlap between the ground-truth and AI detected lesion. For the second objective, the time required to adjust the AI-produced annotation was compared to the time required for annotating the entire CRC without AI assistance. The AI-models were trained using ensemble learning (see our companion abstract for details of the techniques). Results: Our results showed that skipping slices of tumor in training did not alter the accuracy, false positives, or DSC classification of the model. When adjusting the AI-observer segmentation, there was a trend toward decreasing the time required to adjust the annotation compared to full manual segmentation, but the difference was not statistically significant (Table; p=0.121). Conclusions: Our results show that both skipping slices of tumor as well as starting with AI-produced annotation can potentially decrease the effort required to produce high-quality ground truth without compromising the performance of AI. These techniques can help improve the throughput to obtain a large volume of cases to train AI for detecting CRC.[Table: see text]
10
Pehrson, Lea Marie, Dana Li, Alyas Mayar, et al. "Clinicians’ Agreement on Extrapulmonary Radiographic Findings in Chest X-Rays Using a Diagnostic Labelling Scheme." Diagnostics 15, no. 7 (2025): 902. https://doi.org/10.3390/diagnostics15070902.
Texte intégralRésumé :
Objective: Reliable reading and annotation of chest X-ray (CXR) images are essential for both clinical decision-making and AI model development. While most of the literature emphasizes pulmonary findings, this study evaluates the consistency and reliability of annotations for extrapulmonary findings, using a labelling scheme. Methods: Six clinicians with varying experience levels (novice, intermediate, and experienced) annotated 100 CXR images using a diagnostic labelling scheme, in two rounds, separated by a three-week washout period. Annotation consistency was assessed using Randolph’s free-marginal kappa (RK), prevalence- and bias-adjusted kappa (PABAK), proportion positive agreement (PPA), and proportion negative agreement (PNA). Pairwise comparisons and the McNemar’s test were conducted to assess inter-reader and intra-reader agreement. Results: PABAK values indicated high overall grouped labelling agreement (novice: 0.86, intermediate: 0.90, experienced: 0.91). PNA values demonstrated strong agreement on negative findings, while PPA values showed moderate-to-low consistency in positive findings. Significant differences in specific agreement emerged between novice and experienced clinicians for eight labels, but there were no significant variations in RK across experience levels. The McNemar’s test confirmed annotation stability between rounds. Conclusions: This study demonstrates that clinician annotations of extrapulmonary findings in CXR are consistent and reliable across different experience levels using a pre-defined diagnostic labelling scheme. These insights aid in optimizing training strategies for both clinicians and AI models.
11
Hasei, Joe, Ryuichi Nakahara, Yujiro Otsuka, et al. "The Three-Class Annotation Method Improves the AI Detection of Early-Stage Osteosarcoma on Plain Radiographs: A Novel Approach for Rare Cancer Diagnosis." Cancers 17, no. 1 (2024): 29. https://doi.org/10.3390/cancers17010029.
Texte intégralRésumé :
Background/Objectives: Developing high-performance artificial intelligence (AI) models for rare diseases is challenging owing to limited data availability. This study aimed to evaluate whether a novel three-class annotation method for preparing training data could enhance AI model performance in detecting osteosarcoma on plain radiographs compared to conventional single-class annotation. Methods: We developed two annotation methods for the same dataset of 468 osteosarcoma X-rays and 378 normal radiographs: a conventional single-class annotation (1C model) and a novel three-class annotation method (3C model) that separately labeled intramedullary, cortical, and extramedullary tumor components. Both models used identical U-Net-based architectures, differing only in their annotation approaches. Performance was evaluated using an independent validation dataset. Results: Although both models achieved high diagnostic accuracy (AUC: 0.99 vs. 0.98), the 3C model demonstrated superior operational characteristics. At a standardized cutoff value of 0.2, the 3C model maintained balanced performance (sensitivity: 93.28%, specificity: 92.21%), whereas the 1C model showed compromised specificity (83.58%) despite high sensitivity (98.88%). Notably, at the 25th percentile threshold, both models showed identical false-negative rates despite significantly different cutoff values (3C: 0.661 vs. 1C: 0.985), indicating the ability of the 3C model to maintain diagnostic accuracy at substantially lower thresholds. Conclusions: This study demonstrated that anatomically informed three-class annotation can enhance AI model performance for rare disease detection without requiring additional training data. The improved stability at lower thresholds suggests that thoughtful annotation strategies can optimize the AI model training, particularly in contexts where training data are limited.
12
Baur, Tobias, Alexander Heimerl, Florian Lingenfelser, et al. "eXplainable Cooperative Machine Learning with NOVA." KI - Künstliche Intelligenz 34, no. 2 (2020): 143–64. http://dx.doi.org/10.1007/s13218-020-00632-3.
Texte intégralRésumé :
Abstract In the following article, we introduce a novel workflow, which we subsume under the term “explainable cooperative machine learning” and show its practical application in a data annotation and model training tool called NOVA. The main idea of our approach is to interactively incorporate the ‘human in the loop’ when training classification models from annotated data. In particular, NOVA offers a collaborative annotation backend where multiple annotators join their workforce. A main aspect is the possibility of applying semi-supervised active learning techniques already during the annotation process by giving the possibility to pre-label data automatically, resulting in a drastic acceleration of the annotation process. Furthermore, the user-interface implements recent eXplainable AI techniques to provide users with both, a confidence value of the automatically predicted annotations, as well as visual explanation. We show in an use-case evaluation that our workflow is able to speed up the annotation process, and further argue that by providing additional visual explanations annotators get to understand the decision making process as well as the trustworthiness of their trained machine learning models.
13
Bhanu Teja Reddy Maryala. "Global Ethical AI Data Standard: A framework for regulatory harmonization." World Journal of Advanced Engineering Technology and Sciences 15, no. 2 (2025): 2836–41. https://doi.org/10.30574/wjaets.2025.15.2.0851.
Texte intégralRésumé :
The Global Ethical AI Data Standard (GEADS) introduces a comprehensive framework addressing fundamental challenges in artificial intelligence data governance. Production AI systems frequently exhibit inadequate provenance documentation and questionable consent mechanisms, while data annotation practices often lack transparency and fair compensation. The regulatory environment presents significant fragmentation across jurisdictions, with limited convergence in core data protection principles between major frameworks like GDPR and CCPA. Informed consent mechanisms fundamentally fail in AI contexts because traditional notice-and-consent frameworks cannot anticipate how machine learning might derive unexpected insights from data. GEADS addresses these challenges through a three-tier classification system based on data sensitivity and potential impact, implementing core principles of Transparent Provenance, Contextual Consent, Annotator Protections, Derivative Accountability, and Proportional Governance. The framework demonstrates high regulatory compatibility across jurisdictions while maintaining minimal technical overhead. Implementation results show significant improvements in consent comprehension, annotation quality, and ethical issue detection during development phases. Organizations adopting GEADS report reduced compliance costs, fewer regulatory inquiries, and improved stakeholder trust. The framework bridges policy requirements with technical implementation, creating actionable guidelines that harmonize diverse regulatory approaches while introducing AI-specific provisions that address unique challenges in machine learning data governance.
14
Cornwell, Peter. "Progress with Repository-based Annotation Infrastructure for Biodiversity Applications." Biodiversity Information Science and Standards 7 (September 14, 2023): e112707. https://doi.org/10.3897/biss.7.112707.
Texte intégralRésumé :
Rapid development since the 1980s of technologies for analysing texts, has led not only to widespread employment of text 'mining', but also to now-pervasive large language model artificial intelligence (AI) applications. However, building new, concise, data resources from historic, as well as contemporary scientific literature, which can be employed efficiently at scale by automation and which have long-term value for the research community, has proved more elusive.Efforts at codifying analyses, such as the Text Encoding Initiative (TEI), date from the early 1990s and were initially driven by the social sciences and humanities (SSH) and linguistics communities, and extended with multiple XML-based tagging schemes, including in biodiversity (Miller et al. 2012). In 2010, the Bio-Ontologies Special Interest Group (of the International Society for Computational Biology) presented its Annotation Ontology (AO), incorporating JavaScript Object Notation and broadening previous XML-based approaches (Ciccarese et al. 2011). From 2011, the Open Annotation Data Model (OADM) (Sanderson et al. 2013) focused on cross-domain standards with utility for Web 3.0, leading to the W3C Web Annotation Data Model (WADM) Recommendation in February 2017*1 and the potential for unifying the multiplicity of already-in-use tagging approaches.This continual evolution has made the preservation of investment using annotation methods, and in particular of the connections between annotations and their context in source literature, particularly challenging. Infrastructure that entered service during the intervening years does not yet support WADM, and has only recently started to address the parallel emergence of page imagery-based standards such as the International Image Interoperability Framework (IIIF). Notably, IIIF instruments such as Mirador-2, which has been employed widely for manual creation and editing of annotations in SSH, continue to employ the now-deprecated OADM. Although multiple efforts now address combining IIIF and TEI text coordinate systems, they are currently fundamentally incompatible.However, emerging repository technologies enable preservation of annotation investment to be accomplished comprehensively for the first time. Native IIIF support enables interactive previewing of annotations within repository graphical user interfaces and dynamic serialisation technologies provide compatibility with existing XML-based infrastructures. Repository access controls can permit experts to trace annotation sources in original texts even if the literature is not publicly accessible, e.g., due to copyright restriction. This is of paramount importance, not only because surrounding context can be crucial to qualify formal terms that have been annotated, such as collecting country. Also, contemporary automated text mining—essential for operation at the scale of known biodiversity literature—is not 100% accurate and manual checking of uncertainties is currently essential. On-going improvement of language analysis tools through AI integration offers significant future gains from reprocessing literature and updating annotation data resources. Nevertheless, without effective preservation of digitized literature, as well as annotations, this enrichment will not be possible—and today's investments in gathering together, as well as analysing scientific literature will be devalued or lost.We report new functionality included in the InvenioRDM*2 Free and Open Source Software (FOSS) repository software platform, which natively supports IIIF and WADM. InvenioRDM development and maintenance is funded and managed by an international consortium. From late 2023, the InvenioRDM-based ZenodoRDM update*3 will display annotations on biodiversity literature interactively. Significantly, the Biodiversity Literature Repository (BLR) is a Zenodo Community. BLR automatically notifies the Global Biodiversity Information Facility (GBIF) of new taxonomic data and GBIF downloads and integrates this into its service.Moreover, an annotation service based on the WADM-native Mirador-3 FOSS IIIF viewer has now been developed and will enter service with ZenodoRDM. This enables editing of biodiversity annotations from within the repository interface, as well as automated updating of taxonomic information products provided to other major infrastructures such as GBIF.Two aspects of this ZenodoRDM annotation service are presented:dynamic transformation of (preservable) WADM annotations for consumption by contemporary IIIF-compliant applications such as Mirador-3, as well as for Plazi TreatmentBank/GBIF compatibilityauthentication and task organization permitting management of groups of expert contributors performing annotation enrichment tasks directly through the ZenodoRDM graphical user interface (GUI)Workflows for editing existing biodiversity annotations, as well as origination of new annotations, need to be tailored for specific tasks—e.g., unifying geographic collecting location definitions in historic reports—via configurable dialogs for contributors and controlled vocabularies. Selectively populating workflows with annotations according to a task definition is also important to avoid cluttering the editing GUI with non-essential information. Updated annotations are integrated into a new annotation collection upon completion of a task, before updating repository records.Current work on annotation workflows for SSH applications is also reported. The ZenodoRDM biodiversity annotation service implements a generic repository micro-service API, and the implementation of similar services for other repository software platforms is discussed.
15
Valentine, Melissa A., Roger E. Bohn, Amanda L. Pratt, Prachee Jain, Sara J. Singer, and Michael S. Bernstein. "Constructing a Classification Scheme - and its Consequences: A Field Study of Learning to Label Data for Computer Vision in a Hospital Intensive Care Unit." Proceedings of the ACM on Human-Computer Interaction 8, CSCW2 (2024): 1–29. http://dx.doi.org/10.1145/3687029.
Texte intégralRésumé :
Research on data annotation for artificial intelligence (AI) has demonstrated that biases, power, and culture impact the ways that annotators apply labels to data and subsequently affect downstream AI systems. However, annotators can only apply labels that are available to them in the annotation classification scheme. Drawing on a 3-year ethnographic study of an R&D collaboration between medical and AI researchers, we argue that the construction of the classification schema itself -- decisions about what kinds of data can and cannot be collected, what activities can and cannot be detected in the data, what the possible annotation classes ought to be, and the rules by which an item ought to be classified into each class -- dramatically shape the annotation process, and through it, the AI. We draw on Bowker and Star's [9] classification theory to detail how the creation of a training data codebook for a computer vision algorithm in hospital intensive care units (ICUs) evolved from its original, clinically-driven goal of classifying complex clinical activities into a narrower goal of identifying physical objects and simpler activities in the ICU. This work reinforces how trade-offs and decisions made long before annotators begin labeling data are highly consequential to the resulting AI system.
16
Schouten, Gerard, Bas S. H. T. Michielsen, and Barbara Gravendeel. "Data-centric AI approach for automated wildflower monitoring." PLOS ONE 19, no. 9 (2024): e0302958. http://dx.doi.org/10.1371/journal.pone.0302958.
Texte intégralRésumé :
We present the Eindhoven Wildflower Dataset (EWD) as well as a PyTorch object detection model that is able to classify and count wildflowers. EWD, collected over two entire flowering seasons and expert annotated, contains 2,002 top-view images of flowering plants captured ‘in the wild’ in five different landscape types (roadsides, urban green spaces, cropland, weed-rich grassland, marshland). It holds a total of 65,571 annotations for 160 species belonging to 31 different families of flowering plants and serves as a reference dataset for automating wildflower monitoring and object detection in general. To ensure consistent annotations, we define species-specific floral count units and provide extensive annotation guidelines. With a 0.82 mAP (@IoU > 0.50) score the presented baseline model, trained on a balanced subset of EWD, is to the best of our knowledge superior in its class. Our approach empowers automated quantification of wildflower richness and abundance, which helps understanding and assessing natural capital, and encourages the development of standards for AI-based wildflower monitoring. The annotated EWD dataset and the code to train and run the baseline model are publicly available.
17
Lost, Jan, Niklas Tillmans, Sara Merkaj, et al. "NIMG-20. INCORPORATION OF AI-BASED AUTOSEGMENTATION AND CLASSIFICATION INTO NEURORADIOLOGY WORKFLOW: PACS-BASED AI TO BUILD YALE GLIOMA DATASET." Neuro-Oncology 24, Supplement_7 (2022): vii165—vii166. http://dx.doi.org/10.1093/neuonc/noac209.638.
Texte intégralRésumé :
Abstract PURPOSE Translation of AI algorithms into clinical practice is significantly limited by lack of large individual hospital-based datasets with expert annotations. Current methods for generation of annotated imaging data are significantly limited due to inefficient imaging data transfer, complicated annotation software, and time required for experts to generate ground truth information. We incorporated AI tools for auto-segmentation of gliomas into PACS that is used at our institution for reading clinical studies and developed a workflow for annotation of images and development of volumetric segmentations in neuroradiology clinical workflow. Material: 1990 patients from Yale Radiation Oncology Registry (2012-2019) were identified. Segmentations were performed using a UNETR algorithm trained on BRaTS 2021 and an internal dataset of manually segmented tumors. Segmentations were validated by a board-certified neuro-radiologist and natively embedded PyRadiomics in PACS was used for feature extraction. RESULTS In 7 Months (05/2021 - 08/2021, 03/2022 - 05/2022) segmentations and annotations were performed in 835 patients (322 female, 467 male, 46 unknown, mean age 53 yrs). Dataset includes 275 Grade 4 Gliomas (54 Grade 3, 100 Grade 2, 31 Grade 1, 375 unknown). Molecular subtypes include IDH (113 mutated, 498 wildtype, 2 Equivocal, 222 unknown), 1p/19q (87 deleted or co-deleted, 122 intact, 626 unknown), MGMT promotor (182 methylated, 95 partially methylated, 275 unmethylated, 283 unknown), EGFR (76 amplified, 177 not amplified, 582 unknown), ATRX (40 mutated, 157 retained, 638 unknown), Ki-67 (616 known, 219 unknown) and p53 (549 known, 286 unknown). Classification of gliomas between grade 3/4 and grade 1/2, yielded AUC of 0.85. CONCLUSION We developed a method for incorporation of volumetric segmentation, feature extraction, and classification that is easily incorporated into neuroradiology workflow. These tools allowed us to annotate over 100 gliomas per month, thus establishing a proof of concept for rapid development of annotated imaging database for AI applications.
18
Tillmanns, N., J. Lost, S. Merkaj, et al. "P13.05.B INCORPORATION OF AI-BASED AUTOSEGMENTATION AND CLASSIFICATION INTO NEURORADIOLOGY WORKFLOW: PACS-BASED AI TO BUILD YALE GLIOMA DATASET." Neuro-Oncology 25, Supplement_2 (2023): ii101. http://dx.doi.org/10.1093/neuonc/noad137.339.
Texte intégralRésumé :
Abstract BACKGROUND Translation of AI algorithms into clinical practice is significantly limited by lack of large individual hospital-based datasets with expert annotations. Current methods for generation of annotated imaging data are significantly limited due to inefficient imaging data transfer, complicated annotation software, and time required for experts to generate ground truth information. We incorporated AI tools for auto-segmentation of gliomas into PACS that is used at our institution for reading clinical studies and developed a workflow for annotation of images and development of volumetric segmentations in neuroradiology clinical workflow. MATERIAL AND METHODS 1990 patients from Yale Radiation Oncology Registry (2012-2019) were screened. Segmentations were performed using a UNETR algorithm trained on BRaTS 2021 and an internal dataset of manually segmented tumors. AI generated segmentation can be revised and confirmed in PACS. Segmentations were validated by a board-certified neuro-radiologist, after which natively embedded PyRadiomics in PACS was used for direct feature extraction. RESULTS In 7 Months (05/2021 - 08/2021, 03/2022 - 05/2022) segmentations and annotations were performed in 1033 patients (429 female, 604 male, mean age 53 yrs). Dataset includes 595 Grade 4 Gliomas (96 Grade 3, 105 Grade 2, 45 Grade 1, 192 unknown). Molecular subtypes include IDH (129 mutated, 651 wildtype, 253 unknown), 1p/19q (94 deleted or co-deleted, 135 intact, 804 unknown), MGMT promotor (216 methylated, 110 partially methylated, 321 unmethylated, 383 unknown), EGFR (125 amplified, 248 not amplified, 660 unknown), ATRX (41 mutated, 238 retained, 754 unknown), Ki-67 (726 known, 307 unknown) and p53 (639 known, 394 unknown). Classification of gliomas between grade 3/4 and grade 1/2, yielded AUC of 0.85. CONCLUSION We developed a method for incorporation of volumetric segmentation, feature extraction, and classification that is easily incorporated into neuroradiology workflow. These tools allowed us to annotate over 100 gliomas per month, thus establishing a proof of concept for rapid development of annotated imaging database for AI applications.
19
Chandhiramowuli, Srravya, Alex S. Taylor, Sara Heitlinger, and Ding Wang. "Making Data Work Count." Proceedings of the ACM on Human-Computer Interaction 8, CSCW1 (2024): 1–26. http://dx.doi.org/10.1145/3637367.
Texte intégralRésumé :
In this paper, we examine the work of data annotation. Specifically, we focus on the role of counting or quantification in organising annotation work. Based on an ethnographic study of data annotation in two outsourcing centres in India, we observe that counting practices and its associated logics are an integral part of day-to-day annotation activities. In particular, we call attention to the presumption of total countability observed in annotation - the notion that everything, from tasks, datasets and deliverables, to workers, work time, quality and performance, can be managed by applying the logics of counting. To examine this, we draw on sociological and socio-technical scholarship on quantification and develop the lens of a 'regime of counting' that makes explicit the specific counts, practices, actors and structures that underpin the pervasive counting in annotation. We find that within the AI supply chain and data work, counting regimes aid the assertion of authority by the AI clients (also called requesters) over annotation processes, constituting them as reductive, standardised, and homogenous. We illustrate how this has implications for i) how annotation work and workers get valued, ii) the role human discretion plays in annotation, and iii) broader efforts to introduce accountable and more just practices in AI. Through these implications, we illustrate the limits of operating within the logic of total countability. Instead, we argue for a view of counting as partial - located in distinct geographies, shaped by specific interests and accountable in only limited ways. This, we propose, sets the stage for a fundamentally different orientation to counting and what counts in data annotation.
20
Li, Yongqi, Xin Miao, Mayi Xu, and Tieyun Qian. "Strong Empowered and Aligned Weak Mastered Annotation for Weak-to-Strong Generalization." Proceedings of the AAAI Conference on Artificial Intelligence 39, no. 26 (2025): 27437–45. https://doi.org/10.1609/aaai.v39i26.34955.
Texte intégralRésumé :
The super-alignment problem of how humans can effectively supervise super-human AI has garnered increasing attention. Recent research has focused on investigating the weak-to-strong generalization (W2SG) scenario as an analogy for super-alignment. This scenario examines how a pre-trained strong model, supervised by an aligned weak model, can outperform its weak supervisor. Despite good progress, current W2SG methods face two main issues: 1) The annotation quality is limited by the knowledge scope of the weak model; 2) It is risky to position the strong model as the final corrector. To tackle these issues, we propose a ``Strong Empowered and Aligned Weak Mastered'' (SEAM) framework for weak annotations in W2SG. This framework can leverage the vast intrinsic knowledge of the pre-trained strong model to empower the annotation and position the aligned weak model as the annotation master. Specifically, the pre-trained strong model first generates principle fast-and-frugal trees for samples to be annotated, encapsulating rich sample-related knowledge. Then, the aligned weak model picks informative nodes based on the tree's information distribution for final annotations. Experiments on six datasets for preference tasks in W2SG scenarios validate the effectiveness of our proposed method.
21
Kim, Yuna, Ji-Soo Keum, Jie-Hyun Kim, et al. "Real-World Colonoscopy Video Integration to Improve Artificial Intelligence Polyp Detection Performance and Reduce Manual Annotation Labor." Diagnostics 15, no. 7 (2025): 901. https://doi.org/10.3390/diagnostics15070901.
Texte intégralRésumé :
Background/Objectives: Artificial intelligence (AI) integration in colon polyp detection often exhibits high sensitivity but notably low specificity in real-world settings, primarily due to reliance on publicly available datasets alone. To address this limitation, we proposed a semi-automatic annotation method using real colonoscopy videos to enhance AI model performance and reduce manual labeling labor. Methods: An integrated AI model was trained and validated on 86,258 training images and 17,616 validation images. Model 1 utilized only publicly available datasets, while Model 2 additionally incorporated images obtained from real colonoscopy videos of patients through a semi-automatic annotation process, significantly reducing the labeling burden on expert endoscopists. Results: The integrated AI model (Model 2) significantly outperformed the public-dataset-only model (Model 1). At epoch 35, Model 2 achieved a sensitivity of 90.6%, a specificity of 96.0%, an overall accuracy of 94.5%, and an F1 score of 89.9%. All polyps in the test videos were successfully detected, demonstrating considerable enhancement in detection performance compared to the public-dataset-only model. Conclusions: Integrating real-world colonoscopy video data using semi-automatic annotation markedly improved diagnostic accuracy while potentially reducing the need for extensive manual annotation typically performed by expert endoscopists. However, the findings need validation through multicenter external datasets to ensure generalizability.
22
Bartolo, Max, Alastair Roberts, Johannes Welbl, Sebastian Riedel, and Pontus Stenetorp. "Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehension." Transactions of the Association for Computational Linguistics 8 (November 2020): 662–78. http://dx.doi.org/10.1162/tacl_a_00338.
Texte intégralRésumé :
Innovations in annotation methodology have been a catalyst for Reading Comprehension (RC) datasets and models. One recent trend to challenge current RC models is to involve a model in the annotation process: Humans create questions adversarially, such that the model fails to answer them correctly. In this work we investigate this annotation methodology and apply it in three different settings, collecting a total of 36,000 samples with progressively stronger models in the annotation loop. This allows us to explore questions such as the reproducibility of the adversarial effect, transfer from data collected with varying model-in-the-loop strengths, and generalization to data collected without a model. We find that training on adversarially collected samples leads to strong generalization to non-adversarially collected datasets, yet with progressive performance deterioration with increasingly stronger models-in-the-loop. Furthermore, we find that stronger models can still learn from datasets collected with substantially weaker models-in-the-loop. When trained on data collected with a BiDAF model in the loop, RoBERTa achieves 39.9F1 on questions that it cannot answer when trained on SQuAD—only marginally lower than when trained on data collected using RoBERTa itself (41.0F1).
23
Addink, Wouter, Sam Leeflang, and Sharif Islam. "A Simple Recipe for Cooking your AI-assisted Dish to Serve it in the International Digital Specimen Architecture." Biodiversity Information Science and Standards 7 (September 14, 2023): e112678. https://doi.org/10.3897/biss.7.112678.
Texte intégralRésumé :
With the rise of Artificial Intelligence (AI), a large set of new tools and services is emerging that supports specimen data mapping, standards alignment, quality enhancement and enrichment of the data. These tools currently operate in isolation, targeted to individual collections, collection management systems and institutional datasets. To address this challenge, DiSSCo, the Distributed System of Scientific Collections, is developing a new infrastructure for digital specimens, transforming them into actionable information objects. This infrastructure incorporates a framework for annotation and curation that allows the objects to be enriched or enhanced by both experts and machines. This creates the unique possibility to plug-in AI-assisted services that can then leverage digital specimens through this infrastructure, which serves as a harmonised Findable, Accessible, Interoperable and Reusable (FAIR) abstraction layer on top of individual institutional systems or datasets. An early example of such services are the ones developed in the Specimen Data Refinery workflow (Hardisty et al. 2022).The new architecture, DS Arch or Digital Specimen Architecture, is built on the concept of FAIR Digital Objects (FDO) (Islam et al. 2020). All digital specimens and related objects are served with persistent identifiers and machine-readable FDO records with information for machines about the object together with a pointer to its machine-readable type description. The type describes the structure of the object, its attributes and describes allowed operations. The digital specimen type and specimen media type are based on existing Biodiversity Information Standards (TDWG) such as Darwin Core, Access to Biological Collection Data (ABCD) Schema and Audiovisual Core Multimedia Resources Metadata Schema, and include support for annotation operations based on the World Wide Web Consortium (W3C) Annotations Data Model. This enables AI-assisted services registered with DS Arch to autonomously discover digital specimen objects and determine the actions they are authorised to perform. AI-assisted services can facilitate various tasks such as digitisation, extract new information from specimen images, create relations with other objects or standardise data. These operations can be done autonomously, upon user request, or in tandem with expert validation. AI-assisted services registered with DS Arch, can interact in the same way with all digital specimens worldwide when served through DS Arch with their uniform FDO representation, even if the content richness, level of standardisation and scope of the specimen is different. DS Arch has been designed to serve digital specimens for living and preserved specimens, and preserved environmental, earth system and astrogeology samples. With the AI-assisted services, data can be annotated with new data, alternative values, corrections, and with new entity relationships. As a result, the digital specimens become Digital Extended Specimens enabling new science and application (Webster et al. 2021). With the implementation of a sophisticated trust model in DS Arch for community acceptance, these annotations will become part of the data itself and can be made available for inclusion in source systems such as collection management systems and aggregators such as Global Biodiversity Information Facility (GBIF), Geoscience Collections Access Service (GeoCASe) and Catalogue of Life.We aim to demonstrate in the session how AI-assisted services can be registered and used to annotate specimen data. Although the DiSSCo DS Arch is still in development and planned to become operational in 2025, we already have a sandbox environment available in which the concept can be tested and AI-assisted services can be piloted to act on digital specimen data. For testing purposes, the operations on specimens are currently limited to individual specimens and open data, however batch operations will also be possible in the future production environment.
24
Lin, Tai-Pei, Chiou-Ying Yang, Ko-Jiunn Liu, Meng-Yuan Huang, and Yen-Lin Chen. "Immunohistochemical Stain-Aided Annotation Accelerates Machine Learning and Deep Learning Model Development in the Pathologic Diagnosis of Nasopharyngeal Carcinoma." Diagnostics 13, no. 24 (2023): 3685. http://dx.doi.org/10.3390/diagnostics13243685.
Texte intégralRésumé :
Nasopharyngeal carcinoma (NPC) is an epithelial cancer originating in the nasopharynx epithelium. Nevertheless, annotating pathology slides remains a bottleneck in the development of AI-driven pathology models and applications. In the present study, we aim to demonstrate the feasibility of using immunohistochemistry (IHC) for annotation by non-pathologists and to develop an efficient model for distinguishing NPC without the time-consuming involvement of pathologists. For this study, we gathered NPC slides from 251 different patients, comprising hematoxylin and eosin (H&E) slides, pan-cytokeratin (Pan-CK) IHC slides, and Epstein–Barr virus-encoded small RNA (EBER) slides. The annotation of NPC regions in the H&E slides was carried out by a non-pathologist trainee who had access to corresponding Pan-CK IHC slides, both with and without EBER slides. The training process utilized ResNeXt, a deep neural network featuring a residual and inception architecture. In the validation set, NPC exhibited an AUC of 0.896, with a sensitivity of 0.919 and a specificity of 0.878. This study represents a significant breakthrough: the successful application of deep convolutional neural networks to identify NPC without the need for expert pathologist annotations. Our results underscore the potential of laboratory techniques to substantially reduce the workload of pathologists.
25
Tang, Kun, Xu Cao, Zhipeng Cao, et al. "THMA: Tencent HD Map AI System for Creating HD Map Annotations." Proceedings of the AAAI Conference on Artificial Intelligence 37, no. 13 (2023): 15585–93. http://dx.doi.org/10.1609/aaai.v37i13.26848.
Texte intégralRésumé :
Nowadays, autonomous vehicle technology is becoming more and more mature. Critical to progress and safety, high-definition (HD) maps, a type of centimeter-level map collected using a laser sensor, provide accurate descriptions of the surrounding environment. The key challenge of HD map production is efficient, high-quality collection and annotation of large-volume datasets. Due to the demand for high quality, HD map production requires significant manual human effort to create annotations, a very time-consuming and costly process for the map industry. In order to reduce manual annotation burdens, many artificial intelligence (AI) algorithms have been developed to pre-label the HD maps. However, there still exists a large gap between AI algorithms and the traditional manual HD map production pipelines in accuracy and robustness. Furthermore, it is also very resource-costly to build large-scale annotated datasets and advanced machine learning algorithms for AI-based HD map automatic labeling systems. In this paper, we introduce the Tencent HD Map AI (THMA) system, an innovative end-to-end, AI-based, active learning HD map labeling system capable of producing and labeling HD maps with a scale of hundreds of thousands of kilometers. In THMA, we train AI models directly from massive HD map datasets via supervised, self-supervised, and weakly supervised learning to achieve high accuracy and efficiency required by downstream users. THMA has been deployed by the Tencent Map team to provide services to downstream companies and users, serving over 1,000 labeling workers and producing more than 30,000 kilometers of HD map data per day at most. More than 90 percent of the HD map data in Tencent Map is labeled automatically by THMA, accelerating the traditional HD map labeling process by more than ten times.
26
Sklab, Youcef, Hanane Ariouat, Youssef Boujydah, et al. "Towards a Deep Learning-Powered Herbarium Image Analysis Platform." Biodiversity Information Science and Standards 8 (August 28, 2024): e135629. https://doi.org/10.3897/biss.8.135629.
Texte intégralRésumé :
Global digitization efforts have archived millions of specimen scans worldwide in herbarium collections, which are essential for studying plant evolution and biodiversity. ReColNat hosts, at present, over 10 million images. However, analyzing these datasets poses crucial challenges for botanical research. The application of deep learning in biodiversity analyses, particularly in analyzing herbarium scans, has shown promising results across numerous tasks (Ariouat et al. 2023, Ariouat et al. 2024, Groom et al. 2023, Sahraoui et al. 2023).Within the e-Col+project (ANR-21-ESRE-0053), we are developing multiple deep learning models aimed at identifying plant morphological traits. We have developed pipelines and models for cleaning, analyzing, and transforming herbarium images, including models for: i) detecting non-vegetal elements, such as barcodes, envelopes, labels, etc.; ii) detecting plant organs, including leaves, flowers, fruits, etc.; and iii) segmenting to recognize plant parts for image cleaning. We are also developing models for classification tasks related to various morphological traits.To validate these models, improve their generalization, and make them easily usable by end-users, deploying them within a generic platform is crucial. The generic platform called PlantAI, currently under development by the e-Col+ project, should enable easy deployment during development for testing and allow users to load annotations for new traits in order to train a model and add it to the existing catalog. The platform is based on a microservice architecture, allowing users to upload images, create custom datasets, and access various AI models for image analysis.The platform is composed of four main modules, as illustrated in Fig. 1. The first module is the collaborative workspace manager, which allows users to create projects and image datasets and invite other users to collaborate on a project. The second module is the navigation interface and dashboards. This module integrates a search engine using metadata and AI annotations, a navigation interface between projects, datasets, and specimens, as well as dashboards for analysis across datasets, specimens, and AI models.The third module is the dataset manager, which handles metadata and annotations associated with the specimens. These annotations can be produced either by expert users or by AI models. The fourth module is the AI models management module, so that models can be used to generate AI annotations of specimen. During the development lifecycle of an AI model, users can create datasets and annotate them with AI models. These annotations can be in two possible states: validated by experts and non-validated. Users collaborating on a project can indicate errors in the model predictions and leave comments to explain their evaluations. These corrections made by experts can be used to retrain the models and thus improve their performance.This platform, will be highly beneficial for botanists, enhancing the efficiency and effectiveness of biodiversity analyses from herbarium scans. We aim to provide users with a catalog of AI models through this platform and allow them to import their own datasets with their own annotations regarding traits of their choice. Users will be able to select a model from the AI model catalog and train it using their dataset. Ultimately, the model obtained from this training will be automatically deployed to be available for AI annotation. The annotations produced by this model will be automatically available in the filtering and navigation interface, thus allowing for dynamic and automatic integration of the AI annotations into the navigation interface.
27
Zhang, Kai, Ahmad Elalailyi, Luca Perfetti, and Francesco Fassi. "Cost-effective annotation of fisheye images for object detection." International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences XLVIII-2/W8-2024 (December 14, 2024): 491–98. https://doi.org/10.5194/isprs-archives-xlviii-2-w8-2024-491-2024.
Texte intégralRésumé :
Abstract. Nowadays, fisheye image has become commonly used in the 3D reality capturing field. Although AI integration for image recognition has become mature with normal images, providing available annotated dataset and pre-trained models, its application for fisheye images is rarely seen. While the object detection models have generalization ability, dealing with barrel distortion requires specific data for fine-tuning. This paper seeks to acquire prior knowledge from normal image and transfer it to the application that deal with fisheye images. This research is devoted to test the annotation shape that could possibly improve the accuracy when representing the shape of objects. It also seeks a way to prove that the annotation can be converted to fisheye images, resulted into a pre-process, which will facilitate the data preparation process. The tests involve annotations with standard box and quadrilateral polygon, the later turned out to be preserving most of the wanted image content after the conversion. The test result shows that the model trained on converted annotations using quadrilateral polygons, compared to detection model trained on non-converted ones, improves the mean average precision by 8%.
28
Gutierrez Becker, B., E. Giuffrida, M. Mangia, et al. "P069 Artificial intelligence (AI)-filtered Videos for Accelerated Scoring of Colonoscopy Videos in Ulcerative Colitis Clinical Trials." Journal of Crohn's and Colitis 15, Supplement_1 (2021): S173—S174. http://dx.doi.org/10.1093/ecco-jcc/jjab076.198.
Texte intégralRésumé :
Abstract Background Endoscopic assessment is a critical procedure to assess the improvement of mucosa and response to therapy, and therefore a pivotal component of clinical trial endpoints for IBD. Central scoring of endoscopic videos is challenging and time consuming. We evaluated the feasibility of using an Artificial Intelligence (AI) algorithm to automatically produce filtered videos where the non-readable portions of the video are removed, with the aim of accelerating the scoring of endoscopic videos. Methods The AI algorithm was based on a Convolutional Neural Network trained to perform a binary classification task. This task consisted of assigning the frames in a colonoscopy video to one of two classes: “readable” or “unreadable.” The algorithm was trained using annotations performed by two data scientists (BG, FA). The criteria to consider a frame “readable” were: i) the colon walls were within the field of view; ii) contrast and sharpness of the frame were sufficient to visually inspect the mucosa, and iii) no presence of artifacts completely obstructing the visibility of the mucosa. The frames were extracted randomly from 351 colonoscopy videos of the etrolizumab EUCALYPTUS (NCT01336465) Phase II ulcerative colitis clinical trial. Evaluation of the performance of the AI algorithm was performed on colonoscopy videos obtained as part of the etrolizumab HICKORY (NCT02100696) and LAUREL (NCT02165215) Phase III ulcerative colitis clinical trials. Each video was filtered using the AI algorithm, resulting in a shorter video where the sections considered unreadable by the AI algorithm were removed. Each of three annotators (EG, MM and MD) was randomly assigned an equal number of AI-filtered videos and raw videos. The gastroenterologist was tasked to score temporal segments of the video according to the Mayo Clinic Endoscopic Subscore (MCES). Annotations were performed by means of an online annotation platform (Virgo Surgical Video Solutions, Inc). Results We measured the time it took the annotators to score raw and AI-filtered videos. We observed a statistically significant reduction (Mann Whitney U test p-value=0.039) in the median time spent by the annotators scoring raw videos (10.59∓ 0.94 minutes) with respect to the time spent scoring AI-filtered videos (9.51 ∓ 0.92 minutes), with a substantial intra-rater agreement when evaluating highlight and raw videos (Cohen’s kappa 0.92 and 0.55 for experienced and junior gastroenterologists respectively). Conclusion Our analysis shows that AI can be used reliably as an assisting tool to automatically remove non-readable time segments from full colonoscopy videos. The use of our proposed algorithm can lead to reduced annotation times in the task of centrally reading colonoscopy videos.
29
Pennington, Avery, Oliver N. F. King, Win Min Tun, et al. "From Voxels to Viruses: Using Deep Learning and Crowdsourcing to Understand a Virus Factory." Citizen Science: Theory and Practice 9, no. 1 (2024): 37. https://doi.org/10.5334/cstp.739.
Texte intégralRésumé :
Many bioimaging research projects require objects of interest to be identified, located, and then traced to allow quantitative measurement. Depending on the complexity of the system and imaging, instance segmentation is often done manually, and automated approaches still require weeks to months of an individual’s time to acquire the necessary training data for AI models. As such, there is a strong need to develop approaches for instance segmentation that minimize the use of expert annotation while maintaining quality on challenging image analysis problems. Herein, we present our work on a citizen science project we ran called Science Scribbler: Virus Factory on the Zooniverse platform, in which citizen scientists annotated a cryo-electron tomography volume by locating and categorising viruses using point-based annotations instead of manually drawing outlines. One crowdsourcing workflow produced a database of virus locations, and the other workflow produced a set of classifications of those locations. Together, this allowed mask annotation to be generated for training a deep learning–based segmentation model. From this model, segmentations were produced that allowed for measurements such as counts of the viruses by virus class. The application of citizen science–driven crowdsourcing to the generation of instance segmentations of volumetric bioimages is a step towards developing annotation-efficient segmentation workflows for bioimaging data. This approach aligns with the growing interest in citizen science initiatives that combine the collective intelligence of volunteers with AI to tackle complex problems while involving the public with research that is being undertaken in these important areas of science.
30
Qazi, Farheen, Muhammad Naseem, Sonish Aslam, Zainab Attaria, Muhammad Ali Jan, and Syed Salman Junaid. "AnnoVate: Revolutionizing Data Annotation with Automated Labeling Technique." VFAST Transactions on Software Engineering 12, no. 2 (2024): 24–30. http://dx.doi.org/10.21015/vtse.v12i2.1734.
Texte intégralRésumé :
This research introduces AnnoVate, an innovative web application designed to automate the labor-intensive task of object annotation for computer vision applications. Focused on image annotation, the study addresses the escalating demand for data refinement and labeling in the field of artificial intelligence (AI). Leveraging the power of YOLOv8 (You Only Look Once), a high-performance object detection algorithm, AnnoVate minimizes human intervention while achieving an impressive 85% overall accuracy in object detection. The methodology integrates active learning, allowing labelers to selectively prioritize uncertain data during the labeling process. An iterative training approach continuously refines the model, creating a self-improving loop that enhances accuracy over successive loops. The system's flexibility enables users to export labeled datasets for their preferred AI model architectures. AnnoVate not only overcomes the limitations of traditional labeling methods but also establishes a collaborative human-machine interaction paradigm, setting the stage for further advancements in computer vision.
31
Lizzi, Francesca, Abramo Agosti, Francesca Brero, et al. "Quantification of pulmonary involvement in COVID-19 pneumonia by means of a cascade of two U-nets: training and assessment on multiple datasets using different annotation criteria." International Journal of Computer Assisted Radiology and Surgery 17, no. 2 (2021): 229–37. http://dx.doi.org/10.1007/s11548-021-02501-2.
Texte intégralRésumé :
Abstract Purpose This study aims at exploiting artificial intelligence (AI) for the identification, segmentation and quantification of COVID-19 pulmonary lesions. The limited data availability and the annotation quality are relevant factors in training AI-methods. We investigated the effects of using multiple datasets, heterogeneously populated and annotated according to different criteria. Methods We developed an automated analysis pipeline, the LungQuant system, based on a cascade of two U-nets. The first one (U-net$$_1$$ 1 ) is devoted to the identification of the lung parenchyma; the second one (U-net$$_2$$ 2 ) acts on a bounding box enclosing the segmented lungs to identify the areas affected by COVID-19 lesions. Different public datasets were used to train the U-nets and to evaluate their segmentation performances, which have been quantified in terms of the Dice Similarity Coefficients. The accuracy in predicting the CT-Severity Score (CT-SS) of the LungQuant system has been also evaluated. Results Both the volumetric DSC (vDSC) and the accuracy showed a dependency on the annotation quality of the released data samples. On an independent dataset (COVID-19-CT-Seg), both the vDSC and the surface DSC (sDSC) were measured between the masks predicted by LungQuant system and the reference ones. The vDSC (sDSC) values of 0.95±0.01 and 0.66±0.13 (0.95±0.02 and 0.76±0.18, with 5 mm tolerance) were obtained for the segmentation of lungs and COVID-19 lesions, respectively. The system achieved an accuracy of 90% in CT-SS identification on this benchmark dataset. Conclusion We analysed the impact of using data samples with different annotation criteria in training an AI-based quantification system for pulmonary involvement in COVID-19 pneumonia. In terms of vDSC measures, the U-net segmentation strongly depends on the quality of the lesion annotations. Nevertheless, the CT-SS can be accurately predicted on independent test sets, demonstrating the satisfactory generalization ability of the LungQuant.
32
Matuzevičius, Dalius. "A Retrospective Analysis of Automated Image Labeling for Eyewear Detection Using Zero-Shot Object Detectors." Electronics 13, no. 23 (2024): 4763. https://doi.org/10.3390/electronics13234763.
Texte intégralRésumé :
This research presents a retrospective analysis of zero-shot object detectors in automating image labeling for eyeglasses detection. The increasing demand for high-quality annotations in object detection is being met by AI foundation models with open-vocabulary capabilities, reducing the need for labor-intensive manual labeling. There is a notable gap in systematic analyses of foundation models for specialized detection tasks, particularly within the domain of facial accessories. Six state-of-the-art models—Grounding DINO, Detic, OWLViT, OWLv2, YOLO World, and Florence-2—were evaluated across three datasets (FFHQ with custom annotations, CelebAMask-HQ, and Face Synthetics) to assess their effectiveness in zero-shot detection and labeling. Performance metrics, including Average Precision (AP), Average Recall (AR), and Intersection over Union (IoU), were used to benchmark foundation models. The results show that Detic achieved the highest performance scores (AP of 0.97 and AR of 0.98 on FFHQ, with IoU values reaching 0.97), making it highly suitable for automated annotation workflows. Grounding DINO and OWLv2 also showed potential, especially in high-recall scenarios. The results emphasize the importance of prompt engineering. Practical recommendations for using foundation models in specialized dataset annotation are provided.
33
Haunss, Sebastian, Jonas Kuhn, Sebastian Padó, et al. "Integrating Manual and Automatic Annotation for the Creation of Discourse Network Data Sets." Politics and Governance 8, no. 2 (2020): 326–39. http://dx.doi.org/10.17645/pag.v8i2.2591.
Texte intégralRésumé :
This article investigates the integration of machine learning in the political claim annotation workflow with the goal to partially automate the annotation and analysis of large text corpora. It introduces the MARDY annotation environment and presents results from an experiment in which the annotation quality of annotators with and without machine learning based annotation support is compared. The design and setting aim to measure and evaluate: a) annotation speed; b) annotation quality; and c) applicability to the use case of discourse network generation. While the results indicate only slight increases in terms of annotation speed, the authors find a moderate boost in annotation quality. Additionally, with the help of manual annotation of the actors and filtering out of the false positives, the machine learning based annotation suggestions allow the authors to fully recover the core network of the discourse as extracted from the articles annotated during the experiment. This is due to the redundancy which is naturally present in the annotated texts. Thus, assuming a research focus not on the complete network but the network core, an AI-based annotation can provide reliable information about discourse networks with much less human intervention than compared to the traditional manual approach.
34
Edelmers, Edgars, Dzintra Kazoka, Katrina Bolocko, Kaspars Sudars, and Mara Pilmane. "Automatization of CT Annotation: Combining AI Efficiency with Expert Precision." Diagnostics 14, no. 2 (2024): 185. http://dx.doi.org/10.3390/diagnostics14020185.
Texte intégralRésumé :
The integration of artificial intelligence (AI), particularly through machine learning (ML) and deep learning (DL) algorithms, marks a transformative progression in medical imaging diagnostics. This technical note elucidates a novel methodology for semantic segmentation of the vertebral column in CT scans, exemplified by a dataset of 250 patients from Riga East Clinical University Hospital. Our approach centers on the accurate identification and labeling of individual vertebrae, ranging from C1 to the sacrum–coccyx complex. Patient selection was meticulously conducted, ensuring demographic balance in age and sex, and excluding scans with significant vertebral abnormalities to reduce confounding variables. This strategic selection bolstered the representativeness of our sample, thereby enhancing the external validity of our findings. Our workflow streamlined the segmentation process by eliminating the need for volume stitching, aligning seamlessly with the methodology we present. By leveraging AI, we have introduced a semi-automated annotation system that enables initial data labeling even by individuals without medical expertise. This phase is complemented by thorough manual validation against established anatomical standards, significantly reducing the time traditionally required for segmentation. This dual approach not only conserves resources but also expedites project timelines. While this method significantly advances radiological data annotation, it is not devoid of challenges, such as the necessity for manual validation by anatomically skilled personnel and reliance on specialized GPU hardware. Nonetheless, our methodology represents a substantial leap forward in medical data semantic segmentation, highlighting the potential of AI-driven approaches to revolutionize clinical and research practices in radiology.
35
Zubair, Asif, Rich Chapple, Sivaraman Natarajan, et al. "Abstract 456: Jointly leveraging spatial transcriptomics and deep learning models for image annotation achieves better-than-pathologist performance in cell type identification in tumors." Cancer Research 82, no. 12_Supplement (2022): 456. http://dx.doi.org/10.1158/1538-7445.am2022-456.
Texte intégralRésumé :
Abstract For over 100 years, the traditional tools of pathology, such as tissue-marking dyes (e.g. the H&E stain) have been used to study the disorganization and dysfunction of cells within tissues. This has represented a principal diagnostic and prognostic tool in cancer. However, in the last 5 years, new technologies have promised to revolutionize histopathology, with Spatial Transcriptomics technologies allowing us to measure gene expression directly in pathology-stained tissue sections. In parallel with these developments, Artificial Intelligence (AI) applied to histopathology tissue images now approaches pathologist level performance in cell type identification. However, these new technologies still have severe limitations, with Spatial Transcriptomics suffering difficulties distinguishing transcriptionally similar cell types, and AI-based pathology tools often performing poorly on real world out-of-batch test datasets. Thus, century-old techniques still represent standard-of-care in most areas of clinical cancer diagnostics and prognostics. Here, we present a new frontier in digital pathology: describing a conceptually novel computational methodology, based on Bayesian probabilistic modelling, that allows Spatial Transcriptomics data to be leveraged together with the output of deep learning-based AI used to computationally annotate H&E-stained sections of the same tumor. By leveraging cell-type annotations from multiple independent pathologists, we show that this integrated methodology achieves better performance than any given pathologist’s manual tissue annotation in the task of identifying regions of immune cell infiltration in breast cancer, and easily outperforms either technology alone. We also show that on a subset of histopathology slides examined, the methodology can identify regions of clinically relevant immune cell infiltration that were missed entirely by an initial pathologist’s manual annotation. While this use case has clear diagnostic and prognostic value in cancer (e.g. predicting response to immunotherapy), our methodology is generalizable to any type of pathology images and also has broad applications in spatial transcriptomics data analytics, where most applications (such as identifying cell-cell interactions) rely on correct cell type annotations having been established a priori. We anticipate that this work will spur many follow-up studies, including new computational innovations building on the approach. The work sets the stage for better-than-pathologist performance in other cell-type annotation tasks, with relevant applications in diagnostics and prognostics across almost all cancers. Citation Format: Asif Zubair, Rich Chapple, Sivaraman Natarajan, William C. Wright, Min Pan, Hyeong-Min Lee, Heather Tillman, John Easton, Paul Geeleher. Jointly leveraging spatial transcriptomics and deep learning models for image annotation achieves better-than-pathologist performance in cell type identification in tumors [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2022; 2022 Apr 8-13. Philadelphia (PA): AACR; Cancer Res 2022;82(12_Suppl):Abstract nr 456.
36
Santacroce, G., P. Meseguer, I. Zammarchi, et al. "P406 A novel active learning-based digital pathology protocol annotation for histologic assessment in Ulcerative Colitis using PICaSSO Histologic Remission Index (PHRI)." Journal of Crohn's and Colitis 18, Supplement_1 (2024): i843—i844. http://dx.doi.org/10.1093/ecco-jcc/jjad212.0536.
Texte intégralRésumé :
Abstract Background Histologic remission (HR) is a critical treatment target in Ulcerative Colitis (UC). Among several scoring systems, the PICaSSO Histologic Remission Index (PHRI) simplifies HR assessment by evaluating the presence of neutrophils in the bowel tissue. Our artificial intelligence (AI) system built upon PHRI showed remarkable accuracy in HR assessment. PHRI assess neutrophils in four different regions of interest, so segmentation of these compartments is crucial to predict PHRI automatically. However, creating labelled histopathological datasets to train fully-supervised segmentation models takes time and effort. Hence, this study explores the impact of an active learning (AL) algorithm on enhancing image segmentation to alleviate the burden of detailed pathologists’ annotation and to standardise protocol annotation. Methods Biopsy samples from an international real-life prospective UC study were digitised into whole slide images (WSI). Initial annotations of superficial epithelium, lumen and epithelium of crypts and lamina propria for 33 WSI were employed to train a U-Net segmentation model at baseline. An AL framework was employed to iteratively select and annotate 15 unannotated images while selecting those with the highest uncertainty. Uncertainty was calculated using Least confidence sampling, Margin Sampling, and Shannon Entropy. The most informative samples, based on the average of the three uncertainty measure, were selected in consecutive batches of 5 images, and pathologists were enlisted in a human-in-the-loop process to refine annotations. Subsequently, the segmentation model was retrained by incorporating the newly refined annotated samples, and its performance was assessed using a fully annotated test set of 19 WSI. Results Following the baseline model training, the model’s segmentation performance assessed by the Dice score and Intersection over Union (IoU) was 0.622 and 0.386, respectively (see Table 1). Applying the AL algorithm with newly annotated images notably improved model performance, especially with 10 images (Dice=0.651, IoU=0.415). However, training the model with an additional 5 newly annotated images, which exhibited lower uncertainty, did not yield further improvement (Dice = 0.651, IoU = 0.415). Thus, the 10 annotated WSI demonstrating more uncertainty resulted crucial for the AL framework. Conclusion This novel AL-based iterative framework exhibits promise in standardising digital tissue annotation by our PHRI-based AI model. It offers a novel approach for both clinical trial and clinical practice, aiming to alleviate the burden of WSI labelling and reduce the bias of annotation, thereby improving histological assessment in UC.
37
Thakur, Siddhesh, Shahriar Faghani, Mana Moassefi, et al. "TMIC-60. BRATS-PATH: ASSESSING HETEROGENEOUS HISTOPATHOLOGIC REGIONS IN GLIOBLASTOMA." Neuro-Oncology 26, Supplement_8 (2024): viii312. http://dx.doi.org/10.1093/neuonc/noae165.1238.
Texte intégralRésumé :
Abstract Glioblastoma, the most common malignant primary adult brain tumor, poses significant diagnostic and treatment challenges due to its heterogeneous molecular and micro-environmental profiles. To this end, we organize the BraTS-Path challenge to provide a public benchmarking environment and a comprehensive dataset to develop and validate AI models for identifying distinct histopathologic glioblastoma sub-regions in H&E-stained digitized tissue sections. We identified 188 multi-institutional diagnostic slides of glioblastoma (IDH-wt, Gr.4) cases, from the TCGA-GBM and TCGA-LGG data collections, following their reclassification according to the 2021 WHO classification criteria. Sub-regions were selected according to distinctive morphology of histopathologic features and included aggressive tumor biology and areas consistent with potential treatment effect. Selected sub-region annotations included cellular tumor, geographic necrosis, cortical infiltration, pseudopalisading necrosis, microvascular proliferation, white matter penetration, regions dense with macrophages, leptomeningeal infiltration, and presence of lymphocytes. We obtained 107,340 patches of size 512x512 from the 9 sub-regions. A global network of board-certified expert neuropathologists defined and followed a systematic annotation protocol based on clinical definitions and only delineated sub-regions with high confidence, thus ensuring high-quality standardized data. Each tissue section was assigned to an annotator-approver pair, with the annotator delineating sub-regions and the approver ensuring the consistency of the annotations. By crowdsourcing annotations, the BraTS-Path challenge harnesses the collective expertise of clinical neuropathologists and fosters a collaborative environment to advance the neuro-oncology field. The anticipated developed algorithms are expected to integrate state-of-the-art computational methods, achieving high accuracy in identifying diverse histopathologic features and advancing clinical decision-making processes. The BraTS-Path challenge aims to bridge the gap between research and clinical practice by promoting the development of AI-driven tools for precise tumor characterization. This collaborative effort can significantly enhance our understanding of glioblastoma, improve diagnostic accuracy, and inform treatment strategies, thereby contributing to better patient outcomes.
38
Nasim, Md, Xinghang Zhang, Anter El-Azab, and Yexiang Xue. "End-to-End Phase Field Model Discovery Combining Experimentation, Crowdsourcing, Simulation and Learning." Proceedings of the AAAI Conference on Artificial Intelligence 38, no. 21 (2024): 23005–11. http://dx.doi.org/10.1609/aaai.v38i21.30342.
Texte intégralRésumé :
The availability of tera-byte scale experiment data calls for AI driven approaches which automatically discover scientific models from data. Nonetheless, significant challenges present in AI-driven scientific discovery: (i) The annotation of large scale datasets requires fundamental re-thinking in developing scalable crowdsourcing tools. (ii) The learning of scientific models from data calls for innovations beyond black-box neural nets. (iii) Novel visualization & diagnosis tools are needed for the collaboration of experimental and theoretical physicists, and computer scientists. We present Phase-Field-Lab platform for end-to-end phase field model discovery, which automatically discovers phase field physics models from experiment data, integrating experimentation, crowdsourcing, simulation and learning. Phase-Field-Lab combines (i) a streamlined annotation tool which reduces the annotation time (by ~50-75%), while increasing annotation accuracy compared to baseline; (ii) an end-to-end neural model which automatically learns phase field models from data by embedding phase field simulation and existing domain knowledge into learning; and (iii) novel interfaces and visualizations to integrate our platform into the scientific discovery cycle of domain scientists. Our platform is deployed in the analysis of nano-structure evolution in materials under extreme conditions (high temperature and irradiation). Our approach reveals new properties of nano-void defects, which otherwise cannot be detected via manual analysis.
39
Ponnusamy, Pragaash, Alireza Roshan Ghias, Chenlei Guo, and Ruhi Sarikaya. "Feedback-Based Self-Learning in Large-Scale Conversational AI Agents." Proceedings of the AAAI Conference on Artificial Intelligence 34, no. 08 (2020): 13180–87. http://dx.doi.org/10.1609/aaai.v34i08.7022.
Texte intégralRésumé :
Today, most of the large-scale conversational AI agents such as Alexa, Siri, or Google Assistant are built using manually annotated data to train the different components of the system including Automatic Speech Recognition (ASR), Natural Language Understanding (NLU) and Entity Resolution (ER). Typically, the accuracy of the machine learning models in these components are improved by manually transcribing and annotating data. As the scope of these systems increase to cover more scenarios and domains, manual annotation to improve the accuracy of these components becomes prohibitively costly and time consuming. In this paper, we propose a system that leverages customer/system interaction feedback signals to automate learning without any manual annotation. Users of these systems tend to modify a previous query in hopes of fixing an error in the previous turn to get the right results. These reformulations, which are often preceded by defective experiences caused by either errors in ASR, NLU, ER or the application. In some cases, users may not properly formulate their requests (e.g. providing partial title of a song), but gleaning across a wider pool of users and sessions reveals the underlying recurrent patterns. Our proposed self-learning system automatically detects the errors, generate reformulations and deploys fixes to the runtime system to correct different types of errors occurring in different components of the system. In particular, we propose leveraging an absorbing Markov Chain model as a collaborative filtering mechanism in a novel attempt to mine these patterns. We show that our approach is highly scalable, and able to learn reformulations that reduce Alexa-user errors by pooling anonymized data across millions of customers. The proposed self-learning system achieves a win-loss ratio of 11.8 and effectively reduces the defect rate by more than 30% on utterance level reformulations in our production A/B tests. To the best of our knowledge, this is the first self-learning large-scale conversational AI system in production.
40
Ponnusamy, Pragaash, Alireza Ghias, Yi Yi, Benjamin Yao, Chenlei Guo, and Ruhi Sarikaya. "Feedback-Based Self-Learning in Large-Scale Conversational AI Agents." AI Magazine 42, no. 4 (2022): 43–56. http://dx.doi.org/10.1609/aimag.v42i4.15102.
Texte intégralRésumé :
Today, most of the large-scale conversational AI agents such as Alexa, Siri, or Google Assistant are built using manually annotated data to train the different components of the system including automatic speech recognition (ASR), natural language understanding (NLU), and entity resolution (ER). Typically, the accuracy of the machine learning models in these components are improved by manually transcribing and annotating data. As the scope of these systems increase to cover more scenarios and domains, manual annotation to improve the accuracy of these components becomes prohibitively costly and time con-suming. In this paper, we propose a system that leverages customer/system interaction feedback signals to automate learning without any manual annotation. Users of these systems tend to modify a previous query in hopes of fixing an error in the previous turn to get the right results. These reformulations, which are often preceded by defective experiences caused by either errors in ASR, NLU, ER, or the application. In some cases, users may not properly formulate their requests (e.g., providing partial title of a song), but gleaning across a wider pool of users and sessions reveals the underlying recurrent patterns. Our proposed self-learning system automatically detects the errors, generates reformulations, and deploys fixes to the runtime system to correct different types of errors occurring in different components of the system. In particular, we propose leveraging an absorbing Markov Chain model as a collaborative filtering mechanism in a novel attempt to mine these patterns, and coupling it with a guardrail rewrite selection mechanism that reactively evaluates these fixes using feedback friction data. We show that our approach is highly scalable, and able to learn reformulations that reduce Alexa-user errors by pooling anonymized data across millions of customers. The proposed self-learning system achieves a win-loss ratio of 11.8 and effectively reduces the defect rate by more than 30 percent on utterance level reformulations in our production A/B tests. To the best of our knowledge, this is the first self-learning large-scale conversational AI system in production.
41
Ponnusamy, Pragaash, Alireza Ghias, Yi Yi, Benjamin Yao, Chenlei Guo, and Ruhi Sarikaya. "Feedback-Based Self-Learning in Large-Scale Conversational AI Agents." AI Magazine 42, no. 4 (2022): 43–56. http://dx.doi.org/10.1609/aaai.12025.
Texte intégralRésumé :
Today, most of the large-scale conversational AI agents such as Alexa, Siri, or Google Assistant are built using manually annotated data to train the different components of the system including automatic speech recognition (ASR), natural language understanding (NLU), and entity resolution (ER). Typically, the accuracy of the machine learning models in these components are improved by manually transcribing and annotating data. As the scope of these systems increase to cover more scenarios and domains, manual annotation to improve the accuracy of these components becomes prohibitively costly and time con-suming. In this paper, we propose a system that leverages customer/system interaction feedback signals to automate learning without any manual annotation. Users of these systems tend to modify a previous query in hopes of fixing an error in the previous turn to get the right results. These reformulations, which are often preceded by defective experiences caused by either errors in ASR, NLU, ER, or the application. In some cases, users may not properly formulate their requests (e.g., providing partial title of a song), but gleaning across a wider pool of users and sessions reveals the underlying recurrent patterns. Our proposed self-learning system automatically detects the errors, generates reformulations, and deploys fixes to the runtime system to correct different types of errors occurring in different components of the system. In particular, we propose leveraging an absorbing Markov Chain model as a collaborative filtering mechanism in a novel attempt to mine these patterns, and coupling it with a guardrail rewrite selection mechanism that reactively evaluates these fixes using feedback friction data. We show that our approach is highly scalable, and able to learn reformulations that reduce Alexa-user errors by pooling anonymized data across millions of customers. The proposed self-learning system achieves a win-loss ratio of 11.8 and effectively reduces the defect rate by more than 30 percent on utterance level reformulations in our production A/B tests. To the best of our knowledge, this is the first self-learning large-scale conversational AI system in production.
42
Ibodullayev, Sardor Nasriddin o'g'li, and Yaxyobek Sobirjon o'g'li Sodiqjonov. "ARTIFICIAL INTELLIGENCE AND ITS METHODS IN VIRTUAL REALITY." "Yosh mutaxassislar" ilmiy-amaliy jurnali. 2023-06, no. 1 (2023): 57–64. https://doi.org/10.5281/zenodo.8043452.
Texte intégralRésumé :
<strong>Annotation</strong>: Artificial intelligence (AI) has the potential to revolutionize the way we experience virtual reality (VR). By using machine learning algorithms and other AI-powered techniques, developers can create VR experiences that are more immersive, interactive, and personalized. AI can be used to create intelligent avatars that can interact with users in real-time, recognize objects in VR environments, create more sophisticated chatbots, and detect and respond to user emotions in real-time. As AI technology continues to evolve, we can expect to see even more innovative and exciting applications of AI in VR.
43
Bellomo, Tiffany R., Guillaume Goudot, Srihari K. Lella, et al. "Feasibility of Encord Artificial Intelligence Annotation of Arterial Duplex Ultrasound Images." Diagnostics 14, no. 1 (2023): 46. http://dx.doi.org/10.3390/diagnostics14010046.
Texte intégralRésumé :
DUS measurements for popliteal artery aneurysms (PAAs) specifically can be time-consuming, error-prone, and operator-dependent. To eliminate this subjectivity and provide efficient segmentation, we applied artificial intelligence (AI) to accurately delineate inner and outer lumen on DUS. DUS images were selected from a cohort of patients with PAAs from a multi-institutional platform. Encord is an easy-to-use, readily available online AI platform that was used to segment both the inner lumen and outer lumen of the PAA on DUS images. A model trained on 20 images and tested on 80 images had a mean Average Precision of 0.85 for the outer polygon and 0.23 for the inner polygon. The outer polygon had a higher recall score than precision score at 0.90 and 0.85, respectively. The inner polygon had a score of 0.25 for both precision and recall. The outer polygon false-negative rate was the lowest in images with the least amount of blur. This study demonstrates the feasibility of using the widely available Encord AI platform to identify standard features of PAAs that are critical for operative decision making.
44
Huang, Xiaoyuan, Silvia Mirri, and Su-Kit Tang. "Macao-ebird: A Curated Dataset for Artificial-Intelligence-Powered Bird Surveillance and Conservation in Macao." Data 10, no. 6 (2025): 84. https://doi.org/10.3390/data10060084.
Texte intégralRésumé :
Artificial intelligence (AI) currently exhibits considerable potential within the realm of biodiversity conservation. However, high-quality regionally customized datasets remain scarce, particularly within urban environments. The existing large-scale bird image datasets often lack a dedicated focus on endangered species endemic to specific geographic regions, as well as a nuanced consideration of the complex interplay between urban and natural environmental contexts. Therefore, this paper introduces Macao-ebird, a novel dataset designed to advance AI-driven recognition and conservation of endangered bird species in Macao. The dataset comprises two subsets: (1) Macao-ebird-cls, a classification dataset with 7341 images covering 24 bird species, emphasizing endangered and vulnerable species native to Macao; and (2) Macao-ebird-det, an object detection dataset generated through AI-agent-assisted labeling using grounding DETR with improved denoising anchor boxes (DINO), significantly reducing manual annotation effort while maintaining high-quality bounding-box annotations. We validate the dataset’s utility through baseline experiments with the You Only Look Once (YOLO) v8–v12 series, achieving a mean average precision (mAP50) of up to 0.984. Macao-ebird addresses critical gaps in the existing datasets by focusing on region-specific endangered species and complex urban–natural environments, providing a benchmark for AI applications in avian conservation.
45
Xiao, Yi, Xuefei Lin, Tie Ji, Jinhao Qiao, Bowen Ma, and Hao Gong. "AI-Assisted Design: Intelligent Generation of Dong Paper-Cut Patterns." Electronics 14, no. 9 (2025): 1804. https://doi.org/10.3390/electronics14091804.
Texte intégralRésumé :
Intelligent generation technology has been widely applied in the field of design, serving as an essential tool for many designers. This study focuses on the paper-cut patterns of Qin Naishiqing, an inheritor of Dong paper-cutting intangible cultural heritage, and explores the AI-assisted generation of Dong paper-cut patterns under designer–AI collaborative control. It proposes a new role for designers in human–AI collaborative design—the “designer-in-the-loop” model. From the perspective of dataset annotation, designers conduct visual feature analysis, Shape Factor Extraction, and Semantic Factor extraction of paper-cut patterns, actively participating in dataset construction, annotation, and collaborative control methods, including using localized LoRA for detail enhancement and creating controllable collaborative modes through contour lines and structural lines, evaluation of generated results, and iterative optimization. The experimental results demonstrate that the intelligent generation approach under the “designer-in-the-loop” model, combined with designer–AI controllable collaboration, effectively enhances the generation of specific-style Dong paper-cut patterns with limited sample data. This study provides new insights and practical methodologies for the intelligent generation of other stylistic patterns.
46
Bondiau, P., S. Bolle, A. Escande, et al. "PD-0330 AI-based OAR annotation for pediatric brain radiotherapy planning." Radiotherapy and Oncology 170 (May 2022): S293—S295. http://dx.doi.org/10.1016/s0167-8140(22)02823-7.
Texte intégral
47
Radeta, Marko, Ruben Freitas, Claudio Rodrigues, et al. "Man and the Machine: Effects of AI-assisted Human Labeling on Interactive Annotation of Real-Time Video Streams." ACM Transactions on Interactive Intelligent Systems, February 29, 2024. http://dx.doi.org/10.1145/3649457.
Texte intégralRésumé :
AI-assisted interactive annotation is a powerful way to facilitate data annotation – a prerequisite for constructing robust AI models. While AI-assisted interactive annotation has been extensively studied in static settings, less is known about its usage in dynamic scenarios where the annotators operate under time and cognitive constraints, e.g., while detecting suspicious or dangerous activities from real-time surveillance feeds. Understanding how AI can assist annotators in these tasks and facilitate consistent annotation is paramount to ensure high performance for AI models trained on these data. We address this gap in interactive machine learning (IML) research, contributing an extensive investigation of the benefits, limitations, and challenges of AI-assisted annotation in dynamic application use cases. We address both the effects of AI on annotators and the effects of (AI) annotations on the performance of AI models trained on annotated data in real-time video annotations. We conduct extensive experiments that compare annotation performance at two annotator levels (expert and non-expert) and two interactive labelling techniques (with and without AI-assistance). In a controlled study with N = 34 annotators and a follow up study with 51963 images and their annotation labels being input to the AI model, we demonstrate that the benefits of AI-assisted models are greatest for non-expert users and for cases where targets are only partially or briefly visible. The expert users tend to outperform or achieve similar performance as AI model. Labels combining AI and expert annotations result in the best overall performance as the AI reduces overflow and latency in the expert annotations. We derive guidelines for the use of AI-assisted human annotation in real-time dynamic use cases.
48
Krenzer, Adrian, Kevin Makowski, Amar Hekalo, et al. "Fast machine learning annotation in the medical domain: a semi-automated video annotation tool for gastroenterologists." BioMedical Engineering OnLine 21, no. 1 (2022). http://dx.doi.org/10.1186/s12938-022-01001-x.
Texte intégralRésumé :
Abstract Background Machine learning, especially deep learning, is becoming more and more relevant in research and development in the medical domain. For all the supervised deep learning applications, data is the most critical factor in securing successful implementation and sustaining the progress of the machine learning model. Especially gastroenterological data, which often involves endoscopic videos, are cumbersome to annotate. Domain experts are needed to interpret and annotate the videos. To support those domain experts, we generated a framework. With this framework, instead of annotating every frame in the video sequence, experts are just performing key annotations at the beginning and the end of sequences with pathologies, e.g., visible polyps. Subsequently, non-expert annotators supported by machine learning add the missing annotations for the frames in-between. Methods In our framework, an expert reviews the video and annotates a few video frames to verify the object’s annotations for the non-expert. In a second step, a non-expert has visual confirmation of the given object and can annotate all following and preceding frames with AI assistance. After the expert has finished, relevant frames will be selected and passed on to an AI model. This information allows the AI model to detect and mark the desired object on all following and preceding frames with an annotation. Therefore, the non-expert can adjust and modify the AI predictions and export the results, which can then be used to train the AI model. Results Using this framework, we were able to reduce workload of domain experts on average by a factor of 20 on our data. This is primarily due to the structure of the framework, which is designed to minimize the workload of the domain expert. Pairing this framework with a state-of-the-art semi-automated AI model enhances the annotation speed further. Through a prospective study with 10 participants, we show that semi-automated annotation using our tool doubles the annotation speed of non-expert annotators compared to a well-known state-of-the-art annotation tool. Conclusion In summary, we introduce a framework for fast expert annotation for gastroenterologists, which reduces the workload of the domain expert considerably while maintaining a very high annotation quality. The framework incorporates a semi-automated annotation system utilizing trained object detection models. The software and framework are open-source.
49
Rao, Anuradha. "A Radiologist's Perspective of Medical Annotations for AI Programs: The Entire Journey from Its Planning to Execution, Challenges Faced." Indian Journal of Radiology and Imaging, December 11, 2024. https://doi.org/10.1055/s-0044-1800860.
Texte intégralRésumé :
AbstractArtificial intelligence (AI) in radiology and medical science is finding increasing applications with annotations being an integral part of AI development. While annotation may be perceived as passive work of labeling a certain anatomy, the radiologist plays a more important role in this task apart from marking the structures needed. Apart from annotation, more important aspect of their role is planning the anatomies/pathologies needed, type of annotations to be done, choice of the annotation tool, training the annotators, planning the duration of annotation, etc. A close interaction with the technical team is a key factor in the success of the annotations. The quality check of both the internally and externally annotated data, creating a team of good annotators, training them, and periodically reviewing the quality of data become an integral part of their work. Documentation related to the annotation work is another important area where the clinician plays an integral role to comply with the Food and Drug Administration requirements, focused on a clinically explainable and validated AI algorithms. Thus, the clinician becomes an integral part in the ideation, design, implementation/execution of annotations, and its quality control. This article summarizes the experiences gained during planning and executing the annotations for multiple annotation projects involving various imaging modalities with different pathologies.
50
van der Wal, Douwe, Iny Jhun, Israa Laklouk, et al. "Biological data annotation via a human-augmenting AI-based labeling system." npj Digital Medicine 4, no. 1 (2021). http://dx.doi.org/10.1038/s41746-021-00520-6.
Texte intégralRésumé :
AbstractBiology has become a prime area for the deployment of deep learning and artificial intelligence (AI), enabled largely by the massive data sets that the field can generate. Key to most AI tasks is the availability of a sufficiently large, labeled data set with which to train AI models. In the context of microscopy, it is easy to generate image data sets containing millions of cells and structures. However, it is challenging to obtain large-scale high-quality annotations for AI models. Here, we present HALS (Human-Augmenting Labeling System), a human-in-the-loop data labeling AI, which begins uninitialized and learns annotations from a human, in real-time. Using a multi-part AI composed of three deep learning models, HALS learns from just a few examples and immediately decreases the workload of the annotator, while increasing the quality of their annotations. Using a highly repetitive use-case—annotating cell types—and running experiments with seven pathologists—experts at the microscopic analysis of biological specimens—we demonstrate a manual work reduction of 90.60%, and an average data-quality boost of 4.34%, measured across four use-cases and two tissue stain types.