Academic literature on the topic 'Appropriate benchmarks for crowd sourcing'

AbstractAdvancement of understanding in paleontology and biology has always been hindered by difficulty in accessing comparative data. With current and burgeoning technology, the severity of this hindrance can be substantially reduced. Researchers and museum personnel generating three-dimensional (3-D) digital models of museum specimens can archive them using internet repositories that can then be explored and utilized by other researchers and private individuals without a museum trip. We focus on MorphoSource, the largest web archive for 3-D museum data at present. We describe the site, how to use it most effectively in its current form, and best practices for file formats and metadata inclusion to aid the growing community wishing to utilize it for distributing 3-D digital data. The potential rewards of successfully crowd sourcing the digitization of museum collections from the research community are great, as it should ensure rapid availability of the most important datasets. Challenges include long-term governance (i.e., maintaining site functionality, supporting large amounts of digital storage, and monitoring/updating file to prevent bit rot, which is the slow and random corruption of electronic data over time, and data format obsolescence, which is the problem of data becoming unreadable or ineffective because of the loss of functional software necessary for access), and utilization by the community (i.e., detecting and minimizing user error in creating data records, incentivizing data sharing by researchers and institutions alike, and protecting stakeholder rights to data, while maximizing accessibility and discoverability).MorphoSource serves as a proof-of-concept of how these kinds of challenges can be met. Accordingly, it is generally recognized as the most appropriate repository for large, raw datasets of fossil organisms and/or comparative samples. Its existence has begun to transform data transparency standards because journal reviewers, editors, and grant officers now often suggest or require that 3-D data be made available through this site.

Purpose – The purpose of this paper is to conceptualize how digital humanities (DH) projects can be integrated into instructional services programs in libraries. The paper draws on three digital projects from the New York Public Library (NYPL) and explores how librarians can creatively utilize these resources to teach new digital literacy skills such as data analysis and data management. For patrons, they can learn about the content of these crowd-sourcing projects as well. By integrating DH projects into library instruction, the possibilities and opportunities to expand and explore new research and teaching areas are timely and relevant. Design/methodology/approach – The approach of this paper is to explore NYPL’s three digital projects and underscore how they can be integrated into instructional services: “What’s On the Menu,” “Direct Me NYC” and “Map Warper” all offer strengths and limitations but they serve as paradigms to explore how digital resources can serve multipurpose use: they are databases, digital repositories and digital libraries but they can also serve as instructional service tools. Findings – The paper conceptualizes how three DH projects can serve as teaching opportunities for instructional services, particularly teaching digital literacy skills. By exploring the content of each digital project, the paper suggests that users can develop traditional information literacy skills but also digital literacy skills. In addition, as crowdsourcing projects, the Library also benefits from this engagement since users are adding transcriptions or rectified maps to the Library’s site. Patrons develop visual literacy skills as well. The paper addresses how librarians can meet the needs of the scholarly community through these new digital resources. While the paper only addresses the possibilities of these integrations, these ideas can be considered and implemented in any library. Practical implications – The paper addresses positive outcomes with these digital resources to be used for library instructional services. Based on these projects, the paper recommends that DH projects can be integrated into such instructions to introduce new content and digital skills if appropriate. Although, there are limitations with these digital resources, it is possible to maximize their usage if they are used in a different and creative way. It is possible for DH projects to be more than just digital projects but to act as a tool of digital literacy instruction. Librarians must play a creative role to address this gap. However, another limitation is that librarians themselves are “new” to these resources and may find it challenging to understand the importance of DH projects in scholarly research. Originality/value – This paper introduces DH projects produced in a public research library and explores how librarians can use these digital projects to teach patrons on how to analyze data, maps and other content to develop digital literacy skills. The paper conceptualizes the significant roles that these DH projects and librarians can play as critical mediators to introducing and fostering digital literacy in the twenty-first century. The paper can serve as an interest to academic and public libraries with large research collections and digital projects. By offering new innovative ideas of integrating DH into instructional services, the paper addresses how DH projects teaching tools can support specific digital skills such as visual literacy and data analysis.

Online crowdsourcing platforms have proliferated over the last few years and cover a number of important domains, these platforms include from worker-task platforms such Amazon Mechanical Turk, worker-for hire platforms such as TaskRabbit to specialized platforms with specific tasks such as ridesharing like Uber, Lyft, Ola etc. An increasing proportion of human workforce will be employed by these platforms in the near future. The crowdsourcing community has done yeoman’s work in designing effective algorithms for various key components, such as incentive design, task assignment and quality control. Given the increasing importance of these crowdsourcing platforms, it is now time to design mechanisms so that it is easier to evaluate the effectiveness of these platforms. Specifically, we advocate developing benchmarks for crowdsourcing research. Benchmarks often identify important issues for the community to focus and improve upon. This has played a key role in the development of research domains as diverse as databases and deep learning. We believe that developing appropriate benchmarks for crowdsourcing will ignite further innovations. However, crowdsourcing – and future of work, in general – is a very diverse field that makes developing benchmarks much more challenging. Substantial effort is needed that spans across developing benchmarks for datasets, metrics, algorithms, platforms and so on. In this article, we initiate some discussion into this important problem and issue a call-to-arms for the community to work on this important initiative.

Precision medicine aims to treat diseases with special consideration for the individual biological variability. Novel biomarkers (BM) are needed to predict therapeutic responses and to allow for the selection of suitable patients for treatment with certain drugs. However, the identification and validation of appropriate BMs is challenging. Close col-laboration between different partners seems to be a key success factor. While the importance of partnerships and larger, well-established consortia in BM discovery such as the pharmaceutical industry and academic institutions is well un-derstood and has been investigated in the past, the use of open-innovation models, also known as ‘crowd sourcing for biomarkers’, is still in its infancy. Crowd sourcing comprises of a —usually via internet— request for problem solution to an open group of users in a kind of an ‘open call’. The community (crowd) is asked to provide solutions. Since the application of the crowd sourcing method offers the possibility to collect as many as possible novel ideas from a broad community with different expertise, this approach is particularly promising for BM development. In this article we de-scribe the first examples of open-innovation models, such as the ‘grants for targets’ (G4T) and biomarkers initiative ‘InnoCentive’ (innovation/incentive) platform. They may be a fruitful basis for collaborative BM development in the future.