Journal articles on the topic 'Formative assessment framework. Assessment tools. Ames'

The Formative Assessment in Science project was funded by HEFCE as part of the Fund for the Development of Teaching and Learning Programme. Its work was completed in March 2006. The project was centred on a strong collaboration between the Open University and Sheffield Hallam University, but also involved ~20 other universities in action learning activities aimed at improving formative assessment. The project partners used a conceptually and empirically based framework of conditions which, if met, lead to assessment that drives learning. The projects were diverse and the many positive assessment changes achieved demonstrate that the approach of a framework based analysis and careful evaluation can be successful in improving the student experience. I will summarise some of the main conclusions of the project. One concerns the creation of written feedback that promotes learning rather than merely justifying marks. A second covers the effective use of peer assessment. Finally, I will outline the tools that the project has generated. These will remain available for others to undertake similar reform activities.

Purpose The purpose of this paper is to build an assessment-centred blended learning (BL) framework to assess learners, to analyse and to evaluate the impact of the technology support in the form of formative assessment in students’ positive learning. Design/methodology/approach This research proposed an assessment-centred BL framework at the course level to support formative assessment in students by designing a variety of online learning activities combined with e-assessment tools of learning management system (LMS) to analyse and evaluate the impact of the technology application in the form of formative assessment student learning initiative. The author has tested this model in five years with more than 200 courses. Findings Experimental results have shown that formative assessment evaluation form is more efficient when supported by technology such as LMS. Originality/value This research proposed an assessment-centred BL framework at the course level by using LMS tools combined with traditional teaching.

Purpose – The purpose of this paper is to present the evaluation results of a novel performance measurement framework, which has been designed for the particular needs of small- and medium-sized enterprises (SMEs). Prior research indicated that performance measurement is not much engaged in SMEs. The proposed framework aims to address this by creating bottom-up measurement systems based on service delivery processes. The paper presents the framework as well as its tools and gives insights into both the evaluation and its results. Design/methodology/approach – The framework and its tools have been evaluated in a first iteration of prototype implementation according to design science principles. Methodological triangulation was applied engaging participant observation, questionnaires, and formative evaluation. In the course of the evaluation the framework was assessed from a quality, user, and bird’s eye perspective. Findings – The evaluation results show that the application of the Technology Acceptance Model and goal setting theory did contribute to develop a performance measurement system (PMS) that is characterized by usefulness and ease of use. Deficiencies could be noted with regard to renowned quality criteria for PMS, especially concerning balance. First lessons learned are outlined. Originality/value – The main contribution is the theory-driven development of the performance measurement framework and its corresponding tools. The application of this approach resulted in accepted tools and improvements of the motivational power of performance goals. This is important to overcome the current gap between research results and practical application of such frameworks and tools.

Purpose Assessment for learning (AfL) or formative assessment is an idea widely embraced by the education field; however, it is recognised as difficult to practice at classroom. This paper, with a case study, explores how an action research activity, learning study, helps a group of teachers in Hong Kong transcend some assessment boundaries and develop assessment tools that are more conducive for student’s learning. As guided by variation theory, a distinctive feature of learning study is its intensive use of formative assessment tools, including pre- and post-tests and interviews with students, to provide feedback to teaching and evidence of learning. The purpose of this paper is to investigate how teachers learn to use the formative assessment tools to overcome practical problems in teaching and improve their instruction. Design/methodology/approach A case study was conducted. Data collected included field notes on classroom observations, records of meetings, students’ interviews and students’ writings in both pre- and post-tests. The general framework for qualitative data analysis by Miles and Huberman (1994) guided the data analysis process with the foci on the change in teachers’ perceptions and improvement of their skills in adopting formative assessment tools. Findings Although the initial aim of conducting learning study was examination-driven, i.e., to improve the students’ performance in an external examination, it became evident over time that teachers participating in the inquiry process had gained a better understanding of making use of the assessment results as feedback to improve student learning and their instruction. This constituted an ideal case for studying the learning process of the teachers as they practiced AfL and transcend some conventional assessment boundaries in an authentic classroom situation. Originality/value This is one of the scarce studies, which demonstrates how the assessment elements of learning study, i.e. pre/post tests and student interviews, can be integrated into the working practice of teachers as a form of AfL and become an integral aspect of classroom practice.

This article reports on Grade 2 teachers’ perceptions of formative assessment in explaining the phenomenon of the underutilisation of formative assessment practices in mathematics teaching. A qualitative and interpretative case study investigated two Grade 2 teachers’ enactment of formative assessment in priority schools in Gauteng. Data were collected through semi-structured interviews and observations of lessons. The basic principles of qualitative content analysis were applied during data analysis and guided by the formative assessment theoretical framework proposed by Black and Wiliam (2009). The study revealed that teachers’ enactment of formative assessment was limited by their vague understanding of formative assessment and the tensions between formative assessment and curriculum compliance. The study’s central claim is that teachers may know about formative assessment, but if they do not understand how children learn and engage in mathematics learning, then they are unlikely to enact it correctly. While teachers who attended the in-service training programme were able to use some of the strategies as singular tools, they were still unable to implement the combined strategies that constitute the formative assessment pedagogy. Hence, the formative assessment practices of teachers bore limited possible returns on investment to improve learning outcomes in mathematics. The unique contribution of this study is its potential to inform teacher development, policy and practice as it yielded important insights while reinforcing and amplifying existing knowledge.

AbstractA key element of competency-based education is assessment. Effective assessment requires access to a core set of expectations that match a learner's level of preparation. Miller's triangle provides a framework for establishing appropriate expectations that move learners from novice to entry-level clinicians. Formative assessment and feedback are a crucial part of facilitating learning in this context. A pilot investigation was conducted to examine the effects of a formative, video competency on performance in a summative, live competency. Rubrics were used to score performance on two competencies, an oral mechanism exam (OME) and a clinical bedside swallowing examination (CBSE). Performance on the OME was significantly improved in the summative competency, compared with the formative, video competency. Performance on the CBSE did not change from formative to summative competency. Assessment in competency-based education is important as a measure of readiness for entry-level practice. Formative assessment and feedback can improve preparedness and performance on summative competencies. Detailed, criterion-referenced assessment tools are crucial to identifying performance. While the OME rubric used in this investigation appears to meet that standard, it is likely that the CBSE rubric was not specific enough to detect changes.

This paper describes the use of Kane’s validity framework to redevelop a workplace-based assessment program for practicing physicians administered by the College of Physicians and Surgeons of Ontario. The developmental process is presented according to the four inferences in Kane’s model. Scoring was addressed through the creation of specialty-specific assessment criteria and global, narrative-focused reports. Generalization was addressed through standardized sampling protocols and assessor training and consensus-building. Extrapolation was addressed through the use of real-world performance data and an external review of the scoring tools by practicing physicians. Implications were theoretically supported through adherence to formative assessment principles and will be assessed through an evaluation accompanying the implementation of the redeveloped program. Kane’s framework was valuable for guiding the redevelopment process and for systematically collecting validity evidence throughout to support the use of the assessment for its intended purpose. As the use of workplace-based assessment programs for physicians continues to increase, practical examples are needed of how to develop and evaluate these programs using established frameworks. The dissemination of comprehensive validity arguments is vital for sharing knowledge about the development and evaluation of WBA programs and for understanding the effects of these assessments on physician practice improvement.

The given paper focuses on the analysis of a translation project as a powerful assessment tool that can be easily integrated into prospective philologists’ university training in order to enhance its quality and bring its outcomes into line with global industrial needs and requirements. With this aim, substantial translation pedagogy experience of project-based learning is consolidated from the viewpoint of action-oriented, student-centered, situated, experiential and reflective approaches. Translation project properties, design features, role and function distribution peculiarities typical for global professional environment are considered. The received findings are transferred and adjusted to the current terms and conditions of prospective philologists’ training in Ukrainian higher educational setting. As a result, a working classification of translation project varieties to be developed and implemented into training process at different training stages is suggested. Their functions and potential usage as a flexible assessment framework is specified. The translation project opportunity for the arrangement of not only formative but summative assessment as well is substantiated due to its contrasting to the existing examination practices. Finally, the idea of cross-assessment as the complex assessment procedure conducted from different perspectives (hetero, self- and peer-assessment) with the help of versatile tools and instruments (incorporated into single flexible translation project) is introduced.

Background: Technology-enhanced assessment (TEA) is a broad term that encompasses the diverse methods, by which technology can be used to support the management and delivery of assessment in educational institutions, in workplace and lifelong learning, in professional training and development, and so on. Purpose of Study: There is a new initiative at the University of Belgrade, Serbia, to explore computing technology underpinnings of the practice of TEA (computing underpinnings of the practice [CUP] of TEA), at universities and in schools in the country. Sources of Evidence: The CUP of TEA initiative starts from the current insight into current practices of assessment processes in different educational institutions. The data collected include assessment process descriptions from these institutions, specification of technologies used for the purpose, and specification of the educational profiles of participants in the process. Main Argument: The CUP of TEA initiative develops: A novel and comprehensive framework for TEA to lay the foundation for systematic, accurate, effective, and innovative application and use of TEA in educational institutions; specifically designated sets of information and communication technology (ICT) tools and services – ICT toolsets – that support practical application of TEA; several application cases that implement the TEA framework and use different ICT toolsets for TEA in different educational contexts, showcasing the integration of best TEA practices into curricula; and a set of policies that enable educational institutions and their teachers to officially establish the use of the framework for TEA in their courses and daily work. The CUP of TEA initiative covers both formative and summative assessment and supports formal, non-formal, and informal learning and assessment. Conclusions: The ultimate goal of the initiative is not just better, faster, and more informative assessment. It is the creation of a robust teaching and learning assessment approach that improves and advances the quality and efficiency of education. Keywords: Technology Enhanced Assessment; assessment processes; application cases

Introduction. Modern views on the goals and content of education around the world are changing the viewpoints on educational assessment: the goals and functions of assessment are expanded, new assessment forms and tools are being introduced, the roles of students and teachers in assessment practice are revised. Global trends in educational assessment are reflected in national education systems, many of which have made the transition to criteria-based assessment. This process is accompanied by a rapid increase in the volume of scientific and methodological publications on this issue, reducing the ability of scientists and practising educators to comprehend the cumulative evidence in this area of research, which is important at the stage of reform. The solution to this problem can be facilitated by an analytical review of sources, representing a generalisation of international experience in the development of educational assessment.Aim. Based on the analysis of scientific publications, the current research is aimed to comprehend and generalise the current trends prevailing in the world theory and practice of assessment of student learning outcomes, to identify problems and contradictions as a potential basis for the occurrence of risks in assessment practices, to determine the current research directions towards the improvement of assessment in education.Methodology and research methods. The scientific papers of the last thirty years, revealing the issues of the education process assessment, have been used in the research. Sources were selected through the methods of descriptive and bibliometric analysis. The comparative and aspect-based analyses were employed in the research.Results. The wide range of assessment goals confirms the significance of the assessment as the education component; however, it results in some contradictions between the goals, which appear in estimated dichotomies: formative and summative assessment, norm-based and criteria-based assessment. Despite the contrapositive between the formative and summa-tive assessments, the analysis conducted shows the necessity of their integration. The variety of theoretical approaches to the interpretation of assessment forms, the ambiguous definition and use of terminology, the multiplicity of criteria-based assessment models, qualitatively different approaches to the development of criteria and the creation of assessment rubrics on their basis are sources of risks in the practice of educational assessment. In addition, an obstacle to ensuring effective assessment of students’ educational achievements is the insufficient level of competence of teachers as assessors, the unpreparedness of students themselves to take responsibility for the results of their learning.Scientific novelty. From a critical standpoint, the ideas are presented and the results of discussions on the problems of assessing the educational results of students in scientific sources are summarised. The analysis performed can serve as a basis for further research to build an ordered theoretical framework for assessment in education.Practical significance. An analysis of the pros and cons of international experience in the field of educational assessment can help scientists and practising teachers gain a more objective view of new approaches to the assessment of student learning outcomes, thereby contributing to the minimisation of the manifestation of possible risks in the development of assessment tools and procedures and their implementation in educational practice.

This paper proposes an alternative curriculum model to the current Australian Curriculum, which is underpinned by a Systems Thinking methodology (Capra & Luisi 2014). Entitled a Vertical Cubic Curriculum (VCC), this design takes advantage of intelligent design tools whilst drawing on principles from the Australian Vertical Modular Curriculum (Education Department of Victoria, Australia 1980) and the three-dimensional structure proposed by Wragg’s Cubic Curriculum (Wragg, 1997). The VCC proposes an age mixed, multidimensional curriculum space (Carey, 2016) that promotes student voice and student self-efficacy; enabling teachers and students to co-construct a ‘learning curriculum.’ The VCC employs a cubic structure both as a proposed National Framework and as an implemented Cubic Vertical modular design at school level. The VCC is a highly flexible model that fosters metacognitive learning and formative (diagnostic) assessment as a continuum of development.

ObjectiveAcross healthcare systems, there is consensus on the need for independent and impartial assessment of performance. There is less agreement about how measurement and reporting performance improves healthcare. This paper draws on academic theories to develop a conceptual framework—one that classifies in an integrated manner the ways in which change can be leveraged by healthcare performance information.MethodsA synthesis of published frameworks.ResultsThe framework identifies eight levers for change enabled by performance information, spanning internal and external drivers, and emergent and planned processes: (1) cognitive levers provide awareness and understanding; (2) mimetic levers inform about the performance of others to encourage emulation; (3) supportive levers provide facilitation, implementation tools or models of care to actively support change; (4) formative levers develop capabilities and skills through teaching, mentoring and feedback; (5) normative levers set performance against guidelines, standards, certification and accreditation processes; (6) coercive levers use policies, regulations incentives and disincentives to force change; (7) structural levers modify the physical environment or professional cultures and routines; (8) competitive levers attract patients or funders.ConclusionThis framework highlights how performance measurement and reporting can contribute to eight different levers for change. It provides guidance into how to align performance measurement and reporting into quality improvement programme.

Feedback is regularly used in summative assessment to “...reduce the gap between what is now and what should or could be” (Hattie & Timperley, 2007). However, its use in online learning tools, such as H5P, is somewhat limited to responses such as ‘correct’ and ‘incorrect’ which does not inform the learner of how they could improve (Boud and Molloy, 2013). This demonstration presents a framework for designing scaffolded tasks in H5P (http://h5p.org) which provide opportunities for educators to include feedback to guide the learner towards the expected outcome. The case study uses the H5P object ‘Drag and Drop’ object and with the ‘tip’ feature, creates a way for the learner to self-assess and direct their learning, fostering recall and knowledge production. Prompt types include ‘recall reminders’ and ‘eliciting reasoning behind the answer to identify knowledge gaps’ (Walsh and Sattes, 2005). It is suggested that by doing so, the learner is guided towards identifying the correct response and eventually narrowing their knowledge gap (Vygotsky, 1978). In this demonstration, two such H5P activities will be presented and participants will be guided through the stages of Learning Design and pedagogical rationale used in the task creation. This presentation will test the ability of the H5P tool to encompass pedagogical practices such as feedback into its environment. It is expected that future enhancements to the application account for this consideration. References Boud, D. and Molloy, E. (2013). Feedback in Higher and Professional Education, London: Routledge, 1-10. Hattie, J., & Timperley, H. (2007). The Power of Feedback. Review of Educational Research, 77(1), 81–112. (https://journals.sagepub.com/doi/abs/10.3102/003465430298487) Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes. (Edited by M. Cole, J. Scribner, V. John-Steiner, & E. Souberman). Cambridge, MA: Harvard University Press. Walsh, J. A., & Sattes, B. D. (2005). Quality questioning: Research-based practices to engage every learner. Thousand Oaks, CA: Corwin.

This procedures is discussed about development social welfare and justice through education transformation in the 21st century. The development of social welfare and justice particularly in the context of education transformation within the framework of the modern education system demand leadership of teachers and leaders adaptable with the changes and existing condition in the social welfare and justice recognize the needs to be effective education Leaders. In accordance to development social welfare and justice, the requirements to familiar with research findings on the knowledge, skills, implementation tools, methods, and strategies useful for formulating vision direction and positive learning in the 21st century are necessities. This paper also discussed an emerging role and challenging of the 21st century education teachers and leaders required to transform the social welfare and justice. Major transformations required the 21st century leader to be a powerful learner, visionary, instructional and influential people. The 21st century leader is essential to have tools and methods to expect, inspect, direct, respect and reflect. The development social welfare and justice will need to concentrate on curriculum development, formative assessment, instruction, technology adaptability, culture and climate, professional development as well as effective supervision to make a massive improvement in development social welfare and justice

Learning outcomes are the culmination of all teaching and learning activities, because they are part of the benchmarks for the assessment of learners and teachers through various learning patterns and methods within a certain period of time. This is to become a new and further framework for teaching and learning activities (KBM). Learning outcomes are also inspired by methods from the teacher to students and their feedback. Various types of learning methods have been used as a learning formulation, one of which is Jig Saw Learning, this method is an interesting strategy to use if the material to be studied can be divided into several parts and the material is not in the order. The advantages of this strategy are all students learn and teach others. This method is can be implemented in various subjects with supporting capacities and tools such as syllabus, implementation of learning (RPP), student activity sheets (LKS), formative tests, and test validity. In this study the authors conducted a study to what extent the effectiveness of the Jig Saw Learning method was applied in Indonesian language lesson by comparing the theory with the application of the method, not only on learning outcomes, but also on students' understanding and mastery of the subject matter.

Purpose: To analyze personal interactions as one of the possible factors that can generate greater stress in school administrators. For this purpose, we present the design, validation and application of a survey that intends to measure the type of managerial functions that are associated with higher rates of personal interaction and their possible measurement as stressors of those performing administrative functions.Design/methodology/approach: To develop, validate and apply an assessment tool to measure the stress of school administrators. To this end, a mixed analysis methodology has been adopted, involving the selection of a group of experts, the calculation of an “expert proficiency coefficient” (Cabero & Barroso, 2013) and the subsequent validation of the survey using the modified Delphi method in two phases, with a sample of 30 school administrators and educational supervisors. The survey has also been subjected to a process to determine the trustworthiness and validity of the construct through exploratory and confirmatory factor analysis.Findings: The result is a 5-dimension, 31-item survey with a total score of 155 points and three measurements: low stress (1-50 points), medium stress (51-100 points) and high stress (101-155 points). The areas that generate the greatest amount of stress can be summarized in two categories: reprimanding teachers for non-performance of their duties and discipline management at the center (expulsions, bullying, fighting and drugs).Research limitations/implications: The survey was designed according to the socio-educational characteristics of the Spanish context. Further research in other educational contexts would require an adaptation of different items on the scale.Practical implications: Measuring the school administrator’s stress makes it possible to identify those functions that are more susceptible to intervention, both formative and administrative. This improves one of the key areas of school organization.Social implications: The improvement of the school institution and its governance starts with identifying those aspects that require educational interventions, but also social ones that imply a reflection on how schools in the 21st century are managed and organized.Originality/value: There are no Spanish studies that propose assessment tools to measure the stress of school administrators; in this sense, this study provides a framework for the Spanish context.

Objective – Part I of this paper aims to create a framework for an emerging theory of evidence based information literacy instruction. In order to ground this framework in existing theory, a holistic perspective views inquiry as a learning process that synthesizes information searching and knowledge building. An interdisciplinary approach is taken to relate user-centric information behavior theory and constructivist learning theory that supports this synthesis. The substantive theories that emerge serve as a springboard for emerging theory. A second objective of this paper is to define evidence based information literacy instruction by assessing the suitability of performance based assessment and action research as tools of evidence based practice. Methods – An historical review of research grounded in user-centered information behavior theory and constructivist learning theory establishes a body of existing substantive theory that supports emerging theory for evidence based information literacy instruction within an information-to-knowledge approach. A focused review of the literature presents supporting research for an evidence based pedagogy that is performance assessment based, i.e., information users are immersed in real-world tasks that include formative assessments. An analysis of the meaning of action research in terms of its purpose and methodology establishes its suitability for structuring an evidence based pedagogy. Supporting research tests a training model for school librarians and educators which integrates performance based assessment, as well as action research. Results – Findings of an historical analysis of information behavior theory and constructivist teaching practices, and a literature review that explores teaching models for evidence based information literacy instruction, point to two elements of evidence based information literacy instruction: the micro level of information searching behavior and the macro level of the learning task. On the micro level users are confronting information, and searching is seen as the entire process of the interaction of users with a series of information tasks, as described in Kuhlthau’s Information Search Process. The micro level is the level of deep understanding as critical thinking skills craft the connection between information and knowledge. On the macro level the learning task, designed by an instructional team, shapes the inquiry. It is the context for information tasks. The learning task is relevant, engaging, and rigorous to sustain the interest and interaction of the user with information and emerging knowledge. The macro level is the level of instruction whereby the learning task fosters self-reflection, self-correction, and self-regulation. The role of evidence, which is generated by performance assessment and action research, is critical to both levels. On each of these levels the learning task informs the information search. Conclusion – Evidence is the link between information and knowledge in the learning process called inquiry. Sources of this evidence are information tasks embedded in the learning task, which are nested in a teaching and learning culture of inquiry. In order to generate evidence continuously throughout the inquiry unit, the task is characterized by research-based tools such as performance-based assessment and action research. The synergy of these elements in Part 1 of this article establishes the foundation for building a theory that supports further research of evidence based information literacy instruction.

The practice of Blended learning in schooling for almost two decades has shown limited variations from what conventional education had breached hundreds of years ago. In that, continuing the use of large groups teaching, big chunks of required curriculum for learning and instruction, unified mandatory knowledge for all students, the lack of individual achievement pace, unified daily study schedules, and same backward assessment tools and formalities. Yet, ICTs exert every moment profound remote digital penetrating power into the deep privacy of human anatomy, actions, family and daily lives in what could be called the “Age of digital Information Nudity”. Moreover, resource teachers or students’ counselors are able by a click at a digital curriculum to reveal where learners’ locations are, how many times attempted reading, performing assignments, or what study difficulties they are facing. ICTs have dismantled all the psychological, social, educational and physical boundaries separating peoples, businesses and schooling communities. Students living currently in digital environments have a free will through digital means to decide, plan, in addition, to using types, tools, times and settings for their blended and online learnings, even without the pre-consent of school authorities. Hence, a new form of blended education deems crucial, that is “neoBlend-Digit” Schooling. This new hybrid methodology, when well served by developmental appropriate curriculum design, thoughtful treatment of curriculum content, a systemic implementation framework (diagnostic, formative and summative), and optional multi learning tracks, will lead to open ended learning free of failure, outdated teachers and school personnel, and environmental limitations. Consequently civic citizen students are graduating as literates, professional specialists, and academic pioneers.. Realizing as such what the American educator, William Glasser* wished for “Schools without Failure”. In current work however, the long waited dream of Glasser is presented in a concise digital language and concretely accountable educational evidences in field reality.

11030 Background: Current research training in Hematology/Oncology (Heme/Onc) fellowship is typically limited to mentoring relationships, which are highly variable due to the absence of standardized training for mentors and different perceptions of successful mentor-mentee relationships. Formal research productivity assessment is challenging and lacking in our current educational framework. Methods: Electronic surveys were developed to assess research training during fellowship at the University of Wisconsin (UW). Surveys were sent to current Heme/Onc fellows (n = 9), members of fellowship leadership (n = 4), and core research faculty (n = 5) with prior successful mentorship to trainees. Results: Surveys were completed by 6 fellows, 3 fellowship leadership, and 5 core research faculty. Both faculty and trainees recognized the research mentor-mentee relationship to be critical, with 5/6 (83%) fellows and 3/5 (60%) core research faculty identifying research mentors as the most meaningful training source. However, despite 5/6 (83%) fellows having a research mentor identified, there were large variations in the perceived level of proficiency in core research topics. While 3 fellows felt only slightly prepared in basics of clinical trial design, 1 fellow felt very prepared and 2 felt somewhat prepared. Furthermore, 2 fellows each felt very prepared, somewhat prepared, or not at all prepared in assembling grant components. There was also variable confidence reported by core research faculty in the ability of trainees to obtain funding and conduct independent research after fellowship, with 1 extremely confident, 2 somewhat confident, and 2 only a little confident. All individuals surveyed recognized manuscript publication, abstract presentation, and grant acquisition as important measures of research productivity. Beyond that, however, only 1/5 (20%) core research faculty and 0/3 (0%) fellowship leadership noted using rubrics for critical review of trainee academic progress despite 3/6 (50%) fellows believing that such rubrics were being used for their evaluation. Conclusions: The divergent experiences identified among research mentors and mentees and the lack of clearly defined metrics of research productivity highlights the need for a more standardized educational framework and formative evaluation tools. These identified needs have led to a pilot program at UW that intends to create a professional learning community for research mentors, define competencies for research training, and design a research training portfolio and accompanying assessment rubric.

Objective - This study investigated the use of a communities of practice (CoP) model for blended learning in library and information science (LIS) graduate courses. The purposes were to: (1) test the model’s efficacy in supporting student growth related to core LIS concepts, practices, professional identity, and leadership skills, and (2) develop methods for formative and summative assessment using the model. Methods - Using design-based research principles to guide the formative and summative assessments, pre-, mid-, and post-course questionnaires were constructed to test the model and administered to students in three LIS courses taught by the author. Participation was voluntary and anonymous. A total of 34 students completed the three courses; response rate for the questionnaires ranged from 47% to 95%. The pre-course questionnaire addressed attitudes toward technology and the use of technology for learning. The mid-course questionnaire addressed strengths and weaknesses of the course and suggestions for improvement. The post-course questionnaire addressed what students valued about their learning and any changes in attitude toward technology for learning. Data were analyzed on three levels. Micro-level analysis addressed technological factors related to usability and participant skills and attitudes. Meso-level analysis addressed social and pedagogical factors influencing community learning. Macro-level analysis addressed CoP learning outcomes, namely, knowledge of core concepts and practices, and the development of professional identity and leadership skills. Results - The students can be characterized as adult learners who were neither early nor late adopters of technology. At the micro-level, responses indicate that the online tools met high standards of usability and effectively supported online communication and learning. Moreover, the increase in positive attitudes toward the use of technology for learning at the end of the course may indicate that an effective balance between face-to-face and online media was achieved. At the meso-level, students valued learning in community for developing mutual respect, confidence building, risk taking, deeper and more varied learning, learning with and from their peers, and greater enjoyment in the classes. Students found that the online environments were useful for organizing the class objectives and subject matter, “staying connected” between classes, sharing ideas, keeping track of their work, and preparing them for future work in blended environments. At the macro-level, the findings of the effects on student growth related to core LIS concepts, practices, professional identity and leadership skills were inconclusive. However, students’ expressed a high regard for the value of collaboration, and there were indications that the model supported differentiated learning of professional knowledge and skills. Conclusion - The findings strongly suggest that the use of the CoP model had positive effects on the learning process. Students’ high regard for the value of collaboration appears to be a clear effect of using the CoP model. The assessment methods were sufficient for testing the efficacy of most aspects of the model under the limited conditions of this study. Student responses led to refinements in both the model and methods. This study contributes to social constructivist learning approaches and LIS curricular development by presenting an innovative model for supporting professional growth among adult learners, as well as a conceptual framework to guide evidence -based practice. Further testing and refinement of the model in other contexts and by other educators are needed to ensure that the model is robust and broadly applicable.

La presente ponencia procura sistematizar la técnica de resolución de problemas como parte de la evaluación auténtica que se llevó a cabo en la primera práctica de laboratorio, donde se elaboró una mermelada como producto final a partir de fruta fresca, lo cual es parte de las actividades de la asignatura Tecnología Hortifrutícola de Cátedra de Tecnología Agroindustrial, II cuatrimestre del 2013, siendo esto una actividad práctica que ayuda al estudiante a mejorar la calidad y el nivel de aprendizaje significativo en la carrera de Ingeniería Agroindustrial.Dicha estrategia alternativa evaluativa también va acompañada de trabajo colaborativo con la coevaluación, la autoregulación, la autonomía, integración del aprendizaje; lo cual generan, en el estudiante, el desarrollo de pensamiento crítico al tomar decisiones in situ, logrando también que se desarrolle la competencia de elaboración de un informe de laboratorio con el fin de reforzar lo realizado en la práctica, sin dejar de lado la evaluación formativa y cuantitativa, las cuales se involucran en el proceso enseñanza aprendizaje, logrando formar futuros profesionales que logren incluirse de forma exitosa al medio laboral.Como resultado de la aplicación de la técnica, se evidencia que, en la práctica del laboratorio de Tecnología Hortifrutícola de la Cátedra de Tecnología Agroindustrial, se desarrolla la técnica de solución de problemas como parte de la evaluación auténtica, donde se potencian el pensamiento crítico gracias a la técnica de solución de problemas. Con la aplicación de la técnica, se mejoran las habilidades en este caso de elaboración de informes, pues identifican que es una introducción, realizan tanto objetivo general como específicos, una justificación, una marco teórico, realizan de forma adecuada los resultados, analizan estos en la discusión y concluyen de su práctica, en los estudiantes como competencias, se construye por medio de un proceso colaborativo un aprendizaje significativo. Además, se favorece la autonomía, la meta cognición, se valora el error, se genera la autoevaluación y, sin duda, se le enseña al estudiante a enfrentarse a situaciones adversas, similares a las que enfrentará en una planta agroindustrial. También provee al estudiante de herramientas para dar la mejor solución al problema de forma pertinente y analítica, y, además, se potencia la competencia en el estudiante en la elaboración de informes, lo cual le ayudará en su labor como futuro profesional en cualquier industria que se desempeñe.Para efecto de esta ponencia, solamente se centraran en las experiencias vividas por los estudiantes en el primer laboratorio del II cuatrimestre del año 2013, desarrollando la estrategia de resolución de problemas durante la práctica en planta del laboratorio de la asignatura de Tecnología Hortifrutícola.Palabras clave: evaluación auténtica, resolución de problemas, aprendizaje significativo, aprendizaje colaborativo, evaluación formativa.AbstractThis paper attempts to systematize the technical of problem resolution as part of authentic assessment that was carried out in the first lab, where a jam as the final product was made from fresh fruit, which is part of the activities of the subject of Technology Horticulture Technology Department in Agribusiness, II quarter of 2013, this being a practical activity that helps students improve the quality and level of meaningful learning in the career of Agroindustrial Engineering.Such evaluative alternative strategy is also accompanied by collaborative work with peer assessment, self-regulation, autonomy, integration of learning; which generate the student to develop critical thinking decisions taken in situ, also making it the ability to develop a lab report in order to reinforce what has been done in practice to develop, without neglecting the assessment formative and quantitative, which are involved in the teaching-learning process, making train future professionals achieve successful included so the workplace.As a result of the application of the technique, it appears that, in practice Horticulture Technology Laboratory of the Department of Agroindustrial Technology, technical problema resolution as part of authentic assessment, which enhance critical thinking develops thanks technical problem resolution. With the application of the technique, skills are improved in this case reporting, which is identified as an introduction, make both general and specific purpose, a justification, one theoretical framework, done properly the results and analyze these in discussion and conclude from their practice, skills in students as it is constructed through a collaborative process of meaningful learning. Furthermore, autonomy is favored, metacognition, the error value, the self generated and no doubt the student is taught to face adverse similar situations they will face in an agro-industrial plant. It also provides students with tools to give the best solution to the problem of relevant and analytically, and also competition power in the student reporting, which will help them in their future work as a professional in any industry that it performs.For purposes of this paper, only to focus on the experiences of students in the first semester lab II in 2013, developing a strategy for solving problems within the plant laboratory practice of the subject of Horticulture TechnologyKeywords: authentic assessment, problem solving, meaningful learning, collaborative learning, formative assessment.

Background Traditionally, digital health data management has been based on electronic health record (EHR) systems and has been handled primarily by centralized health providers. New mechanisms are needed to give patients more control over their digital health data. Personal health libraries (PHLs) provide a single point of secure access to patients' digital health data and enable the integration of knowledge stored in their digital health profiles with other sources of global knowledge. PHLs can help empower caregivers and health care providers to make informed decisions about patients’ health by understanding medical events in the context of their lives. Objective This paper reports the implementation of a mobile health digital intervention that incorporates both digital health data stored in patients’ PHLs and other sources of contextual knowledge to deliver tailored recommendations for improving self-care behaviors in diabetic adults. Methods We conducted a thematic assessment of patient functional and nonfunctional requirements that are missing from current EHRs based on evidence from the literature. We used the results to identify the technologies needed to address those requirements. We describe the technological infrastructures used to construct, manage, and integrate the types of knowledge stored in the PHL. We leverage the Social Linked Data (Solid) platform to design a fully decentralized and privacy-aware platform that supports interoperability and care integration. We provided an initial prototype design of a PHL and drafted a use case scenario that involves four actors to demonstrate how the proposed prototype can be used to address user requirements, including the construction and management of the PHL and its utilization for developing a mobile app that queries the knowledge stored and integrated into the PHL in a private and fully decentralized manner to provide better recommendations. Results To showcase the main features of the mobile health app and the PHL, we mapped those features onto a framework comprising the user requirements identified in a use case scenario that features a preventive intervention from the diabetes self-management domain. Ongoing development of the app requires a formative evaluation study and a clinical trial to assess the impact of the digital intervention on patient-users. We provide synopses of both study protocols. Conclusions The proposed PHL helps patients and their caregivers take a central role in making decisions regarding their health and equips their health care providers with informatics tools that support the collection and interpretation of the collected knowledge. By exposing the PHL functionality as an open service, we foster the development of third-party applications or services and provide motivational technological support in several projects crossing different domains of interest.

This paper aims to explore the pitfalls of the current evaluation system at the school level in Nepal focusing on the major consequences of educational practices as the output of the evaluation system. Although the provisions of formative assessment tools such as portfolio management and project work were included in the curriculum framework and policy documents, the evaluation process was based on a theoretical basis. Similarly, the judgmental written examination for assessing theoretical aspects and Continuous Assessment System (CAS) with backward-looking nature were involved in the evaluation process. The paper suggests to the educational stakeholders for the practical application and evaluation of soft-skills associated methods such as communication, creativity, cooperation and collaboration of basic and secondary students to promote formative evaluation and the curriculum concept of 'assessment for learning in Nepal.

Introduction: The present study aimed to evaluate an assessment system in residency program based on a competency-based approach. Methods: The establishment of the assessment as a quality improvement of clinical education has progressed since 1396 in Shahid Sadoughi University of Medical Sciences. In this study, the evaluation the assessment system was carried out in 6 steps including the formulation of an evaluation plan, the design of tools, information gathering and data analysis, reporting and feedback. The evaluation was conducted through site visite by using a checklist, reviewing the documentation and interviewing the educational managers. Data were analyzed using descriptive tests. Result: The student assessment system was evaluated in 16 residenty departments. The status of the establishment of a student assessment system in the "objectives", "structure and organization" axes, is desirable, and in "design and implementation framework" and "quality assurance" is in moderate level. In all of the departments were devlepoded assessment bluprinet to determine the capabilities and tools appropriate for assessing them. The results indicated 83% of the capabilities on average were measured in departments. The most diversity in the assessment of learners' abilities was reported in the written tests (cognitive domain) using three tests (Multi-Choice Questions, Essay and Oral Exam. Also, the status of the formative tests was evaluated in "moderate" level. Conclusion: The results of the evaluation show that although key steps have been taken towards implementing the student evaluation system in the residency departments. But, for the establishment of the system need to plan base on the competency-based assessment at the departments and educational administration levels.

Despite the pressure from potential employers and higher education administrators to develop students’ global and intercultural competence, traditional study abroad programs simply are not feasible for many postsecondary students (Berdan & Johannes, 2014; Fischer, 2015). The purpose of this study was to evaluate the effectiveness of an online delivery model for study abroad activities. Building upon the findings of an initial exploratory program using Adobe Connect web conferencing tools, this evaluative case study was the second in a series of design based research studies intended to identify effective practices and develop recommendations to further refine the model through an iterative evaluation process. Using the Online Learning Consortium’s Quality Framework, each of the Five Pillars that support successful online learning was evaluated through a combination of anonymous surveys, pre/post assessments, observations, and student & instructor interviews. Regarding access, 26 students who were enrolled in an intercultural communication course were able to participate in a study abroad experience in Italy; 10 students participated in the traditional study abroad trip while the other 16 participated virtually. The online students were able to join the live meetings, thus expanding their access to international experiences that normally would be closed to them. In terms of student and faculty satisfaction, both groups of students and the instructor reported specific areas of satisfaction, offered critical feedback, and felt that the concept was a viable one. While the students who traveled to Italy had a far more immersive experience, both groups demonstrated gains in learning. Using Morais and Ogden’s (2010) global citizenship pre/post assessment, both groups showed improvement on the self-awareness and intercultural communication scales, and when comparing the two groups the online students improved more on the social responsibility scale while the students who traveled improved more on the global knowledge scale. Both groups submitted assignments of similar quality, engaged in communications between the abroad and online groups, and interacted with the instructor and experts in the field. In terms of differences in student engagement, students had differing opinions on the interaction with the technology and the online group asked more questions during live meetings. The implications of this pilot study should inform the planning of the next case evaluation and are important for other educators who wish to implement a similar approach to internationalizing the curriculum through online instruction.

Background: Canadian residency programs have adopted competency-based medical education, where time-based learning systems are replaced with core competency “milestones” that must be achieved before a student progresses. Assessment tools must be developed to predict performance prior to high-stakes milestones, so interventions can be targeted to improve performance.Objectives: 1. To characterize how well each of three practice written exams predicts passing the Canadian Internal Medicine Royal College (RC) exam. 2. To determine if writing practice exams is perceived to improve performance on the RC exam.Methods: Three 105-question multiple choice question exams were created from a range of internal medicine topics, and offered one month apart to 35 residents. Percentile ranks on each practice exam were compared to the result (pass/fail) on the RC exam. Surveys were completed within 1 month after the RC exam.Results: There were 35 residents invited to participate. Practice exams (PE) 1, 2, and 3 were taken by 33, 26, and 22 residents, for an exam participation rate of 94.3, 74.3, and 62.9%, respectively. Failure on the RC exam could be predicted by percentile ranking <15% on PE1 (OR 19.5, p=0.017) or PE2 (OR 63.0, p=0.006), and by percentile ranking <30% on PE1 (OR 28.8, p=0.003), PE2 (OR 24.0, p=0.010) or PE3 (OR 15.0, p=0.046). The survey was sent out to the 33 participants. Of those, the total number of respondents was 25, with a response rate of 75.5%. Survey takers agreed that practice written exams improved performance on the RC exam (18/25, 88%).Conclusions: Performance in the Canadian Internal Medicine RC Exam can be predicted by performance on any of three practice written exams. This tool can therefore identify trainees for whom additional resources should be invested, to prevent failure of a high-stakes milestone within the competency based medical education framework.RÉSUMÉContexte : Les programmes canadiens de résidence ont choisi de diffuser un enseignement médical axé sur les compétences dans lequel les systèmes d’apprentissage structurés en fonction du temps sont remplacés par des « jalons » liés aux compétences fondamentales que l’étudiant doit atteindre pour aller de l’avant. Il faut élaborer des outils d’évaluation pour prédire la probabilité de résultats escomptés par un étudiant avant que celui-ci ne se présente à certains événements dont les enjeux sont élevés. Ainsi, il devient possible d’intervenir de manière à améliorer les résultats escomptés.Objectifs : 1. Déterminer dans quelle mesure chacun des trois examens de pratique écrits prédit la réussite à l’examen du Collège royal des médecins et chirurgiens du Canada (CRMCC) en médecine interne; 2. Évaluer si le fait de se soumettre à des examens de pratique écrits est perçu comme un élément qui améliore les résultats à l’examen du CRMCC.Méthodologie : Trois examens écrits comportant chacun 105 questions à choix de réponses portant sur un éventail de sujets relatifs à la médecine interne ont été préparés et proposés à 35 résidents à intervalle d’un mois. Les rangs-centiles de chaque examen de pratique ont été comparés avec le résultat obtenu à l’examen du CRMCC (succès/échec). Les sondages ont été effectués dans le mois suivant l’examen du CRMCC.Résultats : Trente-cinq résidents ont été invités aux examens de pratique écrits (EP) 1, 2 et 3. La participation a été respectivement de 33, 26 et 22 résidents, soit de 94,3 %, 74,3 % et 62,9 %. L’échec à l’examen du CRMCC pouvait être prédit par un rang-centile < 15 % à l’EP1 (OR 19,5 et p = 0,017) ou à l’EP2 (OR 63,0 et p = 0,006) et un rang-centile < 30 % à l’EP1 (OR 28,8 et p = 0,003), à l’EP2 (OR 24,0 et p = 0,010) ou à l’EP3 (OR 15,0, et p = 0,046). Le sondage a été envoyé aux 33 participants. Le nombre total de répondants a été de 25, pour un taux de réponse de 75,5 %. La majorité des répondants (18/25, 88 %) sont d’avis que les examens de pratique écrits leur ont permis d’obtenir de meilleurs résultats à l’examen du CRMCC.Conclusions : Les résultats à l’examen du Collège royal des médecins et chirurgiens du Canada (CRMCC) en médecine interne peuvent être prédits par les résultats obtenus à l’un des examens de pratique écrits. Par conséquent, cet outil peut être utilisé dans le cadre de l’enseignement de la médecine axé sur les compétences pour identifier sur qui l’on devrait investir des ressources additionnelles en vue d’éviter un échec à cet événement aux enjeux élevés.Competency-based medical education (CBME) has generated increased attention over the last decade,1–3 and become entrenched within several national medical education frameworks including Canada.4 Proponents of CBME suggest that older medical education models focus on medical knowledge rather than skills, or higher order aspects of practice. 5 Focus on time spent in training can take away from the abilities acquired during that time frame.6 Furthermore, flexible time periods may be more efficient and focused, compared to time-based curriculum.3,6,7 In light of these advantages, the Royal College of Physicians and Surgeons of Canada (RCPSC) has committed to transform medical education to a CBME model for all residency programs by 2017.4While residency programs reorganize toward the CBME model, residents will still be required to perform oral and written exams. It is thus essential that CBME-based programs incorporate assessment tools to predict performance on high-stakes milestones, such as RC exams.We created three written PEs, and evaluated how well each predicted performance on a high-stakes milestone, the RCPSC Internal Medicine exam (RC exam). We also evaluated how well PE were perceived to improve performance on the same high-stakes milestone RC exam.METHODSSetting and ParticipantsThe RC exam contains both written and oral components. All residents sitting both components of the RC exam, within 12 months, who were post-graduate medical residents at Western University (London, Ontario, Canada), were invited to participate. The study was conducted in 2013-2014.Western University Health Sciences Research Ethics Board provided an ethics waiver for this study, since the study was performed as part of the standard operations of an educational program.Intervention: ExamsTwo authors (HS, BT) separately created multiple choice questions (MCQ) reflecting all areas of internal medicine, based on the Objectives of Training of the RC Internal Medicine exam. RC exam questions are not available for purchase, and examinees are forbidden to share RC exam questions. Therefore, PE content and question style was informed by questions purchased for American Board of Internal Medicine (ABIM) course reviews.8,9 MCQ creators had each completed the RC exam within 3 years, and were thus familiar with MCQ and exam format.All authors independently reviewed each PE question to assure quality of content, grammar, spelling, and syntax. Each PE covered all subspecialty areas within internal medicine, including allergy and immunology ( n=4), cardiology (n=13), dermatology (n=2), endocrinology (n=8), gastroenterology (n=10), hematology (n=10), infectious diseases (n=15), nephrology ( n=9), neurology (n=6), oncology (n=4), respiratory and critical care medicine (n=7), rheumatology ( n=14) and statistics (n=3). This topic allocation included 7 questions for JAMA Rational Clinical Exam, and 5 for interpretation of medical images (e. g., chest X-ray, electrocardiogram). PE size (105 questions) and length (3 hours) were chosen to reflect the RC exam.Each PE was offered at two separate times, to assure flexibility within ongoing clinical responsibilities. PE1, PE2, and PE3 were offered approximately 7, 6, and 5 months prior to RC exam, respectively. This timing was chosen so that trainees had sufficient time to improve their performance before the RC exam if a poor PE result was found.Examinees were provided a personalized report for each exam, within 7 days of completing the PE. The personalized report included the examinee’s overall mark, average within each subspecialty, and percentile rank within the entire cohort of examinees. Two separate 1-hour periods were available to review each PE results, with the questions and key, supervised by BT.Intervention: SurveyAll study participants were invited to participate in a survey. The survey assessed how well PE simulated the RC exam, whether the PE were recommended to the next year’s cohort of examinees, and whether the PE improved performance on the RC exam.Outcomes: ExamsEach study participant agreed to provide the RC exam result (“pass” or “fail”) once he or she had received it. Each candidates verbally communicated RC exam result was confirmed online 3 months after the RC exam results were reported (cpso.on.ca).Odds ratios were calculated. The adverse outcome was failure on the RC exam. Exposures evaluated included percentile rank < 15% and <30%. Odds ratios of infinity were prevented by adding 1 adverse outcome to any exposure group without any adverse outcomes; this was performed for 3 exposure groups, but did not impact whether statistical significance was attained. Results are detailed in Table 1.Outcome: SurveySurvey results were on a Likert Scale. The proportion of those respondents who agreed or disagreed were calculated.All data was analyzed using Statistical Package for the Social Sciences (SPSS) version 21.0.RESULTSSetting and ParticipantsThere were 35 residents invited to participate, the total number of possible participants. PE1, PE2, and PE3 were taken by 33, 26, and 22 residents, for an exam participation rate of 94.3, 74.3, and 62.9%, respectively. The majority of invitees took 3 (n=17) or 2 ( n=14), while a minority took 1 (n=2) or 0 (n=2) practice exams.ExamsOf all examinees of the RC exam (n=35), 7 failed. RC exam pass rates were lower when PE1 percentile rank was lower than 15% (40.0 vs. 92.9%, p<0.001) or 30% (44.4 vs. 100%, p<0.004), when PE2 percentile rank was lower than 15% (0.0 vs. 100.0%, p <0.001) or 30% (42.9 vs. 100.0%, p=0.038), and when PE3 percentile rank was lower than 30% (50 vs. 93.75%, p=0.046) (Figure 1). Figure 1. License exam practice pass rate versus percentile rate (PR) on practice exams. Examinees were more likely to fail the RC exam if percentile rank was less than 15% (OR 19.5, p=0.017) or 30% (OR 28.8, p=0.003) in PE1, less than 15% (OR 63.0, p=0.006) or 30% (OR 24.0, p =0.010) in PE2, or less than 30% (OR 15.0, p=0.046) in PE3.SurveyOnly residents who had taken at least 1 practice exam were invited to participate. The survey was sent out to the 33 participants, the total number of possible participants. Of those, the total number of respondents was 25, with a response rate of 75.5%. Most survey respondents agreed that the PEs were an accurate simulation of the written component of the RC examination (20/25, 80%) (Figure 2A). Most survey respondents agreed that the PEs improved performance on the RC written examinations (18/25, 72%) (Figure 2B). Most survey respondents recommended future residents to take the PEs (22/25, 88%) (Figure 2C).DISCUSSIONWe describe the creation of a tool to assess performance on a high-stakes milestone examination, the RC exam. This tool is easy to create, affordable, and is administered on a voluntary basis with high uptake amongst candidates writing the RC exam. The assessment tool has been shown to predict performance well so that resources can be invested in those at risk for failing.There is a possibility that mere participation in the assessment tool itself improves performance on this high-stakes exam. There were insufficient numbers of study participants to determine a correlation between number of exams taken and pass rates. Even still, unwillingness to participate in the study may reflect a general unwillingness to prepare, which means the results would be confounded and correlative rather than causative. One way to look into this is to perform a randomized trial in which half of residents take the assessment tool and the other half doesn’t. Unfortunately, almost all invited residents were anxious to participate, rendering such a possible study impossible. On the other hand, exam takers were able to communicate usefulness of the exam and to provide feedback on how it might be improved for future years.As CBME develops and becomes entrenched, there will continue to be a need to prepare for knowledge based written exams. This exam will continue to be considered a core competency between the stages to transition to practice. Thus, tools are needed to assess exam competence. This study confirms that such tools can and should be developed to assure that trainees are prepared.Ideally, residents with low performance would be identified early enough to intervene to change the outcome. It is uncertain what the ideal time frame is or what the intervention should be. It is reasonable to assume that taking the examination earlier in their training may allow candidates to become aware of their performance and implications thereof and implement earlier changes in learning strategies. For example, in past years, candidates contacted their program directors to ease the clinical workload to allow more study time. Others sought counselling and mentorship from staff, while others were self-directed in their learning and became more motivated to study. On the other hand, poor performance on this formative examination could potentially discourage some residents from studying if they felt their studying was futile. Future research efforts should focus on identifying which intervention is optimal to modify exam performance.The failure rate of 20% on the RC exam the year the study was conducted was unusually high for the program; however, this allowed for a correlation to be established between the PEs and the RC exam. The PEs were able to identify all those who failed the RC exam. However, there were those who scored below the 30th percentile on the PEs and still passed the RC exam. It must be kept in mind that the purpose of these formative exams is to identify those at high risk of failure so they can receive remedial support and improve their chances of passing. Thus, it is possible that through increased remedial support, those candidates who did poorly on the in-training exam managed to pass the RC exam.Only 2 trainees chose to write none of the PEs. While both of these trainees ultimately failed the RC exam, statistical significance could not be established due to the small sample size. It thus remains uncertain whether the act of writing PEs predicts passing the RC exam. However, the study objective was to identify candidates at high risk of failing the RC exam; the next step will be to determine which interventions can improve RC exam result. However, it must be acknowledged that PEs could not only identify candidates at risk of failing RC exam, but also improve their performance. This requires future study before any firm conclusions can be found.This study confirms that formative exams’ results can predict failure on the RC exam. The questions were written by authors who had recently written the RC exam, familiar with its format, and knowledgeable of the current Canadian guidelines, which are a focus of the actual RC exam. Because of confidentiality agreements with the RCPSC, actual RC exam questions can’t be shared, and thus can’t be used as part of the practice exams. However, we attempted to overcome this limitation by having all PE questions reviewed by at least 3 physicians who’d recently successfully completed the RC exam, to assure syntax and format was as similar as possible between PE and RC exam. Furthermore, this limitation does not impact the PEs statistically significant prediction of candidates at risk of failing the RC exam. The study objective was to identify candidates at risk of failing the RC exam, and the PEs are indeed a valid predictor of RC exam performance. There are limitations to this study. Firstly, this was a single centre study. However, Western University has a wide range of subspecialty programs available, and the trainees’ demographics resemble that at other Canadian centers. Secondly, new questions need to be created annually to reflect updated literature and guidelines; this requires ongoing commitment and dedication from staff. These “updated” exams could become more difficult to validate if candidates no longer fail the RC exam. However, if the act of taking the PE predicts passing RC, future research could focus on comparing RC pass rates at programs with and without the PEs. Thirdly, it's entirely possible that the use of questions from old RC exam would be more predictive, but these questions cannot be shared or used for PE due to the confidentiality agreement with the RCPSC. Therefore, creation of independent questions is still required. This is the first study of an assessment tool to predict performance on the Canadian internal medicine examination within the CBME framework. This strategy can easily be replicated and feedback is rapidly provided in a time sensitive manner. This could help trainees direct their preparation and identify knowledge gaps more easily.CONCLUSIONWe report an assessment tool to predict performance on the RC exam that can be a valid and useful form of feedback. This strategy can easily be replicated for other subspecialties or internal medicine programs. Future efforts need to focus on how the results can determine which interventions or learning strategies improve the results of candidates identified to be at risk for failing.DisclaimersThe authors declare they have no competing interest.The authors report no external funding source for this study.The authors declare no previous abstract or poster or research presentation or any online presentation of this study.REFERENCES 1. Caccia N, Nakajima A, Kent N. Competency-based medical education: the wave of the future. J Obstet Gynaecol Can 2015;37:349–53. 2. Carraccio C, Englander R, Gilhooly J, et al. Building a framework of entrustable professional activities, supported by competencies and milestones, to bridge the educational continuum. Acad Med 2016 ;92(3):324–30. doi: 10.1097/ACM.0000000000001141. 3. Carraccio C, Wolfsthal SD, Englander R, Ferentz K, Martin C. Shifting paradigms: from Flexner to competencies. Acad Med 2002;77:361–67. 4. Johnston C. Residents prepare for switch to competency-based medical education. CMAJ2013;185:1029. 5. Talbot M. Monkey see, monkey do: a critique of the competency model in graduate medical education. Med Educ 2004;38:587–92. 6. Long DM. Competency-based residency training: the next advance in graduate medical education. Acad Med 2000;75:1178–83. 7. Bell HS, Kozakowski SM, Winter RO. Competency-based education in family practice. Fam Med 1997;29:701–704.8. Mittman B. Frontrunners 2016: Internal Medicine Q&A Review: Syllabus Companion for Board Review/Practice Questions & Answers for the ABIM Exam. Aliso Viejo, CA: Frontrunners Publishing; 2016.9. Fischer C. Internal Medicine Question Book: Second Edition: Complete Preparation for the American Board of Internal Medicine Exam. New York, NY: Kaplan Publishing; 2009.

Introduction Documentary film has grown significantly in the past decade, with high profile films such as Fahrenheit 9/11, Supersize Me, and An Inconvenient Truth garnering increased attention both at the box office and in the news media. In addition, the rising prominence of web-based media has provided new opportunities for documentary to create social impact. Films are now typically released with websites, Facebook pages, twitter feeds, and web videos to increase both reach and impact. This combination of technology and broader audience appeal has given rise to a current landscape in which documentary films are imbedded within coordinated multi-media campaigns. New media have not only opened up new avenues for communicating with audiences, they have also created new opportunities for data collection and analysis of film impacts. A recent report by McKinsey and Company highlighted this potential, introducing and discussing the implications of increasing consumer information being recorded on the Internet as well as through networked sensors in the physical world. As they found: "Big data—large pools of data that can be captured, communicated, aggregated, stored, and analyzed—is now part of every sector and function of the global economy" (Manyika et al. iv). This data can be mined to learn a great deal about both individual and cultural response to documentary films and the issues they represent. Although film has a rich history in humanities research, this new set of tools enables an empirical approach grounded in the social sciences. However, several researchers across disciplines have noted that limited investigation has been conducted in this area. Although there has always been an emphasis on social impact in film and many filmmakers and scholars have made legitimate (and possibly illegitimate) claims of impact, few have attempted to empirically justify these claims. Over fifteen years ago, noted film scholar Brian Winston commented that "the underlying assumption of most social documentaries—that they shall act as agents of reform and change—is almost never demonstrated" (236). A decade later, Political Scientist David Whiteman repeated this sentiment, arguing that, "despite widespread speculation about the impact of documentaries, the topic has received relatively little systematic attention" ("Evolving"). And earlier this year, the introduction to a special issue of Mass Communication and Society on documentary film stated, "documentary film, despite its growing influence and many impacts, has mostly been overlooked by social scientists studying the media and communication" (Nisbet and Aufderheide 451). Film has been studied extensively as entertainment, as narrative, and as cultural event, but the study of film as an agent of social change is still in its infancy. This paper introduces a systematic approach to measuring the social impact of documentary film aiming to: (1) discuss the context of documentary film and its potential impact; and (2) argue for a social science approach, discussing key issues about conducting such research. Changes in Documentary Practice Documentary film has been used as a tool for promoting social change throughout its history. John Grierson, who coined the term "documentary" in 1926, believed it could be used to influence the ideas and actions of people in ways once reserved for church and school. He presented his thoughts on this emerging genre in his 1932 essay, First Principles of Documentary, saying, "We believe that the cinema's capacity for getting around, for observing and selecting from life itself, can be exploited in a new and vital art form" (97). Richard Barsam further specified the definition of documentary, distinguishing it from non-fiction film, such that all documentaries are non-fiction films but not all non-fiction films are documentaries. He distinguishes documentary from other forms of non-fiction film (i.e. travel films, educational films, newsreels) by its purpose; it is a film with an opinion and a specific message that aims to persuade or influence the audience. And Bill Nichols writes that the definition of documentary may even expand beyond the film itself, defining it as a "filmmaking practice, a cinematic tradition, and mode of audience reception" (12). Documentary film has undergone many significant changes since its inception, from the heavily staged romanticism movement of the 1920s to the propagandist tradition of governments using film to persuade individuals to support national agendas to the introduction of cinéma vérité in the 1960s and historical documentary in the 1980s (cf. Barnouw). However, the recent upsurge in popularity of documentary media, combined with technological advances of internet and computers have opened up a whole new set of opportunities for film to serve as both art and agent for social change. One such opportunity is in the creation of film-based social action campaigns. Over the past decade, filmmakers have taken a more active role in promoting social change by coordinating film releases with action campaigns. Companies such as Participant Media (An Inconvenient Truth, Food Inc., etc.) now create "specific social action campaigns for each film and documentary designed to give a voice to issues that resonate in the films" (Participant Media). In addition, a new sector of "social media" consultants are now offering services, including "consultation, strategic planning for alternative distribution, website and social media development, and complete campaign management services to filmmakers to ensure the content of nonfiction media truly meets the intention for change" (Working Films). The emergence of new forms of media and technology are changing our conceptions of both documentary film and social action. Technologies such as podcasts, video blogs, internet radio, social media and network applications, and collaborative web editing "both unsettle and extend concepts and assumptions at the heart of 'documentary' as a practice and as an idea" (Ellsworth). In the past decade, we have seen new forms of documentary creation, distribution, marketing, and engagement. Likewise, film campaigns are utilizing a broad array of strategies to engage audience members, including "action kits, screening programs, educational curriculums and classes, house parties, seminars, panels" that often turn into "ongoing 'legacy' programs that are updated and revised to continue beyond the film's domestic and international theatrical, DVD and television windows" (Participant Media). This move towards multi-media documentary film is becoming not only commonplace, but expected as a part of filmmaking. NYU film professor and documentary film pioneer George Stoney recently noted, "50 percent of the documentary filmmaker's job is making the movie, and 50 percent is figuring out what its impact can be and how it can move audiences to action" (qtd. in Nisbet, "Gasland"). In his book Convergence Culture, Henry Jenkins, coined the term "transmedia storytelling", which he later defined as "a process where integral elements of a fiction get dispersed systematically across multiple delivery channels for the purpose of creating a unified and coordinated entertainment experience" ("Transmedia"). When applied to documentary film, it is the elements of the "issue" raised by the film that get dispersed across these channels, coordinating, not just an entertainment experience, but a social action campaign. Dimensions of Evaluation It is not unreasonable to assume that such film campaigns, just like any policy or program, have the possibility to influence viewers' knowledge, attitudes, and behavior. Measuring this impact has become increasingly important, as funders of documentary and issue-based films want look to understand the "return on investment" of films in terms of social impact so that they can compare them with other projects, including non-media, direct service projects. Although we "feel" like films make a difference to the individuals who also see them in the broader cultures in which they are embedded, measurement and empirical analysis of this impact are vitally important for both providing feedback to filmmakers and funders as well as informing future efforts attempting to leverage film for social change. This type of systematic assessment, or program evaluation, is often discussed in terms of two primary goals—formative (or process) and summative (or impact) evaluation (cf. Muraskin; Trochim and Donnelly). Formative evaluation studies program materials and activities to strengthen a program, and summative evaluation examines program outcomes. In terms of documentary film, these two goals can be described as follows: Formative Evaluation: Informing the Process As programs (broadly defined as an intentional set of activities with the aim of having some specific impact), the people who interact with them, and the cultures they are situated in are constantly changing, program development and evaluation is an ongoing learning cycle. Film campaigns, which are an intentional set of activities with the aim of impacting individual viewers and broader cultures, fit squarely within this purview. Without formulating hypotheses about the relationships between program activities and goals and then collecting and analyzing data during implementation to test them, it is difficult to learn ways to improve programs (or continue doing what works best in the most efficient manner). Attention to this process enables those involved to learn more about, not only what works, but how and why it works and even gain insights about how program outcomes may be affected by changes to resource availability, potential audiences, or infrastructure. Filmmakers are constantly learning and honing their craft and realizing the impact of their practice can help the artistic process. Often faced with tight budgets and timelines, they are forced to confront tradeoffs all the time, in the writing, production and post-production process. Understanding where they are having impact can improve their decision-making, which can help both the individual project and the overall field. Summative Evaluation: Quantifying Impacts Evaluation is used in many different fields to determine whether programs are achieving their intended goals and objectives. It became popular in the 1960s as a way of understanding the impact of the Great Society programs and has continued to grow since that time (Madaus and Stufflebeam). A recent White House memo stated that "rigorous, independent program evaluations can be a key resource in determining whether government programs are achieving their intended outcomes as well as possible and at the lowest possible cost" and the United States Office of Management and Budget (OMB) launched an initiative to increase the practice of "impact evaluations, or evaluations aimed at determining the causal effects of programs" (Orszag 1). Documentary films, like government programs, generally target a national audience, aim to serve a social purpose, and often do not provide a return on their investment. Participant Media, the most visible and arguably most successful documentary production company in the film industry, made recent headlines for its difficulty in making a profit during its seven-year history (Cieply). Owner and founder Jeff Skoll reported investing hundreds of millions of dollars into the company and CEO James Berk added that the company sometimes measures success, not by profit, but by "whether Mr. Skoll could have exerted more impact simply by spending his money philanthropically" (Cieply). Because of this, documentary projects often rely on grant funding, and are starting to approach funders beyond traditional arts and media sources. "Filmmakers are finding new fiscal and non-fiscal partners, in constituencies that would not traditionally be considered—or consider themselves—media funders or partners" (BRITDOC 6). And funders increasingly expect tangible data about their return on investment. Says Luis Ubiñas, president of Ford Foundation, which recently launched the Just Films Initiative: In these times of global economic uncertainty, with increasing demand for limited philanthropic dollars, assessing our effectiveness is more important than ever. Today, staying on the frontlines of social change means gauging, with thoughtfulness and rigor, the immediate and distant outcomes of our funding. Establishing the need for evaluation is not enough—attention to methodology is also critical. Valid research methodology is a critical component of understanding around the role entertainment can play in impacting social and environmental issues. The following issues are vital to measuring impact. Defining the Project Though this may seem like an obvious step, it is essential to determine the nature of the project so one can create research questions and hypotheses based on a complete understanding of the "treatment". One organization that provides a great example of the integration of documentary film imbedded into a larger campaign or movement is Invisible Children. Founded in 2005, Invisible Children is both a media-based organization as well as an economic development NGO with the goal of raising awareness and meeting the needs of child soldiers and other youth suffering as a result of the ongoing war in northern Uganda. Although Invisible Children began as a documentary film, it has grown into a large non-profit organization with an operating budget of over $8 million and a staff of over a hundred employees and interns throughout the year as well as volunteers in all 50 states and several countries. Invisible Children programming includes films, events, fundraising campaigns, contests, social media platforms, blogs, videos, two national "tours" per year, merchandise, and even a 650-person three-day youth summit in August 2011 called The Fourth Estate. Individually, each of these components might lead to specific outcomes; collectively, they might lead to others. In order to properly assess impacts of the film "project", it is important to take all of these components into consideration and think about who they may impact and how. This informs the research questions, hypotheses, and methods used in evaluation. Film campaigns may even include partnerships with existing social movements and non-profit organizations targeting social change. The American University Center for Social Media concluded in a case study of three issue-based documentary film campaigns: Digital technologies do not replace, but are closely entwined with, longstanding on-the-ground activities of stakeholders and citizens working for social change. Projects like these forge new tools, pipelines, and circuits of circulation in a multiplatform media environment. They help to create sustainable network infrastructures for participatory public media that extend from local communities to transnational circuits and from grassroots communities to policy makers. (Abrash) Expanding the Focus of Impact beyond the Individual A recent focus has shifted the dialogue on film impact. Whiteman ("Theaters") argues that traditional metrics of film "success" tend to focus on studio economic indicators that are far more relevant to large budget films. Current efforts focused on box office receipts and audience size, the author claims, are really measures of successful film marketing or promotion, missing the mark when it comes to understanding social impact. He instead stresses the importance of developing a more comprehensive model. His "coalition model" broadens the range and types of impact of film beyond traditional metrics to include the entire filmmaking process, from production to distribution. Whiteman (“Theaters”) argues that a narrow focus on the size of the audience for a film, its box office receipts, and viewers' attitudes does not incorporate the potential reach of a documentary film. Impacts within the coalition model include both individual and policy levels. Individual impacts (with an emphasis on activist groups) include educating members, mobilizing for action, and raising group status; policy includes altering both agenda for and the substance of policy deliberations. The Fledgling Fund (Barrett and Leddy) expanded on this concept and identified five distinct impacts of documentary film campaigns. These potential impacts expand from individual viewers to groups, movements, and eventually to what they call the "ultimate goal" of social change. Each is introduced briefly below. Quality Film. The film itself can be presented as a quality film or media project, creating enjoyment or evoking emotion in the part of audiences. "By this we mean a film that has a compelling narrative that draws viewers in and can engage them in the issue and illustrate complex problems in ways that statistics cannot" (Barrett and Leddy, 6). Public Awareness. Film can increase public awareness by bringing light to issues and stories that may have otherwise been unknown or not often thought about. This is the level of impact that has received the most attention, as films are often discussed in terms of their "educational" value. "A project's ability to raise awareness around a particular issue, since awareness is a critical building block for both individual change and broader social change" (Barrett and Leddy, 6). Public Engagement. Impact, however, need not stop at simply raising public awareness. Engagement "indicates a shift from simply being aware of an issue to acting on this awareness. Were a film and its outreach campaign able to provide an answer to the question 'What can I do?' and more importantly mobilize that individual to act?" (Barrett and Leddy, 7). This is where an associated film campaign becomes increasingly important, as transmedia outlets such as Facebook, websites, blogs, etc. can build off the interest and awareness developed through watching a film and provide outlets for viewers channel their constructive efforts. Social Movement. In addition to impacts on individuals, films can also serve to mobilize groups focused on a particular problem. The filmmaker can create a campaign around the film to promote its goals and/or work with existing groups focused on a particular issue, so that the film can be used as a tool for mobilization and collaboration. "Moving beyond measures of impact as they relate to individual awareness and engagement, we look at the project's impact as it relates to the broader social movement … if a project can strengthen the work of key advocacy organizations that have strong commitment to the issues raised in the film" (Barrett and Leddy, 7). Social Change. The final level of impact and "ultimate goal" of an issue-based film is long-term and systemic social change. "While we understand that realizing social change is often a long and complex process, we do believe it is possible and that for some projects and issues there are key indicators of success" (Barrett and Leddy, 7). This can take the form of policy or legislative change, passed through film-based lobbying efforts, or shifts in public dialogue and behavior. Legislative change typically takes place beyond the social movement stage, when there is enough support to pressure legislators to change or create policy. Film-inspired activism has been seen in issues ranging from environmental causes such as agriculture (Food Inc.) and toxic products (Blue Vinyl) to social causes such as foreign conflict (Invisible Children) and education (Waiting for Superman). Documentary films can also have a strong influence as media agenda-setters, as films provide dramatic "news pegs" for journalists seeking to either sustain or generation new coverage of an issue (Nisbet "Introduction" 5), such as the media coverage of climate change in conjunction with An Inconvenient Truth. Barrett and Leddy, however, note that not all films target all five impacts and that different films may lead to different impacts. "In some cases we could look to key legislative or policy changes that were driven by, or at least supported by the project... In other cases, we can point to shifts in public dialogue and how issues are framed and discussed" (7). It is possible that specific film and/or campaign characteristics may lead to different impacts; this is a nascent area for research and one with great promise for both practical and theoretical utility. Innovations in Tools and Methods Finally, the selection of tools is a vital component for assessing impact and the new media landscape is enabling innovations in the methods and strategies for program evaluation. Whereas the traditional domain of film impact measurement included box office statistics, focus groups, and exit surveys, innovations in data collection and analysis have expanded the reach of what questions we can ask and how we are able to answer them. For example, press coverage can assist in understanding and measuring the increase in awareness about an issue post-release. Looking directly at web-traffic changes "enables the creation of an information-seeking curve that can define the parameters of a teachable moment" (Hart and Leiserowitz 360). Audience reception can be measured, not only via interviews and focus groups, but also through content and sentiment analysis of web content and online analytics. "Sophisticated analytics can substantially improve decision making, minimize risks, and unearth valuable insights that would otherwise remain hidden" (Manyika et al. 5). These new tools are significantly changing evaluation, expanding what we can learn about the social impacts of film through triangulation of self-report data with measurement of actual behavior in virtual environments. Conclusion The changing media landscape both allows and impels evaluation of film impacts on individual viewers and the broader culture in which they are imbedded. Although such analysis may have previously been limited to box office numbers, critics' reviews, and theater exit surveys, the rise of new media provides both the ability to connect filmmakers, activists, and viewers in new ways and the data in which to study the process. This capability, combined with significant growth in the documentary landscape, suggests a great potential for documentary film to contribute to some of our most pressing social and environmental needs. A social scientific approach, that combines empirical analysis with theory applied from basic science, ensures that impact can be measured and leveraged in a way that is useful for both filmmakers as well as funders. In the end, this attention to impact ensures a continued thriving marketplace for issue-based documentary films in our social landscape. References Abrash, Barbara. "Social Issue Documentary: The Evolution of Public Engagement." American University Center for Social Media 21 Apr. 2010. 26 Sep. 2011 ‹http://www.centerforsocialmedia.org/›. Aufderheide, Patricia. "The Changing Documentary Marketplace." Cineaste 30.3 (2005): 24-28. Barnouw, Eric. Documentary: A History of the Non-Fiction Film. New York: Oxford UP, 1993. Barrett, Diana and Sheila Leddy. "Assessing Creative Media's Social Impact." The Fledgling Fund, Dec. 2008. 15 Sep. 2011 ‹http://www.thefledglingfund.org/media/research.html›. Barsam, Richard M. Nonfiction Film: A Critical History. Bloomington: Indiana UP. 1992. BRITDOC Foundation. The End of the Line: A Social Impact Evaluation. London: Channel 4, 2011. 12 Oct. 2011 ‹http://britdoc.org/news_details/the_social_impact_of_the_end_of_the_line/›. Cieply, Michael. "Uneven Growth for Film Studio with a Message." New York Times 5 Jun. 2011: B1. Ellsworth, Elizabeth. "Emerging Media and Documentary Practice." The New School Graduate Program in International Affairs. Aug. 2008. 22 Sep. 2011. ‹http://www.gpia.info/node/911›. Grierson, John. "First Principles of Documentary (1932)." Imagining Reality: The Faber Book of Documentary. Eds. Kevin Macdonald and Mark Cousins. London: Faber and Faber, 1996. 97-102. Hart, Philip Solomon and Anthony Leiserowitz. "Finding the Teachable Moment: An Analysis of Information-Seeking Behavior on Global Warming Related Websites during the Release of The Day After Tomorrow." Environmental Communication: A Journal of Nature and Culture 3.3 (2009): 355-66. Jenkins, Henry. Convergence Culture: Where Old and New Media Collide. New York: New York UP, 2006. ———. "Transmedia Storytelling 101." Confessions of an Aca-Fan. The Official Weblog of Henry Jenkins. 22 Mar. 2007. 10 Oct. 2011 ‹http://www.henryjenkins.org/2007/03/transmedia_storytelling_101.html›. Madaus, George, and Daniel Stufflebeam. "Program Evaluation: A Historical Overview." Evaluation in Education and Human Services 49.1 (2002): 3-18. Manyika, James, Michael Chui, Jacques Bughin, Brad Brown, Richard Dobbs, Charles Roxburgh, and Angela Hung Byers. Big Data: The Next Frontier for Innovation, Competition, and Productivity. McKinsey Global Institute. May 2011 ‹http://www.mckinsey.com/mgi/publications/big_data/›. Muraskin, Lana. Understanding Evaluation: The Way to Better Prevention Programs. Washington: U.S. Department of Education, 1993. 8 Oct. 2011 ‹http://www2.ed.gov/PDFDocs/handbook.pdf›. Nichols, Bill. "Foreword." Documenting the Documentary: Close Readings of Documentary Film and Video. Eds. Barry Keith Grant and Jeannette Sloniowski. Detroit: Wayne State UP, 1997. 11-13. Nisbet, Matthew. "Gasland and Dirty Business: Documentary Films Shape Debate on Energy Policy." Big Think, 9 May 2011. 1 Oct. 2011 ‹http://bigthink.com/ideas/38345›. ———. "Introduction: Understanding the Social Impact of a Documentary Film." Documentaries on a Mission: How Nonprofits Are Making Movies for Public Engagement. Ed. Karen Hirsch, Center for Social Media. Mar. 2007. 10 Sep. 2011 ‹http://aladinrc.wrlc.org/bitstream/1961/4634/1/docs_on_a_mission.pdf›. Nisbet, Matthew, and Patricia Aufderheide. "Documentary Film: Towards a Research Agenda on Forms, Functions, and Impacts." Mass Communication and Society 12.4 (2011): 450-56. Orszag, Peter. Increased Emphasis on Program Evaluation. Washington: Office of Management and Budget. 7 Oct. 2009. 10 Oct. 2011 ‹http://www.whitehouse.gov/sites/default/files/omb/assets/memoranda_2010/m10-01.pdf›. Participant Media. "Our Mission." 2011. 2 Apr. 2011 ‹http://www.participantmedia.com/company/about_us.php.›. Plantinga, Carl. Rhetoric and Representation in Nonfiction Film. Cambridge: Cambridge UP, 1997. Trochim, William, and James Donnelly. Research Methods Knowledge Base. 3rd ed. Mason: Atomic Dogs, 2007. Ubiñas, Luis. "President's Message." 2009 Annual Report. Ford Foundation, Sep. 2010. 10 Oct. 2011 ‹http://www.fordfoundation.org/about-us/2009-annual-report/presidents-message›. Vladica, Florin, and Charles Davis. "Business Innovation and New Media Practices in Documentary Film Production and Distribution: Conceptual Framework and Review of Evidence." The Media as a Driver of the Information Society. Eds. Ed Albarran, Paulo Faustino, and R. Santos. Lisbon, Portugal: Media XXI / Formal, 2009. 299-319. Whiteman, David. "Out of the Theaters and into the Streets: A Coalition Model of the Political Impact of Documentary Film and Video." Political Communication 21.1 (2004): 51-69. ———. "The Evolving Impact of Documentary Film: Sacrifice and the Rise of Issue-Centered Outreach." Post Script 22 Jun. 2007. 10 Sep. 2011 ‹http://www.allbusiness.com/media-telecommunications/movies-sound-recording/5517496-1.html›. Winston, Brian. Claiming the Real: The Documentary Film Revisited. London: British Film Institute, 1995. Working Films. "Nonprofits: Working Films." Foundation Source Access 31 May 2011. 5 Oct. 2011 ‹http://access.foundationsource.com/nonprofit/working-films/›.

Introduction: Game Culture and GenderAs texts with the potential to help mediate specific forms of identity, video games are rich and complex sites for analysis. A tendency, however, still exists in scholarship to treat video games as just another kind of text, and work that explores the expression of masculine identity persists in drawing from cinematic analysis without proper consideration of game design and how these games are played (Triana). For example, insights from studies into horror cinema may illuminate the relationship between players and game systems in survival horror video games (Habel & Kooyman), but further study is needed to explore how people interact with the game.This article aims to build towards a scholarly definition of the term “Power Fantasy”, a concept that seems well established in wider discourse but is not yet well theorised in the scholarly literature. It does so through a case of the most recent reboot of Doom (2016), a game that in its original incarnation established an enduring tradition for high-action Power Fantasy. In the first-person shooter game Doom, the player fills the role of the “Doom Guy”, a faceless hero who shuttles between Earth and Hell with the sole aim of eviscerating demonic hordes as graphically as possible.How, then, do we begin to theorise the kind of automediation that an iconic game text like Doom facilitates? Substantial work has been done to explore player identification in online games (see Taylor; Yee). Shaw (“Rethinking”) suggests that single-player games are unexplored territory compared to the more social spaces of Massively Multiplayer Online games and other multiplayer experiences, but it is important to distinguish between direct identification with the avatar per se and the ways in which the game text mediates broader gender constructions.Abstract theorisation is not enough, though. To effectively understand this kind of automediation we also need a methodology to gain insights into its processes. The final part of this article, therefore, proposes the analysis of “Let’s Play” videos as a kind of gender identity performance which gives insight into the automediation of dominant masculine gender identities through Power Fantasy video games like Doom. This reflexive performance works to denaturalise gender construction rather than reinforce stable hegemonic identities.Power Fantasy and Gender IdentityPower Fantasy has become an established trope in online critiques and discussions of popular culture. It can be simply defined as “character imagines himself taking revenge on his bullies” (TVTropes). This trope takes on special resonance in video games, where the players themselves live out the violent revenge fantasy in the world of the game.The “power fantasy” of games implies escapism and meaninglessness, evoking outsize explosions and equally outsized displays of dominance. A “power gamer” is one who plays with a single-minded determination to win, at the expense of nuance, social relationships between players, or even their own pleasure in play. (Baker)Many examples apply this concept of Power Fantasy in video games: from God of War to Metroid: Prime and Grand Theft Auto, this prevalent trope of game design uses a kind of “agency mechanics” (Habel & Kooyman) to convince the player that they are becoming increasingly skilful in the game, when in reality the game is simply decreasing in difficulty (PBS Digital Studios). The operation of the Power Fantasy trope is also gendered; in a related trope known as “I Just Want To Be a Badass”, “males are somewhat more prone to harbour [the] wish” to feel powerful (TVTropes). More broadly, even though the game world is obviously not real, playing it requires “an investment in and commitment to a type of masculine performance that is based on the Real (particularly if one is interested in ‘winning’, pummelling your opponent, kicking ass, etc.)” (Burrill 2).Indeed, there is a perceived correlation (if not causation) between the widespread presence of Power Fantasy video games and how “game culture as it stands is shot through with sexism, racism, homophobia, and other biases” (Baker). Golding and van Deventer undertake an extended exploration of this disconcerting side of game culture, concluding that games have “become a venue for some of the more unsophisticated forms of patriarchy” (213) evidenced in the highly-publicised GamerGate movement. This saw an alignment between the label of “gamer” and extreme misogyny, abuse and harassment of women and other minorities in the industry.We have, then, a tentative connection between dominant gameplay forms based on high skill that may be loosely characterised as “Power Fantasy” and some of the most virulent toxic gender expressions seen in recent times. More research is needed to gain a clearer understanding of precisely what Power Fantasy is. Baker’s primary argument is that “power” in games can also be characterised as “power to” or “power with”, as well as the more traditional “power over”. Kurt Squire uses the phrase “Power Fantasy” as a castaway framing for a player who seeks an alternative reward to the usual game progression in Sid Meier’s Civilisation. More broadly, much scholarly work concerning gameplay design and gender identity has been focussed on the hot-button question of videogame violence and its connection to real-world violence, a question that this article avoids since it is well covered elsewhere. Here, a better understanding of the mediation of gender identity through Power Fantasy in Doom can help to illuminate how games function as automedia.Auto-Mediating Gender through Performance in Doom (2016) As a franchise, Doom commands near-incomparable respect as a seminal text of the first-person shooter genre. First released in 1993, it set the benchmark for 3D rendered graphics, energetic sound design, and high-paced action gameplay that was visceral and deeply immersive. It is impossible to mention more recent reboots without recourse to its first seminal instalment and related game texts: Kim Justice suggests a personal identification with it in a 29-minute video analysis entitled “A Personal History of Demon Slaughter”. Doom is a cherished game for many players, possibly because it evokes memories of “boyhood” gaming and all its attendant gender identity formation (Burrill).This identification also arises in livestreams and playthroughs of the game. YouTuber and game reviewer Markiplier describes nostalgically and at lengths his formative experiences playing it (and recounts a telling connection with his father who, he explains, introduced him to gaming), saying “Doom is very important to me […] this was the first game that I sat down and played over and over and over again.” In contrast, Wanderbots confesses that he has never really played Doom, but acknowledges its prominent position in the gaming community by designating himself outside the identifying category of “Doom fan”. He states that he has started playing due to “gushing” recommendations from other gamers. The nostalgic personal connection is important, even in absentia.For the most part, the critical and community response to the 2016 version of Doom was approving: Gilroy admits that it “hit all the right power chords”, raising the signature trope in reference to both gameplay and music (a power chord is a particular technique of playing heavy metal guitar often used in heavy metal music). Doom’s Metacritic score is currently a respectable 85, and, the reception is remarkably consistent between critics and players, especially for such a potentially divisive game (Metacritic). Commentators tend to cite its focus on its high action, mobility, immersion, sound design, and general faithfulness to the spirit of the original Doom as reasons for assessments such as “favourite game ever” (Habel). Game critic Yahtzee’s uncharacteristically approving video review in the iconic Zero Punctuation series is very telling in its assessment of the game’s light narrative framing:Doom seems to have a firm understanding of its audience because, while there is a plot going on, the player-character couldn’t give a half an ounce of deep fried shit; if you want to know the plot then pause the game and read all the fluff text in the character and location database, sipping daintily from your pink teacup full of pussy juice, while the game waits patiently for you to strap your bollocks back on and get back in the fray. (Yahtzee)This is a strident expression of the gendered expectations and response to Doom’s narratological refusal, which is here cast as approvingly masculine and opposed to a “feminine” desire for plot or narrative. It also feeds into a discourse which sees the game as one which demands skill, commitment, and an achievement orientation cast within an exclusivist ideology of “toxic meritocracy” (Paul).In addition to examining reception, approaches to understanding how Doom functions as a “Power Fantasy” or “badass” trope could take a variety of forms. It is tempting to undertake a detailed analysis of its design and gameplay, especially since these feed directly into considerations of player interaction. This could direct a critical focus towards gameplay design elements such as traversal and mobility, difficulty settings, “glory kills”, and cinematic techniques in the same vein as Habel and Kooyman’s analysis of survival horror video games in relation to horror cinema. However, Golding and van Deventer warn against a simplistic analysis of decontextualized gameplay (29-30), and there is a much more intriguing possibility hinted at by Harper’s notion of “Play Practice”.It is useful to analyse a theoretical engagement with a video game as a thought-experiment. But with the rise of gaming as spectacle, and particularly gaming as performance through “Let’s Play” livestreams (or video on demand) on platforms such as TwitchTV and YouTube, it becomes possible to analyse embodied performances of the gameplay of such video games. This kind of analysis allows the opportunity for a more nuanced understanding of how such games mediate gender identities. For Judith Butler, gender is not only performed, it is also performative:Because there is neither an “essence” that gender expresses or externalizes nor an objective ideal to which gender aspires, and because gender is not a fact, the various acts of gender create the idea of gender, and without those acts, there would be no gender at all. (214)Let’s Play videos—that show a player playing a game in real time with their commentary overlaying the on screen action—allow us to see the performative aspects of gameplay. Let’s Plays are a highly popular and developing form: they are not simple artefacts by any means, and can be understood as expressive works in their own right (Lee). They are complex and multifaceted, and while they do not necessarily provide direct insights into the player’s perception of their own identification, with sufficient analysis and unpacking they help us to explore both the construction and denaturalisation of gender identity. In this case, we follow Josef Nguyen’s analysis of Let’s Plays as essential for expression of player identity through performance, but instead focus on how some identity construction may narrow rather than expand the diversity range. T.L. Taylor also has a monograph forthcoming in 2018 titled Watch Me Play: Twitch and the Rise of Game Live Streaming, suggesting the time is ripe for such analysis.These performances are clear in ways we have already discussed: for example, both SplatterCat and Markiplier devote significant time to describing their formative experiences playing Doom as a background to their gameplay performance, while Wanderbots is more distanced. There is no doubt that these videos are popular: Markiplier, for example, has attracted nearly 5 million views of his Doom playthrough. If we see gameplay as automediation, though, these videos become useful artefacts for analysis of gendered performance through gameplay.When SplatterCat discovers the suit of armor for the game’s protagonist, Doom Guy, he half-jokingly remarks “let us be all of the Doom Guy that we can possibly be” (3:20). This is an aspirational mantra, a desire for enacting the game’s Power Fantasy. Markiplier speaks at length about his nostalgia for the game, specifically about how his father introduced him to Doom when he was a child, and he expresses hopes that he will again experience “Doom’s original super-fast pace and just pure unadulterated action; Doom Guy is a badass” (4:59). As the action picks up early into the game, Markiplier expresses the exhilaration and adrenaline that accompany performances of this fastpaced, highly mobile kind of gameplay, implying that he is becoming immersed in the character and, by performing Doom Guy, inhabiting the “badass” role and thus enacting a performance of Power Fantasy:Doom guy—and I hope I’m playing Doom guy himself—is just the embodiment of kickass. He destroys everything and he doesn’t give a fuck about what he breaks in the process. (8:45)This performance of gender through the skilled control of Doom Guy is, initially, unambiguously mediated as Power Fantasy: in control, highly skilled, suffused with Paul’s ‘toxic meritocracy. A similar sentiment is expressed in Wanderbots’ playthrough when the player-character dispenses with narrative/conversation by smashing a computer terminal: “Oh I like this guy already! Alright. Doom Guy does not give a shit. It’s like Wolf Blascowicz [sic], but like, plus plus” (Wanderbots). This is a reference to another iconic first-person shooter franchise, Wolfenstein, which also originated in the 1980s and has experienced a recent successful reboot, and which operates in a similar Power Fantasy mode. This close alignment between these two streamers’ performances suggests significant coherence in both genre and gameplay design and the ways in which players engage with the game as a gendered performative space.Nonetheless, there is no simple one-to-one relationship here—there is not enough evidence to argue that this kind of gameplay experience leads directly to the kind of untrammelled misogyny we see in game culture more broadly. While Gabbiadini et al. found evidence in an experimental study that a masculinist ideology combined with violent video game mechanics could lead to a lack of empathy for women and girls who are victims of violence, Ferguson and Donnellan dispute this finding based on poor methodology, arguing that there is no evidence for a causal relationship between gender, game type and lack of empathy for women and girls. This inconclusiveness in the research is mirrored by an ambiguity in the gendered performance of males playing through Doom, where the Power Fantasy is profoundly undercut in multiple ways.Wanderbots’ Doom playthrough is literally titled ‘I have no idea what I’m Dooming’ and he struggles with particular mechanics and relatively simple progression tools early in the game: this reads against masculinist stereotypes of superior and naturalised gameplay skill. Markiplier’s performance of the “badass” Doom Guy is undercut at various stages: in encountering the iconic challenge of the game, he mentions that “I am halfway decent… not that good at video games” (9:58), and on the verge of the protagonist’s death he admits “If I die this early into my first video I’m going to be very disappointed, so I’m going to have to kick it up a notch” (15:30). This suggests that rather than being an unproblematic and simple expression of male power in a fantasy video game world, the gameplay performances of Power Fantasy games are ambiguous and contested, and not always successfully performed via the avatar. They therefore demonstrate a “kind of gender performance [which] will enact and reveal the performativity of gender itself in a way that destabilizes the naturalized categories of identity and desire” (Butler 211). This cuts across the empowered performance of videogame mastery and physical dominance over the game world, and suggests that the automediation of gender identity through playing video games is a complex phenomenon urgently in need of further theorisation.ConclusionUltimately, this kind of analysis of the mediation of hegemonic gender identities is urgent for a cultural product as ubiquitous as video games. The hyper-empowered “badass” digital avatars of Power Fantasy video games can be expected to have some shaping effect on the identities of those who play them, evidenced by the gendered gameplay performance of Doom briefly explored here. This is by no means a simple or unproblematic process, though. Much further research is needed to test the methodological insights possible by using video performances of gameplay as explorations of the auto-mediation of gender identities through video games.ReferencesBaker, Meguey. “Problematizing Power Fantasy.” The Enemy 1.2 (2015). 18 Feb. 2018 <http://theenemyreader.org/problematizing-power-fantasy/>.Burrill, Derrick. Die Tryin’: Video Games, Masculinity, Culture. New York: Peter Lang, 2008.Butler, Judith. Gender Trouble. 10th ed. London: Routledge, 2002.Ferguson, Christopher, and Brent Donellan. “Are Associations between “Sexist” Video Games and Decreased Empathy toward Women Robust? A Reanalysis of Gabbiadini et al. 2016.” Journal of Youth and Adolescence 46.12 (2017): 2446–2459.Gabbiadini, Alessandro, Paolo Riva, Luca Andrighetto, Chiara Volpato, and Brad J. Bushman. “Acting like a Tough Guy: Violent-Sexist Video Games, Identification with Game Characters, Masculine Beliefs, & Empathy for Female Violence Victims.” PloS One 11.4 (2016). 14 Apr. 2018 <http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0152121>.Gilroy, Joab. “Doom: Review.” IGN, 16 May 2016. 22 Feb. 2018 <http://au.ign.com/articles/2016/05/16/doom-review-2?page=1>.Golding, Dan, and Leena van Deventer. Game Changers: From Minecraft to Misogyny, the Fight for the Future of Videogames. South Melbourne: Affirm, 2016.Habel, Chad, and Ben Kooyman. “Agency Mechanics: Gameplay Design in Survival Horror Video Games”. Digital Creativity 25.1 (2014):1-14.Habel, Chad. “Doom: Review (PS4).” Game Truck Australia, 2017. 22 Feb. 2018 <http://www.gametruckaustralia.com.au/review-doom-2016-ps4/>.Harper, Todd. The Culture of Digital Fighting Games: Performance and Practice. London: Routledge, 2013.Kim Justice. “Doom: A Personal History of Demon Slaughter.” YouTube, 16 Jan. 2017. 22 Feb. 2018 <https://www.youtube.com/watch?v=JtvoENhvkys>.Lee, Patrick. “The Best Let’s Play Videos Offer More than Vicarious Playthroughs.” The A.V. Club, 24 Apr. 2015. 22 Feb. 2018 <https://games.avclub.com/the-best-let-s-play-videos-offer-more-than-vicarious-pl-1798279027>.Markiplier. “KNEE-DEEP IN THE DEAD | DOOM – Part 1.” YouTube, 13 May 2016. 21 Feb. 2018 <https://www.youtube.com/watch?v=pCygvprsgIk>.Metacritic. “Doom (PS4).” 22 Feb. 2018 <http://www.metacritic.com/game/playstation-4/doom>.Nguyen, Josef. “Performing as Video Game Players in Let’s Plays.” Transformative Works and Cultures 22 (2016). <http://journal.transformativeworks.org/index.php/twc/article/view/698>.Paul, Christopher. The Toxic Meritocracy of Video Games: Why Gaming Culture Is the Worst. Minneapolis: U of Minnesota P, 2018.PBS Digital Studios. “Do Games Give Us Too Much Power?” 2017. 18 Feb. 2018 <https://www.youtube.com/watch?v=9COt-_3C0xI>.Shaw, Adrienne. “Do You Identify as a Gamer? Gender, Race, Sexuality, and Gamer Identity.” New Media and Society 14.1 (2012): 28-44.———. “Rethinking Game Studies: A Case Study Approach to Video Game Play and Identification.” Critical Studies in Media Communication 30.5 (2013): 347-361.SplatterCatGaming. “DOOM 2016 PC – Gameplay Intro – #01 Let's Play DOOM 2016 Gameplay.” YouTube, 13 May 2016. 21 Feb. 2018 <https://www.youtube.com/watch?v=tusgsunWEIs>.Squire, Kurt. “Open-Ended Video Games: A Model for Developing Learning for the Interactive Age.” The Ecology of Games: Connecting Youth, Games, and Learning. Ed. Katie Salen. Cambridge, MA: MIT P, 2008. 167–198.Boellstorff, Tom, Bonnie Nardi, Celia Pearce, and T.L. Taylor. Ethnography and Virtual Worlds: A Handbook of Method. Princeton, NJ: Princeton UP, 2012.Triana, Benjamin. “Red Dead Maculinity: Constructing a Conceptual Framework for Analysing the Narrative and Message Found in Video Games.” Journal of Games Criticism 2.2 (2015). 12 Apr. 2018 <http://gamescriticism.org/articles/triana-2-2/>.TVTropes. Playing With / Power Fantasy. 18 Feb. 2018 <http://tvtropes.org/pmwiki/pmwiki.php/PlayingWith/PowerFantasy>.———. I Just Want to Be a Badass. 18 Feb. 2018 <http://tvtropes.org/pmwiki/pmwiki.php/Main/IJustWantToBeBadass>.Wanderbots. “Let’s Play Doom (2016) – PC Gameplay Part I – I Have No Idea What I’m Dooming!” YouTube, 18 June 2016. 14 Apr. 2018 <https://www.youtube.com/watch?v=AQSMNhWAf0o>.Yee, Nick. “Maps of Digital Desires: Exploring the Topography of Gender and Play in Online Games.” Beyond Barbie and Mortal Kombat: New Perspectives on Gender and Gaming. Eds. Yasmin B. Kafai, Carrie Heeter, Jill Denner, and Jennifer Sun. Cambridge, MA: MIT P, 2008. 83-96.Yahtzee. “Doom (2016) Review.” Zero Punctuation, 8 June 2016. 14 Apr. 2018 <https://www.youtube.com/watch?v=HQGxC8HKCD4>.