The 10 most-cited articles in American Journal of Evaluation, 2016

Evaluators often use qualitative research methods, yet there is little evidence on the comparative cost-effectiveness of the two most commonly employed qualitative methods—in-depth interviews (IDIs) and focus groups (FGs). We performed an inductive thematic analysis of data from 40 IDIs and 40 FGs on the health-seeking behaviors of African American men ( N = 350) in Durham, North Carolina. We used a bootstrap simulation to generate 10,000 random samples from each data set and calculated the number of data collection events necessary to reach different levels of thematic saturation. The median number of data collection events required to reach 80% and 90% saturation was 8 and 16, respectively, for IDIs and 3 and 5 for FGs. Interviews took longer but were more cost-effective at both levels. At the median, IDIs cost 20–36% less to reach thematic saturation. Evaluators can consider these empirically based cost-effectiveness data when selecting a qualitative data collection method.

The first article in this series traces the initial development of the concept of evaluation use. As a field, evaluation has always paid attention to the potential for use, both in decision-making and in changing people’s thinking. The broad history of the field as we know it today stemmed from two streams: one focused on tests and measurement, primarily in education, and a second focused on social research methods, primarily concerning knowledge utilization. Evaluation use had its roots in both streams, resulting in three broad categories for discussing the use of evaluation findings: instrumental use, conceptual use or enlightenment, and symbolic use. The additional category of process use, added years later, highlighted the potential utility of people’s participation in the evaluation process.

Since the early 2000s, a significant number of programs and policies have been developed and implemented to prevent and combat human trafficking. At the international, regional and national levels, government, and international, and nongovernment organizations have established plans of action, conducted training, developed policy tools, and conducted a variety of other activities to counter the phenomenon of trafficking in persons. However, only a small number of these anti–human trafficking interventions have been evaluated and an even fewer number have been evaluated rigorously. This article explores the approaches that have been used to evaluate anti–human trafficking interventions. Through a review of 49 evaluations, the study finds that action is required to increase quality evaluations of anti–human trafficking programs in order to ensure that programs are targeted, implemented, and delivered effectively, and the knowledge on the impact of programs is improved.

Fidelity concerns the extent to which a specific evaluation sufficiently incorporates the core characteristics of the overall approach to justify labeling that evaluation by its designated name. Fidelity has traditionally meant implementing a model in exactly the same way each time following the prescribed steps and procedures. The essential principles of developmental evaluation (DE), in contrast, provide high-inference sensitizing guidance that must be interpreted and applied contextually. In lieu of operationalizing DE fidelity criteria, I suggest addressing the degree of manifest sensitivity to essential principles. Principles as sensitizing concepts replace operational rules. This means that sensitivity to essential DE principles should be explicitly and contextually manifest in both processes and outcomes, in both design and use of findings. Eight essential principles of DE are identified and explained. Finally, 10 threats to evaluation model fidelity and/or degree of manifest sensitivity are identified with ways to mitigate those threats.

The authors of this article each bring a different theoretical background to their evaluation practice. The first author has a background of attention to culturally responsive evaluation (CRE), while the second author has a background of attention to systems theories and their application to evaluation. Both have had their own evolution of thinking and application of their respective conceptual traditions over the last 20+ years, influenced considerably by their involvement in the American Evaluation Association. They recently worked together to build evaluation capacity among evaluators of science, technology, engineering, and mathematics (STEM) education programs, in which they explored how these two conceptual and theoretical paths connect. In this article, the authors present their current thinking about the relationship between CRE and systems-oriented evaluation. In a case example, they illustrate the value of integrating the two perspectives to determine the guiding questions for an evaluation of a STEM education project.

The systematic identification of evaluator competency training needs is crucial for the development of evaluators and the establishment of evaluation as a profession. Insight into essential competencies could help align training programs with field-specific needs, therefore clarifying expectations between evaluators, educators, and employers. This investigation of practicing evaluators’ perceptions of competencies addresses the critical need for a competency training gap analysis. Results from an online survey of 403 respondents and a follow-up focus group indicate that the professional practice and systematic inquiry competencies are seen as most important for conducting successful evaluations. Evaluators identified need for additional training in the interpersonal competence and reflective practice competency domains. Trends identified can support the development and modification of programs designed to offer training, education, and professional development to evaluators.

The search for necessary and sufficient causes of some outcome of interest, referred to as configurational comparative research, has long been one of the main preoccupations of evaluation scholars and practitioners. However, only the last three decades have witnessed the evolution of a set of formal methods that are sufficiently elaborate for this purpose. In this article, I provide a hands-on tutorial for qualitative comparative analysis (QCA)—currently the most popular configurational comparative method. In drawing on a recent evaluation of patient follow-through effectiveness in Lynch syndrome tumor-screening programs, I explain the search target of QCA, introduce its core concepts, guide readers through the procedural protocol of this method, and alert them to mistakes frequently made in QCA’s use. An annotated replication file for the QCApro extension package for R accompanies this tutorial.

Making causal claims is central to evaluation practice because we want to know the effects of a program, project, or policy. In the past decade, the conversation about establishing causal claims has become prominent (and problematic). In response to this changing conversation about causality, we argue that evaluators need to take up some new ways of thinking about and examining causal claims in their practices, including (1) being responsive to the situation and intervention, (2) building relevant and defensible causal arguments, (3) being literate in multiple ways of thinking about causality, (4) being familiar with a range of causal designs and methods, (5) layering theories to explain causality at multiple levels; and (6) justifying the causal approach taken to multiple audiences. Drawing on recent literature, we discuss why and how evaluators can take up each of these ideas in practice. We conclude with considerations for evaluator training and future research.

This article describes the development and use of a rapid evaluation approach to meet program accountability and learning requirements in a research for development program operating in five developing countries. The method identifies clusters of outcomes, both expected and unexpected, happening within areas of change. In a workshop, change agents describe the causal connections within outcome clusters to identify outcome trajectories for subsequent verification. Comparing verified outcome trajectories with existing program theory allows program staff to question underlying causal premises and adapt accordingly. The method can be used for one-off evaluations that seek to understand whether, how, and why program interventions are working. Repeated cycles of outcome evidencing can build a case for program contribution over time that can be evaluated as part of any future impact assessment of the program or parts of it.

Many evaluations of programs tend to show few outcomes. One solution to this has been an increasing prominence of the movement that requires programs to implement evidence-based interventions (EBIs). But in a complex world with complex organizations and complex interventions, many challenges have arisen to the implementation of EBIs with fidelity to achieve outcomes at scale. This includes challenges to achieving outcomes in each setting. In this article, we propose the use of empowerment evaluation and one of its major approaches (Getting To Outcomes [GTO]) as a promising method to address the challenges, and GTO can help organizations achieve outcomes by leading them through a set of “accountability questions” for implementing EBIs in their particular setting. These questions can be asked at multiple levels (e.g., national, state, and local organizations) responsible for achieving outcomes. Although we illustrate the possibilities with examples from health care and public health, the potential strategies can be applied to many areas of health and human services and education.