The 10 most-cited articles in American Journal of Evaluation, 2017

The second article in this series on the history of evaluation use has three sections. The first and longest develops a functional definition of the term use , noting that a thorough definition of evaluation use includes the initial stimulus (i.e., evaluation findings or process), the user, the way people use the information, the aspect of the program considered, and the purpose. It then defines evaluation use’s unethical companion, misuse, detailing the distinction between the two. The second section briefly discusses a broadened concept of evaluation impact that expands to include evaluation influence. Finally, the third section summarizes the factors that research has shown to be related to evaluation use.

Qualitative comparative analysis (QCA) is gaining ground in evaluation circles, but the number of applications is still limited. In this article, we consider the challenges that can emerge during a QCA evaluation by drawing on our experience of conducting one in the field of development cooperation. For each stage of the evaluation process, we systematically discuss the challenges we encountered and suggest solutions on how these can be addressed. We believe that sharing this kind of lessons learned can help evaluators become more familiar with QCA, shedding light on what it is to be expected when considering the application of QCA for an evaluation, at the same time reducing unfounded fears and promoting awareness of traps and requirements. The article can be insightful and potentially inspirational for both commissioners and evaluators.

Evaluation is defined by its central task of valuing—the process and product of judging the merit, worth, or significance of a policy or program. However, there are no clear-cut ways to consider values and render value judgments in evaluation practice. There remains contention in the evaluation field about whether and how to make value judgments. No approach to valuing eliminates the uncertainty, plurality, and potential for conflict that comes with considering values. This article explores what critical systems heuristics (CSH), an area of applied systems thinking, might contribute to four long-standing issues regarding valuing: envisioning the social value of evaluation, framing the evaluand and evaluation, selecting and justifying criteria, and determining the roles of the evaluator(s) and stakeholders in valuing. CSH contributes concepts and tools that, in theory, support more reflective, responsible valuing although further practical application is needed.

Although investigations into evaluation theories, methods, and practices have been occurring since the late 1970s, research on evaluation (RoE) has seemingly increased in the past decade. In this review, 257 studies published in 14 evaluation-focused journals over a 10-year period (between 2005 and 2014) were identified as RoE and then classified according to Henry and Mark’s and Mark’s taxonomies of RoE. The majority of RoE published in evaluation-focused journals consists of descriptive studies and, to a lesser extent, comparative studies of evaluation practices. Few investigations, however, address questions pertaining to values or valuing in evaluation, ethics, or evaluation consequences. Gaps in and an agenda for future RoE are discussed.

approach for the evaluation and planning of international development programs. This approach is considered especially suitable for complex interventions. We question this assumption and argue that ToC's focus on cause-effect logic and intended outcomes does not do justice to the recursive nature of complex interventions such as advocacy. Supported by our work as evaluators, and specifically our case study of an advocacy program on child rights, we illustrate how advocacy evolves through recursive interactions, with outcomes that are emergent rather than predictable. We propose putting "practices of change" at the center by emphasizing human interactions, using the analytical lenses of strategies as practice and recursiveness. This provides room for emergent outcomes and implies a different use of ToC. In this article, we make a clear distinction between theoretical reality models and the real world of practices.

Racial framing can have strong effects on programs, policies, and even evaluations. Racial framing developed as a justification for the exploitation of minorities and has been a primary causal factor in the persistence of racism. By being aware of its pattern, structure, origins, and how racial framing generates effects, we can significantly reduce its influence, thus enhancing the rigor of our studies by controlling for a potential bias that’s often covert. Stories play a critical role in framing and reframing processes. They constitute a key part of the vocabulary of action.

Randomization of large number of participants to different treatment groups is often not a feasible or preferable way to answer questions of immediate interest to professional practice. Single case designs (SCDs) are a class of research designs that are experimental in nature but require only a few participants, all of whom receive the treatment(s) of interest. SCDs are particularly relevant when a dependent variable of interest can be measured repeatedly over time across two conditions (e.g., baseline and intervention). Rather than using randomization of large numbers of participants, SCD researchers use careful and prescribed ordering of experimental conditions, which allow researchers to improve internal validity by ruling out alternative explanations for behavior change. This article describes SCD logic, control of threats to internal validity, the use of randomization and counterbalancing, and data analysis in the context of single case research.

Evidence-based interventions hold promise for reducing gaps in health equity across diverse populations, but evidence about effectiveness within these populations lags behind the mainstream, often leaving opportunities to fulfill this promise unrealized. Mismatch between standard intervention outcomes research methods and the cultural and community contexts of populations at greatest risk presents additional challenges in designing and implementing rigorous studies; these challenges too often impede efforts to generate needed evidence. We draw on experiences with American Indian and Alaska Native (AIAN) communities to illustrate how consideration of culture and context can constructively shape intervention research and improve the quality of evidence produced. Case examples from a partnership with one AI community highlight opportunities for increasing alignment in intervention development, research design, and study implementation to maximize both validity and feasibility. We suggest that responsively tailoring intervention outcome research to cultural and community contexts is…

Programs for domestic violence (DV) victims and their families have grown exponentially over the last four decades. The evidence demonstrating the extent of their effectiveness, however, often has been criticized as stemming from studies lacking scientific rigor. A core reason for this critique is the widespread belief that credible evidence can derive only from research grounded in randomized control trials (RCTs). Although the RCT method has its strengths, we argue that it is rarely an optimal—or even a possible—approach for evaluating multifaceted DV programs. This article reviews the reasons that RCT is a poor fit for such programs and argues that a more inclusive conceptualization of credible evidence is critical to expanding our knowledge base about how DV programs affect survivors’ safety and well-being.

Current discussion of the professionalization of the practice of evaluation is focused on matters of competencies and credentialing and largely overlooks several significant issues. These include conflating professionalizing the practice with promoting the value of evaluation as a social good, assuming there is widespread agreement on what the terms profession and professional mean, and not considering how processes of professionalization are affected by the incorporation of evaluative activities in systems of managerial control. Most glaring is the absence of vigorous discussion of what professionalism means in evaluation and what the profession itself aims to add to society, or the social good it seeks to serve. In a word, there is little discussion of the professional ethos of evaluation.