The 10 most-cited articles in American Journal of Evaluation, 2023

Collective impact (CI) is a structured approach that helps drive multi-sector collaborations to address social problems through systems changes. While the CI approach is gaining popularity, practitioners experience challenges in evaluating its implementation and intended outcomes. We conducted a systematic scoping review to understand evaluation methods specific to CI initiatives, identify challenges or limitations with these evaluations, and provide recommendations for the design of CI evaluations. Eighteen studies met the inclusion criteria. Process evaluations were the most frequently used evaluation design. Most studies collected cross-sectional data to evaluate their efforts. The complexity of CI was most frequently cited as the greatest evaluation challenge. Study recommendations primarily focused on improvements during the evaluation planning phase. Taking careful consideration in the planning of CI evaluations, developing context-specific data collection methods, and communicating results intentionally and effectively could prove useful to sufficiently capture and assess…

There is a lack of instructional literature on how to conduct a feminist evaluation to highlight and transform systemic issues in gendered and intersecting power relations. Feminist Participatory Action Research (FPAR) enables a process for conducting community-driven, -led and -owned feminist evaluations that drive social justice actions. By undertaking a critical review of existing literature, this article presents guiding principles and practices in how to conduct a feminist evaluation using FPAR. These principles and practices provide a framework for those who are seeking an evidence base for transformative social justice action in communities, particularly those who are working with complexity in systems-change interventions with multiple stakeholders.

This paper develops a novel approach called Outcome Trajectory Evaluation (OTE) in response to the long-causal-chain problem confronting the evaluation of research for development (R4D) projects. OTE strives to tackle four issues resulting from the common practice of evaluating R4D projects based on theory of change developed at the start. The approach was developed iteratively while conducting four evaluations of policy-related outcomes claimed by the CGIAR, a global R4D organization. The first step is to use a middle-range theory (MRT), based on “grand” social science theory, to help delineate and understand the trajectory that generated the set of outcomes being evaluated. The second step is to then identify project contribution to that trajectory. Other types of theory-driven evaluation are single step: they model how projects achieve outcomes without first considering the overarching causal mechanism—the outcome trajectory—from which the outcomes emerged. The use of an MRT allowed us to accrue learning from one evaluation to the next.

Recognition of the complexity of the world raises questions about new theories, methods, and capabilities for evaluation practice. Approaches originating in the systems sciences show promise; however, many require specialized expertise and resources. This article explores rich pictures as an accessible, low-resource method for facilitating sensemaking amid complexity. The method involves the free-hand drawing of a situation to richly illustrate what is happening and why using images, lines, and few words. The process and product are akin to elaborate childhood doodles, yet are remarkably powerful conveyors of tacit knowledge and issues, differing perspectives, and action possibilities. Drawing together methodological guidance, sensemaking theory, and an application with STEM higher education faculty, this article shows how rich pictures support sensemaking amid complexity and identifies potential applications and directions for research on the method.

This paper unpacks our efforts as external evaluators to work toward decolonizing our evaluation practice. Undertaking this writing exercise as a form of reflective practice demonstrated that decolonization is much more complex than simply translating materials, organizing locals to collect data, and building participants’ capacity around Western modalities. While this complexity is clear in the decolonization literature, practice-based examples that depict barriers and thought processes are rarely presented. Through this paper, we deconstruct our deeply held beliefs around what constitutes good evaluation to assess the effectiveness of our decolonizing approach. Through sharing our critical consciousness-raising dialoguing, this paper reports our progress thus far and provides information and provocations to support others attempting to decolonize their practice.

Free-listing is a quick, semi-quantitative methodology commonly used by anthropologists to uncover information within a cultural domain. In this method note, we review how anthropologists have used free-listing in a variety of research settings. We then apply the social-ecological framework to describe how free-listing can be used for formative, process, outcome, and impact stages of program evaluation. Each type of evaluation includes a set of example free-list prompts to help researchers begin using this tool. We propose that free-listing is a beneficial data collection method in program evaluation. The free-listing method has identified barriers to treatment within our own work with clients recovering from either substance use disorder or opioid use disorder and has aided in providing flexible, individualized services. We conclude by providing recommendations for collecting free-list data and demonstrating the ease of computing free-list results by providing instructions and an example on how to analyze free-list responses.

The field of evaluation has experienced greater professionalization in the areas of evaluator education and training. Growth in these areas included sensitivity to issues of diversity, with efforts to attract and retain evaluators of color. Currently, there is limited scholarship on navigating a world with more opportunity but still dealing with being Black. Motivated by our identities as Black evaluators, we explored the education, training and socialization concerns experienced by Black evaluators and the advice they would offer. Through content analysis of semi-structured interviews and the utilization of Critical Race Theory as a framework, we garnered advice to and from Black evaluators on how to (a) navigate the politics in evaluation; (b) engage in reflection and reflexivity; (c) network and collaborate; (d) mentor and bring others along the way; and (e) develop as a professional evaluator. We offer a synthesis of these findings in the discussion and articulate further implications.

“Lesson study” has garnered considerable attention from educational researchers and practitioners as a promising method for improving instruction. At its core, lesson study reflects a collaborative inquiry process, grounded in cycles of planning, enacting, and reflecting, that promotes teacher learning through active engagement in lesson refinement. In this article, we leverage lesson study as an exemplar case to foreground an examination and application of theory-based evaluation. We overview a novel evaluation framework contextualized to this multidimensional instructional program and draw out the contributions of theory-based evaluation to the study of program effects. To ground the description of this framework, we summarize representative findings from our ongoing evaluation of a lesson study-based instructional program and discuss processes for selecting, using, and managing evidence sources as well as determining causal strands supporting program outcomes. We conclude by discussing lessons learned and implications for theory-based evaluation of complex programs.

The relationship between stability and change is a central paradox of administration that pervades all forms of organizing. Evaluation is not unfamiliar with paradoxical objectives and roles, which can result in tensions for evaluators and stakeholders. In this article, paradoxes between stability and change in the implementation of evaluation, and responses to them, are investigated through the case of social investment funds in Swedish local government. From interviews with staff, managers, and evaluators, findings show how responses to four main paradoxes give priority to top-down summative evaluation that produces instrumental knowledge on outcomes and costs for decision makers. The responses show that the concept of social investment fund evaluation is elastic to contain paradoxes and address different audiences. Also, paradoxes within the structure of the organization develop into paradoxes concerning the roles and goals of evaluation, raising the question of whether individual actors can deal with paradoxes.

The ethical guidelines for the American Evaluation Association and the principles of community-based participatory evaluation both state the importance of equitable stakeholder involvement. Regardless of the evaluation approach, however, evaluators are often confronted with gatekeepers, or those who control the access to stakeholders, information, or resources. Gatekeepers limit both the participation of key community members and, therefore, the exchange of relevant information related to the evaluation—a process called gatekeeping. Little research attention has been placed on studying gatekeeping, resulting in a dearth of knowledge about the influence of gatekeeping on stakeholder-engaged evaluations and social-structural dynamics that potentially perpetuate gatekeeping practices. In this article, we propose a gatekeeping influence theory grounded in the findings from 14 interviews. With a constructed theory of gatekeeping, we document the emergent social-structural and relational dynamics involved in stakeholder-engaged evaluation, with a focus on evaluations that include…