The 10 most-cited articles in American Journal of Evaluation, 2021

Transformative change is needed if the world is to achieve the sustainable development goals. Such change requires attention to culture, ethics, and values. We discuss the need to be responsive to the voices of Indigenous scholars in terms of the assumptions that guide methodological choices in the evaluation of international initiatives. We describe an Indigenous paradigmatic framework and then narrow the focus to a Made in Africa approach to evaluation that is designed to redress the epistemic violence perpetrated by the use of a Western cultural lens to determine evaluation approaches.

Changing health-care climates mean evaluators need to provide findings within shorter time frames, but challenges remain in the creation of rapid research designs capable of delivering quality data to inform decision-making processes. We conducted a review of articles to grapple with these challenges and explore the ways in which rapid evaluations have been used in health care. We found different labels being used to define rapid evaluations and identified a trend in the design of evaluations, where evaluators are moving away from short studies to longer evaluations with multiple feedback loops or cyclical stages. Evaluators are using strategies to speed up evaluations: conducting data collection and analysis in parallel, eliminating the use of transcripts, and utilizing larger evaluation teams to share the workload. Questions persist in relation to the suitability of rapid evaluation designs, the trustworthiness of the data, and the degree to which evaluation findings are used to make changes in practice.

The purpose of this study was to determine professional evaluators’ perceptions of reflective practice (RP) and the extent and manner in which they engage in RP behaviors. Nineteen evaluators with 10 or more years of experience in the evaluation field were interviewed to explore our understanding and practice of RP in evaluation. Findings suggest that RP is a process of self and contextual awareness, involving thinking and questioning, and individual and group meaning-making, focused on facilitating growth in the form of learning and improvement. The roles of individual and collaborative reflection as well as reflection in- and on-action are also discussed. Findings support a call for the further refinement of our understanding of RP in evaluation practice. Evaluators seeking to be better reflective practitioners should be competent in skills such as facilitation and interpersonal skills, as well as budget needed time for RP in evaluation accordingly.

Intervention scientists have used program theory-driven evaluation to design, implement, and assess the success of intervention programs for decades. However, interventions often are designed without the input of the community for which they are intended. The lack of incorporation of community members’ voices that participate in various intervention programs and the de-emphasis of systemic influences on program outcomes may decrease the extent to which participants experience intended positive outcomes. This often leads to interventions that are irrelevant at best and harmful at worst. Theory of change and logic models are two forms of theory-driven evaluation that can be used in tandem to incorporate community voices into program design and implementation while attending to systemic influences on the program. The following paper presents a stage model process for creating and utilizing both a theory of change and logic model for community-based intervention programs aiming to work with ethnically and racially diverse and immigrant populations.

Evaluation policy has been identified as an important means of shaping and influencing organizational evaluation practice, yet, to date, little empirical research has been conducted to deepen our understanding of this relationship. The purpose of this study was to illuminate evaluation policy’s role in leveraging organizational capacity to do and use evaluation. We interviewed 18 published evaluation scholars and practitioners from North America and Europe about this topic. A thematic analysis of findings underscores the importance of context, policy attributes, enablers, and organizational benefits. Based on the findings, we developed an ecological conceptual framework to guide thinking about the role of evaluation policy in capacity building. We discuss these findings in terms of practical implications of understanding context, redressing the imbalance between learning and accountability purposes of evaluation, and organizational leadership, and we conclude with some implications for research.

This study systematically reviewed evidence regarding health program and service evaluations in Indigenous contexts. Following the PRISMA guidelines and combining terms for ‘Indigenous populations’ and ‘health programs and services’. Eight principles emerged: Principle 1: Adopting Indigenous led or co-led approaches is vital to balance power relationships by prioritizing self-determination, Principle 2: Evaluation team should include local Indigenous community members, Principle 3: Indigenous community knowledge and practice should be foundational, Principle 4: Evaluations must be responsive and flexible to meet the needs of the local community, Principle 5: Evaluations should respect and adhere to local Indigenous protocols, culture, wisdom and language, Principle 6: Evaluations should emphasize reciprocity, shared learnings and capacity building, Principle 7: It is important to build strong relationships and trust between and within researcher teams, evaluators and communities, and Principle 8: The evaluation team must acknowledge community capacity and resources by investing in…

Evaluative criteria define a “high quality” or “successful” evaluand and provide the basis for judgment of merit and worth, yet they are often assumed and implicit in the evaluation process. This article presents an empirically supported model that describes and integrates two aspects of criteria: domain and source. Domain identifies the focus or substance of a criterion, while source describes the individual, group, or document from which it is drawn. Developed from a synthesis of evaluation literature and empirical analysis of evaluation reports, the model defines 11 criteria domains and 10 sources and reveals the relationships among them. In this integrated model, the two dimensions can be used together as a thinking tool to guide evaluators in specifying criteria, in empirical research on the valuing process, and as a conceptual framework and language for theorists prescribing criteria selection.

Observations are widely used in research and evaluation to characterize teaching and learning activities. Because conducting observations is typically resource intensive, it is important that inferences from observation data are made confidently. While attention focuses on interrater reliability, the reliability of a single-class measure over the course of a semester receives less attention. We examined the use and limitations of observation for evaluating teaching practices, and how many observations are needed during a typical course to make confident inferences about teaching practices. We conducted two studies based on generalizability theory to calculate reliabilities given class-to-class variation in teaching over a semester. Eleven observations of class periods over the length of a semester were needed to achieve a reliable measure, many more than the one to four class periods typically observed in the literature. Findings suggest practitioners may need to devote more resources than anticipated to achieve reliable measures and comparisons.

Despite the rising popularity of big data, there is speculation that evaluators have been slow adopters of these new statistical approaches. Several possible reasons have been offered for why this is the case: ethical concerns, institutional capacity, and evaluator capacity and values. In this method note, we address one of these barriers and aim to build evaluator capacity to integrate big data analytics into their studies. We focus our efforts on a specific topic modeling technique referred to as latent Dirichlet allocation (LDA) because of the ubiquitousness of qualitative textual data in evaluation. Given current equity debates, both within evaluation and the communities in which we practice, we analyze 1,796 tweets that use the hashtag #equity with the R packages topicmodels and ldatuning to illustrate the use of LDA. Furthermore, a freely available workbook for implementing LDA topic modeling is provided as Supplemental Material Online.

This article describes our experience of conducting a 5-year, culturally responsive evaluation of a federal program with Indigenous communities. It describes how we adapted tenets from “participatory evaluation models” to ensure cultural relevance and empowerment. We provide recommendations for evaluators engaged in similar efforts. The evaluation included stakeholder engagement through a Steering Committee and an Evaluation Working Group in designing and implementing the evaluation. That engagement facilitated attention to Indigenous cultural values in developing a program logic model and medicine wheel and in gathering local perspectives through storytelling to facilitate understanding of community traditions. Our ongoing assessment of program grantees’ needs shaped our approach to evaluation capacity building and development of a diverse array of experiential learning opportunities and user-friendly tools and resources. We present practical strategies from lessons learned during the evaluation design and implementation phases of our project that might be useful for other…