The 10 most-cited articles in American Journal of Evaluation, 2022

Evaluation capacity building (ECB) continues to attract the attention and interest of scholars and practitioners. Over the years, models, frameworks, strategies, and practices related to ECB have been developed and implemented. Although ECB is highly contextual, the evolution of knowledge in this area depends on learning from past efforts in a structured approach. The purpose of the present article is to integrate the ECB literature in evaluation journals. More specifically, the article aims to answer three questions: What types of articles and themes comprise the current literature on ECB? How are current practices of ECB described in the literature? And what is the current status of research on ECB? Informed by the findings of the review, the article concludes with suggestions for future ECB practice and scholarship.

Implementation fidelity is the degree of compliance with which the core elements of program or intervention practices are used as intended. The scientific literature reveals gaps in defining and assessing implementation fidelity in early intervention: lack of common definitions and conceptual framework as well as their lack of application. Through a critical review of the scientific literature, this article aims to identify information that can be used to develop a common language and guidelines for assessing implementation fidelity. An analysis of 46 theoretical and empirical papers about early intervention implementation, published between 1998 and 2018, identified four conceptual frameworks, in addition to that of Dane and Schneider. Following analysis of the conceptual frameworks, a four-component conceptualization of implementation fidelity (adherence, dosage, quality and participant responsiveness) is proposed.

Lesbian, gay, bisexual, transgender, queer, intersex, Two-Spirit, and other sexual and gender minority (LGBTQ+) individuals encounter numerous obstacles to equity across health and healthcare, education, housing, employment, and other domains. Such barriers are even greater for LGBTQ+ individuals who are also Black, Indigenous, and People of Color (BIPOC), as well as those who are disabled, and those who are working-class, poor, and otherwise economically disadvantaged, among other intersecting forms of oppression. Given this, an evaluation cannot be equitable for LGBTQ+ people without meaningfully including our experiences and voices. Unfortunately, all evidence indicates that evaluation has systematically failed to recognize the presence and value of LGBTQ+ populations. Thus, we propose critical action steps and the articulation of a new paradigm of LGBTQ+ Evaluation. Our recommendations are grounded in transformative, equitable, culturally responsive, and decolonial frameworks, as well as our own experiences as LGBTQ+ evaluators and accomplices. We conclude by inviting others to…

Violence, marginalization, oppression, exploitation, erasure, and injustice are cornerstones of the Black experience in the United States. Despite seemingly insurmountable challenges, Black people have worked diligently and competently to earn spaces within the present-day professional arena. While the experiences of Black professionals have been investigated in multiple fields, little is known about Black evaluators. This study aimed to investigate Black evaluators’ experiences in an attempt to understand the intersectionality of their identities, roles, and practice and to ensure voices and contributions of Black evaluators are highlighted. We conducted 26 interviews with Black evaluators across academic, government, nonprofit, philanthropic, and private sectors. We found race and advocate for social justice are central to Black evaluators’ identities, roles, and practice. We developed a thematic framework as a result of our analysis and interpretations of the ways in which Black evaluators’ identities impacted their perceptions of their professional role and practice.

Advocacy evaluation has emerged in the past 20 years as a specialized area of evaluation practice. We offer a review of existing peer-reviewed literature and draw attention to the scarcity of scholarly work on human rights advocacy evaluation in the Global South. The lack of published material in this area is concerning, given the urgent need for human rights advocacy in the Global South and the difficulties of conducting advocacy in contexts in which fundamental human rights are often poorly protected. Based on the review of the literature and our professional experiences in human rights advocacy evaluation in the Global South, we identify themes in the literature that are especially salient in the Global South and warrant more attention. We also offer critical reflections on content areas not addressed in the existing literature and conclude with suggestions as to how activists, evaluators, and other stakeholders can contribute to the development of a field of practice that is responsive to the global challenge of advocacy evaluation.

Empowerment is a concept that has become increasingly used over recent years. However, little research has been undertaken into how empowerment can be evaluated, particularly in the case of young people. The aim of this article is to present an inventory of dimensions and indicators of youth empowerment. The article describes the various phases in the construction and validation of the inventory. These phases were (1) a contrast of the inventory of dimensions and indicators against specialized published writings on youth empowerment; (2) the validation of the resulting inventory by experts; and (3) a contrast with young people through four participatory evaluation processes and six life stories. The tool is scientifically and practically useful and enables the impact of youth empowerment programmes to be evaluated; it also serves to plan and implement socio-educational processes aimed at influencing the empowerment of young people.

Analysis of the differential treatment effects across targeted subgroups and contexts is a critical objective in many evaluations because it delineates for whom and under what conditions particular programs, therapies or treatments are effective. Unfortunately, it is unclear how to plan efficient and effective evaluations that include these moderated effects when the design includes partial nesting (i.e., disparate grouping structures across treatment conditions). In this study, we develop statistical power formulas to identify requisite sample sizes and guide the planning of evaluations probing moderation under two-level partially nested designs. The results suggest that the power to detect moderation effects in partially nested designs is substantially influenced by sample size, moderation effect size, and moderator variance structure (i.e., varies within groups only or within and between groups). We implement the power formulas in the R-Shiny application PowerUpRShiny and demonstrate their use to plan evaluations.

More evaluators have anchored their work in equity-focused, culturally responsive, and social justice ideals. Although we have a sense of approaches that guide evaluators as to how they should attend to culture, diversity, equity, and inclusion (DEI), we have not yet established an empirical understanding of how evaluators measure DEI. In this article, we report an examination of how evaluators and principal investigators (PIs) funded by the National Science Foundation's Advanced Technological Education (ATE) program define and measure DEI within their projects. Evaluators gathered the most evidence related to diversity and less evidence related to equity and inclusion. On average, PIs’ projects engaged in activities designed to increase DEI, with the highest focus on diversity. We believe there continues to be room for improvement and implore the movement of engagement with these important topics from the margins to the center of our field's education, theory, and practice.

Evaluations are considered of key importance for a well-functioning democracy. Against this background, it is vital to assess whether and how evaluation models approach the role of citizens. This paper is the first in presenting a review of citizen involvement in the main evaluation models which are commonly distinguished in the field. We present the results of both a document analysis and an international survey with experts who had a prominent role in developing the models. This overview has not only a theoretical relevance, but can also be helpful for evaluation practitioners or scholars looking for opportunities for citizen involvement. The paper contributes to the evaluation literature in the first place, but also aims to fine-tune available insights on the relationship between evidence informed policy making and citizens.

The impostor phenomenon is a psychological construct referring to a range of negative emotions associated with a person's perception of their own "fraudulent competence" in a field or of their lack of skills necessary to be successful in that field. Anecdotal evidence suggests that many practicing evaluators have experienced impostor feelings, but lack a framework in which to understand their experiences and the forums in which to discuss them. This paper summarizes the literature on the impostor phenomenon, applies it to the field of evaluation, and describes the results of an empirical, quantitatively focused study which included open-ended qualitative questions exploring impostorism in 323 practicing evaluators. The results suggest that impostor phenomenon in evaluators consists of three constructs: Discount, Luck, and Fake. Qualitative data analysis suggests differential coping strategies for men and women. Thematic analysis guided the development of a set of proposed solutions to help lessen the phenomenon's detrimental effects for evaluators.