The 10 most-cited articles in American Journal of Evaluation, 2018

The purpose of this study was to develop triangulation coding methods for a large-scale action research and evaluation project and to examine how practitioners and policy makers interpreted both convergent and divergent data. We created a color-coded system that evaluated the extent of triangulation across methodologies (qualitative and quantitative), data collection methods (observations, interviews, and archival records), and stakeholder groups (five distinct disciplines/organizations). Triangulation was assessed for both specific data points (e.g., a piece of historical/contextual information or qualitative theme) and substantive findings that emanated from further analysis of those data points (e.g., a statistical model or a mechanistic qualitative assertion that links themes). We present five case study examples that explore the complexities of interpreting triangulation data and determining whether data are deemed credible and actionable if not convergent.

The terms “outcome” and “impact” are ubiquitous in evaluation discourse. However, there are many competing definitions that lack clarity and consistency and sometimes represent fundamentally different meanings. This leads to profound confusion, undermines efforts to improve learning and accountability, and represents a challenge for the evaluation profession. This article investigates how the terms are defined and understood by different institutions and communities. It systematically investigates representative sets of definitions, analyzing them to identify 16 distinct defining elements. This framework is then used to compare definitions and assess their usefulness and limitations. Based on this assessment, the article proposes a remedy in three parts: applying good definition practice in future definition updates, differentiating causal perspectives and using appropriate causal language, and employing meaningful qualifiers when using the terms outcome and impact. The article draws on definitions used in international development, but its findings also apply to domestic public…

Our third article on the history of evaluation use affirms its importance in evaluation practice and related literature. It first highlights the centrality of use in the field’s professionalizing documents, extant theories, and the persistence of continuing research. Next, it discusses the challenge of evaluation theories in general, including the prevalence of prescriptive theories, and provides criteria for a good descriptive use theory that even the most detailed use theory does not currently meet. The third section reviews existing use “theories,” nine from Alkin’s evaluation theory tree, two from the literature, and two existing influence theories. This article concludes with a discussion of research on evaluation use past and present, including the effects of competing definitions, along with thoughts for future inquiry.

Culturally and politically science is under attack. The core consequence of perceiving and asserting evaluation as science is that it enhances our credibility and effectiveness in supporting the importance of science in our world and brings us together with other scientists to make common cause in supporting and advocating for science. Other consequences include communicating our role more clearly and credibly to those who value science. Viewing evaluation as science may affect how we are viewed, treated, and positioned in academia, government, and by funders and users of evaluation. The ramifications of evaluation science for evaluation’s status as a profession, branch of applied social science, discipline, and transdiscipline are reviewed. The conclusion offers implications, caveats, and cautions regarding the identity and practice of evaluation science.

The concept of power is a complex and often intangible aspect of the evaluation process that is frequently a focal topic among the conceptual evaluation literature concerning collaborative or culturally responsive evaluation (CRE) contexts. Unfortunately, there remains a significant theory to practice gap as power is often rarely addressed or specifically identified in empirical practice. In the current article, the authors provide a comprehensive review of the empirical literature on CREs, with an emphasis on the manifestation of power in practice. The analysis of the literature selected for review leads to the development of a conceptual framework, which identifies intersecting power dynamics across the relational, political, discursive, and historical/temporal dimensions. The authors conclude with a research synthesis and implications for practice, including a discussion of practical ways that evaluators can begin with considerations of power rather than reflecting on the concept retrospectively.

An increasing number of foundations are embracing racial equity/equity as a core value, and it is influencing how they see themselves and operate. However, evaluation has for the most part remained untouched. Knowing how race/racism has influenced both, philanthropy and evaluation, deepens our understanding of how philanthropic evaluation practice may unintentionally reinforce racism. Equitable evaluation shifts the current evaluation paradigm to one that centers equity/racial equity, so that it is more aligned with the values and intentions of current day philanthropic endeavors.

Culturally responsive evaluation and culturally responsive Indigenous evaluation (CRIE) within the broader field of evaluation are not often included in Western literature nor are they known or used by the majority of mainstream evaluators. In order to address this literature and practice gap, this article offers an overview and a broader origin story of CRIE prior to colonial or European contact in the United States and gives an overview of the historical, theoretical, and practical foundations for conducting CRIE in a contemporary evaluation context. Examples of evidence-based models, theories, and resources are provided to connect CRIE to Western evaluation designs and provide concrete strategies for the field of evaluation going forward. The article concludes with systemic and policy evaluation considerations as agencies from federal (i.e., United States), tribal, and international governments and partners from private or nonprofit sectors collaborate to carry out Indigenous evaluations in the future. Collectively this multijurisdictional, culturally responsive, and…

Recent public events have unnerved the country, touched the consciousness of the populace, and reveal the resurgence of the White supremacist movement, causing many citizens to reflect on race relations. Given the profound impact of racism deeply embedded in the fabric of this country and its complex and destructive influence on the lives of people of color in social programs and in the contexts being evaluated, evaluators and the evaluation profession must not be complicit in the status quo by ignoring issues of race and racism in our work. This article sounds a clarion call for evaluators to use their power and privilege to advance a more equitable society by “calling out” racists’ practices, unmasking inequities, and articulating participants’ experiences within their racialized environments. We unpack how racism is a complex and destructive force in evaluation contexts. In this article, we hope to stimulate evaluative thinking and action by urging evaluators to gain a deeper understanding of racism as a complex interplay of individual attitudes, social values, and institutional…