Evidence for Improvement: An Integrated Approach
Item
Title
Evidence for Improvement: An Integrated Approach
Abstract/Description
OBJECTIVES OR PURPOSES
Across the social sector, a growing number of organizations are joining “continuous improvement” networks in order to solve complex problems (Gates, 2017). Unlike other initiatives, improvement networks are designed to be flexible and evolving. They expect to iterate on their strategies, drawing on evidence gathered from the “frontline” in an ongoing process of learning to improve (Bryk et. al, 2015).
The field of program evaluation offers tools, techniques, and practices that may help improvement efforts in this learning process (Coryn, Noakes, Westine, & Schröter, 2011; Patton, 2011; Weiss, 1997). Evaluative inquiry may enhance an improvement effort’s internal learning processes. At the same time, evaluative approaches may hurt an improvement effort if, for example, evaluators pressure initiatives to pursue “fidelity” of implementation when innovation is needed (Gopalakrishnan, Preskill, & Lu, 2013; Peurach, 2016; Preskill & Beer, 2012).
Little extant work has explored how evaluation and continuous improvement can be brought together. This conceptual paper synthesizes existing scholarship to offer a framework describing how evaluation methods can be applied to continuous improvement networks in an approach we call Evidence for Improvement.
THEORETICAL FRAMEWORK
We use literature on continuous improvement in organizations (Bryk et al., 2015; Peurach, 2016) to conceptualize improvement networks, arguing that these organizational forms have distinct analytic needs. Second, we use literature on program evaluation (Patton, 2011; Weiss, 1997) to identify inquiry approaches that meet these needs.
MODES OF INQUIRY
We wrote this paper by synthesizing: (1) Existing literature in the field of program evaluation and continuous improvement (2) Our experience working in improvement networks, and (3) Opinions of evidence-use and evaluation experts.
DATA SOURCES
Data sources for this paper include: (1) Existing literature in the fields of program evaluation, continuous improvement in organizations, and improvement science, (2) Our own experience working with improvement initiatives and (3) Opinions and views gathered at a convening of evidence-use experts.
RESULTS
We argue that improvement networks move from phases of “exploration,” in which they search broadly to generate new ideas about how to solve a social problem, to phases of “exploitation,” in which they solidify their initial ideas based on evidence gathered from the field and seek to enact these ideas at a larger scale (March, 1991). We also argue that rapid learning is essential for these networks given their inherently iterative nature. Building on this argument, we suggest that an integrated approach to evaluation that combines approaches from developmental, formative, and summative evaluation traditions (Patton, 2011; Weiss, 1997, Scriven, 1967) suits improvement efforts’ evolving analytic needs.
SIGNIFICANCE
Continuous improvement efforts can be ill-served by traditional evaluations because these forms of inquiry do not accommodate the uncertainty associated with innovation (Peurach, 2016; Patton, 2011; Preskill & Beer, 2012). This conceptual paper offers a new, integrated perspective on how to selectively draw from existing evaluation approaches to support improvement networks’ evolving analytic needs.
Across the social sector, a growing number of organizations are joining “continuous improvement” networks in order to solve complex problems (Gates, 2017). Unlike other initiatives, improvement networks are designed to be flexible and evolving. They expect to iterate on their strategies, drawing on evidence gathered from the “frontline” in an ongoing process of learning to improve (Bryk et. al, 2015).
The field of program evaluation offers tools, techniques, and practices that may help improvement efforts in this learning process (Coryn, Noakes, Westine, & Schröter, 2011; Patton, 2011; Weiss, 1997). Evaluative inquiry may enhance an improvement effort’s internal learning processes. At the same time, evaluative approaches may hurt an improvement effort if, for example, evaluators pressure initiatives to pursue “fidelity” of implementation when innovation is needed (Gopalakrishnan, Preskill, & Lu, 2013; Peurach, 2016; Preskill & Beer, 2012).
Little extant work has explored how evaluation and continuous improvement can be brought together. This conceptual paper synthesizes existing scholarship to offer a framework describing how evaluation methods can be applied to continuous improvement networks in an approach we call Evidence for Improvement.
THEORETICAL FRAMEWORK
We use literature on continuous improvement in organizations (Bryk et al., 2015; Peurach, 2016) to conceptualize improvement networks, arguing that these organizational forms have distinct analytic needs. Second, we use literature on program evaluation (Patton, 2011; Weiss, 1997) to identify inquiry approaches that meet these needs.
MODES OF INQUIRY
We wrote this paper by synthesizing: (1) Existing literature in the field of program evaluation and continuous improvement (2) Our experience working in improvement networks, and (3) Opinions of evidence-use and evaluation experts.
DATA SOURCES
Data sources for this paper include: (1) Existing literature in the fields of program evaluation, continuous improvement in organizations, and improvement science, (2) Our own experience working with improvement initiatives and (3) Opinions and views gathered at a convening of evidence-use experts.
RESULTS
We argue that improvement networks move from phases of “exploration,” in which they search broadly to generate new ideas about how to solve a social problem, to phases of “exploitation,” in which they solidify their initial ideas based on evidence gathered from the field and seek to enact these ideas at a larger scale (March, 1991). We also argue that rapid learning is essential for these networks given their inherently iterative nature. Building on this argument, we suggest that an integrated approach to evaluation that combines approaches from developmental, formative, and summative evaluation traditions (Patton, 2011; Weiss, 1997, Scriven, 1967) suits improvement efforts’ evolving analytic needs.
SIGNIFICANCE
Continuous improvement efforts can be ill-served by traditional evaluations because these forms of inquiry do not accommodate the uncertainty associated with innovation (Peurach, 2016; Patton, 2011; Preskill & Beer, 2012). This conceptual paper offers a new, integrated perspective on how to selectively draw from existing evaluation approaches to support improvement networks’ evolving analytic needs.
Author/creator
Date
At conference
AERA Annual Meeting Conference
Place presented
Resource type
Research/Scholarly Media
Resource status/form
Presentation/Poster
Scholarship genre
Synthesis/Overview
IRE Approach/Concept
Other related resources/entities
Citation
Sherer, D. G., Norman, J. R., Peurach, D. J., Bryk, A. S., & Vasudeva, A. (2019). Evidence for Improvement: An Integrated Approach. AERA Annual Meeting, Toronto, ON.
Linked resources
Filter by property
| Title | Alternate label | Class |
|---|---|---|
 The Scholarship of Improvement: Building Community Around an Emerging Tradition of Practice-Focused Research |
Session |
Comments
No comment yet! Be the first to add one!