Strategies for Making Digital Learning System Data Usable: A Design Workshop Approach
Item
Title
Strategies for Making Digital Learning System Data Usable: A Design Workshop Approach
Abstract/Description
Objectives
In this paper, we describe the ways in which researchers at SRI International and the Carnegie Foundation for the Advancement of Teaching worked with practitioners (i.e., administrators and instructors) in the Carnegie Math Pathways NIC to develop data products that helped practitioners measure changes to the system organized around promoting students’ productive persistent in developmental math classrooms (Silva & White, 2013).
Theoretical Perspective
Based on findings from literature on data driven decision making, improvement science, and the emerging field of data science, SRI and Carnegie developed a series of conjectures that guided the development, iterative refinement, and use of data products to be used by practitioners (Krumm et al., in press). Across development activities, the data product needed to (1) address an important need; (2) provide relative advantage; (3) show trends over time; (4) be aligned to valued outcomes; and (5) be tied to follow-up action.
Productive persistence entails tenacity plus the use of good learning strategies (Yeager et al., 2013). As members of the NIC, faculty were engaged in testing change ideas directected at promoting productive persistence (i.e., need). Relative advantage (Rogers, 2003) came about by providing data related to productive persistence on more timely intervals using data students were generating on a day-to-day basis through the online learning system (i.e., Instructure’s Canvas Learning Management System). For improvement purposes, data need to be followed over time and aligned to changes that are made at particular points in time (Provost & Murray, 2011). Moreover, data need to be aligned to valued outcomes either through theory-based conjectures or through historical data analyses that provide an empirical warrant. Lastly, data need to be tied to follow-up actions in order to be actionable.
Methods and Data
To help identify the antecedents of actionable data, we engaged Pathways faculty in a series of design workshops where data from the online learning system were presented and they were taken through a structured design process for tieing actions to data products. Data for this paper come from two sources: (1) historical data from the online learning system that were used to develop data products and (2) faculty's reactions to data products captured in notes made by faculty, brainstormed data products developed by faculty, and researchers’ field notes from design sessions.
Results
We present results on three levels: (1) the student-level patterns of behaviors that were presented to faculty, (2) faculty members’ interpretations of patterns, and (3) the follow-on actions developed by faculty. For example, we identified that students who worked until successful on course assessments where they could attempt an assessment multiple times, were more successful overall in their respective courses. Faculty generated sample messages that they could provide to students. Future work is focused on collecting evidence related to the data products’ effectiveness for supporting faculty.
Significance
Making data useful is a canonical problem facing practitioners at all levels of the educational system. This paper provides a concrete approach for making data useful that has high potential for being tested in other settings.
In this paper, we describe the ways in which researchers at SRI International and the Carnegie Foundation for the Advancement of Teaching worked with practitioners (i.e., administrators and instructors) in the Carnegie Math Pathways NIC to develop data products that helped practitioners measure changes to the system organized around promoting students’ productive persistent in developmental math classrooms (Silva & White, 2013).
Theoretical Perspective
Based on findings from literature on data driven decision making, improvement science, and the emerging field of data science, SRI and Carnegie developed a series of conjectures that guided the development, iterative refinement, and use of data products to be used by practitioners (Krumm et al., in press). Across development activities, the data product needed to (1) address an important need; (2) provide relative advantage; (3) show trends over time; (4) be aligned to valued outcomes; and (5) be tied to follow-up action.
Productive persistence entails tenacity plus the use of good learning strategies (Yeager et al., 2013). As members of the NIC, faculty were engaged in testing change ideas directected at promoting productive persistence (i.e., need). Relative advantage (Rogers, 2003) came about by providing data related to productive persistence on more timely intervals using data students were generating on a day-to-day basis through the online learning system (i.e., Instructure’s Canvas Learning Management System). For improvement purposes, data need to be followed over time and aligned to changes that are made at particular points in time (Provost & Murray, 2011). Moreover, data need to be aligned to valued outcomes either through theory-based conjectures or through historical data analyses that provide an empirical warrant. Lastly, data need to be tied to follow-up actions in order to be actionable.
Methods and Data
To help identify the antecedents of actionable data, we engaged Pathways faculty in a series of design workshops where data from the online learning system were presented and they were taken through a structured design process for tieing actions to data products. Data for this paper come from two sources: (1) historical data from the online learning system that were used to develop data products and (2) faculty's reactions to data products captured in notes made by faculty, brainstormed data products developed by faculty, and researchers’ field notes from design sessions.
Results
We present results on three levels: (1) the student-level patterns of behaviors that were presented to faculty, (2) faculty members’ interpretations of patterns, and (3) the follow-on actions developed by faculty. For example, we identified that students who worked until successful on course assessments where they could attempt an assessment multiple times, were more successful overall in their respective courses. Faculty generated sample messages that they could provide to students. Future work is focused on collecting evidence related to the data products’ effectiveness for supporting faculty.
Significance
Making data useful is a canonical problem facing practitioners at all levels of the educational system. This paper provides a concrete approach for making data useful that has high potential for being tested in other settings.
Author/creator
Date
At conference
AERA Anual Meeting
Resource type
Research/Scholarly Media
Resource status/form
Presentation/Poster
Scholarship genre
Empirical
Open access/full-text available
No
Peer reviewed
No
Citation
Krumm, A. E., & Beattie, R. (2017). Strategies for Making Digital Learning System Data Usable: A Design Workshop Approach. AERA Anual Meeting, San Antonio, TX.
Comments
No comment yet! Be the first to add one!