Key takeaways
-
Direct assessment, competency-based education requires a different set of student success metrics, and the UW Flexible Option developed an alternate metrics framework in response.
Read more...
Semester-based metrics don’t apply when students can start at the beginning of any month of the year. Traditional metrics are based on time spent learning. In UW Flex, standards of academic outcomes are constant; time spent learning is variable. Students start their academic work any month of the year and set their own pace in this faculty-led highly structured program as they demonstrate mastery over the academic outcomes specified for their program. In addition, students identify their own reasons for enrolling in UW Flex (career advancement, personal learning, etc.), another important variable in the evaluation of student and program success.
-
The framework used by the UW Flexible Option defines student success as students moving through a program at their own pace, demonstrating mastery of subject matter (academic outcomes), and meeting their self-defined educational goals.
Read more...
Students progress at their own pace. Pace reflects the rate of progress compared to the student’s Individualized Learning Plan, as measured by assessment/competency completion within subscription periods.
Academic outcomes are both quantitative (competencies mastered in a subscription period) and qualitative (perceived value and impact).
Goals (for example, to get a job, promotion, acceptance to a higher degree program) drive students’ enrollment decisions. Upon enrollment, students establish personal learning goals and the Academic Success Coach includes those in the learning plan. Measures address not only goals achieved but students’ satisfaction with the educational experience.
-
New information about program success is derived through aggregation and analyses of individual students’ pace, outcomes, and goals.
Read more...
The Metrics Framework supports new ways to examine program success, such as measures of program pace by types of students; attempted versus completed competencies; and a program’s ability to meet students’ employment goals.
UW Flex uses data from our LMS to evaluate how students are responding to each of our competency projects and assessments. What do patterns of completion look like? Are some projects and assessments too easy or too difficult?Aaron Brower, UW-Extension provost and vice chancellor, in this Navigating the CBE Frontier March 2017 post in The Evolllution
-
Strategic management of resources is required to overcome challenges inherent in implementing the UW Flexible Option Metrics Framework (see “More detail” section below).
Read more...
Challenges are introduced below and further detailed in “Measuring What Matters,” a chapter in the “Handbook of Research on Competency-Based Education in University Settings,” edited by Karen Rasmussen, Pamela Northrup, and Robin Colson (IGI Global, 2017).
A first step is to build awareness of these challenges and then account for them early in the CBE development process.
Save
Challenges
-
New metrics required new data and new sources of data.
Read more...
UW Flex early response: When UW Flex launched in 2013-14, no off-the-shelf student records systems available at that time met needs of this non-term direct assessment CBE program. (This topic is addressed in the Enrollment Management and Technology content area.) As a result, UW Flex staff had to manually record data while they created operations to systematize collection of student records. They spent time and money creating a solution that applied uniquely to UW Flex.
Looking forward from 2017: UW Flex staff are developing a Student Engagement System to enable efficient and scalable data collection and utilization. (Details on this system are included in the Enrollment Management and Technology content area.) In the future and ideally, this system will use a standardized student records database that will allow modules (such as for recruitment, enrollment, financial aid, and transcription) to easily “plug in” and “plug out.”
-
Aggregation of student goals and pace required institutional research capacity that was not immediately available.
Read more...
UW Flex early response: Initial efforts focused on enrollment management and student retention. Efforts in 2015 turned to creating data systems and structures that allow the capture of student pace, outcomes, and goals. Student outcomes are captured through the learning management system (D2L).
Data analysis requires research capacity. Looking forward from 2017: UW Flex staff continue to develop the capacity, data systems, and structures to capture and aggregate student goals, outcomes, and pace articulated in the Metrics Framework and with the methodological and analytic capacity to mine “big data” for these student metrics.
For example, to find measures specific to the direct assessment, competency-based UW Flexible Option, our research staff designed a survey testing 27 measures to determine which best distinguished graduates from non-returning students (those who stop out for six months or more). (In prior research, we found that students who stop out for six months or more are significantly more likely not to complete their degree. See Mayeshiba & Brower, “Student success and retention using new definitions created for non-term, direct assessment CBE.” Journal of Competency-Based Education, vol 2, issue 1.)
The survey was piloted with students who left UW Flex between January 2014 and February 2017. Response rates—85.7% of graduates and 44.6% of non-returning students—were well above industry norms for the type of survey and population.
Surveys covered demographics, educational goals, and perceptions of various attributes of the UW Flexible Option. Preliminary findings indicate that demographics and educational goals are not predictive of attrition.
Two measures of student perceptions of UW Flexible Option powerfully distinguished graduates from non-returning students. Students who agreed or strongly agreed with the following statements were far more likely to persist through to graduation than those who were either neutral or disagreed with the statements:
- “I was able to complete the competency sets within the 3-month time limit.”
- “The assessments accurately evaluated my knowledge and
As a result of our research, we will begin to intervene with students who answer in the neutral to negative range on the above two items.
Analysis on responses to survey open-ended questions continues. When complete, we expect to share additional findings in this section of the website.
-
UW Flex senior leaders recognize the critical importance of directly comparing student outcomes for different kinds of students in different modalities.
Read more...
UW Flex early response: A three-way partnership supported a pilot study comparing student outcomes. The partners included Lumina Foundation, UW-Milwaukee Flexible Option faculty, and the National Research Center for Distance Education and Technological Advancements (DETA). Completed in 2017, this study compares select learning outcomes among students in the RN to BSN degree-completion programs in the traditional online and UW Flex formats.
Looking forward from 2017: The study findings indicate that UW Flex student outcomes are comparable with those of students in the traditional course. However, the population studied was small, warranting further research for conclusive evidence. See Mayeshiba, Jansen & Mihlbauer, “An Evaluation of Critical Thinking in Competency-Based and Traditional Online Learning Environments,” Online Learning Consortium Special Issue: National Research Center for Distance Education and Technological Advancements (DETA), vol 22, issue 2 (2018).
UW Flex senior leaders look forward to directly comparing student outcomes for different kinds of students (varying by demographics, employment status and history, prior educational and work experiences, and goals) by different modalities (such as traditional classroom, traditional online, and direct assessment CBE) to improve, develop, and grow programs.
The American Institutes for Research 2017 report “Measuring Student Success in Postsecondary Competency-Based Education Programs: Toward a Student Outcomes Metrics Framework” also notes the challenge in contrasting progression between CBE and other programs, concluding, “Researchers can continue to rely on credit hours when making comparisons for now, but there is broad interest in finding other, better standards to use as the field matures.”
-
Data governance and data sharing across institutions within the university system were not yet developed and implemented.
Read more...
UW Flex early response: Due to variables across institutions, UW Flex relied on case-by-case resolution on data sharing across institutions within UW System.
Looking forward from 2017: UW Flex staff support the development of common data structures and systems for UW System that would facilitate cross-institution data governance.
We gather…data primarily from our CRM, LMS and SIS, and we also use post-graduation surveys to capture information about graduates…We do not manage all of these data in one giant database, but we are working towards that goal.UW-Extension Provost Aaron Brower in a post in The Evolllution.
