Quantification of students' learning through reflection on doing based on text similarity

In this paper we address the following question: How can instructors leverage assessment instruments in design, build, and test courses to simultaneously improve student outcomes and assess student learning well enough to improve courses for future students? A learning statement is a structured text-based construct for students to record what they learned by reflecting on authentic immersive experiences in a semester-long engineering design course. The immersive experiences include lectures, assignments, reviews, building, testing, and a post-analysis of an electro-mechanical device to address a given customer need. Over the past three years, in the School of Aerospace and Mechanical Engineering at the University of Oklahoma, Norman, we have collected almost 30,000 learning statements from almost 400 students. In the past few years, we have analyzed this data to improve our understanding of what students have learned by reflecting on doing and thence how we might improve the delivery of the course. In an earlier paper, we described a text mining framework to facilitate the analysis of a vast number of learning statements. Our focus, in the earlier paper, was on describing the functionalities (i.e., data cleaning, data management, text analysis, and visualization results) of the framework and demonstrating one of the text quantification methods-term frequency-using the learning statements. In this paper, we focus on demonstrating another text quantification method, namely, text similarity, to facilitate instructors' gaining new insights from students' learning statements. In the method of text similarity, we measure the cosine distance between two text vectors and is typically used to compare the semantic similarity between documents. In this paper, we compare the similarity between what students learned (embodied in learning statements) and what instructors expected the students to learn (embodied in the course booklet), thus providing evidence-based guidance to instructors on how to improve the delivery of AME4163-Principles of Engineering Design.