Assessment of Student Learning Through Reflection on Doing Using the Latent Dirichlet Algorithm

Can we provide evidence-based guidance to instructors to improve the delivery of the course based on students’ reflection on doing? Over three years at the University of Oklahoma, Norman, USA, we have collected about 18,000 Take-aways from almost 400 students who participated in an undergraduate design, build, and test course. In this paper, we illustrate the efficacy of using the Latent Dirichlet Algorithm to respond to the question posed above. We describe a method to analyze the Take-aways using a Latent Dirichlet Allocation (LDA) algorithm to extract topics from the Take-away data and then relate the extracted topics to instructors’ expectations using text similarity. The advantage of the LDA algorithm is anchored in that it provides a means for summarizing large amount of take-away data into several key topics so that instructors can eliminate the labor-intensive evaluation of it. By connecting and comparing what students learned (embodied in Take-aways) and what instructors expected the students to learn (embodied in stated Principles of Engineering Design), we provide evidence-based guidance to instructors on how to improve the delivery of AME4163: Principles of Engineering Design. Our objective in this paper is to introduce a method for quantifying text data to facilitate an instructor to modify the content and delivery of the next version of the course. The proposed method can be extended to other courses patterned after AME4163 to generate similar data sets covering student learning and instructor expectations, and the LDA algorithm can be used for dealing with the large amount of textual data embodied in students’ Take-aways.