Student Program Classification

Overview

This was my project for CP3209: Undergraduate Research Opportunities Program (UROP) at NUS; the topic was initially proposed by my supervisor, but the project scope changed quite a bit since its initial conception. I was under the supervision of Prof. Martin Henz, and I was also fortunate to have worked with Prof Boyd Anderson as well. This project was also meant for further development of Source Academy, an online learning platform for CS1101S: Programming Methodology.

This project focused on classifying student programs initially for use in the Source Academy auto-grader, but eventually became the foundations for the AI chatbot system within Source Academy, especially given the release of ChatGPT (GPT-3) a few months prior to the beginning of the project. After reviewing methods of program classification, we decided that the goals of the project did not align with prior works (which focused more on software engineering or debugging, and less relevant for educational contexts). We came up with the idea to extract specific function calls, operations, and data flow from programs; through flow analysis, we extracted features from the Abstract Syntax Trees (AST) of student programs, which were then used to train tree-based and LLM-based clustering methods. By comparing patterns found between these two clustering models, we managed to identify distinct patterns of what we termed "algorithmic processes", i.e. different implementations of the same algorithms that demonstrate how students fundamentally understood the algorithm. (An example is the iterative vs. recursive processes for recursive functions; you can refer to our textbook, Structure and Interpretation of Computer Programs Ch. 1.2.1, for more information.)

After we developed these clustering models, we also conducted a round of surveys with other CS1101S teaching assistants, and found that teaching assistants would indeed make the same classification as the clustering methods did - even though these were not based on known algorithms and programming paradigms (e.g. functional programming vs OOP). This suggested to us that there is a categorization that can be implicitly found by teaching assistants and tutors, as representations of meaningful differences in how students conceptually understood the course materials - and our clustering models were able to classify these successfully! This was also rather novel given the recency of LLMs back then. The report can be made available upon request!

This project was a catalyst for me to delve deeper into what we mean by "student understanding", and eventually led me to pursue a double major in psychology alongside computer science, as I wanted to understand the psychological mechanisms of the human mind and potentially combine it with machine learning.

Project Timeline

Project Gallery