Deriving Instructional Insights from Human–LLM Co-Evaluation of Student Collaboration in Data-Centric Programming
This study combines a large language model (LLM) with expert qualitative coding to investigate how variations in computer-supported collaborative learning (CSCL) instructional designs shape collaboration dynamics in data-centric programming. Using an automated LLM-powered labeling pipeline, we classified and quantified indicators of collaborative competency across 73 team transcripts collected from two CSCL activities designed for data science courses. This analysis revealed that instructional design variations yield statistically significant differences in how teams co-construct shared knowledge—particularly in developing shared understanding of problems and solutions. Subsequent expert qualitative coding of these collaborative exchanges confirmed and enriched the LLM findings by revealing significant variations in both frequency and depth of collaboration attributable to design differences. Drawing on established CSCL frameworks and prior research, we derive actionable instructional insights for calibrating scripted collaboration activities in the context of data-centric programming, with the potential to inform similar designs in broader collaborative programming tasks. In addition, this work demonstrates a replicable methodology whereby learning scientists can effectively combine LLM-based automation with human qualitative analysis to evaluate and enhance CSCL in computer science education.