Deriving Instructional Insights from Human–LLM Co-Evaluation of Student Collaboration in Data-Centric ProgrammingGlobal
This study combines a large language model (LLM) with expert qualitative coding to investigate how variations in computer-supported collaborative learning (CSCL) instructional designs shape collaboration dynamics in data-centric programming. Using an automated LLM-powered labeling pipeline, we classified and quantified indicators of collaborative competency across 73 team transcripts collected from two CSCL activities designed for data science courses. This analysis revealed that instructional design variations yield statistically significant differences in how teams co-construct shared knowledge—particularly in developing shared understanding of problems and solutions. Subsequent expert qualitative coding of these collaborative exchanges confirmed and enriched the LLM findings by revealing significant variations in both frequency and depth of collaboration attributable to design differences. Drawing on established CSCL frameworks and prior research, we derive actionable instructional insights for calibrating scripted collaboration activities in the context of data-centric programming, with the potential to inform similar designs in broader collaborative programming tasks. In addition, this work demonstrates a replicable methodology whereby learning scientists can effectively combine LLM-based automation with human qualitative analysis to evaluate and enhance CSCL in computer science education.
Fri 20 FebDisplayed time zone: Central Time (US & Canada) change
13:40 - 15:00 | From Unplugged Activities to LLM Insights: Rethinking How We Teach Intro Computing CoursesPapers at Meeting Room 275 Chair(s): Miranda Parker | ||
13:40 20mTalk | CS Unplugged in Gateway Computing Courses: A Collaborative, Active Learning Approach in Introductory Computing Papers Hyesung Park Georgia Gwinnett College, Evelyn Brannock Georgia Gwinnett College, Tacksoo Im Georgia Gwinnett College, Wei Jin Georgia Gwinnett College, Sunae Shin Georgia Gwinnett College, Xin Xu Georgia Gwinnett College, David Kerven Georgia Gwinnett College | ||
14:00 20mTalk | Deriving Instructional Insights from Human–LLM Co-Evaluation of Student Collaboration in Data-Centric ProgrammingGlobal Papers Marshall An Carnegie Mellon University, Christine Kwon Carnegie Mellon University, Yoonjae Lee Seoul National University, Ji-Hyeon Hur Seoul National University, Dongho LEE Dalhousie University, Vincent Huai Carnegie Mellon University, Barry Zheng Carnegie Mellon University, Matthew Yu Carnegie Mellon University, Joana Liu Carnegie Mellon University, Jenny Pugh Carnegie Mellon University, Gahgene Gweon Graduate School of Convergence Science and Technology, Seoul National University, John Stamper Carnegie Mellon University | ||
14:20 20mTalk | Repetition Meets Context: Teaching CS1 Through Two Scientific DomainsGlobal Papers Meiying Qin York University, Jade Atallah York University, Hovig Kouyoumdjian York University, Jonatan Schroeder York University, Larry Yueli Zhang York University, May Haidar York University | ||
14:40 20mTalk | Systems for Scaling Accessibility Efforts in Large Computing Courses Papers Ritesh Kanchi Harvard University, Miya Natsuhara University of Washington, Seattle, Matt Wang University of Washington Media Attached | ||