The proliferation of generative AI tools like ChatGPT and GitHub Copilot has fundamentally challenged project-based learning and code-based assessments in CS education. Students can generate sophisticated solutions without genuine understanding, making it difficult to distinguish between authentic learning and AI-assisted completion. Educators have responded by returning to oral examinations to verify comprehension, but conducting oral assessments at scale remains impractical for most courses.

This talk discusses AI-driven interactive chat-based oral examinations as a scalable, authentic assessment tool for large CS courses. We built a tool, integrated into Classmoji, a GitHub-native learning management system, that conducts automated conversational assessments that probe students’ understanding of their own code submissions. Rather than simply verifying functional correctness, as traditional autograders do, the system evaluates conceptual understanding through adaptive dialogue.

Drawing on pedagogical frameworks including the Depth of Knowledge model and Socratic questioning methods employed by tools like Stanford’s Sherpa and Georgia Tech’s Socratic Mind, our system generates targeted questions about implementation choices, algorithmic reasoning, and underlying concepts. The AI adapts follow-up questions based on student responses, creating personalized assessment experiences that scaffold understanding through iterative attempts, which may also reduce some challenges associated with traditional oral exams, including anxiety for shy students and examiner biases.

We demonstrate the system’s integration with GitHub-based workflows and discuss implications for maintaining academic integrity while supporting authentic learning in an AI-ubiquitous environment. Our approach addresses the paradox of using AI to combat AI-generated work, transforming a threat into an opportunity for scaled, personalized evaluation.