To Tell or to Ask? Comparing the Effects of Targeted vs. Socratic AI Hints
As enrollment in CS1 courses continues to increase, extensive research has focused on autonomous support to offer personalized assistance for struggling students at scale. However, it is crucial that these intervention techniques do not inadvertently hinder the development of higher-order, computational thinking skills for novice programmers. This poster extends upon the research on LLM-based support by assessing the short- and long-term student outcomes from two carefully prompt-engineered, LLM-generated hint styles: Targeted and Socratic hints. A randomized controlled trial with 178 students was conducted over two semesters in a CS1 course at a large university, allowing students to interact with a hint generation AI agent while attempting course coding assignments. In the short-term, students receiving Socratic hints spent more time, took more attempts, and used more keystrokes to solve coding questions, while committing more repeat errors. Furthermore, this short-term loss in debugging efficiency is not counteracted by any evidence of an improvement in long-term student outcomes. Further research is being conducted to quantify the tradeoff between short-term performance and long-term, higher-order coding skill improvement in the development of educational AI agents.