Peers can offer valuable advice, but effectively matching students with relevant peer recommendations remains a challenge. Prior research proposed a reflection recommender system that shares students’ responses to reflection questions about challenges they faced with peers who shared similar challenges. To refine the responses, we incorporated an LLM approach to generate personalized responses, filter irrelevant reflections, and provide advice directly relevant to the students’ challenges. We also formatted the output in a conversational-style response in contrast to the unmodified list of student solutions. We compare the original student-challenge recommender system (SCS) to our LLM-integrated student-challenge recommender system (LLM-SCS) in a comparative study of three computer science courses using A/B testing to limit the impact of order. We asked students to rate solutions from both systems on a 7-point Likert scale scale and provide free-response feedback. Based on participants from three computer science courses with 142 total solution ratings, we found that the average rating of LLM-SCS solutions was 5.23, and SCS solutions with an average rating of 4.87. A t-test demonstrated that the differences were not statistically significant, demonstrating that students find the quality of LLM-SCS and SCS responses are comparable. An investigation of free-response feedback reveals that students express diverse needs and preferences; some preferred the conversational style of the LLM-SCS response, and others valued the uniquely tailored advice of the original response. In this work, we explore these differences in great depth. In conclusion, we find that the LLMs can help organize and refine peer student advice.