# Hints-In-Browser: Benchmarking Language Models for Programming Feedback Generation

**Authors:** Nachiket Kotalwar, Alkis Gotovos, Adish Singla  
**arXiv:** [2406.05053](https://arxiv.org/abs/2406.05053)  
**Date:** 2024-06-07  
**Summary page:** https://www.summarizepaper.com/en/paper/2406.05053v1/

## Key points

- Authors explore generative AI and large language models for improving programming education
- Focus on generating personalized feedback and hints for learners
- Benchmarking various language models for programming feedback generation across multiple criteria
- Leveraging advancements in in-browser inference technology for cost-effectiveness and data privacy protection
- Introducing fine-tuning pipeline based on synthetic data generated by GPT-4 to enhance feedback quality of smaller models

## Summary

, , , , 

In their paper titled "Hints-In-Browser: Benchmarking Language Models for Programming Feedback Generation," authors Nachiket Kotalwar, Alkis Gotovos, and Adish Singla explore the potential of generative AI and large language models in improving programming education. The focus is on generating personalized feedback and hints for learners, with recent efforts aiming to enhance the quality of generated feedback to match that of human tutors. The study benchmarks various language models for programming feedback generation across multiple performance criteria, including quality, cost, time efficiency, and data privacy. A key innovation lies in leveraging advancements in in-browser inference technology, enabling these models to run directly within web browsers. This approach offers advantages in terms of cost-effectiveness and data privacy protection. To enhance the feedback quality of smaller models compatible with in-browser inference engines, the authors introduce a fine-tuning pipeline based on synthetic data generated by GPT-4. They demonstrate the effectiveness of fine-tuned Llama3-8B and Phi3-3.8B 4-bit quantized models using WebLLM's in-browser inference engine on three distinct Python programming datasets. In addition to presenting their findings, the authors plan to release the full implementation of their work along with a web application and datasets to support further research on in-browser language models. By addressing a range of performance criteria beyond just quality, this study contributes valuable insights into optimizing programming education through innovative AI-driven approaches.