CSTutorBench: A Benchmark for Evaluating Small Language Models as Programming Tutors

Large language models are increasingly being viewed as AI tutors, but their use in schools raises concerns due to privacy issues, high costs, and reliance on proprietary solutions. Small language models (SLMs) are emerging as a promising alternative, but choosing the right model for a specific educational context is challenging—especially in niche areas such as block-based programming, which is virtually absent from the models’ training data.

How CSTutorBench Works

The new CSTutorBench benchmark evaluates language models as computer science teachers in VEX VR—a block-based programming environment for controlling robots. The benchmark includes:

• 17 scenarios based on real-world situations;
• evaluation using a pedagogical rubric grounded in teaching research and student feedback;
• a pipeline involving human evaluators and LLM judges for automated assessment.

Test Results

Preliminary tests covered 11 models ranging from 4 to 120 billion parameters. It turned out that the models perform well on superficial criteria—vocabulary and conversational tone—but face serious difficulties with more nuanced pedagogical behavior. In particular, they struggle to avoid giving direct answer-based prompts and to take the student’s learning history into account.

In the sample studied, the model family and the approach to instruction-tuning proved to be more significant predictors of teaching quality than the sheer number of parameters. However, due to the small number of models tested, the authors note that their confidence in this conclusion is limited.

The Impact of Prompt Engineering

Targeted prompt refinement, based on recent research in the field of educational prompt engineering, improved performance in 10 out of 11 models. This confirms that context-oriented benchmarks based on pedagogical principles play a key role in selecting SLMs for educational applications.

Source: cs.AI updates on arXiv.org