Error Understanding in Program Code: A Systematic Study of LLM-DL Combinations for Multi-label Classification

Abstract

Programming is a core skill in CS and SE, yet identifying and resolving code errors remains challenging for practitioners. LLMs have shown remarkable capabilities in NL understanding, but how code-specialized LLMs behave when paired with DL sequence decoders, and which component of such a pipeline drives performance, remains insufficiently explored. This study presents a systematic evaluation of LLM-DL combinations for multi-label error classification (MLEC) of source code. Eight fine-tuned LLMs, including CodeT5, GraphCodeBERT, CodeT5+, UniXcoder, RoBERTa, RoBERTa with a narrowed learning-rate range, PLBART, and CoTexT, are integrated with GRU, LSTM, BiLSTM, and BiLSTM with an additive attention mechanism decoder on a real-world Python code error dataset. The resulting 32 model variants, tuned with Optuna, are assessed on a comprehensive multi-label metric suite. In single-run evaluation, CodeT5+ GRU performs best, with a weighted F1-score of 0.8243, average accuracy of 91.84%, exact match accuracy of 53.78%, Hamming loss of 0.0816, and one-error of 0.0708. To identify where this performance originates, seed-controlled baselines and component ablations are added with paired significance testing. Encoder choice has the largest effect: across four encoders sharing an identical linear classification head, the weighted F1-score spans 0.7846 to 0.8263, ordered by code specialization. On CodeT5+, the linear head exceeds the GRU hybrid under matched seeds by 0.0040 weighted F1 (p = 0.0013) while training about 24% faster. Max pooling outperforms mean and attention pooling, and explicitly modeling label interactions does not improve weighted F1 despite substantial label co-occurrence. These results identify encoder quality, rather than decoder complexity, as the primary lever for MLEC and support the development of scalable automated feedback tools for programming education and SE.