arXiv subjects

Keyang Qian

Publications and source records attributed to Keyang Qian.

5 recordsLinked to original sources

arXiv2025

Dean of LLM Tutors: A Framework for Automated Quality Review of AI-generated Feedback

Large language model (LLM) tutors are increasingly used to generate educational feedback, but existing research has focused mainly on feedback generation rather than feedback evaluation. As a result, LLM-generated feedback may offer limited pedagogical value and carry risks of hallucination. The current study introduces DeanLLM, an automated review framework for comprehensively evaluating feedback generated by LLM tutors before it is shared with students. We developed a 16-dimension evaluation framework covering feedback content, educational effectiveness, and hallucination risks, and validated it using using human-expert annotations of LLM-generated tutor feedback on synthetic computer science assignment submissions derived from real coursework. We then examined whether LLMs could serve as automated LLM-generated tutor feedback reviewers, and used the best-performing reviewer to benchmark tutor feedback generated by 10 commercial LLMs. Psychometric analyses supported the reliability of the proposed framework and showed that human reviewers tended to evaluate feedback holistically, whereas the LLM reviewer separated rubric dimensions more mechanically. Standard zero-shot and few-shot prompting showed limited agreement with human experts for content-quality judgments. Supervised fine-tuning of GPT-4.1 with human-labelled examples containing scores only, without explanatory rationales, achieved the strongest alignment with expert judgments. Reasoning LLMs were particularly effective at hallucination detection and produced automated tutor feedback with stronger educational effectiveness and factuality than lightweight models. The findings indicate that DeanLLM offers a scalable way for automatically improving the reliability and safety of LLM tutor feedback, while also demonstrating that reviewer calibration and model choice remain critical for educational deployment.

cs.CY↗

arXiv2025

SCALEFeedback: A Large-Scale Dataset of Synthetic Computer Science Assignments for LLM-generated Educational Feedback Research

Using Large Language Models (LLMs) to give educational feedback to students for their assignments has attracted much attention in the AI in Education (AIED) field. Yet, there is currently no large-scale open-source dataset of student assignments that includes detailed assignment descriptions, rubrics, and student submissions across various courses. As a result, research on generalisable methodology for automatic generation of effective and responsible educational feedback remains limited. In this paper, we introduce a synthetic computer science university assignment dataset for LLM-based educational feedback research, called SCALEFeedback (Synthetic Computer science Assignments for LLM Educational Feedback Research). The dataset is generated via Sophisticated Assignment Mimicry (SAM) framework specifically designed to synthesise this dataset and that utilizes one-to-one LLM-based imitation from real assignment descriptions, rubrics, and student submissions. Our open-source dataset contains 10,000 synthetic student submissions spanning 155 assignments across 59 university-level computer science courses. Technical validation confirmed that the synthetic dataset closely resembles real data while successfully eliminating personally identifiable information present in the source material. The creation of this dataset is a valuable contribution to researchers who aim to develop LLM-based generalisable methods for offering high-quality, automated educational feedback in a scalable way.

cs.CY↗

arXiv2025

Towards Reliable Generative AI-Driven Scaffolding: Reducing Hallucinations and Enhancing Quality in Self-Regulated Learning Support

Generative Artificial Intelligence (GenAI) holds a potential to advance existing educational technologies with capabilities to automatically generate personalised scaffolds that support students' self-regulated learning (SRL). While advancements in large language models (LLMs) promise improvements in the adaptability and quality of educational technologies for SRL, there remain concerns about the hallucinations in content generated by LLMs, which can compromise both the learning experience and ethical standards. To address these challenges, we proposed GenAI-enabled approaches for evaluating personalised SRL scaffolds before they are presented to students, aiming for reducing hallucinations and improving the overall quality of LLM-generated personalised scaffolds. Specifically, two approaches are investigated. The first approach involved developing a multi-agent system approach for reliability evaluation to assess the extent to which LLM-generated scaffolds accurately target relevant SRL processes. The second approach utilised the "LLM-as-a-Judge" technique for quality evaluation that evaluates LLM-generated scaffolds for their helpfulness in supporting students. We constructed evaluation datasets, and compared our results with single-agent LLM systems and machine learning approach baselines. Our findings indicate that the reliability evaluation approach is highly effective and outperforms the baselines, showing almost perfect alignment with human experts' evaluations. Moreover, both proposed evaluation approaches can be harnessed to effectively reduce hallucinations. Additionally, we identified and discussed bias limitations of the "LLM-as-a-Judge" technique in evaluating LLM-generated scaffolds. We suggest incorporating these approaches into GenAI-powered personalised SRL scaffolding systems to mitigate hallucination issues and improve the overall scaffolding quality.

cs.CY↗

arXiv2023

Blockchain-enabled Parametric Solar Energy Insurance via Remote Sensing

Despite its popularity, the nature of solar energy is highly uncertain and weather dependent, affecting the business viability and investment of solar energy generation, especially for household users. To stabilize the income from solar energy generation, there have been limited traditional options, such as using energy storage to pool excessive solar energy in off-peak periods or financial derivatives from future markets to hedge energy prices. In this paper, we explore a novel idea of "parametric solar energy insurance", by which solar panel owners can insure their solar energy generation based on a verifiable geographically specific index (surface solar irradiation). Parametric solar energy insurance offers opportunities of financial subsidies for insufficient solar energy generation and amortizes the fluctuations of renewable energy generation geographically. Furthermore, we propose to leverage blockchain and remote sensing (satellite imagery) to provide a publicly verifiable platform for solar energy insurance, which not only automates the underwriting and claims of a solar energy insurance policy, but also improves its accountability and transparency. We utilize the state-of-the-art succinct zero-knowledge proofs (zk-SNARK) to realize privacy-preserving blockchain-based solar energy insurance on real-world permissionless blockchain platform Ethereum.

cs.CR↗

arXiv2023

Privacy-preserving Blockchain-enabled Parametric Insurance via Remote Sensing and IoT

Traditional Insurance, a popular approach of financial risk management, has suffered from the issues of high operational costs, opaqueness, inefficiency and a lack of trust. Recently, blockchain-enabled "parametric insurance" through authorized data sources (e.g., remote sensing and IoT) aims to overcome these issues by automating the underwriting and claim processes of insurance policies on a blockchain. However, the openness of blockchain platforms raises a concern of user privacy, as the private user data in insurance claims on a blockchain may be exposed to outsiders. In this paper, we propose a privacy-preserving parametric insurance framework based on succinct zero-knowledge proofs (zk-SNARKs), whereby an insuree submits a zero-knowledge proof (without revealing any private data) for the validity of an insurance claim and the authenticity of its data sources to a blockchain for transparent verification. Moreover, we extend the recent zk-SNARKs to support robust privacy protection for multiple heterogeneous data sources and improve its efficiency to cut the incurred gas cost by 80%. As a proof-of-concept, we implemented a working prototype of bushfire parametric insurance on real-world blockchain platform Ethereum, and present extensive empirical evaluations.

cs.CR↗