Searcharxiv⌕ Search
Home/Archive/arXiv

arXiv · 2609.28484

Measuring the Professional Educational Competence of Foundation Models

Keqian Li·Mingzi Zhang·Xiaolong Wang·Aimin Zhou
View original source ↗Read source document ↗Export metadata ↓

Abstract

Foundation models tutor, assess, and instruct at population scale, requiring externally defined measures of educational competence. Existing benchmarks emphasize difficult academic problems or isolated synthetic educational tasks rather than authentic teacher-entry standards. We introduce EDU 1.0 (Educational Due Diligence for Foundation Models), which uses teacher-entry assessments as proxies for educational competence rather than substitutes for human qualification. EDU 1.0 comprises 10,012 questions from teacher certification and recruitment examinations in the United States, China, and India, including the U.S. Praxis series, China's National Teacher Qualification Examination, and India's Kendriya Vidyalaya Sangathan examinations. It covers foundational literacy and knowledge, pedagogical principles, and subject-specific pedagogical expertise across language arts, mathematics, science, social science, and education practice. Across 36 foundation-model variants, the strongest proprietary model reaches a response-balanced score of 96.5%; the leading open-weight model trails by 2.1 percentage points, while the leading system deployable on a single accelerator reaches 92.2%. These aggregates conceal a shared limitation: all three systems score higher on general pedagogical principles than on assessments requiring pedagogy to be applied within a discipline. Their subject-assessment scores span 4.1-8.8 points, and their shortfall relative to general pedagogy widens from 2.6 to 4.1 points as capability declines. The outstanding requirement is therefore pedagogical content knowledge, the capacity to make particular subject matter teachable to particular learners, rather than general pedagogy or model scale.

Explore related subjects

cs.CY

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Keqian Li, Mingzi Zhang, Xiaolong Wang, Aimin Zhou. 2026-08-09. Measuring the Professional Educational Competence of Foundation Models. https://arxiv.org/abs/2609.28484

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

AI out-persuades expert humans in text-based dialogue

Many societal decisions are settled by contests of persuasion. Conversational AI is a powerful new entrant in these contests, but whether it can out-persuade skilled and highly incentivized humans has remained unclear. Here, in a series of four preregistered experiments (n = 18,978 conversations from 6,923 people), we pitted AI systems against a range of human persuaders, including laypeople, winners of a separately preregistered four-round online persuasion tournament, professional canvassers, and world championship debaters. We found that AI systems were reliably more persuasive than expert humans, even when expert humans chose their issues, researched in advance, underwent hours of live, structured practice, and were incentivized with £1,000 cash bonuses. In a follow-up study, AI's advantage persisted after experts received a coaching tool that let them practice against the AI that beat them, review their performance history, and see what AI would have said at key moments. We found converging evidence that AI's advantage stemmed from rapidly deploying larger quantities of information: after coaching, expert humans could tie an AI constrained to respond at human speeds and with human-length messages. In a final study, we show that AI's advantage extends to consequential real-world behavior: AI was more than 3x more effective than professional canvassers from a UK fundraising firm at raising real-money donations to Save the Children. Together, these results establish that frontier AI systems out-persuade expert humans in text-based conversation, with significant implications for political communication.

cs.CY↗
arXiv2026

Same Facts, Different Remedy: Linguistic Profiling and Arbitrariness in LLM and System One Decisions

Large language models are increasingly used to resolve consumer complaints, and the way complainants write varies with age, education and socioeconomic status (SES). We audit three remedy-decision systems using sociolinguistic rewrites of US consumer financial complaints that are instructed to preserve every fact: two generative LLMs that return a single label (GPT-4o-mini and GPT-5-mini) and a probabilistic "System One" model that returns a distribution over remedies (Jev). Using pre-registered hypotheses tested once on held-out complaints, we separate two phenomena that single-label audits conflate. At the group scale, older, college-educated and higher-SES styles receive slightly more generous remedies; these effects replicate but are small, with every monetary-remedy gap within 0.5 percentage points. At the individual scale, two style versions of the same complaint receive different monetary decisions 2.6-6.3% of the time. In a variance-component analysis of the probabilistic model, persona components shared across the two rewrite models account for about 1.6% of within-complaint variation. The persona gaps are associated mainly with delivery (vocabulary, grammar, politeness) rather than content. Positive controls show that all three systems respond 49-387 times more strongly to remedy-relevant facts than to style. Rewriting every complaint into one standard style reduced GPT-4o-mini's education and SES gaps by 94% and 87% while retaining fact sensitivity, but reversed two of the probabilistic model's gaps and did not make individual decisions more stable. Independent LLM auditors flagged material fact changes in 3.3% and 11.7% of rewrites, and instability persisted among unflagged rewrites. Fairness audits of LLM decision systems should report group disparities and individual instability as separate outcomes.

cs.CY↗
arXiv2026

Uniqueness-Aware Feature Engineering for Representing Humans in Algorithmic Decision Making

Fairness analyses of algorithmic decision systems typically treat the feature representation of individuals as an a priori fixed input. How well a feature representation captures what is actually decision-relevant for different people and their unique backgrounds, however, is typically left unexamined. We address this gap by modeling feature engineering as a stochastic selection process over the pooled characteristics as candidate features for a finite training population. Based on this model, we develop a new framework for uniqueness-aware feature engineering which operates in two layers. The first layer measures population heterogeneity using a multivariate hypergeometric model of feature prevalence. It shows how well an individual is represented by any given input representation and which individual of the population is "least represented" compared to their personal, reference characteristics. The second layer acts on this assessment using set-similarity measures to search for an optimum between representational fit of the population and predictive accuracy. Without assuming prior knowledge about unseen individuals, our framework grants represented humans agency over what counts as decision-relevant in their case, gives deploying institutions accountability through a publishable representation fit report, and provides each person subject to a decision transparency about their personal representation fit. We operationalize our framework as a six-step pipeline and show a proof-of-principle using synthetic data. Our approach offers a blueprint for making algorithmic decision systems accountable for how well they represent individuals during development and in production.

cs.CY↗