Searcharxiv⌕ Search
Home/Archive/arXiv

arXiv · 2609.31976

Toward Embedding-Based Psychometrics: Structural Modeling of Assessment-Item Semantics With Contextual Scores

Jinsong Chen·Shi-Ting Chen
View original source ↗Read source document ↗Export metadata ↓

Abstract

Contextual scores represent assessment items through their similarities to reference words in an external corpus. We examine the semantic structure of scores for 40 TIMSS mathematics scored units using a partially specified two-step factor procedure. A search across factor counts identifies a persistent seven-group structure under the featured construction. Subsequent comparisons consistently favor a general dimension alongside group associations, although individual group memberships remain sensitive to some specification choices. Item examples distinguish recurring, cross-domain, sensitive, and imposed associations. Simpler and unrestricted references clarify the contribution and limits of the anchored representation: it improves on a single factor but does not achieve the lowest working Bayesian information criterion (BIC). A separate response benchmark compares three initial Q constructions and their Hull-PVAF revisions under higher-order and saturated attribute distributions. Among these diagnostic models, BIC favors the official content framework and the Akaike information criterion (AIC) favors its direct four-factor augmentation, but a matched unidimensional two-parameter logistic model has lower AIC and BIC than all twelve conditions. These findings support a conditional semantic representation while limiting direct diagnostic interpretation. We discuss learned text-assisted response calibration as a prospective application requiring a larger calibrated item bank and independent evaluation.

Explore related subjects

cs.CL

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Jinsong Chen, Shi-Ting Chen. 2026-09-25. Toward Embedding-Based Psychometrics: Structural Modeling of Assessment-Item Semantics With Contextual Scores. https://arxiv.org/abs/2609.31976

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

LaSEr-Edit: Localized Span-level Error Editing with Energy-based Localization

As large language models (LLMs) are widely adopted in real-world applications, it has become critical to ensure LLMs satisfy safety constraints, such as non-toxicity and logical consistency, as well as task- and situation-specific constraints. Controlling the output through instructions is a simple and tempting approach; however, it remains brittle, is opaque in how it influences model behavior, and thus cannot reliably ensure constraint satisfaction. Moreover, most recent controlled text generation (CTG) methods require access to the internal components of language models--such as weights or logits--making them incompatible with popular API-based LLMs. In this work, we propose LaSEr-Edit, a constraint-satisfying text revision method that can be applied to any LLMs, black- or white-box. We first find that lightweight, task-specific energy-based models (EBMs) achieve error-localization performance competitive with or even better than that of much larger LLMs, while operating substantially faster. Based on this finding, we propose two variants of text revision methods that incorporate energy-based error localization: LaSEr-LLM Edit, which instructs an LLM to edit text given EBM-predicted error spans, and LaSEr-EBM Edit, which uses the EBM not only for localization but also for editing by reranking edit candidates. Through experiments in diverse single-constraint control tasks, we show that LaSEr-LLM Edit controls text better than plain LLM-based editing in most of the tasks. We also find that LaSEr-EBM Edit further improves the control performance of LaSEr-LLM Edit and achieves among the strongest controllability across all tasks. Furthermore, we find that LaSEr-Edit, especially LaSEr-EBM Edit, performs well even when multiple constraints are controlled simultaneously.

cs.CL↗
arXiv2026

No Free Labels: Limitations of LLM-as-a-Judge Without Human Grounding

Reliable evaluation of large language models (LLMs) is critical as their deployment rapidly expands, particularly in high-stakes domains such as business and finance. The LLM-as-a-Judge framework, which uses prompted LLMs to evaluate response quality, is appealing due to its scalability, low cost, and strong correlations with human stylistic preferences. However, it remains unclear how accurately these methods can assess response quality in domains where correctness matters more than style. To address this gap, we introduce the Business and Finance Fundamentals Benchmark (BFF-Bench), a dataset of 160 challenging questions and long-form responses authored by financial professionals. These experts subsequently evaluated the correctness of 1,200 responses generated by a diverse set of LLMs on both BFF-Bench and a challenging subset of MT-Bench. With this expert-annotated dataset of judgments (VERDICTS), we analyze the agreement between a suite of automated grading methods and human experts. While we observe that LLM Judges are more reliable than other grading methods, our findings reveal a clear pattern in LLM Judge performance: when not provided with a correct reference, judges show high agreement with human experts only on questions the judges were able to correctly answer themselves. We demonstrate that providing the judges with expert-written references largely mitigates this issue, highlighting the limits of using LLM-as-a-Judge without any form of human verification.

cs.CL↗
arXiv2026

Adaptive Activation Steering for Efficient LLM Reasoning via Closed-Loop PID Control

Reasoning LLMs trained with long chain-of-thought often overthink: they spend tokens on redundant reflection and transitions that inflate cost without improving accuracy. Static activation steering (e.g.\ SEAL) suppresses such content with a fixed vector, but applies the same strength regardless of how redundant the current chunk actually is. We describe PID-steering, a training-free, decoding-time method that modulates the steering strength with a PID controller driven by a lightweight chunk-level redundancy classifier. On a subset of GSM8K with DeepSeek-R1-Distill-Qwen-1.5B, the method improves accuracy from 85.7\% to 89.6\% (+3.9 pp) while cutting average output length from 1026 to 790 tokens ($-$23\%). We report it as a small-scale proof of concept rather than a benchmark result.

cs.CL↗