Searcharxiv⌕ Search

arXiv subjects

Cassandra Handan-Nader

Publications and source records attributed to Cassandra Handan-Nader.

2 recordsLinked to original sources

DOCKET: A Pluralistic Benchmark for Evaluating AI-Assisted Policy Feedback in U.S. Rulemaking

As AI models advance and impact deliberative democracies, there is a growing need to assess their support of pluralistic viewpoints. Most pluralism benchmarks are insufficient for this: they are not grounded in diversity (i.e., stakeholders across different frames, skills and needs), quality (i.e., rigorous and technical debates), and impact (i.e., invested deliberators and real-world outcomes). To address this, we first (i) curate DOCKET, the most comprehensive dataset of the U.S. Federal Notice & Comment process to date. DOCKET consists of 42.3M public comments on 22,400 regulations across 44 government agencies between 2016 and 2026 and contains millions of policy debates; we link public comments to particular policy items proposed by regulators to track the public's attention and the responsiveness of regulators to public input. (ii) Next, we demonstrate the pluralistic utility of DOCKET - we show that public commenters have a wide breadth of perspectives with significant depth in policy expertise and that regulators are often responsive. (iii) Finally, we present DOCKETBench, the first systematic evaluation of LLMs' ability to represent human policy perspectives in federal rulemaking. Only 18-30% of LLM-generated feedback aligns with real human comments. LLMs critique proposals broadly but generally, lacking the specific grounding, citations, and lived-experience arguments that humans bring to bear; in addition, most models we test systematically agree with government perspectives (overlapping with up to 68% of their claims) and agree least with labor unions (as little as 26% of their claims), demonstrating a fundamental skew in perspectives that even tools like persona modeling do not solve. Taken together, we reveal fundamental gaps in how language models navigate real policy issues, and DOCKETBench serves as a testbed for closing those gaps in a crucial real-world domain.

cs.CY↗

Statistical Uncertainty in Word Embeddings: GloVe-V

Static word embeddings are ubiquitous in computational social science applications and contribute to practical decision-making in a variety of fields including law and healthcare. However, assessing the statistical uncertainty in downstream conclusions drawn from word embedding statistics has remained challenging. When using only point estimates for embeddings, researchers have no streamlined way of assessing the degree to which their model selection criteria or scientific conclusions are subject to noise due to sparsity in the underlying data used to generate the embeddings. We introduce a method to obtain approximate, easy-to-use, and scalable reconstruction error variance estimates for GloVe (Pennington et al., 2014), one of the most widely used word embedding models, using an analytical approximation to a multivariate normal model. To demonstrate the value of embeddings with variance (GloVe-V), we illustrate how our approach enables principled hypothesis testing in core word embedding tasks, such as comparing the similarity between different word pairs in vector space, assessing the performance of different models, and analyzing the relative degree of ethnic or gender bias in a corpus using different word lists.

cs.CL↗