arXiv subjects

Joshua Kavner

Publications and source records attributed to Joshua Kavner.

8 recordsLinked to original sources

arXiv2026

Judge Reliability Harness: Stress Testing the Reliability of LLM Judges

We present the Judge Reliability Harness, an open source library for constructing validation suites that test the reliability of LLM judges. As LLM based scoring is widely deployed in AI benchmarks, more tooling is needed to efficiently assess the reliability of these methods. Given a benchmark dataset and an LLM judge configuration, the harness generates reliability tests that evaluate both binary judgment accuracy and ordinal grading performance for free-response and agentic task formats. We evaluate four state-of-the-art judges across four benchmarks spanning safety, persuasion, misuse, and agentic behavior, and find meaningful variation in performance across models and perturbation types, highlighting opportunities to improve the robustness of LLM judges. No judge that we evaluated is uniformly reliable across benchmarks using our harness. For example, our preliminary experiments on judges revealed consistency issues as measured by accuracy in judging another LLM's ability to complete a task due to simple text formatting changes, paraphrasing, changes in verbosity, and flipping the ground truth label in LLM-produced responses. The code for this tool is available at: https://github.com/RANDCorporation/judge-reliability-harness

cs.AI↗

arXiv2026

Stabilizing Welfare-Maximizing Decisions via Endogenous Transfers

Many multi-agent systems depend on collective decisions made by self-interested agents, which raises deep questions about coalition formation and stability. We study social choice with endogenous, outcome-contingent transfers, where agents voluntarily form contracts that redistribute utility depending on the collective decision, allowing for fully strategic coalition formation. We show that under consensus rules, individually rational strong Nash equilibria (IR-SNE) always exist, implementing welfare-maximizing outcomes with feasible transfers, and provide a simple, efficient algorithm to construct them. For more general anonymous, monotonic, and resolute rules, we identify necessary conditions for viable deviations, significantly limiting the possibility of destabilizing coalitions. By bridging cooperative and noncooperative perspectives, our approach shows that transferable utility can achieve core-like stability, restoring efficiency and budget balance even where classical impossibility results generally apply. Overall, this framework offers a practical and robust way to coordinate large-scale strategic multi-agent systems.

econ.TH↗

arXiv2025

Bridging Theory and Perception in Fair Division: A Study on Comparative and Fair Share Notions

The allocation of resources among multiple agents is a fundamental problem in both economics and computer science. In these settings, fairness plays a crucial role in ensuring social acceptability and practical implementation of resource allocation algorithms. Traditional fair division solutions have given rise to a variety of approximate fairness notions, often as a response to the challenges posed by non-existence or computational intractability of exact solutions. However, the inherent incompatibility among these notions raises a critical question: which concept of fairness is most suitable for practical applications? In this paper, we examine two broad frameworks -- threshold-based and comparison-based fairness notions -- and evaluate their perceived fairness through a comprehensive human subject study. Our findings uncover novel insights into the interplay between perception of fairness, theoretical guarantees, the role of externalities and subjective valuations, and underlying cognitive processes, shedding light on the theory and practice of fair division.

cs.GT↗

arXiv2024

Average-Case Analysis of Iterative Voting

Iterative voting is a natural model of repeated strategic decision-making in social choice theory when agents have the opportunity to update their votes prior to finalizing the group decision. Prior work has analyzed the efficacy of iterative plurality on the welfare of the chosen outcome at equilibrium, relative to the truthful vote profile, via an adaptation of the price of anarchy. However, prior analyses have only studied the worst- and average-case performances when agents' preferences are distributed by the impartial culture. This work extends average-case analysis comprehensively across three alternatives and distinguishes under which of agents' preference distributions iterative plurality improves or degrades asymptotic welfare.

cs.GT↗

arXiv2023

Distribution of Chores with Information Asymmetry

A well-regarded fairness notion when dividing indivisible chores is envy-freeness up to one item (EF1), which requires that pairwise envy can be eliminated by the removal of a single item. While an EF1 and Pareto optimal (PO) allocation of goods can always be found via well-known algorithms, even the existence of such solutions for chores remains open, to date. We take an epistemic approach utilizing information asymmetry by introducing dubious chores--items that inflict no cost on receiving agents but are perceived costly by others. On a technical level, dubious chores provide a more fine-grained approximation of envy-freeness than EF1. We show that finding allocations with minimal number of dubious chores is computationally hard. Nonetheless, we prove the existence of envy-free and fractional PO allocations for $n$ agents with only $2n-2$ dubious chores and strengthen it to $n-1$ dubious chores in four special classes of valuations. Our experimental analysis demonstrates that often only a few dubious chores are needed to achieve envy-freeness.

cs.GT↗

arXiv2023

Convergence of Multi-Issue Iterative Voting under Uncertainty

We study the effect of strategic behavior in iterative voting for multiple issues under uncertainty. We introduce a model synthesizing simultaneous multi-issue voting with Meir, Lev, and Rosenschein (2014)'s local dominance theory and determine its convergence properties. After demonstrating that local dominance improvement dynamics may fail to converge, we present two sufficient model refinements that guarantee convergence from any initial vote profile for binary issues: constraining agents to have O-legal preferences and endowing agents with less uncertainty about issues they are modifying than others. Our empirical studies demonstrate that although cycles are common when agents have no uncertainty, introducing uncertainty makes convergence almost guaranteed in practice.

cs.GT↗

arXiv2022

Epistemic vs. Counterfactual Fairness in Allocation of Resources

Resource allocation is fundamental to a variety of societal decision-making settings, ranging from the distribution of charitable donations to assigning limited public housing among interested families. A central challenge in this context is ensuring fair outcomes, which often requires balancing conflicting preferences of various stakeholders. While extensive research has been conducted on theoretical and algorithmic solutions within the fair division framework, much of this work neglects the subjective perception of fairness by individuals. This study focuses on the fairness notion of envy-freeness (EF), which ensures that no agent prefers the allocation of another agent according to their own preferences. While the existence of exact EF allocations may not always be feasible, various approximate relaxations, such as counterfactual and epistemic EF, have been proposed. Through a series of experiments with human participants, we compare perceptions of fairness between three widely studied counterfactual and epistemic relaxations of EF. Our findings indicate that allocations based on epistemic EF are perceived as fairer than those based on counterfactual relaxations. Additionally, we examine a variety of factors, including scale, balance of outcomes, and cognitive effort involved in evaluating fairness and their role in the complexity of reasoning across treatments.

cs.GT↗

arXiv2021

Strategic Behavior is Bliss: Iterative Voting Improves Social Welfare

Recent work in iterative voting has defined the additive dynamic price of anarchy (ADPoA) as the difference in social welfare between the truthful and worst-case equilibrium profiles resulting from repeated strategic manipulations. While iterative plurality has been shown to only return alternatives with at most one less initial votes than the truthful winner, it is less understood how agents' welfare changes in equilibrium. To this end, we differentiate agents' utility from their manipulation mechanism and determine iterative plurality's ADPoA in the worst- and average-cases. We first prove that the worst-case ADPoA is linear in the number of agents. To overcome this negative result, we study the average-case ADPoA and prove that equilibrium winners have a constant order welfare advantage over the truthful winner in expectation. Our positive results illustrate the prospect for social welfare to increase due to strategic manipulation.

cs.GT↗