arXiv subjects

Daniel G. Goldstein

Publications and source records attributed to Daniel G. Goldstein.

13 recordsLinked to original sources

arXiv2026

Use and Effects of LLMs in Peer Review: A Randomized Experiment and Survey at ICML 2026

LLMs are rapidly reshaping peer review, making it important to understand how reviewers use them in practice and how different LLM-use policies affect review outcomes. We investigate these questions through a randomized experiment and an anonymous post-survey at ICML 2026, a major machine learning conference involving over 24,000 papers and 17,000 reviewers. Reviewers were assigned to either a conservative policy prohibiting all LLM use or a permissive policy allowing limited assistance, with randomization among a subset of main-track papers and reviewers. Policy assignment had near-zero effects on final paper decisions, paper scores, and reviewer confidence, although reviews under the permissive policy were 5.5-7% longer. Post-survey responses (N=1,486) revealed diverse attitudes toward LLMs and substantial noncompliance: 22.5% of conservative-policy reviewers reported using an LLM despite the prohibition, and 36.5% of permissive-policy reviewers reported at least one explicitly disallowed use. We discuss implications for future peer-review policy and tool design.

cs.HC↗

arXiv2026

Bringing Everyone to the Table: An Experimental Study of LLM-Facilitated Group Decision Making

Group decision-making often suffers from uneven information sharing, hindering decision quality. While large language models (LLMs) have been widely studied as aids for individuals, their potential to support groups of users, potentially as facilitators, is relatively underexplored. We present a pre-registered randomized experiment with 1,475 participants assigned to 281 live groups completing a hidden profile task--selecting an optimal city for a hypothetical sporting event--under one of four facilitation conditions: no facilitation, a one-time message prompting information sharing, a human facilitator, or an LLM (GPT-4o) facilitator. We find that LLM facilitation increased information shared within a discussion by raising the minimum level of engagement with the task among group members, and that these gains came at limited cost in terms of participants' attitudes towards the task, their group, or their facilitator. Whether by human or AI, there was no significant effect of facilitation on the final decision outcome, suggesting that even substantial but partial increases in information sharing were insufficient to overcome the hidden profile effect studied. To support the design and evaluation of LLM-mediated group decision-making systems, we release our data and our experimental platform, the Group-AI Interaction Laboratory (GRAIL), as an open-source tool.

cs.HC↗

arXiv2026

From Augmentation to Reconstruction: Guiding the AI Disruption to the Good Place

Artificial intelligence feels omnipresent, yet the disruption many expect has not fully arrived. The main reason is not model capability, nor even the tools built to harness those models. Rather, most organizations are still using AI to accelerate workflows designed for a pre-AI world. We offer a three-stage lens: Augmentation, Automation, and Reconstruction, and argue that the most consequential disruption resides in the third stage where workflows and markets are rebuilt around delegation, machine-to-machine interaction, continuous monitoring, and auditable constraints. Achieving this system-level transformation takes time: it requires trust and accountability infrastructure, machine-legible and interoperable data and interfaces, the design and adoption of these new workflows, and economic incentives that favor reconstruction rather than local optimization: the complementary investments that produce the familiar "productivity J-curve" of general-purpose technologies. We illustrate this transition through examples in consumer markets, education, news, and coding. Finally, we emphasize a normative point: the agentic future is not predetermined. Leaders must both skate to where the puck is going and actively steer it toward a good place, ensuring innovation delivers welfare gains felt by businesses and consumers around the world.

econ.GN↗

arXiv2026

Agentic Markets: Equilibrium Effects of Improving Consumer Search

Motivated by agentic markets -- two-sided markets in which consumers and businesses are assisted by AI tools that facilitate consumers' search -- we study the impact of improved search technology on learning and welfare in markets. We put forth a model where consumers engage in costly search to acquire signals of product fit prior to purchase. The market tracks indications of fit for searched products and indications of quality for chosen products, thereby guiding searches. We characterize the long-run steady-state of the resulting dynamics as well as the impact of improving search technology. We find cheaper search improves learning and consumer surplus, whereas more informative search can degrade both unless the market learns as much as consumers about the products by, for example, ``reading the transcripts'' of agentic conversations. Finally, we consider the impact of search improvements on how businesses set prices. At equilibrium prices in symmetric markets, consumer surplus is improved by cheaper search but may be decreased by more informative search, due to weakened inter-business competition.

cs.GT↗

arXiv2026

Coach not crutch: Evidence that AI can improve writing skill despite reducing effort

In a series of highly-powered empirical studies, we examine the intuition that by sparing effort, using AI inevitably hinders learning. First, in a nationally representative survey of young adults, the majority expressed the view that using AI makes people lazier and less capable. Next, in a random-assignment experiment, we gave participants a tutorial on best practices in professional writing, then provided one group with access to an AI writing tool and asked another to practice writing on their own. Those who practiced with AI indeed exerted less effort while practicing -- yet wrote better cover letters in no-AI writing tests. In a second experiment with more rigorous control conditions, access to AI improved writing more than either googling cover letter examples and tips or receiving personalized feedback on their practice letters from experienced human editors. A third experiment explained these learning gains by showing that AI can teach by example: participants who viewed a cover letter that had been revised by the AI tool but did no further practice improved their writing as much as those who practiced writing with the original AI tool. Collectively, these pre-registered experiments suggest that AI can exert opposing effects on effort and learning rate -- making it possible in at least some cases to work less and learn more.

cs.HC↗

arXiv2025

Magentic Marketplace: An Open-Source Environment for Studying Agentic Markets

As LLM agents advance, they are increasingly mediating economic decisions, ranging from product discovery to transactions, on behalf of users. Such applications promise benefits but also raise many questions about agent accountability and value for users. Addressing these questions requires understanding how agents behave in realistic market conditions. However, previous research has largely evaluated agents in constrained settings, such as single-task marketplaces (e.g., negotiation) or structured two-agent interactions. Real-world markets are fundamentally different: they require agents to handle diverse economic activities and coordinate within large, dynamic ecosystems where multiple agents with opaque behaviors may engage in open-ended dialogues. To bridge this gap, we investigate two-sided agentic marketplaces where Assistant agents represent consumers and Service agents represent competing businesses. To study these interactions safely, we develop Magentic-Marketplace -- a simulated environment where Assistants and Services can operate. This environment enables us to study key market dynamics: the utility agents achieve, behavioral biases, vulnerability to manipulation, and how search mechanisms shape market outcomes. Our experiments show that frontier models can approach optimal welfare -- but only under ideal search conditions. Performance degrades sharply with scale, and all models exhibit severe first-proposal bias, creating 10-30x advantages for response speed over quality. These findings reveal how behaviors emerge across market conditions, informing the design of fair and efficient agentic marketplaces.

cs.MA↗

arXiv2025

The Agentic Economy

Generative AI has transformed human-computer interaction by enabling natural language interfaces and the emergence of autonomous agents capable of acting on users' behalf. While early applications have improved individual productivity, these gains have largely been confined to predefined tasks within existing workflows. We argue that the more profound economic impact lies in reducing communication frictions between consumers and businesses. This shift could reorganize markets, redistribute power, and catalyze the creation of new products and services. We explore the implications of an agentic economy, where assistant agents act on behalf of consumers and service agents represent businesses, interacting programmatically to facilitate transactions. A key distinction we draw is between unscripted interactions -- enabled by technical advances in natural language and protocol design -- and unrestricted interactions, which depend on market structures and governance. We examine the current limitations of siloed and end-to-end agents, and explore future scenarios shaped by technical standards and market dynamics. These include the potential tension between agentic walled gardens and an open web of agents, implications for advertising and discovery, the evolution of micro-transactions, and the unbundling and rebundling of digital goods. Ultimately, we argue that the architecture of agentic communication will determine the extent to which generative AI democratizes access to economic opportunity.

cs.CY↗

arXiv2023

Comparing Traditional and LLM-based Search for Consumer Choice: A Randomized Experiment

Recent advances in the development of large language models are rapidly changing how online applications function. LLM-based search tools, for instance, offer a natural language interface that can accommodate complex queries and provide detailed, direct responses. At the same time, there have been concerns about the veracity of the information provided by LLM-based tools due to potential mistakes or fabrications that can arise in algorithmically generated text. In a set of online experiments we investigate how LLM-based search changes people's behavior relative to traditional search, and what can be done to mitigate overreliance on LLM-based output. Participants in our experiments were asked to solve a series of decision tasks that involved researching and comparing different products, and were randomly assigned to do so with either an LLM-based search tool or a traditional search engine. In our first experiment, we find that participants using the LLM-based tool were able to complete their tasks more quickly, using fewer but more complex queries than those who used traditional search. Moreover, these participants reported a more satisfying experience with the LLM-based search tool. When the information presented by the LLM was reliable, participants using the tool made decisions with a comparable level of accuracy to those using traditional search, however we observed overreliance on incorrect information when the LLM erred. Our second experiment further investigated this issue by randomly assigning some users to see a simple color-coded highlighting scheme to alert them to potentially incorrect or misleading information in the LLM responses. Overall we find that this confidence-based highlighting substantially increases the rate at which users spot incorrect information, improving the accuracy of their overall decisions while leaving most other measures unaffected.

cs.HC↗

arXiv2023

Comparing scalable strategies for generating numerical perspectives

Numerical perspectives help people understand extreme and unfamiliar numbers (e.g., \$330 billion is about \$1,000 per person in the United States). While research shows perspectives to be helpful, generating them at scale is challenging both because it is difficult to identify what makes some analogies more helpful than others, and because what is most helpful can vary based on the context in which a given number appears. Here we present and compare three policies for large-scale perspective generation: a rule-based approach, a crowdsourced system, and a model that uses Wikipedia data and semantic similarity (via BERT embeddings) to generate context-specific perspectives. We find that the combination of these three approaches dominates any single method, with different approaches excelling in different settings and users displaying heterogeneous preferences across approaches. We conclude by discussing our deployment of perspectives in a widely-used online word processor.

cs.HC↗

arXiv2021

Manipulating and Measuring Model Interpretability

With machine learning models being increasingly used to aid decision making even in high-stakes domains, there has been a growing interest in developing interpretable models. Although many supposedly interpretable models have been proposed, there have been relatively few experimental studies investigating whether these models achieve their intended effects, such as making people more closely follow a model's predictions when it is beneficial for them to do so or enabling them to detect when a model has made a mistake. We present a sequence of pre-registered experiments (N=3,800) in which we showed participants functionally identical models that varied only in two factors commonly thought to make machine learning models more or less interpretable: the number of features and the transparency of the model (i.e., whether the model internals are clear or black box). Predictably, participants who saw a clear model with few features could better simulate the model's predictions. However, we did not find that participants more closely followed its predictions. Furthermore, showing participants a clear model meant that they were less able to detect and correct for the model's sizable mistakes, seemingly due to information overload. These counterintuitive findings emphasize the importance of testing over intuition when developing interpretable models.

cs.AI↗

arXiv2020

How good is good enough for COVID19 apps? The influence of benefits, accuracy, and privacy on willingness to adopt

A growing number of contact tracing apps are being developed to complement manual contact tracing. A key question is whether users will be willing to adopt these contact tracing apps. In this work, we survey over 4,500 Americans to evaluate (1) the effect of both accuracy and privacy concerns on reported willingness to install COVID19 contact tracing apps and (2) how different groups of users weight accuracy vs. privacy. Drawing on our findings from these first two research questions, we (3) quantitatively model how the amount of public health benefit (reduction in infection rate), amount of individual benefit (true-positive detection of exposures to COVID), and degree of privacy risk in a hypothetical contact tracing app may influence American's willingness to install. Our work takes a descriptive ethics approach toward offering implications for the development of policy and app designs related to COVID19.

cs.CY↗

arXiv2017

Learning in the Repeated Secretary Problem

In the classical secretary problem, one attempts to find the maximum of an unknown and unlearnable distribution through sequential search. In many real-world searches, however, distributions are not entirely unknown and can be learned through experience. To investigate learning in such a repeated secretary problem we conduct a large-scale behavioral experiment in which people search repeatedly from fixed distributions. In contrast to prior investigations that find no evidence for learning in the classical scenario, in the repeated setting we observe substantial learning resulting in near-optimal stopping behavior. We conduct a Bayesian comparison of multiple behavioral models which shows that participants' behavior is best described by a class of threshold-based models that contains the theoretically optimal strategy. Fitting such a threshold-based model to data reveals players' estimated thresholds to be surprisingly close to the optimal thresholds after only a small number of games.

cs.GT↗

arXiv2017

Simple rules for complex decisions

From doctors diagnosing patients to judges setting bail, experts often base their decisions on experience and intuition rather than on statistical models. While understandable, relying on intuition over models has often been found to result in inferior outcomes. Here we present a new method, select-regress-and-round, for constructing simple rules that perform well for complex decisions. These rules take the form of a weighted checklist, can be applied mentally, and nonetheless rival the performance of modern machine learning algorithms. Our method for creating these rules is itself simple, and can be carried out by practitioners with basic statistics knowledge. We demonstrate this technique with a detailed case study of judicial decisions to release or detain defendants while they await trial. In this application, as in many policy settings, the effects of proposed decision rules cannot be directly observed from historical data: if a rule recommends releasing a defendant that the judge in reality detained, we do not observe what would have happened under the proposed action. We address this key counterfactual estimation problem by drawing on tools from causal inference. We find that simple rules significantly outperform judges and are on par with decisions derived from random forests trained on all available features. Generalizing to 22 varied decision-making domains, we find this basic result replicates. We conclude with an analytical framework that helps explain why these simple decision rules perform as well as they do.

stat.AP↗