Report on the Designing Accountable Software Systems Workshop

Catherine Albiston; Travis Breaux; Kat Dearstyne; Jane Cleland-Huang; Serge Egelman; Joan Feigenbaum; Lu Feng; Max Lindquist; Stephen Miner; Ruzica Piskac; Sarah Santos; Jordan Schmerge; Anmol Singhal; Maria Smith; Daniel Weitzner; Christopher Yoo

arXiv · 2606.02804

Report on the Designing Accountable Software Systems Workshop

Catherine Albiston·Travis Breaux·Kat Dearstyne·Jane Cleland-Huang·Serge Egelman·Joan Feigenbaum·Lu Feng·Max Lindquist·Stephen Miner·Ruzica Piskac·Sarah Santos·Jordan Schmerge·Anmol Singhal·Maria Smith·Daniel Weitzner·Christopher Yoo

View original source ↗Read source document ↗Export metadata ↓

Abstract

The Workshop on Designing Accountable Software Systems (DASS) was convened in November 2024 with support from the U.S. National Science Foundation to engage a wide range of current and future stakeholders from government, academia, and industry on the cross-disciplinary topic of accountability in software systems. Over two days, attendees engaged in a series of panels, invited talks, and breakout sessions covering: (1) the dimensions of accountability, including legal compliance as well as business and societal aspects and drivers; (2) a conceptual model of the various structures needed to realize accountability; (3) the sources of legal requirements that affect software; (4) the operationalization of legal requirements in software; (5) the requirements to preserve evidence needed to conduct investigations; and (6) a range of challenges and contextual factors beyond software that affect why some accountability structures succeed, while others fail. The workshop was conducted as a collaborative systematization of knowledge that culminated in several research directions. The findings include the importance of clarifying definitions and responsibilities within accountable organizations, which can affect whether those researching accountability are making assumptions that limit the generalizability of findings. Further research was also identified as needed to study the ways to improve the translation of accountability structures into the software design process while improving engagement with stakeholders, such as legislators, regulators, business executives and system developers. Finally, a key finding was the high demands that DASS-like research projects place on interdisciplinary teams: both in terms of team formation and sustainment, as well as, the specific demands of cross-disciplinary learning that covers both research methods, research dissemination, and career development.

Explore related subjects

cs.SE

Keep this discovery

BibTeX RIS

Catherine Albiston, Travis Breaux, Kat Dearstyne, Jane Cleland-Huang, Serge Egelman, Joan Feigenbaum, Lu Feng, Max Lindquist, Stephen Miner, Ruzica Piskac, Sarah Santos, Jordan Schmerge, Anmol Singhal, Maria Smith, Daniel Weitzner, Christopher Yoo. 2026-06-01. Report on the Designing Accountable Software Systems Workshop. https://arxiv.org/abs/2606.02804

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

LLMVul: A Vulnerability-Labeled Dataset of LLM-Generated C/C++ Functions from Real Production Repositories

Large language models (LLMs) are increasingly used to generate and assist with software development, yet existing vulnerability datasets largely focus on human-written code or controlled prompting environments. This limits the ability to study security weaknesses in LLM-generated code as it appears in real-world software projects. We present LLMVul, a vulnerability-labeled dataset of LLM-generated C/C++ functions mined from real production repositories. We mine AI-assisted development activity from GitHub over a 4 year period, from November 13, 2022 to September 3, 2026, using provenance signals such as commit metadata and AI-related authorship evidence. After filtering and deduplication, LLMVul contains 21,430 unique C/C++ functions from 226 repositories, together with repository, commit, function, provenance, and AI-tool metadata. We establish vulnerability labels using an ensemble of complementary static-analysis and pattern-based techniques and assign Common Weakness Enumeration (CWE) categories to confirmed vulnerable functions. To assess labeling reliability, we additionally conduct independent manual annotation and measure inter-rater agreement using Cohen's kappa ($k=0.79$). LLMVul contains 1,540 ensemble-vulnerable functions spanning 17 unique CWE categories, providing substantially more real-world LLM-generated vulnerable C/C++ functions than existing vulnerability-oriented LLM code benchmarks. By preserving both code-level vulnerability labels and generation/provenance metadata, LLMVul enables reproducible research on vulnerability detection, security evaluation of LLM-generated code, and analysis of vulnerability patterns in AI-assisted software development. The LLMVul dataset is publicly available at https://doi.org/10.5281/zenodo.22668216.

cs.SE↗

arXiv2026

What a Random Draw from the MCP Registry Contains, and What Tool-Use Benchmarks Contain Instead

Studies of the Model Context Protocol (MCP) server ecosystem draw their samples in ways that quietly select for servers that work: reference sets, popularity lists, hand-curated frames, or pipelines that repair a server until it starts. We report what an unrepaired probability sample actually contains. From a 24,135-server registry census we draw 400 npm/stdio servers with a published seed and probe each one over the wire. Only 48.8% complete an initialize handshake, against 66.7% for a hand-curated frame measured with the same instrument, and the dominant failure is not missing credentials (13.3%) but servers that never start at all (37.5%). Among the 195 that do run, hard conformance is total: zero fatal JSON Schema violations across 2,766 advertised tools. Optional safety annotations are the real variance, and the tool-level omission rate on a random draw is 58.8% against 41.5% on the curated frame, so curation flatters this figure too. We then compare the tool descriptions these servers advertise against two tool-use benchmark corpora using one method held constant. Real MCP tools show 2.8% near-duplication at cosine 0.70, and all of it lies within single servers: cross-author near-duplication is 0.0% at every threshold tested. BFCL v4 shows 16.7%, of which 16.4 points lie between independently presented tasks. UltraTool shows 0.3%, cleaner than real tools, so this is a property of BFCL and not of synthetic corpora as a class. Separately, 68.8% of raw BFCL rows and 85.6% of raw UltraTool rows are exact name-plus-description repeats, against 0.4% for real MCP, so any statistic computed over these releases without global deduplication measures repetition rather than tools. All figures regenerate from released scripts and a published seed.

cs.SE↗

arXiv2026

Engineering Reliable Commit Gates for Agentic AI: Cost-Aware Verification Portfolios under Common-Mode Data Failures

Agentic systems commit state-changing actions, but additional verifiers can inherit the same upstream fault. We present VP-CONTROL, a runtime-assurance design and deterministic benchmark for cost-aware commit gates. Its 48 task templates yield 2,880 scenarios across six fault regimes. A fixed-call 2 x 2 experiment separates verifier-model diversity from evidence-source diversity. On frozen proposals from two local actor families, a cross-model vote over shared evidence approves 62.9% of unsafe proposals, versus 22.9% with an independent source. The source effect is 40.9 percentage points, compared with 11.3 for model diversity. A portfolio controller selects verification plans using only deployment-observable metadata. Approximate cluster-adjusted calibration at a nominal 5% per-task target yields 1.9% unsafe execution and 38.2% automated safe coverage on the locked test. Matched-budget portfolios also improve on fixed verification policies. Transfer remains conditional: unseen fault families yield 16-26% risk, and a FinQA check fails to reproduce the source effect with the tested small verifiers. A preregistered live HTTP/SQLite study tests concurrent writes and lost responses. After-check races defeat verifier-only gates; transactional partial guards prevent only covered failures, while a full atomic guard records no unsafe effects across 216 episodes. Idempotent request identifiers prevent duplicate effects after lost responses. The results motivate explicit evidence lineage, cost-aware selection, and commit-time enforcement, while exposing the limits of approximate calibration and local-tool generalization.

cs.SE↗