M3MAD-Bench: Multi-Dimensional Evaluation of Multi-Agent Debate Across Domains and Modalities

Abstract

As an agent-level reasoning and coordination paradigm, Multi-Agent Debate (MAD) orchestrates multiple agents through structured debate to improve answer quality and support complex reasoning. However, existing research on MAD suffers from two fundamental limitations: evaluations are conducted under fragmented and inconsistent settings, making fair comparison difficult, and are largely confined to text-only scenarios, leaving its effectiveness in multimodal settings underexplored. To address these gaps, we introduce M3MAD-Bench, a unified and extensible benchmark for evaluating MAD methods across Multi-domain tasks, Multi-modal inputs, and Multi-dimensional metrics. M3MAD-Bench establishes standardized protocols over five core task domains, including Knowledge, Mathematics, Medicine, Natural Sciences, and Complex Reasoning, covering a total of 13 datasets, and systematically includes both pure text and vision-language data, enabling controlled cross-modality comparison. We evaluate MAD methods on 9 base models spanning different architectures, scales, and modality capabilities. Beyond accuracy, M3MAD-Bench incorporates efficiency-oriented metrics such as token consumption and inference time, providing a holistic view of performance--cost trade-offs. Through extensive experiments, we derive nine key insights, revealing that MAD is not uniformly effective: collaborative methods are generally more robust than adversarial ones, especially on reasoning-intensive and multimodal tasks, but often incur substantial efficiency costs. These findings provide practical guidance for selecting and designing MAD strategies in real-world applications. We believe M3MAD-Bench offers a reliable foundation for future research on standardized and reproducible MAD evaluation. The code is available at https://github.com/liaolea/M3MAD-Bench.