Closing a 17-Year Gap: Algorithmic Detection and Empirical Prevalence of Rank Reversal in Multi-Criteria Decision Analysis

Abstract

Rank Reversal, where the relative order of alternatives changes in ways that violate axioms of rational decision-making, is a well-documented threat to the reliability of Multi-Criteria Decision Analysis (MCDA) methods. Wang and Triantaphyllou (2008) proposed three systematic test criteria to detect this phenomenon, but despite more than 700 citations, no validated, open-source implementation has closed the gap between theory and practice, a 17-year absence we trace to the non-trivial algorithmic challenges of operationalizing these tests for real-world pipelines. We present an algorithmic framework, implemented in the open-source Scikit-Criteria library, that translates Wang and Triantaphyllou (2008)'s three criteria into concrete, pipeline-compatible procedures: a controlled degradation strategy with hierarchical tie-breaking and graceful handling of preprocessing filters (RRT1), and a dominance-graph construction with exact condensation and transitive reduction for detecting transitivity violations and recomposition inconsistencies (RRT2/RRT3). We demonstrate the framework through two case studies: an application to a cryptocurrency evaluation problem (Van Heerden et al., 2021), and a large-scale audit of 27 pipeline/dataset combinations reproduced from 20 published MCDM methods. The audit shows top-alternative stability (RRT1) is nearly universal (96.3%), but transitivity (RRT2) fails for 14.8% and recomposition consistency (RRT3), the strictest criterion, fails for nearly half (48.1%) of published examples-evidence that rank reversal is a pervasive, measurable feature of the current MCDM literature, not a marginal or adversarial concern.