arXiv subjects

Nazar Protsiv

Publications and source records attributed to Nazar Protsiv.

3 recordsLinked to original sources

arXiv2025

VirnyFlow: Optimizing ML Pipelines for Accuracy, Fairness, and Stability at Scale

Developing machine learning (ML) systems for real-world deployment requires navigating context-dependent trade-offs among accuracy, fairness, stability, and other objectives. Existing AutoML frameworks optimize pipelines efficiently, but they fix the optimization objective up front, leave it outside the developer's control during search, and rarely scale beyond a single node. We present VirnyFlow, a system that optimizes ML pipelines jointly for accuracy, fairness, and stability at scale. A user-defined evaluation protocol, with fairness measured over binary and intersectional groups, drives every layer of the optimizer: multi-objective Bayesian optimization of physical pipelines, cost-aware bandit selection of logical pipelines, and multi-criterion pruning. The architecture combines asynchronous execution over Apache Kafka with database-backed experiment management, providing fine-grained parallelism, fault tolerance, and interactive inspection of trade-offs. On six real-world datasets, VirnyFlow achieves competitive or superior performance compared to state-of-the-art AutoML systems (auto-sklearn, Alpine Meadow, FLAML) under identical resource constraints, scales to 128 workers across four nodes on datasets of up to 2.6M records, and achieves up to 7x higher speedup than the best-scaling single-node baseline, while maintaining stable accuracy and fairness as parallelism increases. A clinical case study on distribution shift and an IRB-approved user study demonstrate human-in-the-loop navigation of trade-offs in practice: rather than returning a single "best" model, VirnyFlow lets data scientists define, inspect, and iteratively refine the objectives of the search to fit their deployment context.

cs.LG↗

arXiv2024

Still More Shades of Null: An Evaluation Suite for Responsible Missing Value Imputation

Data missingness is a practical challenge of sustained interest to the scientific community. In this paper, we present Shades-of-Null, an evaluation suite for responsible missing value imputation. Our work is novel in two ways (i) we model realistic and socially-salient missingness scenarios that go beyond Rubin's classic Missing Completely at Random (MCAR), Missing At Random (MAR) and Missing Not At Random (MNAR) settings, to include multi-mechanism missingness (when different missingness patterns co-exist in the data) and missingness shift (when the missingness mechanism changes between training and test) (ii) we evaluate imputers holistically, based on imputation quality and imputation fairness, as well as on the predictive performance, fairness and stability of the models that are trained and tested on the data post-imputation. We use Shades-of-Null to conduct a large-scale empirical study involving 29,736 experimental pipelines, and find that while there is no single best-performing imputation approach for all missingness types, interesting trade-offs arise between predictive performance, fairness and stability, based on the combination of missingness scenario, imputer choice, and the architecture of the predictive model. We make Shades-of-Null publicly available, to enable researchers to rigorously evaluate missing value imputation methods on a wide range of metrics in plausible and socially meaningful scenarios.

cs.AI↗

arXiv2023

An Epistemic and Aleatoric Decomposition of Arbitrariness to Constrain the Set of Good Models

Recent research reveals that machine learning (ML) models are highly sensitive to minor changes in their training procedure, such as the inclusion or exclusion of a single data point, leading to conflicting predictions on individual data points; a property termed as arbitrariness or instability in ML pipelines in prior work. Drawing from the uncertainty literature, we show that stability decomposes into epistemic and aleatoric components, capturing the consistency and confidence in prediction, respectively. We use this decomposition to provide two main contributions. Our first contribution is an extensive empirical evaluation. We find that (i) epistemic instability can be reduced with more training data whereas aleatoric instability cannot; (ii) state-of-the-art ML models have aleatoric instability as high as 79% and aleatoric instability disparities among demographic groups as high as 29% in popular fairness benchmarks; and (iii) fairness pre-processing interventions generally increase aleatoric instability more than in-processing interventions, and both epistemic and aleatoric instability are highly sensitive to data-processing interventions and model architecture. Our second contribution is a practical solution to the problem of systematic arbitrariness. We propose a model selection procedure that includes epistemic and aleatoric criteria alongside existing accuracy and fairness criteria, and show that it successfully narrows down a large set of good models (50-100 on our datasets) to a handful of stable, fair and accurate ones. We built and publicly released a python library to measure epistemic and aleatoric multiplicity in any ML pipeline alongside existing confusion-matrix-based metrics, providing practitioners with a rich suite of evaluation metrics to use to define a more precise criterion during model selection.

cs.LG↗