arXiv subjects

Claudia Raibulet

Publications and source records attributed to Claudia Raibulet.

4 recordsLinked to original sources

arXiv2025

Monitoring and Observability of Machine Learning Systems: Current Practices and Gaps

Production machine learning (ML) systems fail silently -- not with crashes, but through wrong decisions. While observability is recognized as critical for ML operations, there is a lack empirical evidence of what practitioners actually capture. This study presents empirical results on ML observability in practice through seven focus group sessions in several domains. We catalog the information practitioners systematically capture across ML systems and their environment and map how they use it to validate models, detect and diagnose faults, and explain observed degradations. Finally, we identify gaps in current practice and outline implications for tooling design and research to establish ML observability practices.

cs.SE↗

arXiv2025

Tracing Distribution Shifts with Causal System Maps

Monitoring machine learning (ML) systems is hard, with standard practice focusing on detecting distribution shifts rather than their causes. Root-cause analysis often relies on manual tracing to determine whether a shift is caused by software faults, data-quality issues, or natural change. We propose ML System Maps -- causal maps that, through layered views, make explicit the propagation paths between the environment and the ML system's internals, enabling systematic attribution of distribution shifts. We outline the approach and a research agenda for its development and evaluation.

cs.SE↗

arXiv2025

From Tea Leaves to System Maps: A Survey and Framework on Context-aware Machine Learning Monitoring

Machine learning (ML) models in production fail when their broader systems -- from data pipelines to deployment environments -- deviate from training assumptions, not merely due to statistical anomalies in input data. Despite extensive work on data drift, data validation, and out-of-distribution detection, ML monitoring research remains largely model-centric while neglecting contextual information: auxiliary signals about the system around the model (external factors, data pipelines, downstream applications). Incorporating this context turns statistical anomalies into actionable alerts and structured root-cause analysis. Drawing on a systematic review of 94 primary studies, we identify three dimensions of contextual information for ML monitoring: the system element concerned (natural environment or technical infrastructure); the aspect of that element (runtime states, structural relationships, prescriptive properties); and the representation used (formal constructs or informal formats). This forms the Contextual System-Aspect-Representation (C-SAR) framework, a descriptive model synthesizing our findings. We identify 20 recurring triplets across these dimensions and map them to the monitoring activities they support. This study provides a holistic perspective on ML monitoring: from interpreting "tea leaves" (i.e., isolated data and performance statistics) to constructing and managing "system maps" (i.e., end-to-end views that connect data, models, and operating context).

cs.SE↗

arXiv2024

Expert-Driven Monitoring of Operational ML Models

We propose Expert Monitoring, an approach that leverages domain expertise to enhance the detection and mitigation of concept drift in machine learning (ML) models. Our approach supports practitioners by consolidating domain expertise related to concept drift-inducing events, making this expertise accessible to on-call personnel, and enabling automatic adaptability with expert oversight.

cs.LG↗