arXiv subjects

Lorenzo Testa

Publications and source records attributed to Lorenzo Testa.

14 recordsLinked to original sources

arXiv2026

A Doubly Robust Machine Learning Approach for Disentangling Treatment Effect Heterogeneity with Functional Outcomes

Causal inference is paramount for understanding the effects of interventions, yet extracting personalized insights from increasingly complex data remains a significant challenge for modern machine learning. This is the case, in particular, when considering functional outcomes observed over a continuous domain (e.g., time, or space). Estimation of heterogeneous treatment effects, known as CATE, has emerged as a crucial tool for personalized decision-making, but existing meta-learning frameworks are largely limited to scalar outcomes, failing to provide satisfying results in scientific applications that leverage the rich, continuous information encoded in functional data. Here, we introduce FOCaL (Functional Outcome Causal Learning), a novel, doubly robust meta-learner specifically engineered to estimate a functional heterogeneous treatment effect (F-CATE). FOCaL integrates advanced functional regression techniques for both outcome modeling and functional pseudo-outcome reconstruction, thereby enabling the direct and robust estimation of F-CATE. We provide a rigorous theoretical derivation of FOCaL, demonstrate its performance and robustness compared to existing non-robust functional methods through comprehensive simulation studies, and illustrate its practical utility on diverse real-world functional datasets. FOCaL advances the capabilities of machine intelligence to infer nuanced, individualized causal effects from complex data, paving the way for more precise and trustworthy AI systems in personalized medicine, adaptive policy design, and fundamental scientific discovery.

stat.ME↗

arXiv2025

Towards Efficient Inference under Nonmonotone Missingness with General Imputation

Missing data are ubiquitous in classical survey and longitudinal studies as well as modern multi-modality data analysis. A longstanding challenge arises under nonmonotone missingness, where different units may observe arbitrary subsets of all variables. We study parameter estimation and inference problem under this setting. Semiparametric efficiency theory characterizes the efficient estimator through inversion of an operator constructed from pattern-specific conditional expectations. However, this estimator is generally not tractable due to compositions of conditional expectations across patterns. We introduce the Restricted ANOVA hierarchY (RAY), a functional decomposition that reveals an almost-eigen structure of the operator under missing completely at random. This structure yields a closed-form, computable approximation to the efficient estimator. RAY estimator is applicable to general Z-estimation problems, and it remains unbiased for arbitrary independent imputation functions. In theory, we establish verifiable sufficient conditions where RAY attains the efficiency lower bound, and offer a general bound for the efficiency gap otherwise. We further develop adaptive RAY estimator, which attains the minimal asymptotic variance within a broader class containing RAY and other existing estimators. Finally, we investigate the extension of RAY under missing at random mechanism. Simulations and a single-cell multi-omics application demonstrate the efficiency gains of the proposed estimators.

stat.ME↗

arXiv2025

Efficient Difference-in-Differences Estimation when Outcomes are Missing at Random

The Difference-in-Differences (DiD) method is a fundamental tool for causal inference, yet its application is often complicated by missing data. Although recent work has developed robust DiD estimators for complex settings like staggered treatment adoption, these methods typically assume complete data and fail to address the critical challenge of outcomes that are missing at random (MAR) -- a common problem that invalidates standard estimators. We develop a rigorous framework, rooted in semiparametric theory, for identifying and efficiently estimating the Average Treatment Effect on the Treated (ATT) when either pre- or post-treatment (or both) outcomes are missing at random. We first establish nonparametric identification of the ATT under two minimal sets of sufficient conditions. For each, we derive the semiparametric efficiency bound, which provides a formal benchmark for asymptotic optimality. We then propose novel estimators that are asymptotically efficient, achieving this theoretical bound. A key feature of our estimators is their multiple robustness, which ensures consistency even if some nuisance function models are misspecified. We validate the properties of our estimators and showcase their broad applicability through an extensive simulation study.

stat.ME↗

arXiv2025

Rescuing double robustness: safe estimation under complete misspecification

Double robustness is a major selling point of semiparametric and missing data methodology. Its virtues lie in protection against partial nuisance misspecification and asymptotic semiparametric efficiency under correct nuisance specification. However, in many applications, complete nuisance misspecification should be regarded as the norm (or at the very least the expected default), and thus doubly robust estimators may behave fragilely. In fact, it has been amply verified empirically that these estimators can perform poorly when all nuisance functions are misspecified. Here, we first characterize this phenomenon of double fragility, and then propose a solution based on adaptive correction clipping (DR+ACC). We argue that our DR+ACC proposal is safe, in that it inherits the favorable properties of doubly robust estimators under correct nuisance specification, but its error is guaranteed to be bounded by a convex combination of the individual nuisance model errors, which prevents the instability caused by the compounding product of errors of doubly robust estimators. We also show that our proposal comes with no reduction in semiparametric efficiency compared to doubly robust estimators, and thus valid inference based on asymptotic normality can be conducted when nuisances are well-specified. We showcase the efficacy of our DR+ACC estimator both through extensive simulations and by applying it to the analysis of Alzheimer's disease proteomics data.

stat.ME↗

arXiv2025

Semiparametric semi-supervised learning for general targets under distribution shift and decaying overlap

In modern scientific applications, large volumes of covariate data are readily available, while outcome labels are costly, sparse, and often subject to distribution shift. This asymmetry has spurred interest in semi-supervised (SS) learning, but most existing approaches rely on strong assumptions -- such as missing completely at random (MCAR) labeling or strict positivity -- that put substantial limitations on their practical usefulness. In this work, we introduce a general semiparametric framework for estimation, inference, and efficiency benchmarking in SS settings where labels are missing at random (MAR) and the overlap may vanish as sample size increases. Our framework, that we label D2S3, accommodates a wide range of smooth statistical targets -- including means, linear regression coefficients, quantiles, and causal effects -- and remains valid under high-dimensional nuisance estimation and distributional shift between labeled and unlabeled samples. We extend the theoretical guarantees of augmented inverse probability weighting estimators to preserve double robustness, asymptotic normality, and semiparametric efficiency under this challenging D2S3 regime. A key insight is that classical root-n convergence fails under vanishing overlap; we instead provide corrected asymptotic rates that capture the impact of the decay in overlap. We validate our theory through simulations and demonstrate practical utility in real-world applications on the internet of things and public health where labeled data are scarce.

math.ST↗

arXiv2025

Doubly-Robust Functional Average Treatment Effect Estimation

Understanding causal relationships in the presence of complex, structured data remains a central challenge in modern statistics and science in general. While traditional causal inference methods are well-suited for scalar outcomes, many scientific applications demand tools capable of handling functional data -- outcomes observed as functions over continuous domains such as time or space. Motivated by this need, we propose DR-FoS, a novel method for estimating the Functional Average Treatment Effect (FATE) in observational studies with functional outcomes. DR-FoS exhibits double robustness properties, ensuring consistent estimation of FATE even if either the outcome or the treatment assignment model is misspecified. By leveraging recent advances in functional data analysis and causal inference, we establish the asymptotic properties of the estimator, proving its convergence to a Gaussian process. This guarantees valid inference with simultaneous confidence bands across the entire functional domain. Through extensive simulations, we show that DR-FoS achieves robust performance under a wide range of model specifications. Finally, we illustrate the utility of DR-FoS in a real-world application, analyzing functional outcomes to uncover meaningful causal insights in the SHARE ({\em Survey of Health, Aging and Retirement in Europe}) dataset.

stat.ME↗

arXiv2024

Climate Impact Assessment Requires Weighting: Introducing the Weighted Climate Dataset

High-resolution gridded climate data are readily available from multiple sources, yet climate research and decision-making increasingly require country and region-specific climate information weighted by socio-economic factors. Moreover, the current landscape of disparate data sources and inconsistent weighting methodologies exacerbates the reproducibility crisis and undermines scientific integrity. To address these issues, we have developed a globally comprehensive dataset at both country (GADM0) and region (GADM1) levels, encompassing various climate indicators (precipitation, temperature, SPEI, wind gust). Our methodology involves weighting gridded climate data by population density, night-time light intensity, cropland area, and concurrent population count -- all proxies for socio-economic activity -- before aggregation. We process data from multiple sources, offering daily, monthly, and annual climate variables spanning from 1900 to 2023. A unified framework streamlines our preprocessing steps, and rigorous validation against leading climate impact studies ensures data reliability. The resulting Weighted Climate Dataset is publicly accessible through an online dashboard at https://weightedclimatedata.streamlit.app/.

stat.AP↗

arXiv2023

A unified repository for pre-processed climate data weighted by gridded economic activity

Although high-resolution gridded climate variables are provided by multiple sources, the need for country and region-specific climate data weighted by indicators of economic activity is becoming increasingly common in environmental and economic research. We process available information from different climate data sources to provide spatially aggregated data with global coverage for both countries (GADM0 resolution) and regions (GADM1 resolution) and for a variety of climate indicators (average precipitations, average temperatures, average SPEI). We weigh gridded climate data by population density or by night light intensity -- both proxies of economic activity -- before aggregation. Climate variables are measured daily, monthly, and annually, covering (depending on the data source) a time window from 1900 (at the earliest) to 2023. We pipeline all the preprocessing procedures in a unified framework, which we share in the open-access Weighted Climate Data Repository web app. Finally, we validate our data through a systematic comparison with those employed in leading climate impact studies.

econ.GN↗

arXiv2023

Contrasting pre-vaccine COVID-19 waves in Italy through Functional Data Analysis

We use data from 107 Italian provinces to characterize and compare mortality patterns in the first two COVID-19 epidemic waves, which occurred prior to the introduction of vaccines. We also associate these patterns with mobility, timing of government restrictions, and socio-demographic, infrastructural, and environmental covariates. Notwithstanding limitations in the accuracy and reliability of publicly available data, we are able to exploit information in curves and shapes through Functional Data Analysis techniques. Specifically, we document differences in magnitude and variability between the two waves; while both were characterized by a co-occurrence of 'exponential' and 'mild' mortality patterns, the second spread much more broadly and asynchronously through the country. Moreover, we find evidence of a significant positive association between local mobility and mortality in both epidemic waves and corroborate the effectiveness of timely restrictions in curbing mortality. The techniques we describe could capture additional signals of interest if applied, for instance, to data on cases and positivity rates. However, we show that the quality of such data, at least in the case of Italian provinces, was too poor to support meaningful analyses.

stat.AP↗

arXiv2023

FAStEN: An Efficient Adaptive Method for Feature Selection and Estimation in High-Dimensional Functional Regressions

Functional regression analysis is an established tool for many contemporary scientific applications. Regression problems involving large and complex data sets are ubiquitous, and feature selection is crucial for avoiding overfitting and achieving accurate predictions. We propose a new, flexible and ultra-efficient approach to perform feature selection in a sparse high dimensional function-on-function regression problem, and we show how to extend it to the scalar-on-function framework. Our method, called FAStEN, combines functional data, optimization, and machine learning techniques to perform feature selection and parameter estimation simultaneously. We exploit the properties of Functional Principal Components and the sparsity inherent to the Dual Augmented Lagrangian problem to significantly reduce computational cost, and we introduce an adaptive scheme to improve selection accuracy. In addition, we derive asymptotic oracle properties, which guarantee estimation and selection consistency for the proposed FAStEN estimator. Through an extensive simulation study, we benchmark our approach to the best existing competitors and demonstrate a massive gain in terms of CPU time and selection performance, without sacrificing the quality of the coefficients' estimation. The theoretical derivations and the simulation study provide a strong motivation for our approach. Finally, we present an application to brain fMRI data from the AOMIC PIOP1 study. Complete FAStEN code is provided at https://github.com/IBM/funGCN.

stat.ME↗

arXiv2022

Aggregate Processes as Distributed Adaptive Services for the Industrial Internet of Things

The Industrial Internet of Things (IIoT) promises to bring many benefits, including increased productivity, reduced costs, and increased safety to new generation manufacturing plants. The main ingredients of IIoT are the connected, communicating devices directly located in the workshop floor (far edge devices), as well as edge gateways that connect such devices to the Internet and, in particular, to cloud servers. The field of Edge Computing advocates that keeping computations as close as possible to the sources of data can be an effective means of reducing latency, preserving privacy, and improve the overall efficiency of the system, although building systems where (far) edge and cloud nodes cooperate is quite challenging. In the present work we propose the adoption of the Aggregate Programming (AP) paradigm (and, in particular, the "aggregate process" construct) as a way to simplify building distributed, intelligent services at the far edge of an IIoT architecture. We demonstrate the feasibility and efficacy of the approach with simulated experiments on FCPP (a C++ library for AP), and with some basic experiments on physical IIoT boards running an ad-hoc porting of FCPP.

cs.SE↗

arXiv2022

Venture Capital investments through the lens of Network and Functional Data Analysis

In this paper we characterize the performance of venture capital-backed firms based on their ability to attract investment. The aim of the study is to identify relevant predictors of success built from the network structure of firms' and investors' relations. Focusing on deal-level data for the health sector, we first create a bipartite network among firms and investors, and then apply functional data analysis (FDA) to derive progressively more refined indicators of success captured by a binary, a scalar and a functional outcome. More specifically, we use different network centrality measures to capture the role of early investments for the success of the firm. Our results, which are robust to different specifications, suggest that success has a strong positive association with centrality measures of the firm and of its large investors, and a weaker but still detectable association with centrality measures of small investors and features describing firms as knowledge bridges. Finally, based on our analyses, success is not associated with firms' and investors' spreading power (harmonic centrality), nor with the tightness of investors' community (clustering coefficient) and spreading ability (VoteRank).

stat.AP↗

arXiv2021

Can you always reap what you sow? Network and functional data analysis of VC investments in health-tech companies

"Success" of firms in venture capital markets is hard to define, and its determinants are still poorly understood. We build a bipartite network of investors and firms in the healthcare sector, describing its structure and its communities. Then, we characterize "success" introducing progressively more refined definitions, and we find a positive association between such definitions and the centrality of a company. In particular, we are able to cluster funding trajectories of firms into two groups capturing different "success" regimes and to link the probability of belonging to one or the other to their network features (in particular their centrality and the one of their investors). We further investigate this positive association by introducing scalar as well as functional "success" outcomes, confirming our findings and their robustness.

cs.SI↗

arXiv2020

The shapes of an epidemic: using Functional Data Analysis to characterize COVID-19 in Italy

We investigate patterns of COVID-19 mortality across 20 Italian regions and their association with mobility, positivity, and socio-demographic, infrastructural and environmental covariates. Notwithstanding limitations in accuracy and resolution of the data available from public sources, we pinpoint significant trends exploiting information in curves and shapes with Functional Data Analysis techniques. These depict two starkly different epidemics; an "exponential" one unfolding in Lombardia and the worst hit areas of the north, and a milder, "flat(tened)" one in the rest of the country -- including Veneto, where cases appeared concurrently with Lombardia but aggressive testing was implemented early on. We find that mobility and positivity can predict COVID-19 mortality, also when controlling for relevant covariates. Among the latter, primary care appears to mitigate mortality, and contacts in hospitals, schools and work places to aggravate it. The techniques we describe could capture additional and potentially sharper signals if applied to richer data.

stat.AP↗