arXiv subjects

Munther Dahleh

Publications and source records attributed to Munther Dahleh.

At least 19 recordsLinked to original sources

arXiv2026

Synthetic Nearest Neighbors: Extending Synthetic Controls for Matrix Completion with Missing Not at Random Data

We develop a causal framework for matrix completion under missing not at random (MNAR) data. Drawing on synthetic controls from the econometric panel data literature, our approach relaxes two assumptions common in MNAR matrix completion: positivity and independence of observation indicators. Unlike traditional panel data models, which often require prescribed block-sparse geometries, our framework accommodates flexible, heterogeneous observation patterns through target-specific local information structures. We propose synthetic nearest neighbors (SNN), a local synthetic-controls-inspired estimator, and establish finite-sample entrywise error bounds and consistency for mean recovery under suitable conditions. We further derive asymptotic normality under heteroskedastic noise and develop feasible entrywise inference. To estimate entry-specific noise variances, we apply the same local principle to squared outcomes, obtaining consistency under bounded noise and asymptotic unbiasedness under general subgaussian noise. Simulation studies corroborate the theoretical findings across a range of missingness designs and observation patterns.

stat.ME↗

arXiv2025

VITA: Variational Pretraining of Transformers for Climate-Robust Crop Yield Forecasting

Accurate crop yield forecasting is essential for global food security. However, current AI models systematically underperform when yields deviate from historical trends. We attribute this to the lack of rich, physically grounded datasets directly linking atmospheric states to yields. To address this, we introduce VITA (Variational Inference Transformer for Asymmetric Data), a variational pretraining framework that learns representations from large satellite-based weather datasets and transfers to the ground-based limited measurements available for yield prediction. VITA is trained using detailed meteorological variables as proxy targets during pretraining and learns to predict latent atmospheric states under a seasonality-aware sinusoidal prior. This allows the model to be fine-tuned using limited weather statistics during deployment. Applied to 763 counties in the US Corn Belt, VITA achieves state-of-the-art performance in predicting corn and soybean yields across all evaluation scenarios, particularly during extreme years, with statistically significant improvements (paired t-test, p < 0.0001). Importantly, VITA outperforms prior frameworks like GNN-RNN without soil data, and larger foundational models (e.g., Chronos-Bolt) with less compute, making it practical for real-world use, especially in data-scarce regions. This work highlights how domain-aware AI design can overcome data limitations and support resilient agricultural forecasting in a changing climate.

cs.LG↗

arXiv2025

Finite Sample Analysis of Tensor Decomposition for Learning Mixtures of Linear Systems

We study the problem of learning mixtures of linear dynamical systems (MLDS) from input-output data. The mixture setting allows us to leverage observations from related dynamical systems to improve the estimation of individual models. Building on spectral methods for mixtures of linear regressions, we propose a moment-based estimator that uses tensor decomposition to estimate the impulse response parameters of the mixture models. The estimator improves upon existing tensor decomposition approaches for MLDS by utilizing the entire length of the observed trajectories. We provide sample complexity bounds for estimating MLDS in the presence of noise, in terms of both the number of trajectories $N$ and the trajectory length $T$, and demonstrate the performance of the estimator through simulations.

eess.SY↗

arXiv2024

Towards Data Auctions with Externalities

The design of data markets has gained importance as firms increasingly use machine learning models fueled by externally acquired training data. A key consideration is the externalities firms face when data, though inherently freely replicable, is allocated to competing firms. In this setting, we demonstrate that a data seller's optimal revenue increases as firms can pay to prevent allocations to others. To do so, we first reduce the combinatorial problem of allocating and pricing multiple datasets to the auction of a single digital good by modeling utility for data through the increase in prediction accuracy it provides. We then derive welfare and revenue maximizing mechanisms, highlighting how the form of firms' private information - whether the externalities one exerts on others is known, or vice-versa - affects the resulting structures. In all cases, under appropriate assumptions, the optimal allocation rule is a single threshold per firm, where either all data is allocated or none is.

cs.GT↗

arXiv2024

Sample Efficient Reinforcement Learning with Partial Dynamics Knowledge

The problem of sample complexity of online reinforcement learning is often studied in the literature without taking into account any partial knowledge about the system dynamics that could potentially accelerate the learning process. In this paper, we study the sample complexity of online Q-learning methods when some prior knowledge about the dynamics is available or can be learned efficiently. We focus on systems that evolve according to an additive disturbance model of the form $S_{h+1} = f(S_h, A_h) + W_h$, where $f$ represents the underlying system dynamics, and $W_h$ are unknown disturbances independent of states and actions. In the setting of finite episodic Markov decision processes with $S$ states, $A$ actions, and episode length $H$, we present an optimistic Q-learning algorithm that achieves $\tilde{\mathcal{O}}(\text{Poly}(H)\sqrt{T})$ regret under perfect knowledge of $f$, where $T$ is the total number of interactions with the system. This is in contrast to the typical $\tilde{\mathcal{O}}(\text{Poly}(H)\sqrt{SAT})$ regret for existing Q-learning methods. Further, if only a noisy estimate $\hat{f}$ of $f$ is available, our method can learn an approximately optimal policy in a number of samples that is independent of the cardinalities of state and action spaces. The sub-optimality gap depends on the approximation error $\hat{f}-f$, as well as the Lipschitz constant of the corresponding optimal value function. Our approach does not require modeling of the transition probabilities and enjoys the same memory complexity as model-free methods.

cs.LG↗

arXiv2024

WeatherFormer: A Pretrained Encoder Model for Learning Robust Weather Representations from Small Datasets

This paper introduces WeatherFormer, a transformer encoder-based model designed to learn robust weather features from minimal observations. It addresses the challenge of modeling complex weather dynamics from small datasets, a bottleneck for many prediction tasks in agriculture, epidemiology, and climate science. WeatherFormer was pretrained on a large pretraining dataset comprised of 39 years of satellite measurements across the Americas. With a novel pretraining task and fine-tuning, WeatherFormer achieves state-of-the-art performance in county-level soybean yield prediction and influenza forecasting. Technical innovations include a unique spatiotemporal encoding that captures geographical, annual, and seasonal variations, adapting the transformer architecture to continuous weather data, and a pretraining strategy to learn representations that are robust to missing weather features. This paper for the first time demonstrates the effectiveness of pretraining large transformer encoder models for weather-dependent applications across multiple domains.

cs.CV↗

arXiv2023

SAMoSSA: Multivariate Singular Spectrum Analysis with Stochastic Autoregressive Noise

The well-established practice of time series analysis involves estimating deterministic, non-stationary trend and seasonality components followed by learning the residual stochastic, stationary components. Recently, it has been shown that one can learn the deterministic non-stationary components accurately using multivariate Singular Spectrum Analysis (mSSA) in the absence of a correlated stationary component; meanwhile, in the absence of deterministic non-stationary components, the Autoregressive (AR) stationary component can also be learnt readily, e.g. via Ordinary Least Squares (OLS). However, a theoretical underpinning of multi-stage learning algorithms involving both deterministic and stationary components has been absent in the literature despite its pervasiveness. We resolve this open question by establishing desirable theoretical guarantees for a natural two-stage algorithm, where mSSA is first applied to estimate the non-stationary components despite the presence of a correlated stationary AR component, which is subsequently learned from the residual time series. We provide a finite-sample forecasting consistency bound for the proposed algorithm, SAMoSSA, which is data-driven and thus requires minimal parameter tuning. To establish theoretical guarantees, we overcome three hurdles: (i) we characterize the spectra of Page matrices of stable AR processes, thus extending the analysis of mSSA; (ii) we extend the analysis of AR process identification in the presence of arbitrary bounded perturbations; (iii) we characterize the out-of-sample or forecasting error, as opposed to solely considering model identification. Through representative empirical studies, we validate the superior performance of SAMoSSA compared to existing baselines. Notably, SAMoSSA's ability to account for AR noise structure yields improvements ranging from 5% to 37% across various benchmark datasets.

cs.LG↗

arXiv2023

Estimation of Models with Limited Data by Leveraging Shared Structure

Modern data sets, such as those in healthcare and e-commerce, are often derived from many individuals or systems but have insufficient data from each source alone to separately estimate individual, often high-dimensional, model parameters. If there is shared structure among systems however, it may be possible to leverage data from other systems to help estimate individual parameters, which could otherwise be non-identifiable. In this paper, we assume systems share a latent low-dimensional parameter space and propose a method for recovering $d$-dimensional parameters for $N$ different linear systems, even when there are only $T<d$ observations per system. To do so, we develop a three-step algorithm which estimates the low-dimensional subspace spanned by the systems' parameters and produces refined parameter estimates within the subspace. We provide finite sample subspace estimation error guarantees for our proposed method. Finally, we experimentally validate our method on simulations with i.i.d. regression data and as well as correlated time series data.

cs.LG↗

arXiv2023

Learning Good State and Action Representations via Tensor Decomposition

The transition kernel of a continuous-state-action Markov decision process (MDP) admits a natural tensor structure. This paper proposes a tensor-inspired unsupervised learning method to identify meaningful low-dimensional state and action representations from empirical trajectories. The method exploits the MDP's tensor structure by kernelization, importance sampling and low-Tucker-rank approximation. This method can be further used to cluster states and actions respectively and find the best discrete MDP abstraction. We provide sharp statistical error bounds for tensor concentration and the preservation of diffusion distance after embedding. We further prove that the learned state/action abstractions provide accurate approximations to latent block structures if they exist, enabling function approximation in downstream tasks such as policy evaluation.

stat.ML↗

arXiv2022

Selling Information in Competitive Environments

Data buyers compete in a game of incomplete information about which a single data seller owns some payoff-relevant information. The seller faces a joint information- and mechanism-design problem: deciding which information to sell, while eliciting the buyers' types and imposing payments. We derive the welfare- and revenue-optimal mechanisms for a class of games with binary actions and states. Our results highlight the critical properties of selling information in competitive environments: (i) the negative externalities arising from buyer competition increase the profitability of recommending the correct action to one buyer exclusively; (ii) for the buyers to follow the seller's recommendations, the degree of exclusivity must be limited; (iii) the buyers' obedience constraints also limit the distortions in the allocation of information introduced by a monopolist seller; (iv) as competition becomes fiercer, these limitations become more severe, weakening the impact of market power on the allocation of information.

cs.GT↗

arXiv2021

Causal Matrix Completion

Matrix completion is the study of recovering an underlying matrix from a sparse subset of noisy observations. Traditionally, it is assumed that the entries of the matrix are "missing completely at random" (MCAR), i.e., each entry is revealed at random, independent of everything else, with uniform probability. This is likely unrealistic due to the presence of "latent confounders", i.e., unobserved factors that determine both the entries of the underlying matrix and the missingness pattern in the observed matrix. For example, in the context of movie recommender systems -- a canonical application for matrix completion -- a user who vehemently dislikes horror films is unlikely to ever watch horror films. In general, these confounders yield "missing not at random" (MNAR) data, which can severely impact any inference procedure that does not correct for this bias. We develop a formal causal model for matrix completion through the language of potential outcomes, and provide novel identification arguments for a variety of causal estimands of interest. We design a procedure, which we call "synthetic nearest neighbors" (SNN), to estimate these causal estimands. We prove finite-sample consistency and asymptotic normality of our estimator. Our analysis also leads to new theoretical results for the matrix completion literature. In particular, we establish entry-wise, i.e., max-norm, finite-sample consistency and asymptotic normality results for matrix completion with MNAR data. As a special case, this also provides entry-wise bounds for matrix completion with MCAR data. Across simulated and real data, we demonstrate the efficacy of our proposed estimator.

econ.EM↗

arXiv2021

Eliciting Social Knowledge for Creditworthiness Assessment

Access to capital is a major constraint for economic growth in the developing world. Yet those attempting to lend in this space face high defaults due to their inability to distinguish creditworthy borrowers from the rest. In this paper, we propose two novel scoring mechanisms that incentivize community members to truthfully report their signal on the creditworthiness of others in their community. We first design a truncated asymmetric scoring-rule for a setting where the lender has no liquidity constraints. We then derive a novel, strictly-proper VCG scoring mechanism for the liquidity-constrained setting. Whereas Chen et al. [2011] give an impossibility result for an analogous setting in which sequential reports are made in the context of decision markets, we achieve a positive result through appeal to interim beliefs about the reports of others in a setting with simultaneous reports.Moreover, the use of VCG methods allows for the integration of linear belief aggregation methods.

cs.GT↗

arXiv2021

QuickFlex: a Fast Algorithm for Flexible Region Construction for the TSO-DSO Coordination

Most of the new technological changes in power systems are expected to take place in distribution grids. The enormous potential for distribution flexibility could meet the transmission system's needs, changing the paradigm of generator-centric energy and ancillary services provided to a demand-centric one, by placing more importance on smaller resources, such as flexible demands and electric vehicles. For unlocking such capabilities, it is essential to understand the aggregated flexibility that can be harvested from the large population of new technologies located in distribution grids. Distribution grids, therefore, could provide aggregated flexibility at the transmission level. To date, most computational methods for estimating the aggregated flexibility at the interface between distribution grids and transmission grids have the drawback of requiring significant computational time, which hinders their applicability. This paper presents a new algorithm, coined as QuickFlex} for constructing the flexibility domain of distribution grids. Contrary to previous methods, a priory flexibility domain accuracy can be selected. Our method requires few iterations for constructing the flexibility region. The number of iterations needed is mainly independent of the distribution grid's input size and flexible elements. Numerical experiments are performed in four grids ranging from 5 nodes to 123 nodes. It is shown that QuickFlex outperforms existing proposals in the literature in both speed and accuracy.

eess.SY↗

arXiv2021

Network Consensus with Privacy: A Secret Sharing Method

In this work, inspired by secret sharing schemes, we introduce a privacy-preserving approach for network consensus, by which all nodes in a network can reach an agreement on their states without exposing the individual state to neighbors. With the privacy degree defined for the agents, the proposed method makes the network resistant to the collusion of any given number of neighbors, and protects the consensus procedure from communication eavesdropping. Unlike existing works, the proposed privacy-preserving algorithm is resilient to node failures. When a node fails, the method offers the possibility of rebuilding the lost node via the information kept in its neighbors, even though none of the neighbors knows the exact state of the failing node. Moreover, it is shown that the proposed method can achieve consensus and average consensus almost surely, when the agents have arbitrary privacy degrees and a common privacy degree, respectively. To illustrate the theory, two numerical examples are presented.

eess.SY↗

arXiv2020

Consensus with Preserved Privacy against Neighbor Collusion

This paper proposes a privacy-preserving algorithm to solve the average consensus problem based on Shamir's secret sharing scheme, in which a network of agents reach an agreement on their states without exposing their individual state until an agreement is reached. Unlike other methods, the proposed algorithm renders the network resistant to the collusion of any given number of neighbors (even with all neighbors' colluding). Another virtue of this work is that such a method can protect the network consensus procedure from eavesdropping.

cs.CR↗

arXiv2019

Zorro: A Model Agnostic System to Price Consumer Data

Personal data is essential in showing users targeted ads - the economic backbone of the web. Still, there are major inefficiencies in how data is transacted online: (1) users don't decide what information is released nor get paid for this privacy loss; (2) algorithmic advertisers are stuck in inefficient long-term contracts where they purchase user data without knowing the value it provides. This paper proposes a system, Zorro, which aims to rectify aforementioned two problems. As the main contribution, we provide a natural, 'absolute' definition of 'Value of Data' (VoD) - for any quantity of interest, it is the delta between an individual's value and population mean. The challenge remains how to operationalize this definition, independently of a buyer's model for VoD. We propose a model-agnostic solution, relying on matrix estimation, and use it to estimate click-through-rate (CTR), as an example. Regarding (2), Zorro empowers advertisers to measure value of user data on a query-by-query basis and based only on the increase in accuracy it provides in estimating CTR. In contrast advertisers currently engage in inefficient long-term data contracts with third party data sellers. We highlight two results on a large ad-click dataset: (i) our system has R^2=0.58, in line with best-in-class results for related problems (e.g. content recommendation). Crucially, our system is model-agnostic - we estimate CTR without accessing an advertiser's proprietary models, a required property of any such pricing system;(ii) our experiments show selling user data has incremental value ranging from 30%-69% depending on ad category. Roughly, this translates to at least USD 16 Billion loss in value for advertisers if user data is not provided. Regarding (1), in addition to allowing users to get paid for data sharing, we extend our mathematical framework to when users provide explicit intent.

cs.SI↗

arXiv2019

A Marketplace for Data: An Algorithmic Solution

In this work, we aim to design a data marketplace; a robust real-time matching mechanism to efficiently buy and sell training data for Machine Learning tasks. While the monetization of data and pre-trained models is an essential focus of industry today, there does not exist a market mechanism to price training data and match buyers to sellers while still addressing the associated (computational and other) complexity. The challenge in creating such a market stems from the very nature of data as an asset: (i) it is freely replicable; (ii) its value is inherently combinatorial due to correlation with signal in other data; (iii) prediction tasks and the value of accuracy vary widely; (iv) usefulness of training data is difficult to verify a priori without first applying it to a prediction task. As our main contributions we: (i) propose a mathematical model for a two-sided data market and formally define the key associated challenges; (ii) construct algorithms for such a market to function and analyze how they meet the challenges defined. We highlight two technical contributions: (i) a new notion of 'fairness' required for cooperative games with freely replicable goods; (ii) a truthful, zero regret mechanism to auction a class of combinatorial goods based on utilizing Myerson's payment function and the Multiplicative Weights algorithm. These might be of independent interest.

cs.GT↗

arXiv2018

Between-Ride Routing for Private Transportation Services

Spurred by the growth of transportation network companies and increasing data capabilities, vehicle routing and ride-matching algorithms can improve the efficiency of private transportation services. However, existing routing solutions do not address where drivers should travel after dropping off a passenger and before receiving the next passenger ride request, i.e., during the between-ride period. We address this problem by developing an efficient algorithm to find the optimal policy for drivers between rides in order to maximize driver profits. We model the road network as a graph, and we show that the between-ride routing problem is equivalent to a stochastic shortest path problem, an infinite dynamic program with no discounting. We prove under reasonable assumptions that an optimal routing policy exists that avoids cycles; policies of this type can be efficiently found. We present an iterative approach to find an optimal routing policy. Our approach can account for various factors, including the frequency of passenger ride requests at different locations, traffic conditions, and surge pricing. We demonstrate the effectiveness of the approach by implementing it on road network data from Boston and New York City.

eess.SY↗