arXiv subjects

Abbas El Gamal

Publications and source records attributed to Abbas El Gamal.

At least 19 recordsLinked to original sources

arXiv2026

Information Theory and Statistical Learning

This manuscript contains preprint of a chapter under consideration for inclusion in the forthcoming third edition of {\em Cover and Thomas's Elements of Information Theory}, posted with permission from Wiley. The table of contents EIT-3 ToC of the new edition can be found at: https://docs.google.com/document/d/1L-m4oQEJw1PJhoxBeMwrrBD8S_HmvzMEkPbYvS24980/edit?usp=sharing . For feedback, please contact abbas@ee.stanford.edu Learning and information theory intersect in both model training and the characterization of fundamental performance limits. This manuscript provides a concise and accessible treatment of the first intersection, requiring only basic background in information theory and statistics at the senior undergraduate or first-year graduate level. End-of-chapter exercises make the material well suited for classroom use as well as self-study. The chapter focuses on the role of divergence measures in model training, with examples ranging from linear and logistic regression to autoregressive models, variational autoencoders, diffusion models, generative adversarial networks, and score-based models. It introduces the evidence lower bound (ELBO), f-divergences, and the Fisher divergence. In particular, the treatment of the generative diffusion model provides a more systematic and explicit derivation than is typical in the literature.

cs.IT↗

arXiv2026

Information-theoretic Limits of Learning and Estimation

Information theory plays a central role in establishing fundamental limits on what any learning or estimation algorithm can -- and cannot -- achieve, regardless of computational power. In this chapter, we provide an introduction to these connections. End-of-chapter exercises makes the material suitable for both classroom use and self-study. We begin by introducing concentration inequalities along with the notions of covering and packing in metric spaces, and the associated concept of metric entropy. These tools are essential for our analysis. We then introduce the learning-theoretic framework and derive upper bounds on generalization error in terms of metric entropy, Rademacher complexity, and the VC dimension, as well as mutual information and relative entropy. Finally we discuss the minimax estimation framework and establish lower bounds on minimax risk using Fano's inequality, yielding bounds in terms of relative entropy and covering and packing numbers. This manuscript contains preprint of a chapter under consideration for inclusion in the forthcoming third edition of Cover and Thomas's Elements of Information Theory, posted with permission from Wiley. It would follow the chapter posted at arXiv:2605.02989 . The table of contents of the new edition can be found at: https://docs.google.com/document/d/1L-m4oQEJw1PJhoxBeMwrrBD8S_HmvzMEkPbYvS24980/edit?usp=sharing . For feedback, please contact abbas@ee.stanford.edu.

cs.IT↗

arXiv2024

GPU Accelerated Security Constrained Optimal Power Flow

We propose a GPU accelerated proximal message passing algorithm for solving contingency-constrained DC optimal power flow problems (OPF). We consider a highly general formulation of OPF that uses a sparse device-node model and supports a broad range of devices and constraints, e.g., energy storage and ramping limits. Our algorithm is a variant of the alternating direction method multipliers (ADMM) that does not require solving any linear systems and only consists of sparse incidence matrix multiplies and vectorized scalar operations. We develop a pure PyTorch implementation of our algorithm that runs entirely on the GPU. The implementation is also end-to-end differentiable, i.e., all updates are automatic differentiation compatible. We demonstrate the performance of our method using test cases of varying network sizes and time horizons. Relative to a CPU-based commercial optimizer, our implementation achieves well over 100x speedups on large test cases, solving problems with over 500 million variables in under a minute on a single GPU.

math.OC↗

arXiv2024

Scalable and Interactive Electricity Grid Expansion Planning

Large scale grid expansion planning studies are essential to rapidly and efficiently decarbonizing the electricity sector. These studies help policy makers and grid participants understand which renewable generation, storage, and transmission assets should be built and where they will be most cost effective or have the highest emissions impact. However, these studies are often either too computationally expensive to run repeatedly or too coarsely modeled to give actionable decision information. In this study, we present an implicit gradient descent algorithm to solve expansion planning studies at scale, i.e., problems with many scenarios and large network models. Our algorithm is also interactive: given a base plan, planners can modify assumptions and data then quickly receive an updated plan. This allows the planner to study expansion outcomes for a wide variety of technology cost, weather, and electrification assumptions. We demonstrate the scalability of our tool, solving a case with over a hundred million variables. Then, we show that using warm starts can speed up subsequent runs by as much as 100x. We highlight how this can be used to quickly conduct storage cost uncertainty analysis.

math.OC↗

arXiv2024

Fast Grid Emissions Sensitivities using Parallel Decentralized Implicit Differentiation

Marginal emissions rates -- the sensitivity of carbon emissions to electricity demand -- are important for evaluating the impact of emissions mitigation measures. Like locational marginal prices, locational marginal emissions rates (LMEs) can vary geographically, even between nearby locations, and may be coupled across time periods because of, for example, storage and ramping constraints. This temporal coupling makes computing LMEs computationally expensive for large electricity networks with high storage and renewable penetrations. Recent work demonstrates that decentralized algorithms can mitigate this problem by decoupling timesteps during differentiation. Unfortunately, we show these potential speedups are negated by the sparse structure inherent in power systems problems. We address these limitations by introducing a parallel, reverse-mode decentralized differentiation scheme that never explicitly instantiates the solution map Jacobian. We show both theoretically and empirically that parallelization is necessary to achieve non-trivial speedups when computing grid emissions sensitivities. Numerical results on a 500 node system indicate that our method can achieve greater than 10x speedups over centralized and serial decentralized approaches.

eess.SY↗

arXiv2024

Gradient Methods for Scalable Multi-value Electricity Network Expansion Planning

We consider multi-value expansion planning (MEP), a general bilevel optimization model in which a planner optimizes arbitrary functions of the dispatch outcome in the presence of a partially controllable, competitive electricity market. The MEP problem can be used to jointly plan various grid assets, such as transmission, generation, and battery storage capacities; examples include identifying grid investments that minimize emissions in the absence of a carbon tax, maximizing the profit of a portfolio of renewable investments and long-term energy contracts, or reducing price inequities between different grid stakeholders. The MEP problem, however, is in general nonconvex, making it difficult to solve exactly for large real-world systems. Therefore, we propose a fast stochastic implicit gradient-based heuristic method that scales well to large networks with many scenarios. We use a strong duality reformulation and the McCormick envelope to provide a lower bound on the performance of our algorithm via convex relaxation. We test the performance of our method on a large model of the U.S. Western Interconnect and demonstrate that it scales linearly with network size and number of scenarios and can be efficiently parallelized on large machines. We find that for medium-sized 16 hour cases, gradient descent on average finds a 5.3x lower objective value in 16.5x less time compared to a traditional reformulation-based approach solved with an interior point method. We conclude with a large example in which we jointly plan transmission, generation, and storage for a 768 hour case on 100 node system, showing that emissions penalization leads to additional 40.0% reduction in carbon intensity at an additional cost of $17.1/MWh.

math.OC↗

arXiv2023

Coordination of DERs for Grid Reliability via Day-ahead Demand-Supply Power Bounds

A previous study has shown that coordinating DERs to protect the distribution grid can significantly reduce the infrastructure upgrades needed to address future increases in DER and electrification penetrations. Implementing such coordination in the real world, however, is challenging due the temporal and spatial uncertainties about the loads and renewable generation, smart meter and network delays, incomplete information about the grid, different consumer objectives and privacy constraints, and scalability of the coordination scheme. This paper describes a day-ahead 2-layer DER coordination scheme that addresses these challenges. A global controller uses historical load data to compute day-ahead hourly demand upper and lower bounds for each consumer node. It then solves a largest volume axis-aligned box optimization problem to determine corresponding supply power bounds which if followed, ensures grid reliability. A local controller at each consumer node then determines the DER power injections which satisfy the consumer's objectives while obeying its supply bounds. Simulation results demonstrate, for example, that this scheme can capture 62% of the reduction in transformer violations achievable by the perfect-foresight centralized controller used in the aforementioned previous study.

eess.SY↗

arXiv2023

Coordinating Distributed Energy Resources for Reliability can Significantly Reduce Future Distribution Grid Upgrades and Peak Load

Current DER coordination schemes, such as demand-response and VPPs, aim to reduce electricity costs during peak demand events with no consideration of distribution grid reliability. We show that coordinating DERs for grid reliability can significantly reduce both the infrastructure upgrades needed to support future increases in DER and electrification penetrations and peak load. Specifically, using a power flow driven simulation-optimization methodology, we compare the potential reliability improvements with a perfect-foresight centralized DER controller that minimizes reliability violations to a local controller that minimizes consumer electricity cost. We find, for example, that by 2050 with local control, on average 81% of the transformers in a distribution grid experience violations, compared to 28% with centralized control, which simultaneously reduces peak load by 17%. These reductions are achieved with only 5.1% increase in electricity cost. These findings suggest that future incentives for DER adoption should include reliability coordination.

eess.SY↗

arXiv2023

Dynamic locational marginal emissions via implicit differentiation

Locational marginal emissions rates (LMEs) estimate the rate of change in emissions due to a small change in demand in a transmission network, and are an important metric for assessing the impact of various energy policies or interventions. In this work, we develop a new method for computing the LMEs of an electricity system via implicit differentiation. The method is model agnostic; it can compute LMEs for any convex optimization-based dispatch model, including some of the complex dispatch models employed by system operators in real electricity systems. In particular, this method lets us derive LMEs for dynamic dispatch models, i.e., models with temporal constraints such as ramping and storage. Using real data from the U.S. electricity system, we validate the proposed method against a state-of-the-art merit-order-based method and show that incorporating dynamic constraints improves model accuracy by 8.2%. Finally, we use simulations on a realistic 240-bus model of WECC to demonstrate the flexibility of the tool and the importance of incorporating dynamic constraints. Namely, static LMEs and dynamic LMEs exhibit a normalized average RMS deviation of 28.40%, implying dynamic constraints are essential to accurately modeling emissions rates.

eess.SY↗

arXiv2022

A Strengthened Cutset Upper Bound on the Capacity of the Relay Channel and Applications

We develop a new upper bound on the capacity of the relay channel that is tighter than previously known upper bounds. This upper bound is proved using traditional weak converse techniques involving mutual information inequalities and Gallager-type explicit identification of auxiliary random variables. We show that the new upper bound is strictly tighter than all previous bounds for the Gaussian relay channel with non-zero channel gains. When specialized to the relay channel with orthogonal receiver components, the bound resolves a conjecture by Kim on a class of deterministic relay channels. When further specialized to the class of product-form relay channels with orthogonal receiver components, the bound resolves a generalized version of Cover's relay channel problem, recovers the recent upper bound for the Gaussian case by Wu et al., and improves upon the recent bounds for the binary symmetric case by Wu et al. and Barnes et al., which were obtained using non-traditional geometric proof techniques. For the special class of a relay channel with orthogonal receiver components, we develop another upper bound on the capacity which utilizes an auxiliary receiver and show that it is strictly tighter than the bound by Tandon and Ulukus. Finally, we show through the Gaussian relay channel with i.i.d. relay output sequence that the bound with the auxiliary receiver can be strictly tighter than our main bound.

cs.IT↗

arXiv2020

DER Information Unaware Coordination via Day-ahead Dynamic Power Bounds

Reliability and voltage quality in distribution networks have been achieved via a combination of transformer power rating satisfaction and voltage management asset control. To maintain reliable operation under this paradigm, however, future grids with deep DER penetrations would require costly equipment upgrades. These upgrades can be mitigated via judicious coordination of DER operation. Earlier work has assumed a hierarchical control architecture in which a global controller (GC) uses detailed power injection and DER data and knowledge of DER owners' objectives to determine setpoints that local controllers should follow in order to achieve reliable and cost effective grid operation. Having such data and assuming knowledge of DER owners' objectives, however, are often not desirable or possible. In an earlier work, a 2-layer DER coordination architecture was shown to achieve close to optimal performance despite infrequent communication to a global controller. Motivated by this work, this paper proposes a day-ahead coordination scheme that uses forecasted power profile ranges to generate day-ahead dynamic power rating bounds at each transformer. Novel features of this scheme include: (i) the GC knows only past node power injection data and does not impose or know DER owner objectives, (ii) we use bounds that ensure reliable operation to guide the local controllers rather than setpoint tracking, and (iii) we consider electric vehicle (EV) charging in addition to storage. Simulations using the IEEE 123-bus network show that with 50% solar, 50% EVs and 10% storage penetrations, the uncoordinated approach incurs rating violations at nearly all 86 transformers and results in 10 times higher voltage deviation, while our approach incurs only 12 rating violations and maintains almost the same voltage deviations as before the addition of solar and EVs.

eess.SY↗

arXiv2020

Network Information Theoretic Security

Shannon showed that to achieve perfect secrecy in point-to-point communication, the message rate cannot exceed the shared secret key rate giving rise to the simple one-time pad encryption scheme. In this paper, we extend this work from point-to-point to networks. We consider a connected network with pairwise communication between the nodes. We assume that each node is provided with a certain amount of secret bits before communication commences. An eavesdropper with unlimited computing power has access to all communication and can hack a subset of the nodes not known to the rest of the nodes. We investigate the limits on information-theoretic secure communication for this network. We establish a tradeoff between the secure channel rate (for a node pair) and the secure network rate (sum over all node pair rates) and show that perfect secrecy can be achieved if and only if the sum rate of any subset of unhacked channels does not exceed the shared unhacked-secret-bit rate of these channels. We also propose two practical and efficient schemes that achieve a good balance of network and channel rates with perfect secrecy guarantee. This work has a wide range of potential applications for which perfect secrecy is desired, such as cyber-physical systems, distributed-control systems, and ad-hoc networks.

cs.IT↗

arXiv2018

Minimax Learning for Remote Prediction

The classical problem of supervised learning is to infer an accurate predictor of a target variable $Y$ from a measured variable $X$ by using a finite number of labeled training samples. Motivated by the increasingly distributed nature of data and decision making, in this paper we consider a variation of this classical problem in which the prediction is performed remotely based on a rate-constrained description $M$ of $X$. Upon receiving $M$, the remote node computes an estimate $\hat Y$ of $Y$. We follow the recent minimax approach to study this learning problem and show that it corresponds to a one-shot minimax noisy source coding problem. We then establish information theoretic bounds on the risk-rate Lagrangian cost and a general method to design a near-optimal descriptor-estimator pair, which can be viewed as a rate-constrained analog to the maximum conditional entropy principle used in the classical minimax learning problem. Our results show that a naive estimate-compress scheme for rate-constrained prediction is not in general optimal.

cs.IT↗

arXiv2018

A Two-layer Decentralized Control Architecture for DER Coordination

This paper presents a two-layer distributed energy resource (DER) coordination architecture that allows for separate ownership of data, operates with data subjected to a large buffering delay, and employs a new measure of power quality. The two-layer architecture comprises a centralized model predictive controller (MPC) and several decentralized MPCs each operating independently with no direct communication between them and with infrequent communication with the centralized controller. The goal is to minimize a combination of total energy cost and a measure of power quality while obeying cyber-physical constraints. The global controller utilizes a fast AC optimal power flow (OPF) solver and extensive parallelization to scale the solution to large networks. Each local controller attempts to maximize arbitrage profit while following the load profile and constraints dictated by the global controller. Extensive simulations are performed for two distribution networks under a wide variety of possible storage and solar penetrations enabled by the controller speed. The simulations show that (i) the two-layer architecture can achieve tenfold improvement in power quality relative to no coordination, while capturing nearly all of the available arbitrage profit for a moderate amount of storage penetration, and (ii) both power quality and arbitrage profits are optimized when the solar and storage are distributed more widely over the network, hence it is more effective to install storage closer to the consumer.

eess.SP↗

arXiv2018

State-Adaptive Coded Caching for Symmetric Broadcast Channels

Coded-caching delivery is considered over a symmetric noisy broadcast channel whose state is unknown at the transmitter during the cache placement phase. In particular, the delivery phase is modeled by a state-dependent broadcast channel where the state remains constant over each transmission block and is learned by the transmitter (and the receivers) only at the beginning of each block. A state-adaptive coded caching scheme is proposed that improves either on rate or decoding latency over two baseline schemes that are based on standard coded caching.

cs.IT↗

arXiv2018

Strong Functional Representation Lemma and Applications to Coding Theorems

This paper shows that for any random variables $X$ and $Y$, it is possible to represent $Y$ as a function of $(X,Z)$ such that $Z$ is independent of $X$ and $I(X;Z|Y)\le\log(I(X;Y)+1)+4$ bits. We use this strong functional representation lemma (SFRL) to establish a bound on the rate needed for one-shot exact channel simulation for general (discrete or continuous) random variables, strengthening the results by Harsha et al. and Braverman and Garg, and to establish new and simple achievability results for one-shot variable-length lossy source coding, multiple description coding and Gray-Wyner system. We also show that the SFRL can be used to reduce the channel with state noncausally known at the encoder to a point-to-point channel, which provides a simple achievability proof of the Gelfand-Pinsker theorem.

cs.IT↗

arXiv2017

Extended Gray-Wyner System with Complementary Causal Side Information

We establish the rate region of an extended Gray-Wyner system for 2-DMS $(X,Y)$ with two additional decoders having complementary causal side information. This extension is interesting because in addition to the operationally significant extreme points of the Gray-Wyner rate region, which include Wyner's common information, G{á}cs-K{ö}rner common information and information bottleneck, the rate region for the extended system also includes the K{ö}rner graph entropy, the privacy funnel and excess functional information, as well as three new quantities of potential interest, as extreme points. To simplify the investigation of the 5-dimensional rate region of the extended Gray-Wyner system, we establish an equivalence of this region to a 3-dimensional mutual information region that consists of the set of all triples of the form $(I(X;U),\,I(Y;U),\,I(X,Y;U))$ for some $p_{U|X,Y}$. We further show that projections of this mutual information region yield the rate regions for many settings involving a 2-DMS, including lossless source coding with causal side information, distributed channel synthesis, and lossless source coding with a helper.

cs.IT↗

arXiv2016

Distributed Simulation of Continuous Random Variables

We establish the first known upper bound on the exact and Wyner's common information of $n$ continuous random variables in terms of the dual total correlation between them (which is a generalization of mutual information). In particular, we show that when the pdf of the random variables is log-concave, there is a constant gap of $n^{2}\log e+9n\log n$ between this upper bound and the dual total correlation lower bound that does not depend on the distribution. The upper bound is obtained using a computationally efficient dyadic decomposition scheme for constructing a discrete common randomness variable $W$ from which the $n$ random variables can be simulated in a distributed manner. We then bound the entropy of $W$ using a new measure, which we refer to as the erosion entropy.

cs.IT↗