arXiv subjects

Alexandre Street

Publications and source records attributed to Alexandre Street.

At least 19 recordsLinked to original sources

arXiv2026

How optimistic inflow forecasts distort dispatch, prices, and contracts in hydro-dominated power systems: evidence from Brazil

Centralized hydrothermal planning models determine generation schedules and electricity spot prices based on inflow forecasts in audited-cost power systems, such as those prevalent in Latin America, and provide operational benchmarks and decision support in hydro-dominated competitive electricity markets. Consequently, biased forecasts can propagate directly into both operational decisions and market outcomes. This paper studies how persistent optimistic inflow-forecast bias propagates through the Brazilian hydrothermal power system and market. For a stylized hydrothermal model, we show analytically that optimistic bias weakly reduces water values and weakly increases first-stage hydro discharge relative to the unbiased optimum, thereby lowering reservoir storage and postponing thermal commitment. Using official Brazilian planning and operational data, we provide empirical evidence consistent with this mechanism. We then conduct a controlled SDDP experiment to compare policies trained under biased and bias-corrected inflow-forecast processes, evaluating both under the same bias-corrected inflow scenarios. The policy trained under biased forecasts produces lower reservoir levels, delayed dry-season thermal dispatch, sharper spot-price peaks, higher reliability risk, and higher expected operating costs. Finally, we show that these distortions increase the price-quantity risk for hydropower producers and reduce their willingness to contract. The results indicate that inflow-forecast bias is not merely a statistical forecasting problem, but can be a source of operational inefficiency, reliability risk, and distorted market incentives in hydro-dominated power systems. We argue that the insights and policy implications drawn in this paper may be relevant beyond Brazil to other hydro-dominated systems and electricity markets that are increasingly reliant on energy storage.

econ.GN↗

arXiv2025

Penalty-Free SDDP: Feasibility Cuts for Robust Multi-Stage Stochastic Optimization in Energy Planning

Multi-stage decision problems under uncertainty can be efficiently solved with the Stochastic Dual Dynamic Programming (SDDP) algorithm. However, traditional implementations require all stage problems to be feasible. Feasibility is usually enforced by adding slack variables and penalizing them in the objective function, a process that depends on case-specific calibration and often distorts the economic interpretation of results. This paper proposes the Penalty-Free SDDP, an extension that introduces a Future Feasibility Function alongside the traditional Future Cost Function. The new recursion handles infeasibilities automatically, distinguishing between temporary and truly infeasible cases, and propagates feasibility information across stages through dedicated feasibility cuts. The approach was validated in a large-scale deterministic case inspired by the Brazilian hydrothermal system, achieving equivalent feasibility to the benchmark solution while eliminating miscalibrated artificial penalties. Results confirm its robustness and practicality as a foundation for future stochastic, multi-stage applications.

math.OC↗

arXiv2025

Two-Stage Data-Driven Contextual Robust Optimization: An End-to-End Learning Approach for Online Energy Applications

Traditional end-to-end contextual robust optimization models are trained for specific contextual data, requiring complete retraining whenever new contextual information arrives. This limitation hampers their use in online decision-making problems such as energy scheduling, where multiperiod optimization must be solved every few minutes. In this paper, we propose a novel Data-Driven Contextual Uncertainty Set, which gives rise to a new End-to-End Data-Driven Contextual Robust Optimization model. For right-hand-side uncertainty, we introduce a reformulation scheme that enables the development of a variant of the Column-and-Constraint Generation (CCG) algorithm. This new CCG method explicitly considers the contextual vector within the cut expressions, allowing previously generated cuts to remain valid for new contexts, thereby significantly accelerating convergence in online applications. Numerical experiments on energy and reserve scheduling problems, based on classical test cases and large-scale networks (with more than 10,000 nodes), demonstrate that the proposed method reduces computation times and operational costs while capturing context-dependent risk structures. The proposed framework (model and method), therefore, offers a unified, scalable, and prescriptive approach to robust decision-making under uncertainty, effectively bridging data-driven learning and optimization.

math.OC↗

arXiv2025

A Causation-Based Framework for Pricing and Cost Allocation of Energy, Reserves, and Transmission in Modern Power Systems

The increasing vulnerability of power systems has heightened the need for operating reserves to manage contingencies such as generator outages, line failures, and sudden load variations. Unlike energy costs, driven by consumer demand, operating reserve costs arise from addressing the most critical credible contingencies - prompting the question: how should these costs be allocated through efficient pricing mechanisms? As an alternative to previously reported schemes, this paper presents a new causation-based pricing framework for electricity markets based on contingency-constrained energy and reserve scheduling models. Major salient features include a novel security charge mechanism along with the explicit definition of prices for up-spinning reserves, down-spinning reserves, and transmission services. These features ensure more comprehensive and efficient cost-reflective market operations. Moreover, the proposed nodal pricing scheme yields revenue adequacy and neutrality while promoting reliability incentives for generators based on the cost-causation principle. An additional salient aspect of the proposed framework is the economic incentive for transmission assets, which are remunerated based on their use to deliver energy and reserves across all contingency states. Numerical results from two case studies illustrate the performance of the proposed pricing scheme.

eess.SY↗

arXiv2024

Assessing the Optimistic Bias in the Natural Inflow Forecasts: A Call for Model Monitoring in Brazil

Hydroelectricity accounted for roughly 61.4% of Brazil's total generation in 2024 and addressed most of the intermittency of wind and solar generation. Thus, inflow forecasting plays a critical role in the operation, planning, and market in this country, as well as in any other hydro-dependent power system. These forecasts influence generation schedules, reservoir management, and market pricing, shaping the dynamics of the entire electricity sector. The objective of this paper is to measure and present empirical evidence of a systematic optimistic bias in the official inflow forecast methodology, which is based on the PAR(p)-A model. Additionally, we discuss possible sources of this bias and recommend ways to mitigate it. By analyzing 14 years of historical data from the Brazilian system through rolling-window multistep (out-of-sample) forecasts, results indicate that the official forecast model exhibits statistically significant biases of 1.28, 3.83, 5.39, and 6.73 average GW for 1-, 6-, 12-, and 24-step-ahead forecasts in the Southeast subsystem, and 0.54, 1.66, 2.32, and 3.17 average GW in the Northeast subsystem. These findings uncover the limitations of current inflow forecasting methodologies used in Brazil and call for new governance and monitoring policies.

eess.SY↗

arXiv2024

Time Series Analysis by State Space Learning

Time series analysis by state-space models is widely used in forecasting and extracting unobservable components like level, slope, and seasonality, along with explanatory variables. However, their reliance on traditional Kalman filtering frequently hampers their effectiveness, primarily due to Gaussian assumptions and the absence of efficient subset selection methods to accommodate the multitude of potential explanatory variables in today's big-data applications. Our research introduces the State Space Learning (SSL), a novel framework and paradigm that leverages the capabilities of statistical learning to construct a comprehensive framework for time series modeling and forecasting. By utilizing a regularized high-dimensional regression framework, our approach jointly extracts typical time series unobservable components, detects and addresses outliers, and selects the influence of exogenous variables within a high-dimensional space in polynomial time and global optimality guarantees. Through a controlled numerical experiment, we demonstrate the superiority of our approach in terms of subset selection of explanatory variables accuracy compared to relevant benchmarks. We also present an intuitive forecasting scheme and showcase superior performances relative to traditional time series models using a dataset of 48,000 monthly time series from the M4 competition. We extend the applicability of our approach to reformulate any linear state space formulation featuring time-varying coefficients into high-dimensional regularized regressions, expanding the impact of our research to other engineering applications beyond time series analysis. Finally, our proposed methodology is implemented within the Julia open-source package, ``StateSpaceLearning.jl".

stat.ML↗

arXiv2024

Efficiently Training Deep-Learning Parametric Policies using Lagrangian Duality

Constrained Markov Decision Processes (CMDPs) are critical in many high-stakes applications, where decisions must optimize cumulative rewards while strictly adhering to complex nonlinear constraints. In domains such as power systems, finance, supply chains, and precision robotics, violating these constraints can result in significant financial or societal costs. Existing Reinforcement Learning (RL) methods often struggle with sample efficiency and effectiveness in finding feasible policies for highly and strictly constrained CMDPs, limiting their applicability in these environments. Stochastic dual dynamic programming is often used in practice on convex relaxations of the original problem, but they also encounter computational challenges and loss of optimality. This paper introduces a novel approach, Two-Stage Deep Decision Rules (TS-DDR), to efficiently train parametric actor policies using Lagrangian Duality. TS-DDR is a self-supervised learning algorithm that trains general decision rules (parametric policies) using stochastic gradient descent (SGD); its forward passes solve {\em deterministic} optimization problems to find feasible policies, and its backward passes leverage duality theory to train the parametric policy with closed-form gradients. TS-DDR inherits the flexibility and computational performance of deep learning methodologies to solve CMDP problems. Applied to the Long-Term Hydrothermal Dispatch (LTHD) problem using actual power system data from Bolivia, TS-DDR is shown to enhance solution quality and to reduce computation times by several orders of magnitude when compared to current state-of-the-art methods.

cs.LG↗

arXiv2024

Long-term Hydrothermal Bid-based Market Simulator

Simulating long-term hydrothermal bid-based markets considering strategic agents is a challenging task. The representation of strategic agents considering intertemporal constraints within a stochastic framework brings additional complexity to the already difficult single-period bilevel, thus, non-convex, optimal bidding problem. Thus, we propose a simulation methodology that effectively addresses these challenges for large-scale hydrothermal power systems. We demonstrate the effectiveness of the framework through a case study with real data from the large-scale Brazilian power system. In the case studies, we show the effects of market concentration in power systems and how contracts can be used to mitigate them. In particular, we show how market power might affect the current setting in Brazil. The developed method can strongly benefit policymakers, market monitors, and market designers as simulations can be used to understand existing power systems and experiment with alternative designs.

math.OC↗

arXiv2023

Distribution System Operation Amidst Wildfire-Prone Climate Conditions Under Decision-Dependent Line Availability Uncertainty

Wildfires can severely damage electricity grids leading to long periods of power interruption. Climate change will exacerbate this threat by increasing the frequency of dry climate conditions. Under these climate conditions, human-related actions that initiate wildfires should be avoided, including those induced by power systems operation. In this paper, we propose a novel optimization model that is capable of determining appropriate network topology changes (via switching actions) to alleviate the levels of power flows through vulnerable parts of the grid so as to decrease the probability of wildfire ignition. Within this framework, the proposed model captures the relationship between failure probabilities and line-flow decisions by explicitly considering the former as a function of the latter. The resulting formulation is a two-stage model with endogenous decision-dependent probabilities, where the first stage determines the optimal switching actions and the second stage evaluates the worst-case expected operation cost. We propose an exact iterative method to deal with this intricate problem and the methodology is illustrated with a 54-bus and a 138-bus distribution system.

math.OC↗

arXiv2023

Comparing BilevelJuMP.jl Formulations: Support Vector Regression Hyperparameter Tuning

In this technical report, we compare multiple reformulation techniques and solvers that can be used with the Julia package BilevelJuMP. We focus on the special case of Hyperparameter Tuning for Support Vector Regression. We describe a bilevel model for the problem in question. Then we present code for generating data and models that solve the problem. Finally, we present results and a brief analysis.

math.OC↗

arXiv2022

BilevelJuMP.jl: Modeling and Solving Bilevel Optimization in Julia

In this paper we present BilevelJuMP, a new Julia package to support bilevel optimization within the JuMP framework. The package is a Julia library that enables the user to describe both upper and lower-level optimization problems using the JuMP algebraic syntax. Due to the generality and flexibility our library inherits from JuMP's syntax, our package allows users to model bilevel optimization problems with conic constraints in the lower level and all JuMP supported constraints in the upper level (Conic, Quadratic, Non-Linear, Integer, etc.). Moreover, the user-defined problem can be subsequently solved by various techniques relying on mathematical program with equilibrium constraints (MPEC) reformulations. Manipulations on the original problem data are possible due to MathOptInterface.jl's structures and Dualization.jl features. Hence, the proposed package allows quickly model, deploy, and thereby experiment bilevel models based on off-the-shelf mixed integer linear programming and nonlinear solvers.

math.OC↗

arXiv2021

Solving Multistage Stochastic Linear Programming via Regularized Linear Decision Rules: An Application to Hydrothermal Dispatch Planning

The solution of multistage stochastic linear problems (MSLP) represents a challenge for many application areas. Long-term hydrothermal dispatch planning (LHDP) materializes this challenge in a real-world problem that affects electricity markets, economies, and natural resources worldwide. No closed-form solutions are available for MSLP and the definition of non-anticipative policies with high-quality out-of-sample performance is crucial. Linear decision rules (LDR) provide an interesting simulation-based framework for finding high-quality policies for MSLP through two-stage stochastic models. In practical applications, however, the number of parameters to be estimated when using an LDR may be close to or higher than the number of scenarios of the sample average approximation problem, thereby generating an in-sample overfit and poor performances in out-of-sample simulations. In this paper, we propose a novel regularized LDR to solve MSLP based on the AdaLASSO (adaptive least absolute shrinkage and selection operator). The goal is to use the parsimony principle, as largely studied in high-dimensional linear regression models, to obtain better out-of-sample performance for LDR applied to MSLP. Computational experiments show that the overfit threat is non-negligible when using classical non-regularized LDR to solve the LHDP, one of the most studied MSLP with relevant applications. Our analysis highlights the following benefits of the proposed framework in comparison to the non-regularized benchmark: 1) significant reductions in the number of non-zero coefficients (model parsimony), 2) substantial cost reductions in out-of-sample evaluations, and 3) improved spot-price profiles.

math.OC↗

arXiv2021

An Integrated Progressive Hedging and Benders Decomposition with Multiple Master Method to Solve the Brazilian Generation Expansion Problem

This paper exploits the decomposition structure of the large-scale hydrothermal generation expansion planning problem with an integrated modified Benders Decomposition and Progressive Hedging approach. We consider detailed and realistic data from the Brazilian power system to represent hourly chronological constraints based on typical days per month and year. Also, we represent the multistage stochastic nature of the optimal hydrothermal operational policy through co-optimized linear decision rules for individual reservoirs. Therefore, we ensure investment decisions compatible with a nonanticipative (implementable) operational policy. To solve the large-scale optimization problem, we propose an improved Benders Decomposition method with multiple instances of the master problem, each of which strengthened by primal cuts and new Benders cuts generated by each master's trial solution. Additionally, our new approach allows using Progressive Hedging penalization terms for accelerating the convergence of the method. We show that our method is 60\% faster than the benchmark. Finally, the consideration of a nonanticipative operational policy can save 7.64\% of the total cost (16.18\% of the investment costs) and significantly improve spot price profiles.

math.OC↗

arXiv2021

Assessing the Cost of Network Simplifications in Long-Term Hydrothermal Dispatch Planning Models

The sustainable utilization of hydro energy relies on accurate estimates of the opportunity cost of the water. This value is calculated through long-term hydrothermal dispatch problems (LTHDP), and the recent literature has raised awareness about the consequences of modeling simplifications in these problems. The inaccurate representation of Kirchhoff's voltage law under the premise of a DC power flow is an example. Under a non-linear AC model, however, the LTHDP becomes intractable, and the literature lacks an accurate evaluation method of different modeling alternatives. In this paper, we extend the state-of-the-art cost-assessment framework of network approximations for LTHDP and bring relevant and practical new insights. First, we increase the quality of the assessment by using an AC power flow to simulate and compare the performance of five policies based on different network approximations. Second, we find that the tightest network relaxation (based on semidefinite programming) is not the one exhibiting the best performance. Results show that the DC {power flow} with quadratic losses approximation exhibits the lowest expected cost and inconsistency gaps. Finally, its computational burden is lower than that exhibited by the semidefinite relaxation, whereas market distortions are significantly reduced in comparison to previously published benchmarks based on DC power flow.

math.OC↗

arXiv2021

Application-Driven Learning: A Closed-Loop Prediction and Optimization Approach Applied to Dynamic Reserves and Demand Forecasting

Forecasting and decision-making are generally modeled as two sequential steps with no feedback, following an open-loop approach. In this paper, we present application-driven learning, a new closed-loop framework in which the processes of forecasting and decision-making are merged and co-optimized through a bilevel optimization problem. We present our methodology in a general format and prove that the solution converges to the best estimator in terms of the expected cost of the selected application. Then, we propose two solution methods: an exact method based on the KKT conditions of the second-level problem and a scalable heuristic approach suitable for decomposition methods. The proposed methodology is applied to the relevant problem of defining dynamic reserve requirements and conditional load forecasts, offering an alternative approach to current ad hoc procedures implemented in industry practices. We benchmark our methodology with the standard sequential least-squares forecast and dispatch planning process. We apply the proposed methodology to an illustrative system and to a wide range of instances, from dozens of buses to large-scale realistic systems with thousands of buses. Our results show that the proposed methodology is scalable and yields consistently better performance than the standard open-loop approach.

math.OC↗

arXiv2020

ScoreDrivenModels.jl: a Julia Package for Generalized Autoregressive Score Models

Score-driven models, also known as generalized autoregressive score models, represent a class of observation-driven time series models. They possess powerful properties, such as the ability to model different conditional distributions and to consider time-varying parameters within a flexible framework. In this paper, we present ScoreDrivenModels.jl, an open-source Julia package for modeling, forecasting, and simulating time series using the framework of score-driven models. The package is flexible with respect to model definition, allowing the user to specify the lag structure and which parameters are time-varying or constant. It is also possible to consider several distributions, including Beta, Exponential, Gamma, Lognormal, Normal, Poisson, Student's t, and Weibull. The provided interface is flexible, allowing interested users to implement any desired distribution and parametrization.

stat.CO↗

arXiv2020

Short-Term Covid-19 Forecast for Latecomers

The number of Covid-19 cases is increasing dramatically worldwide. Therefore, the availability of reliable forecasts for the number of cases in the coming days is of fundamental importance. We propose a simple statistical method for short-term real-time forecasting of the number of Covid-19 cases and fatalities in countries that are latecomers -- i.e., countries where cases of the disease started to appear some time after others. In particular, we propose a penalized (LASSO) regression with an error correction mechanism to construct a model of a latecomer in terms of the other countries that were at a similar stage of the pandemic some days before. By tracking the number of cases and deaths in those countries, we forecast through an adaptive rolling-window scheme the number of cases and deaths in the latecomer. We apply this methodology to Brazil, and show that (so far) it has been performing very well. These forecasts aim to foster a better short-run management of the health system capacity.

stat.AP↗

arXiv2020

A Multi-Quantile Regression Time Series Model with Interquantile Lipschitz Regularization for Wind Power Probabilistic Forecasting

Modern decision-making processes require uncertainty-aware models, especially those relying on non-symmetric costs and risk-averse profiles. The objective of this work is to propose a dynamic model for the conditional non-parametric distribution function (CDF) to generate probabilistic forecasts for a renewable generation time series. To do that, we propose an adaptive non-parametric time-series model driven by a regularized multiple-quantile-regression (MQR) framework. In our approach, all regression models are jointly estimated through a single linear optimization problem that finds the global-optimal parameters in polynomial time. An innovative feature of our work is the consideration of a Lipschitz regularization of the first derivative of coefficients in the quantile space, which imposes coefficient smoothness. The proposed regularization induces a coupling effect among quantiles creating a single non-parametric CDF model with improved out-of-sample performance. A case study with realistic wind-power generation data from the Brazilian system shows: 1) the regularization model is capable to improve the performance of MQR probabilistic forecasts, and 2) our MQR model outperforms five relevant benchmarks: two based on the MQR framework, and three based on parametric models, namely, SARIMA, and GAS with Beta and Weibull CDF.

stat.AP↗