arXiv subjects

Joaquim Dias Garcia

Publications and source records attributed to Joaquim Dias Garcia.

At least 19 recordsLinked to original sources

arXiv2026

How optimistic inflow forecasts distort dispatch, prices, and contracts in hydro-dominated power systems: evidence from Brazil

Centralized hydrothermal planning models determine generation schedules and electricity spot prices based on inflow forecasts in audited-cost power systems, such as those prevalent in Latin America, and provide operational benchmarks and decision support in hydro-dominated competitive electricity markets. Consequently, biased forecasts can propagate directly into both operational decisions and market outcomes. This paper studies how persistent optimistic inflow-forecast bias propagates through the Brazilian hydrothermal power system and market. For a stylized hydrothermal model, we show analytically that optimistic bias weakly reduces water values and weakly increases first-stage hydro discharge relative to the unbiased optimum, thereby lowering reservoir storage and postponing thermal commitment. Using official Brazilian planning and operational data, we provide empirical evidence consistent with this mechanism. We then conduct a controlled SDDP experiment to compare policies trained under biased and bias-corrected inflow-forecast processes, evaluating both under the same bias-corrected inflow scenarios. The policy trained under biased forecasts produces lower reservoir levels, delayed dry-season thermal dispatch, sharper spot-price peaks, higher reliability risk, and higher expected operating costs. Finally, we show that these distortions increase the price-quantity risk for hydropower producers and reduce their willingness to contract. The results indicate that inflow-forecast bias is not merely a statistical forecasting problem, but can be a source of operational inefficiency, reliability risk, and distorted market incentives in hydro-dominated power systems. We argue that the insights and policy implications drawn in this paper may be relevant beyond Brazil to other hydro-dominated systems and electricity markets that are increasingly reliant on energy storage.

econ.GN↗

arXiv2026

QUBO.jl: A Julia Ecosystem for Quadratic Unconstrained Binary Optimization

We present QUBO.jl, an end-to-end Julia package for working with QUBO (Quadratic Unconstrained Binary Optimization) instances. This tool aims to convert a broad range of optimization problems in JuMP, Julia's mathematical programming package, for straightforward application in many physics and physics-inspired solution methods whose standard model form is equivalent to QUBO. These methods include quantum annealing, quantum gate-circuit optimization algorithms (Quantum Optimization Alternating Ansatz, Variational Quantum Eigensolver), other hardware-accelerated platforms, such as Coherent Ising Machines and Simulated Bifurcation Machines, and more traditional methods such as simulated annealing. In addition to working with reformulations, QUBO.jl allows its users to interface with the aforementioned hardware, sending QUBO models to these devices and retrieving results for subsequent analysis. QUBO.jl was written as a JuMP / MathOptInterface (MOI) layer that automatically maps between the input and output frames, thus providing a smooth modeling experience.

math.OC↗

arXiv2026

MathOptInterface: a data structure for mathematical optimization problems

We introduce MathOptInterface, an abstract data structure for representing mathematical optimization problems based on combining pre-defined functions and sets. MathOptInterface is significantly more general than existing data structures in the literature, encompassing, for example, a spectrum of problems classes from integer programming with indicator constraints to bilinear semidefinite programming. We also outline an automated rewriting system between equivalent formulations of a constraint. MathOptInterface has been implemented in practice, forming the foundation of a recent rewrite of JuMP, an open-source algebraic modeling language in the Julia language. The regularity of the MathOptInterface representation leads naturally to a general file format for mathematical optimization we call MathOptFormat. In addition, the automated rewriting system provides modeling power to users while making it easy to connect new solvers to JuMP.

math.OC↗

arXiv2026

An Efficient Hybrid Heuristic for the Transmission Expansion Planning under Uncertainty

We address the stochastic transmission expansion planning (STEP) problem under uncertainty in renewable generation capacity and demand. STEP's objective is to minimize total transmission investment and generation costs. To tackle the computational challenges posed by large-scale systems, we propose a heuristic strategy that combines the progressive hedging (PH) algorithm for scenario-wise decomposition with an integrated approach for solving the resulting subproblems. The latter combines a destroy-and-repair operator, a beam search procedure, and a mixed-integer programming solver. The proposed framework is evaluated on large-scale systems from the literature with up to 10000 nodes, adapted to stochastic scenarios using parameters from the California test system (CATS). Compared with a non-trivial baseline algorithm that includes the same integrated approach, the proposed PH-based method consistently improved solution quality for the six systems considered (including CATS), achieving an average cost improvement of 5.28% within a 2-hour time limit.

math.OC↗

arXiv2026

Integrated Investment and Operational Planning for Sugarcane-Based Biofuels and Bioelectricity under Market Uncertainty

Sugarcane biomass is a strategic resource for the energy transition, particularly in Brazil, where it underpins electricity and ethanol production. Investment planning is challenged by diverse production pathways, price volatility, and feedstock variability. This work develops a two-stage stochastic optimization model integrating investment and operational decisions for sugarcane facilities. The model aims to support robust planning for diversified biomass plants, aiding the sector's decarbonization. The first stage defines capacity expansion under economies of scale through a power-law cost function. The second stage defines operational decisions under price and feedstock uncertainty, modeled via scenarios and Conditional Value-at-Risk. \answer{From an investor's perspective, the objective is to minimize risk-adjusted net costs. In addition to its methodological contributions, this work also provides an open-source implementation of the proposed capacity expansion planning framework, referred to as \textit{OptBio}.} A Brazilian case study shows risk-neutral strategies favor sugar/ethanol but are vulnerable, whereas risk-averse strategies promote diversification. Sensitivity analyses indicate biomethane and hydrogen may become viable with favorable prices, while biochar boost productivity and profitability.

math.OC↗

arXiv2025

A General and Streamlined Differentiable Optimization Framework

Differentiating through constrained optimization problems is increasingly central to learning, control, and large-scale decision-making systems, yet practical integration remains challenging due to solver specialization and interface mismatches. This paper presents a general and streamlined framework-an updated DiffOpt.jl-that unifies modeling and differentiation within the Julia optimization stack. The framework computes forward - and reverse-mode solution and objective sensitivities for smooth, potentially nonconvex programs by differentiating the KKT system under standard regularity assumptions. A first-class, JuMP-native parameter-centric API allows users to declare named parameters and obtain derivatives directly with respect to them - even when a parameter appears in multiple constraints and objectives - eliminating brittle bookkeeping from coefficient-level interfaces. We illustrate these capabilities on convex and nonconvex models, including economic dispatch, mean-variance portfolio selection with conic risk constraints, and nonlinear robot inverse kinematics. Two companion studies further demonstrate impact at scale: gradient-based iterative methods for strategic bidding in energy markets and Sobolev-style training of end-to-end optimization proxies using solver-accurate sensitivities. Together, these results demonstrate that differentiable optimization can be deployed as a routine tool for experimentation, learning, calibration, and design-without deviating from standard JuMP modeling practices and while retaining access to a broad ecosystem of solvers.

cs.LG↗

arXiv2025

Assessing the Optimistic Bias in the Natural Inflow Forecasts: A Call for Model Monitoring in Brazil

Hydroelectricity accounted for roughly 61.4% of Brazil's total generation in 2024 and addressed most of the intermittency of wind and solar generation. Thus, inflow forecasting plays a critical role in the operation, planning, and market in this country, as well as in any other hydro-dependent power system. These forecasts influence generation schedules, reservoir management, and market pricing, shaping the dynamics of the entire electricity sector. The objective of this paper is to measure and present empirical evidence of a systematic optimistic bias in the official inflow forecast methodology, which is based on the PAR(p)-A model. Additionally, we discuss possible sources of this bias and recommend ways to mitigate it. By analyzing 14 years of historical data from the Brazilian system through rolling-window multistep (out-of-sample) forecasts, results indicate that the official forecast model exhibits statistically significant biases of 1.28, 3.83, 5.39, and 6.73 average GW for 1-, 6-, 12-, and 24-step-ahead forecasts in the Southeast subsystem, and 0.54, 1.66, 2.32, and 3.17 average GW in the Northeast subsystem. These findings uncover the limitations of current inflow forecasting methodologies used in Brazil and call for new governance and monitoring policies.

eess.SY↗

arXiv2025

Sobolev Training of End-to-End Optimization Proxies

Optimization proxies - machine learning models trained to approximate the solution mapping of parametric optimization problems in a single forward pass - offer dramatic reductions in inference time compared to traditional iterative solvers. This work investigates the integration of solver sensitivities into such end to end proxies via a Sobolev training paradigm and does so in two distinct settings: (i) fully supervised proxies, where exact solver outputs and sensitivities are available, and (ii) self supervised proxies that rely only on the objective and constraint structure of the underlying optimization problem. By augmenting the standard training loss with directional derivative information extracted from the solver, the proxy aligns both its predicted solutions and local derivatives with those of the optimizer. Under Lipschitz continuity assumptions on the true solution mapping, matching first order sensitivities is shown to yield uniform approximation error proportional to the training set covering radius. Empirically, different impacts are observed in each studied setting. On three large Alternating Current Optimal Power Flow benchmarks, supervised Sobolev training cuts mean squared error by up to 56 percent and the median worst case constraint violation by up to 400 percent while keeping the optimality gap below 0.22 percent. For a mean variance portfolio task trained without labeled solutions, self supervised Sobolev training halves the average optimality gap in the medium risk region (standard deviation above 10 percent of budget) and matches the baseline elsewhere. Together, these results highlight Sobolev training whether supervised or self supervised as a path to fast reliable surrogates for safety critical large scale optimization workloads.

cs.LG↗

arXiv2024

Long-term Hydrothermal Bid-based Market Simulator

Simulating long-term hydrothermal bid-based markets considering strategic agents is a challenging task. The representation of strategic agents considering intertemporal constraints within a stochastic framework brings additional complexity to the already difficult single-period bilevel, thus, non-convex, optimal bidding problem. Thus, we propose a simulation methodology that effectively addresses these challenges for large-scale hydrothermal power systems. We demonstrate the effectiveness of the framework through a case study with real data from the large-scale Brazilian power system. In the case studies, we show the effects of market concentration in power systems and how contracts can be used to mitigate them. In particular, we show how market power might affect the current setting in Brazil. The developed method can strongly benefit policymakers, market monitors, and market designers as simulations can be used to understand existing power systems and experiment with alternative designs.

math.OC↗

arXiv2024

Application-Driven Learning: A Closed-Loop Prediction and Optimization Approach Applied to Dynamic Reserves and Demand Forecasting

Forecasting and decision-making are generally modeled as two sequential steps with no feedback, following an open-loop approach. In this paper, we present application-driven learning, a new closed-loop framework in which the processes of forecasting and decision-making are merged and co-optimized through a bilevel optimization problem. We present our methodology in a general format and prove that the solution converges to the best estimator in terms of the expected cost of the selected application. Then, we propose two solution methods: an exact method based on the KKT conditions of the second-level problem and a scalable heuristic approach suitable for decomposition methods. The proposed methodology is applied to the relevant problem of defining dynamic reserve requirements and conditional load forecasts, offering an alternative approach to current ad hoc procedures implemented in industry practices. We benchmark our methodology with the standard sequential least-squares forecast and dispatch planning process. We apply the proposed methodology to an illustrative system and to a wide range of instances, from dozens of buses to large-scale realistic systems with thousands of buses. Our results show that the proposed methodology is scalable and yields consistently better performance than the standard open-loop approach.

math.OC↗

arXiv2024

Regularization and Optimization in Model-Based Clustering

Due to their conceptual simplicity, k-means algorithm variants have been extensively used for unsupervised cluster analysis. However, one main shortcoming of these algorithms is that they essentially fit a mixture of identical spherical Gaussians to data that vastly deviates from such a distribution. In comparison, general Gaussian Mixture Models (GMMs) can fit richer structures but require estimating a quadratic number of parameters per cluster to represent the covariance matrices. This poses two main issues: (i) the underlying optimization problems are challenging due to their larger number of local minima, and (ii) their solutions can overfit the data. In this work, we design search strategies that circumvent both issues. We develop more effective optimization algorithms for general GMMs, and we combine these algorithms with regularization strategies that avoid overfitting. Through extensive computational analyses, we observe that optimization or regularization in isolation does not substantially improve cluster recovery. However, combining these techniques permits a completely new level of performance previously unachieved by k-means algorithm variants, unraveling vastly different cluster structures. These results shed new light on the current status quo between GMM and k-means methods and suggest the more frequent use of general GMMs for data exploration. To facilitate such applications, we provide open-source code as well as Julia packages (UnsupervisedClustering.jl and RegularizedCovarianceMatrices.jl) implementing the proposed techniques.

cs.LG↗

arXiv2023

Flexible Differentiable Optimization via Model Transformations

We introduce DiffOpt.jl, a Julia library to differentiate through the solution of optimization problems with respect to arbitrary parameters present in the objective and/or constraints. The library builds upon MathOptInterface, thus leveraging the rich ecosystem of solvers and composing well with modeling languages like JuMP. DiffOpt offers both forward and reverse differentiation modes, enabling multiple use cases from hyperparameter optimization to backpropagation and sensitivity analysis, bridging constrained optimization with end-to-end differentiable programming. DiffOpt is built on two known rules for differentiating quadratic programming and conic programming standard forms. However, thanks ability to differentiate through model transformation, the user is not limited to these forms and can differentiate with respect to the parameters of any model that can be reformulated into these standard forms. This notably includes programs mixing affine conic constraints and convex quadratic constraints or objective function.

cs.LG↗

arXiv2023

A Multicut Approach to Compute Upper Bounds for Risk-Averse SDDP

Stochastic Dual Dynamic Programming (SDDP) is a widely used and fundamental algorithm for solving multistage stochastic optimization problems. Although SDDP has been frequently applied to solve risk-averse models with the Conditional Value-at-Risk (CVaR), it is known that the estimation of upper bounds is a methodological challenge, and many methods are computationally intensive. In practice, this leaves most SDDP implementations without a practical and clear stopping criterion. In this paper, we propose using the information already contained in a multicut formulation of SDDP to solve this problem with a simple and computationally efficient methodology. The multicut version of SDDP, in contrast with the typical average cut, preserves the information about which scenarios give rise to the worst costs, thus contributing to the CVaR value. We use this fact to modify the standard sampling method on the forward step so the average of multiple paths approximates the nested CVaR cost. We highlight that minimal changes are required in the SDDP algorithm and there is no additional computational burden for a fixed number of iterations. We present multiple case studies to empirically demonstrate the effectiveness of the method. First, we use a small hydrothermal dispatch test case, in which we can write the deterministic equivalent of the entire scenario tree to show that the method perfectly computes the correct objective values. Then, we present results using a standard approximation of the Brazilian operation problem and a real hydrothermal dispatch case based on data from Colombia. Our numerical experiments showed that this method consistently calculates upper bounds higher than lower bounds for those risk-averse problems and that lower bounds are improved thanks to the better exploration of the scenarios tree.

math.OC↗

arXiv2023

Comparing BilevelJuMP.jl Formulations: Support Vector Regression Hyperparameter Tuning

In this technical report, we compare multiple reformulation techniques and solvers that can be used with the Julia package BilevelJuMP. We focus on the special case of Hyperparameter Tuning for Support Vector Regression. We describe a bilevel model for the problem in question. Then we present code for generating data and models that solve the problem. Finally, we present results and a brief analysis.

math.OC↗

arXiv2023

JuMP 1.0: Recent improvements to a modeling language for mathematical optimization

JuMP is an algebraic modeling language embedded in the Julia programming language. JuMP allows users to model optimization problems of a variety of kinds, including linear programming, integer programming, conic optimization, semidefinite programming, and nonlinear programming, and handles the low-level details of communicating with solvers. After nearly 10 years in development, JuMP 1.0 was released in March, 2022. In this short communication, we highlight the improvements to JuMP from recent releases up to and including 1.0.

cs.PL↗

arXiv2022

BilevelJuMP.jl: Modeling and Solving Bilevel Optimization in Julia

In this paper we present BilevelJuMP, a new Julia package to support bilevel optimization within the JuMP framework. The package is a Julia library that enables the user to describe both upper and lower-level optimization problems using the JuMP algebraic syntax. Due to the generality and flexibility our library inherits from JuMP's syntax, our package allows users to model bilevel optimization problems with conic constraints in the lower level and all JuMP supported constraints in the upper level (Conic, Quadratic, Non-Linear, Integer, etc.). Moreover, the user-defined problem can be subsequently solved by various techniques relying on mathematical program with equilibrium constraints (MPEC) reformulations. Manipulations on the original problem data are possible due to MathOptInterface.jl's structures and Dualization.jl features. Hence, the proposed package allows quickly model, deploy, and thereby experiment bilevel models based on off-the-shelf mixed integer linear programming and nonlinear solvers.

math.OC↗

arXiv2021

An Integrated Progressive Hedging and Benders Decomposition with Multiple Master Method to Solve the Brazilian Generation Expansion Problem

This paper exploits the decomposition structure of the large-scale hydrothermal generation expansion planning problem with an integrated modified Benders Decomposition and Progressive Hedging approach. We consider detailed and realistic data from the Brazilian power system to represent hourly chronological constraints based on typical days per month and year. Also, we represent the multistage stochastic nature of the optimal hydrothermal operational policy through co-optimized linear decision rules for individual reservoirs. Therefore, we ensure investment decisions compatible with a nonanticipative (implementable) operational policy. To solve the large-scale optimization problem, we propose an improved Benders Decomposition method with multiple instances of the master problem, each of which strengthened by primal cuts and new Benders cuts generated by each master's trial solution. Additionally, our new approach allows using Progressive Hedging penalization terms for accelerating the convergence of the method. We show that our method is 60\% faster than the benchmark. Finally, the consideration of a nonanticipative operational policy can save 7.64\% of the total cost (16.18\% of the investment costs) and significantly improve spot price profiles.

math.OC↗

arXiv2021

Assessing the Cost of Network Simplifications in Long-Term Hydrothermal Dispatch Planning Models

The sustainable utilization of hydro energy relies on accurate estimates of the opportunity cost of the water. This value is calculated through long-term hydrothermal dispatch problems (LTHDP), and the recent literature has raised awareness about the consequences of modeling simplifications in these problems. The inaccurate representation of Kirchhoff's voltage law under the premise of a DC power flow is an example. Under a non-linear AC model, however, the LTHDP becomes intractable, and the literature lacks an accurate evaluation method of different modeling alternatives. In this paper, we extend the state-of-the-art cost-assessment framework of network approximations for LTHDP and bring relevant and practical new insights. First, we increase the quality of the assessment by using an AC power flow to simulate and compare the performance of five policies based on different network approximations. Second, we find that the tightest network relaxation (based on semidefinite programming) is not the one exhibiting the best performance. Results show that the DC {power flow} with quadratic losses approximation exhibits the lowest expected cost and inconsistency gaps. Finally, its computational burden is lower than that exhibited by the semidefinite relaxation, whereas market distortions are significantly reduced in comparison to previously published benchmarks based on DC power flow.

math.OC↗