arXiv subjects

Sebastian Sager

Publications and source records attributed to Sebastian Sager.

At least 19 recordsLinked to original sources

arXiv2026

Tucker Attention: A generalization of approximate attention mechanisms

The pursuit of reducing the memory footprint of the self-attention mechanism in multi-headed self attention (MHA) spawned a rich portfolio of methods, e.g., group-query attention (GQA) and multi-head latent attention (MLA). The methods leverage specialized low-rank factorizations across embedding dimensions or attention heads. From the point of view of classical low-rank approximation, these methods are unconventional and raise questions of which objects they really approximate and how to interpret the low-rank behavior of the resulting representations. To answer these questions, this work proposes a generalized view on the weight objects in the self-attention layer and a factorization strategy, which allows us to construct a parameter efficient scheme, called Tucker Attention. Tucker Attention requires an order of magnitude fewer parameters for comparable validation metrics, compared to GQA and MLA, as evaluated in LLM and ViT test cases. Additionally, Tucker Attention~encompasses GQA, MLA, MHA as special cases and is fully compatible with flash-attention and rotary position embeddings (RoPE). This generalization strategy yields insights of the actual ranks achieved by MHA, GQA, and MLA, and further enables simplifications for MLA.

cs.LG↗

arXiv2026

Adjoint-Compatible Surrogates of the Expected Information Gain for Optimal Experimental Design

We consider optimal experimental design for parameter estimation in dynamical systems governed by controlled ordinary differential equations. In such problems, Fisher-based criteria are attractive because they lead to time-additive objectives compatible with adjoint-based optimal control, but they remain intrinsically local and may perform poorly under strong nonlinearities or non-Gaussian prior uncertainty. By contrast, the expected information gain (EIG) provides a principled Bayesian objective, yet it is typically too costly to evaluate and does not naturally admit an adjoint-compatible formulation. In this work, we introduce adjoint-compatible surrogates of the EIG based on an exact chain-rule decomposition and tractable approximations of the posterior distribution of the unknown parameter. This leads to two surrogate criteria: an instantaneous surrogate, obtained by replacing the posterior with the prior, and a Gaussian tilting surrogate, obtained by reweighting the prior through a design-driven quadratic information factor. We establish theoretical properties of these surrogates, including exactness of the Gaussian tilting surrogate in the linear-Gaussian setting, and illustrate their behavior on benchmark controlled dynamical systems. The results show that the proposed surrogates remain competitive in nearly Gaussian regimes and provide clearer benefits over Fisher-based designs when prior uncertainty is non-Gaussian or multimodal.

math.OC↗

arXiv2025

Mitigating Subject Dependency in EEG Decoding with Subject-Specific Low-Rank Adapters

Subject-specific distribution shifts represent a fundamental obstacle to developing foundation models for brain decoding. We propose the Subject-Specific Low-Rank Adapter (SuLoRA), a drop-in replacement for standard linear or convolutional layers that captures inter-subject variability by decomposing weights into a shared, subject-invariant component and a lightweight, low-rank correction unique to each subject. This explicit separation enables existing architectures to become robust to subject shifts without architectural redesign. We evaluate SuLoRA on MEG speech perception and EEG motor imagery tasks across CNN and transformer architectures. In the speech decoding task, SuLoRA exceeds the baseline performance with half of the parameters. On motor imagery dataset, SuLoRA outperforms both subject-agnostic models and independently trained subject-specific models. SuLoRA offers a practical path towards effective cross-subject foundation models for brain signal applications.

cs.LG↗

arXiv2025

Superstructure Optimization with Embedded Neural Networks for Sustainable Aviation Fuel Production

This study presents a multi-objective optimization framework for sustainable aviation fuel (SAF) production, integrating artificial neural networks (ANNs) within a mixed-integer quadratically constrained programming (MIQCP) formulation. By embedding data-driven surrogate models into the mathematical optimization structure, the proposed methodology addresses key limitations of conventional superstructure-based approaches, enabling simultaneous optimization of discrete process choices and continuous operating parameters. The framework captures variable input and output stream compositions, facilitating the joint optimization of target product composition and system design. Application to Fischer-Tropsch (FT) kerosene production demonstrates that cost-minimizing configurations under unconstrained CO2 emissions are dominated by the fossil-based autothermal reforming (ATR) route. Imposing carbon emission constraints necessitates the integration of biomass gasification and direct air capture coupled with carbon sequestration (DAC-CS), resulting in substantially reduced net emissions but higher production costs. At the zero-emission limit, hybrid configurations combining ATR and biomass gasification achieve the lowest costs (~2.38 \$/kg-kerosene), followed closely by biomass gasification-only (~2.43 \$/kg), both of which outperform the ATR-only pathway with DAC-CS (~2.65 \$/kg). In contrast, DAC-only systems relying exclusively on atmospheric CO2 and water electrolysis are prohibitively expensive (~10.8 \$/kg). The results highlight the critical role of the embedded ANNs: optimal process conditions, such as FT reactor pressure and gasification temperature, adapt to changing circumstances, consistently outperforming fixed setups and achieving up to 20% cost savings.

cs.CE↗

arXiv2025

An analysis of optimization problems involving ReLU neural networks

Solving mixed-integer optimization problems with embedded neural networks with ReLU activation functions is challenging. Big-M coefficients that arise in relaxing binary decisions related to these functions grow exponentially with the number of layers. We survey and propose different approaches to analyze and improve the run time behavior of mixed-integer programming solvers in this context. Among them are clipped variants and regularization techniques applied during training as well as optimization-based bound tightening and a novel scaling for given ReLU networks. We numerically compare these approaches for three benchmark problems from the literature. We use the number of linear regions, the percentage of stable neurons, and overall computational effort as indicators. As a major takeaway we observe and quantify a trade-off between the often desired redundancy of neural network models versus the computational costs for solving related optimization problems.

math.OC↗

arXiv2024

Optimal Experimental Design for Universal Differential Equations

Complex dynamic systems are typically either modeled using expert knowledge in the form of differential equations or via data-driven universal approximation models such as artificial neural networks (ANN). While the first approach has advantages with respect to interpretability, transparency, data-efficiency, and extrapolation, the second approach is able to learn completely unknown functional relations from data and may result in models that can be evaluated more efficiently. To combine the complementary advantages, universal differential equations (UDE) have been suggested. They replace unknown terms in the differential equations with ANN. Such hybrid models allow to both encode prior domain knowledge, such as first principles, and to learn unknown mechanisms from data. Often, data for the training of UDE can only be obtained via costly experiments. We consider optimal experimental design (OED) for planning of experiments and generating data needed to train UDE. The number of weights in the embedded ANN usually leads to an overfitting of the regression problem. To make the OED problem tractable for optimization, we propose and compare dimension reduction methods that are based on lumping of weights and singular value decomposition of the Fisher information matrix (FIM), respectively. They result in lower-dimensional variational differential equations, which are easier to solve and yield regular FIM. Our numerical results showcase the advantages of OED for UDE, such as increased data-efficiency and better extrapolation properties.

math.OC↗

arXiv2024

Generalized Inverse Optimal Control and its Application in Biology

Living organisms exhibit remarkable adaptations across all scales, from molecules to ecosystems. We believe that many of these adaptations correspond to optimal solutions driven by evolution, training, and underlying physical and chemical laws and constraints. While some argue against such optimality principles due to their potential ambiguity, we propose generalized inverse optimal control to infer them directly from data. This novel approach incorporates multi-criteria optimality, nestedness of objective functions on different scales, the presence of active constraints, the possibility of switches of optimality principles during the observed time horizon, maximization of robustness, and minimization of time as important special cases, as well as uncertainties involved with the mathematical modeling of biological systems. This data-driven approach ensures that optimality principles are not merely theoretical constructs but are firmly rooted in experimental observations. Furthermore, the inferred principles can be used in forward optimal control to predict and manipulate biological systems, with possible applications in bio-medicine, biotechnology, and agriculture. As discussed and illustrated, the well-posed problem formulation and the inference are challenging and require a substantial interdisciplinary effort in the development of theory and robust numerical methods.

q-bio.QM↗

arXiv2024

A Sequential Benders-based Mixed-Integer Quadratic Programming Algorithm and Its Implementation in the CAMINO Toolbox

Sequential quadratic programming and sequential convex programming efficiently solve nonlinear programs (NLPs) by linearizing inner nonlinearities while preserving the outer convex structure. This paper introduces a sequential mixed-integer quadratic programming (MIQP) algorithm to extend this methodology to mixed-integer nonlinear problems (MINLPs), leveraging the efficiency of modern MIQP solvers. The algorithm uses a three-step iterative process. First, the MINLP is linearized around the current iterate. Second, an MIQP is formulated and solved, with its feasible region restricted to a specific area around the linearization point. This region is defined using objective values and derivatives from previous iterations, drawing on concepts from generalized Benders' decomposition. Third, the integer variables from the MIQP solution are fixed, and an NLP involving only the continuous variables is solved. The best solution among all iterates becomes the linearization point for the next iteration. A fallback strategy based on a mixed-integer linear program (MILP) is used when MIQP progress stalls. This guarantees convergence to the global optimal solution for convex MINLPs. For nonconvex problems, the algorithm functions as a heuristic without global optimality guarantees. Numerical experiments show its competitiveness with other MINLP solvers on benchmark problems. In addition, the algorithm was successfully applied to mixed-integer optimal control problems, demonstrating its effectiveness in handling challenging nonlinear equality constraints. The proposed algorithm is publicly available at https://github.com/minlp-toolbox/CAMINO with the name s-b-miqp.

math.OC↗

arXiv2024

Sparse convex relaxations in polynomial optimization

We present a novel, general, and unifying point of view on sparse approaches to polynomial optimization. Solving polynomial optimization problems to global optimality is a ubiquitous challenge in many areas of science and engineering. Historically, different approaches on how to solve nonconvex polynomial optimization problems based on convex relaxations have been developed in different scientific communities. Here, we introduce the concept of monomial patterns. A pattern determines what monomials are to be linked by convex constraints in a convex relaxation of a polynomial optimization problem. This concept helps to understand existing approaches from different schools of thought, to develop novel relaxation schemes, and to derive a flexible duality theory, which can be specialized to many concrete situations that have been considered in the literature. We unify different approaches to polynomial optimization including polyhedral approximations, dense semidefinite relaxations, SONC, SAGE, and TSSOS in a self-contained exposition. We also carry out computational experiments to demonstrate the practical advantages of a flexible usage of pattern-based sparse relaxations of polynomial optimization problems.

math.OC↗

arXiv2024

Refined TSSOS

The moment-sum of squares hierarchy by Lasserre has become an established technique for solving polynomial optimization problems. It provides a monotonically increasing series of tight bounds, but has well-known scalability limitations. For structured optimization problems, the term-sparsity SOS (TSSOS) approach scales much better due to block-diagonal matrices, obtained by completing the connected components of adjacency graphs. This block structure can be exploited by semidefinite programming solvers, for which the overall runtime then depends heavily on the size of the largest block. However, already the first step of the TSSOS hierarchy may result in large diagonal blocks. We suggest a new approach that refines TSSOS iterations using combinatorial optimization and results in block-diagonal matrices with reduced maximum block sizes. Numerical results on a benchmark library show the large potential for computational speedup for unconstrained and constrained polynomial optimization problems, while obtaining almost identical bounds in comparison to established methods.

math.OC↗

arXiv2022

On Convergence of Binary Trust-Region Steepest Descent

Binary trust-region steepest descent (BTR) and combinatorial integral approximation (CIA) are two recently investigated approaches for the solution of optimization problems with distributed binary-/discrete-valued variables (control functions). We show improved convergence results for BTR by imposing a compactness assumption that is similar to the convergence theory of CIA. As a corollary we conclude that BTR also constitutes a descent algorithm on the continuous relaxation and its iterates converge weakly-$^*$ to stationary points of the latter. We provide computational results that validate our findings. In addition, we observe a regularizing effect of BTR, which we explore by means of a hybridization of CIA and BTR.

math.OC↗

arXiv2021

Liftings of polynomial systems decreasing the mixed volume

The BKK theorem states that the mixed volume of the Newton polytopes of a system of polynomial equations upper bounds the number of isolated torus solutions of the system. Homotopy continuation solvers make use of this fact to pick efficient start systems. For systems where the mixed volume bound is not attained, such methods are still tracking more paths than necessary. We propose a strategy of improvement by lifting a system to an equivalent system with a strictly lower mixed volume at the expense of more variables. We illustrate this idea providing lifting constructions for arbitrary bivariate systems and certain dense-enough systems.

math.AG↗

arXiv2020

A finite element / neural network framework for modeling suspensions of non-spherical particles. Concepts and medical applications

An accurate prediction of the translational and rotational motion of particles suspended in a fluid is only possible if a complete set of correlations for the force coefficients of fluid-particle interaction is known. The present study is thus devoted to the derivation and validation of a new framework to determine the drag, lift, rotational and pitching torque coefficients for different non-spherical particles in a fluid flow. The motivation for the study arises from medical applications, where particles may have an arbitrary and complex shape. Here, it is usually not possible to derive accurate analytical models for predicting the different hydrodynamic forces. However, considering for example the various components of blood, their shape takes an important role in controlling several body functions such as control of blood viscosity or coagulation. Therefore, the presented model is designed to be applicable to a broad range of shapes. Another important feature of the suspensions occurring in medical and biological applications is the high number of particles. The modelling approach we propose can be efficiently used for simulations of solid-liquid suspensions with numerous particles. Based on resolved numerical simulations of prototypical particles we generate data to train a neural network which allows us to quickly estimate the hydrodynamic forces experienced by a specific particle immersed in a fluid.

physics.flu-dyn↗

arXiv2019

Model-based simulation of maintenance therapy of childhood acute lymphoblastic leukemia

Acute lymphoblastic leukemia is the most common malignancy in childhood. Successful treatment requires initial high-intensity chemotherapy, followed by low-intensity oral maintenance therapy with oral 6-mercaptopurine (6MP) and methotrexate (MTX) until 2-3 years after disease onset. However, intra- and interindividual variability in the pharmacokinetics (PK) and pharmacodynamics (PD) of 6MP and MTX make it challenging to balance the desired antileukemic effects with undesired excessive myelosuppression during maintenance therapy. A model to simulate the dynamics of different cell types, especially neutrophils, would be a valuable contribution to improving treatment protocols (6MP and MTX dosing regimens) and a further step to understanding the heterogeneity in treatment efficacy and toxicity. We applied and modified a recently developed semi-mechanistic PK/PD model to neutrophils and analyzed their behavior using a nonlinear mixed-effects modeling approach and clinical data obtained from 116 patients. The PK model of 6MP influenced the accuracy of absolute neutrophil count (ANC) predictions, whereas the PD effect of MTX did not. Predictions based on ANC were more accurate than those based on white blood cell counts. Using the new cross-validated mathematical model, simulations of different treatment protocols showed a linear dose-effect relationship and reduced ANC variability for constant dosages. Advanced modeling allows the identification of optimized control criteria and the weighting of specific influencing factors for protocol design and individually adapted therapy to exploit the optimal effect of maintenance therapy on survival.

q-bio.QM↗

arXiv2019

Model-based optimal AML consolidation treatment

Neutropenia is an adverse event commonly arising during intensive chemotherapy of acute myeloid leukemia (AML). It is often associated with infectious complications. Mathematical modeling, simulation, and optimization of the treatment process would be a valuable tool to support clinical decision making, potentially resulting in less severe side effects and deeper remissions. However, until now, there has been no validated mathematical model available to simulate the effect of chemotherapy treatment on white blood cell (WBC) counts and leukemic cells simultaneously. We developed a population pharmacokinetic/pharmacodynamic (PK/PD) model combining a myelosuppression model considering endogenous granulocyte-colony stimulating factor (G-CSF), a PK model for cytarabine (Ara-C), a subcutaneous absorption model for exogenous G-CSF, and a two-compartment model for leukemic blasts. This model was fitted to data of 44 AML patients during consolidation therapy with a novel Ara-C plus G-CSF schedule from a phase II controlled clinical trial. Additionally, we were able to optimize treatment schedules with respect to disease progression, WBC nadirs, and the amount of Ara-C and G-CSF. The developed PK/PD model provided good prediction accuracies and an interpretation of the interaction between WBCs, G-CSF, and blasts. For 14 patients (those with available bone marrow blast counts), we achieved a median 4.2-fold higher WBC count at nadir, which is the most critical time during consolidation therapy. The simulation results showed that relative bone marrow blast counts remained below the clinically important threshold of 5%, with a median of 60% reduction in Ara-C.

q-bio.QM↗

arXiv2019

The asymptotic error of chaos expansion approximations for stochastic differential equations

In this paper we present a numerical scheme for stochastic differential equations based upon the Wiener chaos expansion. The approximation of a square integrable stochastic differential equation is obtained by cutting off the infinite chaos expansion in chaos order and in number of basis elements. We derive an explicit upper bound for the $L^2$ approximation error associated with our method. The proofs are based upon an application of Malliavin calculus.

math.PR↗

arXiv2019

Convexification of box-constrained polynomial optimization problems via monomial patterns

Convexification is a core technique in global polynomial optimization. Currently, there are two main approaches competing in theory and practice: the approach of nonlinear programming and the approach based on positivity certificates from real algebra. The former are comparatively cheap from a computational point of view, but typically do not provide tight relaxations with respect to bounds for the original problem. The latter are typically computationally expensive, but do provide tight relaxations. We embed both kinds of approaches into a unified framework of monomial relaxations. We develop a convexification strategy that allows to trade off the quality of the bounds against computational expenses. Computational experiments show very encouraging results.

math.OC↗

arXiv2018

Solving Quadratic Programs to High Precision using Scaled Iterative Refinement

Quadratic optimization problems (QPs) are ubiquitous, and solution algorithms have matured to a reliable technology. However, the precision of solutions is usually limited due to the underlying floating-point operations. This may cause inconveniences when solutions are used for rigorous reasoning. We contribute on three levels to overcome this issue. First, we present a novel refinement algorithm to solve QPs to arbitrary precision. It iteratively solves refined QPs, assuming a floating-point QP solver oracle. We prove linear convergence of residuals and primal errors. Second, we provide an efficient implementation, based on SoPlex and qpOASES that is publicly available in source code. Third, we give precise reference solutions for the Maros and M\'esz\'aros benchmark library.

math.OC↗