arXiv subjects

Vyacheslav Kungurtsev

Publications and source records attributed to Vyacheslav Kungurtsev.

At least 55 records · Page 3Linked to original sources

arXiv2024

Learning Dynamic Bayesian Networks from Data: Foundations, First Principles and Numerical Comparisons

In this paper, we present a guide to the foundations of learning Dynamic Bayesian Networks (DBNs) from data in the form of multiple samples of trajectories for some length of time. We present the formalism for a generic as well as a set of common types of DBNs for particular variable distributions. We present the analytical form of the models, with a comprehensive discussion on the interdependence between structure and weights in a DBN model and their implications for learning. Next, we give a broad overview of learning methods and describe and categorize them based on the most important statistical features, and how they treat the interplay between learning structure and weights. We give the analytical form of the likelihood and Bayesian score functions, emphasizing the distinction from the static case. We discuss functions used in optimization to enforce structural requirements. We briefly discuss more complex extensions and representations. Finally we present a set of comparisons in different settings for various distinct but representative algorithms across the variants.

cs.LG↗

arXiv2024

Empirical Bayes for Dynamic Bayesian Networks Using Generalized Variational Inference

In this work, we demonstrate the Empirical Bayes approach to learning a Dynamic Bayesian Network. By starting with several point estimates of structure and weights, we can use a data-driven prior to subsequently obtain a model to quantify uncertainty. This approach uses a recent development of Generalized Variational Inference, and indicates the potential of sampling the uncertainty of a mixture of DAG structures as well as a parameter posterior.

cs.LG↗

arXiv2024

Optimal Control of Two-Phase Membrane Problem

We consider an optimal control problem where the state is governed by a free boundary problem called the two-phase membrane problem and the control appears in the coefficients of the characteristic function of the positivity and negativity parts of the solution. Our investigation focuses on various properties associated with the control-to-state map. Due to the non-differentiability of this map, we regularize the state equation. The existence, uniqueness, and characterization of the optimal pairs are established.

math.OC↗

arXiv2024

Stochastic Langevin Differential Inclusions with Applications to Machine Learning

Stochastic differential equations of Langevin-diffusion form have received significant attention, thanks to their foundational role in both Bayesian sampling algorithms and optimization in machine learning. In the latter, they serve as a conceptual model of the stochastic gradient flow in training over-parameterized models. However, the literature typically assumes smoothness of the potential, whose gradient is the drift term. Nevertheless, there are many problems for which the potential function is not continuously differentiable, and hence the drift is not Lipschitz continuous everywhere. This is exemplified by robust losses and Rectified Linear Units in regression problems. In this paper, we show some foundational results regarding the flow and asymptotic properties of Langevin-type Stochastic Differential Inclusions under assumptions appropriate to the machine-learning settings. In particular, we show strong existence of the solution, as well as an asymptotic minimization of the canonical free-energy functional.

math.OC↗

arXiv2024

Robust Quantum Gate Complexity: Foundations

Optimal control of closed quantum systems is a well studied geometrically elegant set of computational theory and techniques that have proven pivotal in the implementation and understanding of quantum computers. The design of a circuit itself corresponds to an optimal control problem of choosing the appropriate set of gates (which appear as control operands) in order to steer a qubit from an initial, easily prepared state, to one that is informative to the user in some sense, for e.g., an oracle whose evaluation is part of the circuit. However, contemporary devices are known to be noisy, and it is not certain that a circuit will behave as intended. Yet, although the computational tools exist in broader optimal control theory, robustness of adequate operation of a quantum control system with respect to uncertainty and errors has not yet been broadly studied in the literature. In this paper, we propose a new approach inspired by the closed quantum optimal control and its connection to geometric interpretations. To this end, we present the appropriate problem definitions of robustness in the context of quantum control, focusing on its broader implications for gate complexity.

quant-ph↗

arXiv2024

Efficient Dataset Distillation via Minimax Diffusion

Dataset distillation reduces the storage and computational consumption of training a network by generating a small surrogate dataset that encapsulates rich information of the original large-scale one. However, previous distillation methods heavily rely on the sample-wise iterative optimization scheme. As the images-per-class (IPC) setting or image resolution grows larger, the necessary computation will demand overwhelming time and resources. In this work, we intend to incorporate generative diffusion techniques for computing the surrogate dataset. Observing that key factors for constructing an effective surrogate dataset are representativeness and diversity, we design additional minimax criteria in the generative training to enhance these facets for the generated images of diffusion models. We present a theoretical model of the process as hierarchical diffusion control demonstrating the flexibility of the diffusion process to target these criteria without jeopardizing the faithfulness of the sample to the desired distribution. The proposed method achieves state-of-the-art validation performance while demanding much less computational resources. Under the 100-IPC setting on ImageWoof, our method requires less than one-twentieth the distillation time of previous methods, yet yields even better performance. Source code and generated data are available in https://github.com/vimar-gu/MinimaxDiffusion.

cs.CV↗

arXiv2024

The stochastic Ravine accelerated gradient method with general extrapolation coefficients

In a real Hilbert space domain setting, we study the convergence properties of the stochastic Ravine accelerated gradient method for convex differentiable optimization. We consider the general form of this algorithm where the extrapolation coefficients can vary with each iteration, and where the evaluation of the gradient is subject to random errors. This general treatment models a breadth of practical algorithms and numerical implementations. We show that, under a proper tuning of the extrapolation parameters, and when the error variance associated with the gradient evaluations or the step-size sequences vanish sufficiently fast, the Ravine method provides fast convergence of the values both in expectation and almost surely. We also improve the convergence rates from O(.) to o(.). Moreover, we show almost sure summability property of the gradients, which implies the fast convergence of the gradients towards zero. This property reflects the fact that the high-resolution ODE of the Ravine method includes a Hessian-driven damping term. When the space is also separable, our analysis allows also to establish almost sure weak convergence of the sequence of iterates provided by the algorithm. We finally specialize the analysis to consider different parameter choices, including vanishing and constant (heavy ball method with friction) damping parameter, and present a comprehensive landscape of the tradeoffs in speed and accuracy associated with these parameter choices and statistical properties on the sequence of errors in the gradient computations. We provide a thorough discussion of the similarities and differences with the Nesterov accelerated gradient which satisfies similar asymptotic convergence rates.

math.OC↗

arXiv2024

Spectral Methods for Quantum Optimal Control: Artificial Boundary Conditions

The problem of quantum state preparation is one of the main challenges in achieving the quantum advantage. Furthermore, classically, for multi-level problems, our ability to solve the corresponding quantum optimal control problems is rather limited. The ability of the latter to feed into the former may result in significant progress in quantum computing. To address this challenge, we propose a formulation of quantum optimal control that makes use of artificial boundary conditions for the Schrödinger equation in combination with spectral methods. The resulting formulations are well suited for investigating periodic potentials and lend themselves to direct numerical treatment using conventional methods for bounded domains.

quant-ph↗

arXiv2024

A Stochastic-Gradient-based Interior-Point Algorithm for Solving Smooth Bound-Constrained Optimization Problems

A stochastic-gradient-based interior-point algorithm for minimizing a continuously differentiable objective function (that may be nonconvex) subject to bound constraints is presented, analyzed, and demonstrated through experimental results. The algorithm is unique from other interior-point methods for solving smooth nonconvex optimization problems since the search directions are computed using stochastic gradient estimates. It is also unique in its use of inner neighborhoods of the feasible region -- defined by a positive and vanishing neighborhood-parameter sequence -- in which the iterates are forced to remain. It is shown that with a careful balance between the barrier, step-size, and neighborhood sequences, the proposed algorithm satisfies convergence guarantees in both deterministic and stochastic settings. The results of numerical experiments show that in both settings the algorithm can outperform projection-based methods.

math.OC↗

arXiv2024

The Effects of Transmission-Rights Pricing on Multi-Stage Electricity Markets

Cross-border transmission infrastructure is pivotal in balancing modern power systems, but requires fair allocation of cross-border transmission capacity, possibly via fair pricing thereof. This requirement can be implemented using multi-stage market mechanisms for Physical Transmission Rights (PTRs). We analyse the related dynamics, and show prisoner's dilemma arises. Understanding these dynamics enables the development of novel market-settlement mechanisms to enhance market efficiency and incentivize renewable energy use.

math.OC↗

arXiv2024

Time-Varying Semidefinite Programming: Path Following a Burer-Monteiro Factorization

We present an online algorithm for time-varying semidefinite programs (TV-SDPs), based on the tracking of the solution trajectory of a low-rank matrix factorization, also known as the Burer-Monteiro factorization, in a path-following procedure. There, a predictor-corrector algorithm solves a sequence of linearized systems. This requires the introduction of a horizontal space constraint to ensure the local injectivity of the low-rank factorization. The method produces a sequence of approximate solutions for the original TV-SDP problem, for which we show that they stay close to the optimal solution path if properly initialized. Numerical experiments for a time-varying max-cut SDP relaxation demonstrate the computational advantages of the proposed method for tracking TV-SDPs in terms of runtime compared to off-the-shelf interior point methods.

math.OC↗

arXiv2023

Scheduling a Multi-Product Pipeline: A Discretized MILP Formulation

Multi-product pipelines are a highly efficient means of transporting liquids. Traditionally used to transport petroleum, its products and derivatives, they are now being repurposed to transport liquified natural gas admixed with hydrogen of various colors. We propose a novel mixed-integer linear programming (MILP) formulation, which optimizes efficiency while satisfying a wide range of real-world constraints developed to meet the needs of the Czech national pipeline operator CEPRO. We provide tests on well-known synthetic (path-graph) networks and demonstrate the formulation's scaling properties using open-source and commercial MILP solvers.

math.OC↗

arXiv2023

Unlocking the Potential of Federated Learning: The Symphony of Dataset Distillation via Deep Generative Latents

Data heterogeneity presents significant challenges for federated learning (FL). Recently, dataset distillation techniques have been introduced, and performed at the client level, to attempt to mitigate some of these challenges. In this paper, we propose a highly efficient FL dataset distillation framework on the server side, significantly reducing both the computational and communication demands on local devices while enhancing the clients' privacy. Unlike previous strategies that perform dataset distillation on local devices and upload synthetic data to the server, our technique enables the server to leverage prior knowledge from pre-trained deep generative models to synthesize essential data representations from a heterogeneous model architecture. This process allows local devices to train smaller surrogate models while enabling the training of a larger global model on the server, effectively minimizing resource utilization. We substantiate our claim with a theoretical analysis, demonstrating the asymptotic resemblance of the process to the hypothetical ideal of completely centralized training on a heterogeneous dataset. Empirical evidence from our comprehensive experiments indicates our method's superiority, delivering an accuracy enhancement of up to 40% over non-dataset-distillation techniques in highly heterogeneous FL contexts, and surpassing existing dataset-distillation methods by 18%. In addition to the high accuracy, our framework converges faster than the baselines because rather than the server trains on several sets of heterogeneous data distributions, it trains on a multi-modal distribution. Our code is available at https://github.com/FedDG23/FedDG-main.git

cs.LG↗

arXiv2023

Trilevel and Multilevel Optimization using Monotone Operator Theory

We consider rather a general class of multi-level optimization problems, where a convex objective function is to be minimized subject to constraints of optimality of nested convex optimization problems. As a special case, we consider a trilevel optimization problem, where the objective of the two lower layers consists of a sum of a smooth and a non-smooth term.~Based on fixed-point theory and related arguments, we present a natural first-order algorithm and analyze its convergence and rates of convergence in several regimes of parameters.

math.OC↗

arXiv2023

Parametric Semidefinite Programming: Geometry of the Trajectory of Solutions

In many applications, solutions of convex optimization problems are updated on-line, as functions of time. In this paper, we consider parametric semidefinite programs, which are linear optimization problems in the semidefinite cone whose coefficients (input data) depend on a time parameter. We are interested in the geometry of the solution (output data) trajectory, defined as the set of solutions depending on the parameter. We propose an exhaustive description of the geometry of the solution trajectory. As our main result, we show that only six distinct behaviors can be observed at a neighborhood of a given point along the solution trajectory. Each possible behavior is then illustrated by an example.

math.OC↗

arXiv2023

Inexact Direct-Search Methods for Bilevel Optimization Problems

In this work, we introduce new direct search schemes for the solution of bilevel optimization (BO) problems. Our methods rely on a fixed accuracy black box oracle for the lower-level problem, and deal both with smooth and potentially nonsmooth true objectives. We thus analyze for the first time in the literature direct search schemes in these settings, giving convergence guarantees to approximate stationary points, as well as complexity bounds in the smooth case. We also propose the first adaptation of mesh adaptive direct search schemes for BO. Some preliminary numerical results on a standard set of bilevel optimization problems show the effectiveness of our new approaches.

math.OC↗

arXiv2023

Hybrid Methods in Polynomial Optimisation

The Moment/Sum-of-squares hierarchy provides a way to compute the global minimizers of polynomial optimization problems (POP), at the cost of solving a sequence of increasingly large semidefinite programs (SDPs). We consider large-scale POPs, for which interior-point methods are no longer able to solve the resulting SDPs. We propose an algorithm that combines a first-order method for solving the SDP relaxation, and a second-order method on a non-convex problem obtained from the POP. The switch from the first to the second-order method is based on a quantitative criterion, whose satisfaction ensures that Newton's method converges quadratically from its first iteration. This criterion leverages the point-estimation theory of Smale and the active-set identification. We illustrate the methodology to obtain global minimizers of large-scale optimal power flow problems.

math.OC↗

arXiv2023

Quantum Solutions to the Privacy vs. Utility Tradeoff

In this work, we propose a novel architecture (and several variants thereof) based on quantum cryptographic primitives with provable privacy and security guarantees regarding membership inference attacks on generative models. Our architecture can be used on top of any existing classical or quantum generative models. We argue that the use of quantum gates associated with unitary operators provides inherent advantages compared to standard Differential Privacy based techniques for establishing guaranteed security from all polynomial-time adversaries.

quant-ph↗