arXiv subjects

Vladimir Kolmogorov

Publications and source records attributed to Vladimir Kolmogorov.

At least 37 records · Page 2Linked to original sources

arXiv2023

Parameter estimation for integer-valued Gibbs distributions

A central problem in computational statistics is to convert a procedure for sampling combinatorial from an objects into a procedure for counting those objects, and vice versa. Weconsider sampling problems coming from *Gibbs distributions*, which are probability distributions of the form $μ^Ω_β(ω) \propto e^{βH(ω)}$ for $β$ in an interval $[β_\min, β_\max]$ and $H( ω) \in \{0 \} \cup [1, n]$. The *partition function* is the normalization factor $Z(β)=\sum_{ω\inΩ}e^{βH(ω)}$. Two important parameters are the log partition ratio $q = \log \tfrac{Z(β_\max)}{Z(β_\min)}$ and the vector of counts $c_x = |H^{-1}(x)|$. Our first result is an algorithm to estimate the counts $c_x$ using roughly $\tilde O( \frac{q}{ε^2})$ samples for general Gibbs distributions and $\tilde O( \frac{n^2}{ε^2} )$ samples for integer-valued distributions (ignoring some second-order terms and parameters). We show this is optimal up to logarithmic factors. We illustrate with improved algorithms for counting connected subgraphs and perfect matchings in a graph. We develop a key subroutine for global estimation of the partition function. Specifically, we produce a data structure to estimate $Z(β)$ for \emph{all} values $β$, without further samples. Constructing the data structure requires $O(\frac{q \log n}{ε^2})$ samples for general Gibbs distributions and $O(\frac{n^2 \log n}{ε^2} + n \log q)$ samples for integer-valued distributions. This improves over a prior algorithm of Kolmogorov (2018) which computes the single point estimate $Z(β_\max)$ using $\tilde O(\frac{q}{ε^2})$ samples. We also show that this complexity is optimal as a function of $n$ and $q$ up to logarithmic terms.

math.PR↗

arXiv2023

Solving relaxations of MAP-MRF problems: Combinatorial in-face Frank-Wolfe directions

We consider the problem of solving LP relaxations of MAP-MRF inference problems, and in particular the method proposed recently in (Swoboda, Kolmogorov 2019; Kolmogorov, Pock 2021). As a key computational subroutine, it uses a variant of the Frank-Wolfe (FW) method to minimize a smooth convex function over a combinatorial polytope. We propose an efficient implementation of this subproutine based on in-face Frank-Wolfe directions, introduced in (Freund et al. 2017) in a different context. More generally, we define an abstract data structure for a combinatorial subproblem that enables in-face FW directions, and describe its specialization for tree-structured MAP-MRF inference subproblems. Experimental results indicate that the resulting method is the current state-of-art LP solver for some classes of problems. Our code is available at https://pub.ist.ac.at/~vnk/papers/IN-FACE-FW.html.

math.OC↗

arXiv2021

One-sided Frank-Wolfe algorithms for saddle problems

We study a class of convex-concave saddle-point problems of the form $\min_x\max_y \langle Kx,y\rangle+f_{\cal{P}}(x)-h^\ast(y)$ where $K$ is a linear operator, $f_{\cal{P}}$ is the sum of a convex function $f$ with a Lipschitz-continuous gradient and the indicator function of a bounded convex polytope $\cal{P}$, and $h^\ast$ is a convex (possibly nonsmooth) function. Such problem arises, for example, as a Lagrangian relaxation of various discrete optimization problems. Our main assumptions are the existence of an efficient linear minimization oracle ($lmo$) for $f_{\cal{P}}$ and an efficient proximal map for $h^*$ which motivate the solution via a blend of proximal primal-dual algorithms and Frank-Wolfe algorithms. In case $h^*$ is the indicator function of a linear constraint and function $f$ is quadratic, we show a $O(1/n^2)$ convergence rate on the dual objective, requiring $O(n \log n)$ calls of $lmo$. If the problem comes from the constrained optimization problem $\min_{x\in\mathbb R^d}\{f_{\cal{P}}(x)\:|\:Ax-b=0\}$ then we additionally get bound $O(1/n^2)$ both on the primal gap and on the infeasibility gap. In the most general case, we show a $O(1/n)$ convergence rate of the primal-dual gap again requiring $O(n\log n)$ calls of $lmo$. To the best of our knowledge, this improves on the known convergence rates for the considered class of saddle-point problems. We show applications to labeling problems frequently appearing in machine learning and computer vision.

math.OC↗

arXiv2020

Extensions of the Algorithmic Lovasz Local Lemma

We consider recent formulations of the algorithmic Lovasz Local Lemma by Achlioptas-Iliopoulos-Kolmogorov [2] and by Achlioptas-Iliopoulos-Sinclair [3]. These papers analyze a random walk algorithm for finding objects that avoid undesired "bad events" (or "flaws"), and prove that under certain conditions the algorithm is guaranteed to find a "flawless" object quickly. We show that conditions proposed in these papers are incomparable, and introduce a new family of conditions that includes those in [2, 3] as special cases. We also consider another condition that appeared in [3] in the context of sparse k-SAT formulas. This condition imposes a constraint for each variable of the problem, whereas traditional LLL formulations impose a constraint for each clause. Achlioptas et al. handled the variable-based condition via a reduction to a different condition and then applying a single-clause backtracking algorithm. We propose a new condition that directly captures the sparse k-SAT application considered in [3], and allows the use of the standard local search algorithm (which offers important advantages such as parallelization). Finally, we extend our previous notion of "commutativity" from [20] and prove several implications of commutativity using some new tools that we develop. In particular, we simplify the result of Iliopoulos [16] about approximating the LLL distribution.

cs.DS↗

arXiv2019

Testing the complexity of a valued CSP language

A Valued Constraint Satisfaction Problem (VCSP) provides a common framework that can express a wide range of discrete optimization problems. A VCSP instance is given by a finite set of variables, a finite domain of labels, and an objective function to be minimized. This function is represented as a sum of terms where each term depends on a subset of the variables. To obtain different classes of optimization problems, one can restrict all terms to come from a fixed set $Γ$ of cost functions, called a language. Recent breakthrough results have established a complete complexity classification of such classes with respect to language $Γ$: if all cost functions in $Γ$ satisfy a certain algebraic condition then all $Γ$-instances can be solved in polynomial time, otherwise the problem is NP-hard. Unfortunately, testing this condition for a given language $Γ$ is known to be NP-hard. We thus study exponential algorithms for this meta-problem. We show that the tractability condition of a finite-valued language $Γ$ can be tested in $O(\sqrt[3]{3}^{\,|D|}\cdot poly(size(Γ)))$ time, where $D$ is the domain of $Γ$ and $poly(\cdot)$ is some fixed polynomial. We also obtain a matching lower bound under the Strong Exponential Time Hypothesis (SETH). More precisely, we prove that for any constant $δ<1$ there is no $O(\sqrt[3]{3}^{\,δ|D|})$ algorithm, assuming that SETH holds.

cs.CC↗

arXiv2019

MAP inference via Block-Coordinate Frank-Wolfe Algorithm

We present a new proximal bundle method for Maximum-A-Posteriori (MAP) inference in structured energy minimization problems. The method optimizes a Lagrangean relaxation of the original energy minimization problem using a multi plane block-coordinate Frank-Wolfe method that takes advantage of the specific structure of the Lagrangean decomposition. We show empirically that our method outperforms state-of-the-art Lagrangean decomposition based algorithms on some challenging Markov Random Field, multi-label discrete tomography and graph matching problems.

cs.LG↗

arXiv2018

Commutativity in the Algorithmic Lovasz Local Lemma

We consider the recent formulation of the Algorithmic Lovász Local Lemma [10,2,3] for finding objects that avoid `bad features', or `flaws'. It extends the Moser-Tardos resampling algorithm [17] to more general discrete spaces. At each step the method picks a flaw present in the current state and goes to a new state according to some prespecified probability distribution (which depends on the current state and the selected flaw). However, it is less flexible than the Moser-Tardos method since [10,2,3] require a specific flaw selection rule, whereas [17] allows an arbitrary rule (and thus can potentially be implemented more efficiently). We formulate a new "commutativity" condition, and prove that it is sufficient for an arbitrary rule to work. It also enables an efficient parallelization under an additional assumption. We then show that existing resampling oracles for perfect matchings and permutations do satisfy this condition.

cs.DS↗

arXiv2018

A Local Lemma for Focused Stochastic Algorithms

We develop a framework for the rigorous analysis of focused stochastic local search algorithms. These are algorithms that search a state space by repeatedly selecting some constraint that is violated in the current state and moving to a random nearby state that addresses the violation, while hopefully not introducing many new ones. An important class of focused local search algorithms with provable performance guarantees has recently arisen from algorithmizations of the Lovász Local Lemma (LLL), a non-constructive tool for proving the existence of satisfying states by introducing a background measure on the state space. While powerful, the state transitions of algorithms in this class must be, in a precise sense, perfectly compatible with the background measure. In many applications this is a very restrictive requirement and one needs to step outside the class. Here we introduce the notion of \emph{measure distortion} and develop a framework for analyzing arbitrary focused stochastic local search algorithms, recovering LLL algorithmizations as the special case of no distortion. Our framework takes as input an arbitrary such algorithm and an arbitrary probability measure and shows how to use the measure as a yardstick of algorithmic progress, even for algorithms designed independently of the measure.

cs.DM↗

arXiv2018

Even Delta-Matroids and the Complexity of Planar Boolean CSPs

The main result of this paper is a generalization of the classical blossom algorithm for finding perfect matchings. Our algorithm can efficiently solve Boolean CSPs where each variable appears in exactly two constraints (we call it edge CSP) and all constraints are even $Δ$-matroid relations (represented by lists of tuples). As a consequence of this, we settle the complexity classification of planar Boolean CSPs started by Dvorak and Kupec. Using a reduction to even $Δ$-matroids, we then extend the tractability result to larger classes of $Δ$-matroids that we call efficiently coverable. It properly includes classes that were known to be tractable before, namely co-independent, compact, local, linear and binary, with the following caveat: we represent $Δ$-matroids by lists of tuples, while the last two use a representation by matrices. Since an $n\times n$ matrix can represent exponentially many tuples, our tractability result is not strictly stronger than the known algorithm for linear and binary $Δ$-matroids.

cs.CC↗

arXiv2018

Efficient Optimization for Rank-based Loss Functions

The accuracy of information retrieval systems is often measured using complex loss functions such as the average precision (AP) or the normalized discounted cumulative gain (NDCG). Given a set of positive and negative samples, the parameters of a retrieval system can be estimated by minimizing these loss functions. However, the non-differentiability and non-decomposability of these loss functions does not allow for simple gradient based optimization algorithms. This issue is generally circumvented by either optimizing a structured hinge-loss upper bound to the loss function or by using asymptotic methods like the direct-loss minimization framework. Yet, the high computational complexity of loss-augmented inference, which is necessary for both the frameworks, prohibits its use in large training data sets. To alleviate this deficiency, we present a novel quicksort flavored algorithm for a large class of non-decomposable loss functions. We provide a complete characterization of the loss functions that are amenable to our algorithm, and show that it includes both AP and NDCG based loss functions. Furthermore, we prove that no comparison based algorithm can improve upon the computational complexity of our approach asymptotically. We demonstrate the effectiveness of our approach in the context of optimizing the structured hinge loss upper bound of AP and NDCG loss for learning models for a variety of vision tasks. We show that our approach provides significantly better results than simpler decomposable loss functions, while requiring a comparable training time.

cs.CV↗

arXiv2017

A Faster Approximation Algorithm for the Gibbs Partition Function

We consider the problem of estimating the partition function $Z(β)=\sum_x \exp(-β(H(x))$ of a Gibbs distribution with a Hamilton $H(\cdot)$, or more precisely the logarithm of the ratio $q=\ln Z(0)/Z(β)$. It has been recently shown how to approximate $q$ with high probability assuming the existence of an oracle that produces samples from the Gibbs distribution for a given parameter value in $[0,β]$. The current best known approach due to Huber [9] uses $O(q\ln n\cdot[\ln q + \ln \ln n+\varepsilon^{-2}])$ oracle calls on average where $\varepsilon$ is the desired accuracy of approximation and $H(\cdot)$ is assumed to lie in $\{0\}\cup[1,n]$. We improve the complexity to $O(q\ln n\cdot\varepsilon^{-2})$ oracle calls. We also show that the same complexity can be achieved if exact oracles are replaced with approximate sampling oracles that are within $O(\frac{\varepsilon^2}{q\ln n})$ variation distance from exact oracles. Finally, we prove a lower bound of $Ω(q\cdot \varepsilon^{-2})$ oracle calls under a natural model of computation.

cs.DS↗

arXiv2017

The Complexity of General-Valued CSPs

An instance of the Valued Constraint Satisfaction Problem (VCSP) is given by a finite set of variables, a finite domain of labels, and a sum of functions, each function depending on a subset of the variables. Each function can take finite values specifying costs of assignments of labels to its variables or the infinite value, which indicates infeasible assignments. The goal is to find an assignment of labels to the variables that minimizes the sum. We study (assuming that P $\ne$ NP) how the complexity of this very general problem depends on the set of functions allowed in the instances, the so-called constraint language. The case when all allowed functions take values in $\{0,\infty\}$ corresponds to ordinary CSPs, where one deals only with the feasibility issue and there is no optimization. This case is the subject of the Algebraic CSP Dichotomy Conjecture predicting for which constraint languages CSPs are tractable and for which NP-hard. The case when all allowed functions take only finite values corresponds to finite-valued CSP, where the feasibility aspect is trivial and one deals only with the optimization issue. The complexity of finite-valued CSPs was fully classified by Thapper and Živný. An algebraic necessary condition for tractability of a general-valued CSP with a fixed constraint language was recently given by Kozik and Ochremiak. As our main result, we prove that if a constraint language satisfies this algebraic necessary condition, and the feasibility CSP corresponding to the VCSP with this language is tractable, then the VCSP is tractable. The algorithm is a simple combination of the assumed algorithm for the feasibility CSP and the standard LP relaxation. As a corollary, we obtain that a dichotomy for ordinary CSPs would imply a dichotomy for general-valued CSPs.

cs.CC↗

arXiv2017

Inference algorithms for pattern-based CRFs on sequence data

We consider Conditional Random Fields (CRFs) with pattern-based potentials defined on a chain. In this model the energy of a string (labeling) $x_1...x_n$ is the sum of terms over intervals $[i,j]$ where each term is non-zero only if the substring $x_i...x_j$ equals a prespecified pattern $α$. Such CRFs can be naturally applied to many sequence tagging problems. We present efficient algorithms for the three standard inference tasks in a CRF, namely computing (i) the partition function, (ii) marginals, and (iii) computing the MAP. Their complexities are respectively $O(n L)$, $O(n L \ell_{max})$ and $O(n L \min\{|D|,\log (\ell_{max}+1)\})$ where $L$ is the combined length of input patterns, $\ell_{max}$ is the maximum length of a pattern, and $D$ is the input alphabet. This improves on the previous algorithms of (Ye et al., 2009) whose complexities are respectively $O(n L |D|)$, $O(n |Γ| L^2 \ell_{max}^2)$ and $O(n L |D|)$, where $|Γ|$ is the number of input patterns. In addition, we give an efficient algorithm for sampling. Finally, we consider the case of non-positive weights. (Komodakis & Paragios, 2009) gave an $O(n L)$ algorithm for computing the MAP. We present a modification that has the same worst-case complexity but can beat it in the best case.

cs.LG↗

arXiv2017

A new look at reweighted message passing

We propose a new family of message passing techniques for MAP estimation in graphical models which we call {\em Sequential Reweighted Message Passing} (SRMP). Special cases include well-known techniques such as {\em Min-Sum Diffusion} (MSD) and a faster {\em Sequential Tree-Reweighted Message Passing} (TRW-S). Importantly, our derivation is simpler than the original derivation of TRW-S, and does not involve a decomposition into trees. This allows easy generalizations. We present such a generalization for the case of higher-order graphical models, and test it on several real-world problems with promising results.

cs.AI↗

arXiv2016

Superconcentrators of Density 25.3

An $N$-superconcentrator is a directed, acyclic graph with $N$ input nodes and $N$ output nodes such that every subset of the inputs and every subset of the outputs of same cardinality can be connected by node-disjoint paths. It is known that linear-size and bounded-degree superconcentrators exist. We prove the existence of such superconcentrators with asymptotic density $25.3$ (where the density is the number of edges divided by $N$). The previously best known densities were $28$ \cite{Scho2006} and $27.4136$ \cite{YuanK12}.

cs.DM↗

arXiv2016

Total variation on a tree

We consider the problem of minimizing the continuous valued total variation subject to different unary terms on trees and propose fast direct algorithms based on dynamic programming to solve these problems. We treat both the convex and the non-convex case and derive worst case complexities that are equal or better than existing methods. We show applications to total variation based 2D image processing and computer vision problems based on a Lagrangian decomposition approach. The resulting algorithms are very efficient, offer a high degree of parallelism and come along with memory requirements which are only in the order of the number of image pixels.

cs.CV↗

arXiv2015

Effectiveness of Structural Restrictions for Hybrid CSPs

Constraint Satisfaction Problem (CSP) is a fundamental algorithmic problem that appears in many areas of Computer Science. It can be equivalently stated as computing a homomorphism $\mbox{$\bR \rightarrow \bGamma$}$ between two relational structures, e.g.\ between two directed graphs. Analyzing its complexity has been a prominent research direction, especially for {\em fixed template CSPs} in which the right side $\bGamma$ is fixed and the left side $\bR$ is unconstrained. Far fewer results are known for the {\em hybrid} setting that restricts both sides simultaneously. It assumes that $\bR$ belongs to a certain class of relational structures (called a {\em structural restriction} in this paper). We study which structural restrictions are {\em effective}, i.e.\ there exists a fixed template $\bGamma$ (from a certain class of languages) for which the problem is tractable when $\bR$ is restricted, and NP-hard otherwise. We provide a characterization for structural restrictions that are {\em closed under inverse homomorphisms}. The criterion is based on the {\em chromatic number} of a relational structure defined in this paper; it generalizes the standard chromatic number of a graph. As our main tool, we use the algebraic machinery developed for fixed template CSPs. To apply it to our case, we introduce a new construction called a "lifted language." We also give a characterization for structural restrictions corresponding to minor-closed families of graphs, extend results to certain Valued CSPs (namely conservative valued languages), and state implications for CSPs with ordered variables, (valued) CSPs on structures with large girth, and for the maximum weight independent set problem on some restricted families of graphs including graphs with large girth.

cs.CC↗

arXiv2014

The power of linear programming for general-valued CSPs

Let $D$, called the domain, be a fixed finite set and let $Γ$, called the valued constraint language, be a fixed set of functions of the form $f:D^m\to\mathbb{Q}\cup\{\infty\}$, where different functions might have different arity $m$. We study the valued constraint satisfaction problem parametrised by $Γ$, denoted by VCSP$(Γ)$. These are minimisation problems given by $n$ variables and the objective function given by a sum of functions from $Γ$, each depending on a subset of the $n$ variables. Finite-valued constraint languages contain functions that take on only rational values and not infinite values. Our main result is a precise algebraic characterisation of valued constraint languages whose instances can be solved exactly by the basic linear programming relaxation (BLP). For a valued constraint language $Γ$, BLP is a decision procedure for $Γ$ if and only if $Γ$ admits a symmetric fractional polymorphism of every arity. For a finite-valued constraint language $Γ$, BLP is a decision procedure if and only if $Γ$ admits a symmetric fractional polymorphism of some arity, or equivalently, if $Γ$ admits a symmetric fractional polymorphism of arity 2. Using these results, we obtain tractability of several novel classes of problems, including problems over valued constraint languages that are: (1) submodular on arbitrary lattices; (2) $k$-submodular on arbitrary finite domains; (3) weakly (and hence strongly) tree-submodular on arbitrary trees.

cs.CC↗