arXiv subjects

Marek Cygan

Publications and source records attributed to Marek Cygan.

At least 37 records · Page 2Linked to original sources

arXiv2024

MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts

State Space Models (SSMs) have become serious contenders in the field of sequential modeling, challenging the dominance of Transformers. At the same time, Mixture of Experts (MoE) has significantly improved Transformer-based Large Language Models, including recent state-of-the-art open models. We propose that to unlock the potential of SSMs for scaling, they should be combined with MoE. We showcase this on Mamba, a recent SSM-based model that achieves remarkable performance. Our model, MoE-Mamba, outperforms both Mamba and baseline Transformer-MoE. In particular, MoE-Mamba reaches the same performance as Mamba in $2.35\times$ fewer training steps while preserving the inference performance gains of Mamba against Transformer.

cs.LG↗

arXiv2024

Scaling Laws for Fine-Grained Mixture of Experts

Mixture of Experts (MoE) models have emerged as a primary solution for reducing the computational cost of Large Language Models. In this work, we analyze their scaling properties, incorporating an expanded range of variables. Specifically, we introduce a new hyperparameter, granularity, whose adjustment enables precise control over the size of the experts. Building on this, we establish scaling laws for fine-grained MoE, taking into account the number of training tokens, model size, and granularity. Leveraging these laws, we derive the optimal training configuration for a given computational budget. Our findings not only show that MoE models consistently outperform dense Transformers but also highlight that the efficiency gap between dense and MoE models widens as we scale up the model size and training budget. Furthermore, we demonstrate that the common practice of setting the size of experts in MoE to mirror the feed-forward layer is not optimal at almost any computational budget.

cs.LG↗

arXiv2023

On Many-Actions Policy Gradient

We study the variance of stochastic policy gradients (SPGs) with many action samples per state. We derive a many-actions optimality condition, which determines when many-actions SPG yields lower variance as compared to a single-action agent with proportionally extended trajectory. We propose Model-Based Many-Actions (MBMA), an approach leveraging dynamics models for many-actions sampling in the context of SPG. MBMA addresses issues associated with existing implementations of many-actions SPG and yields lower bias and comparable variance to SPG estimated from states in model-simulated rollouts. We find that MBMA bias and variance structure matches that predicted by theory. As a result, MBMA achieves improved sample efficiency and higher returns on a range of continuous action environments as compared to model-free, many-actions, and model-based on-policy SPG baselines.

cs.LG↗

arXiv2023

Grasping Student: semi-supervised learning for robotic manipulation

Gathering real-world data from the robot quickly becomes a bottleneck when constructing a robot learning system for grasping. In this work, we design a semi-supervised grasping system that, on top of a small sample of robot experience, takes advantage of images of products to be picked, which are collected without any interactions with the robot. We validate our findings both in the simulation and in the real world. In the regime of a small number of robot training samples, taking advantage of the unlabeled data allows us to achieve performance at the level of 10-fold bigger dataset size used by the baseline. The code and datasets used in the paper will be released at https://github.com/nomagiclab/grasping-student.

cs.RO↗

arXiv2022

Polite Teacher: Semi-Supervised Instance Segmentation with Mutual Learning and Pseudo-Label Thresholding

We present Polite Teacher, a simple yet effective method for the task of semi-supervised instance segmentation. The proposed architecture relies on the Teacher-Student mutual learning framework. To filter out noisy pseudo-labels, we use confidence thresholding for bounding boxes and mask scoring for masks. The approach has been tested with CenterMask, a single-stage anchor-free detector. Tested on the COCO 2017 val dataset, our architecture significantly (approx. +8 pp. in mask AP) outperforms the baseline at different supervision regimes. To the best of our knowledge, this is one of the first works tackling the problem of semi-supervised instance segmentation and the first one devoted to an anchor-free detector.

cs.CV↗

arXiv2022

One Simple Trick to Fix Your Bayesian Neural Network

One of the most popular estimation methods in Bayesian neural networks (BNN) is mean-field variational inference (MFVI). In this work, we show that neural networks with ReLU activation function induce posteriors, that are hard to fit with MFVI. We provide a theoretical justification for this phenomenon, study it empirically, and report the results of a series of experiments to investigate the effect of activation function on the calibration of BNNs. We find that using Leaky ReLU activations leads to more Gaussian-like weight posteriors and achieves a lower expected calibration error (ECE) than its ReLU-based counterpart.

stat.ML↗

arXiv2022

n-CPS: Generalising Cross Pseudo Supervision to n Networks for Semi-Supervised Semantic Segmentation

We present n-CPS - a generalisation of the recent state-of-the-art cross pseudo supervision (CPS) approach for the task of semi-supervised semantic segmentation. In n-CPS, there are n simultaneously trained subnetworks that learn from each other through one-hot encoding perturbation and consistency regularisation. We also show that ensembling techniques applied to subnetworks outputs can significantly improve the performance. To the best of our knowledge, n-CPS paired with CutMix outperforms CPS and sets the new state-of-the-art for Pascal VOC 2012 with (1/16, 1/8, 1/4, and 1/2 supervised regimes) and Cityscapes (1/16 supervised).

cs.CV↗

arXiv2020

Randomized contractions meet lean decompositions

We show an algorithm that, given an $n$-vertex graph $G$ and a parameter $k$, in time $2^{O(k \log k)} n^{O(1)}$ finds a tree decomposition of $G$ with the following properties: * every adhesion of the tree decomposition is of size at most $k$, and * every bag of the tree decomposition is $(i,i)$-unbreakable in $G$ for every $1 \leq i \leq k$. Here, a set $X \subseteq V(G)$ is $(a,b)$-unbreakable in $G$ if for every separation $(A,B)$ of order at most $b$ in $G$, we have $|A \cap X| \leq a$ or $|B \cap X| \leq a$. The resulting tree decomposition has arguably best possible adhesion size boundsand unbreakability guarantees. Furthermore, the parametric factor in the running time bound is significantly smaller than in previous similar constructions. These improvements allow us to present parameterized algorithms for Minimum Bisection, Steiner Cut, and Steiner Multicut with improved parameteric factor in the running time bound. The main technical insight is to adapt the notion of lean decompositions of Thomas and the subsequent construction algorithm of Bellenbaum and Diestel to the parameterized setting.

cs.DS↗

arXiv2020

Tight Bounds on Subexponential Time Approximation of Set Cover and Related Problems

We show that Set Cover on instances with $N$ elements cannot be approximated within $(1-γ)\ln N$-factor in time exp($N^{γ-δ})$, for any $0 < γ< 1$ and any $δ> 0$, assuming the Exponential Time Hypothesis. This essentially matches the best upper bound known by Cygan et al.\ (IPL, 2009) of $(1-γ)\ln N$-factor in time $exp(O(N^γ))$. The lower bound is obtained by extracting a standalone reduction from Label Cover to Set Cover from the work of Moshkovitz (Theory of Computing, 2015), and applying it to a different PCP theorem than done there. We also obtain a tighter lower bound when conditioning on the Projection Games Conjecture. We also treat three problems (Directed Steiner Tree, Submodular Cover, and Connected Polymatroid) that strictly generalize Set Cover. We give a $(1-γ)\ln N$-approximation algorithm for these problems that runs in $exp(\tilde{O}(N^γ))$ time, for any $1/2 \le γ< 1$.

cs.DS↗

arXiv2019

On problems equivalent to (min,+)-convolution

In recent years, significant progress has been made in explaining the apparent hardness of improving upon the naive solutions for many fundamental polynomially solvable problems. This progress has come in the form of conditional lower bounds -- reductions from a problem assumed to be hard. The hard problems include 3SUM, All-Pairs Shortest Path, SAT, Orthogonal Vectors, and others. In the $(\min,+)$-convolution problem, the goal is to compute a sequence $(c[i])^{n-1}_{i=0}$, where $c[k] = $ $\min_{i=0,\ldots,k} $ $\{a[i] $ $+$ $b[k-i]\}$, given sequences $(a[i])^{n-1}_{i=0}$ and $(b[i])_{i=0}^{n-1}$. This can easily be done in $O(n^2)$ time, but no $O(n^{2-\varepsilon})$ algorithm is known for $\varepsilon > 0$. In this paper, we undertake a systematic study of the $(\min,+)$-convolution problem as a hardness assumption. First, we establish the equivalence of this problem to a group of other problems, including variants of the classic knapsack problem and problems related to subadditive sequences. The $(\min,+)$-convolution problem has been used as a building block in algorithms for many problems, notably problems in stringology. It has also appeared as an ad hoc hardness assumption. Second, we investigate some of these connections and provide new reductions and other results. We also explain why replacing this assumption with the SETH might not be possible for some problems.

cs.DS↗

arXiv2018

On subexponential running times for approximating directed Steiner tree and related problems

This paper concerns proving almost tight (super-polynomial) running times, for achieving desired approximation ratios for various problems. To illustrate, the question we study, let us consider the Set-Cover problem with n elements and m sets. Now we specify our goal to approximate Set-Cover to a factor of (1-d)ln n, for a given parameter 0 = 2^{n^{c d}}, for some constant 0 = exp((1+o(1)){log^{d-c}n}), for any c>0, unless the ETH is false. Our result follows by analyzing the work of Halperin and Krauthgamer [STOC, 2003]. The same lower and upper bounds hold for CST.

cs.DS↗

arXiv2018

Online Facility Location with Deletions

In this paper we study three previously unstudied variants of the online Facility Location problem, considering an intrinsic scenario when the clients and facilities are not only allowed to arrive to the system, but they can also depart at any moment. We begin with the study of a natural fully-dynamic online uncapacitated model where clients can be both added and removed. When a client arrives, then it has to be assigned either to an existing facility or to a new facility opened at the client's location. However, when a client who has been also one of the open facilities is to be removed, then our model has to allow to reconnect all clients that have been connected to that removed facility. In this model, we present an optimal O(log n_act / log log n_act)-competitive algorithm, where n_act is the number of active clients at the end of the input sequence. Next, we turn our attention to the capacitated Facility Location problem. We first note that if no deletions are allowed, then one can achieve an optimal competitive ratio of O(log n/ log log n), where n is the length of the sequence. However, when deletions are allowed, the capacitated version of the problem is significantly more challenging than the uncapacitated one. We show that still, using a more sophisticated algorithmic approach, one can obtain an online O(log m + log c log n)-competitive algorithm for the capacitated Facility Location problem in the fully dynamic model, where m is number of points in the input metric and c is the capacity of any open facility.

cs.DS↗

arXiv2018

Hardness of approximation for H-free edge modification problems

The $H$-Free Edge Deletion problem asks, for a given graph $G$ and an integer $k$, whether it is possible to delete at most $k$ edges from $G$ to make it $H$-free, that is, not containing $H$ as an induced subgraph. The $H$-Free Edge Completion problem is defined similarly, but we add edges instead of deleting them. The study of these two problem families has recently been the subject of intensive studies from the point of view of parameterized complexity and kernelization. In particular, it was shown that the problems do not admit polynomial kernels (under plausible complexity assumptions) for almost all graphs $H$, with several important exceptions occurring when the class of $H$-free graphs exhibits some structural properties. In this work we complement the parameterized study of edge modification problems to $H$-free graphs by considering their approximability. We prove that whenever $H$ is $3$-connected and has at least two non-edges, then both $H$-Free Edge Deletion and $H$-Free Edge Completion are very hard to approximate: they do not admit $\mathrm{poly}(\mathsf{OPT})$-approximation in polynomial time, unless $\mathrm{P}=\mathrm{NP}$, or even in time subexponential in $\mathsf{OPT}$, unless the Exponential Time Hypothesis fails. The assumption of the existence of two non-edges appears to be important: we show that whenever $H$ is a complete graph without one edge, then $H$-Free Edge Deletion is tightly connected to the Min Horn problem, whose approximability is still open. Finally, in an attempt to extend our hardness results beyond $3$-connected graphs, we consider the cases of $H$ being a path or a cycle, and we achieve an almost complete dichotomy there.

cs.DS↗

arXiv2018

Improved GQ-CNN: Deep Learning Model for Planning Robust Grasps

Recent developments in the field of robot grasping have shown great improvements in the grasp success rates when dealing with unknown objects. In this work we improve on one of the most promising approaches, the Grasp Quality Convolutional Neural Network (GQ-CNN) trained on the DexNet 2.0 dataset. We propose a new architecture for the GQ-CNN and describe practical improvements that increase the model validation accuracy from 92.2% to 95.8% and from 85.9% to 88.0% on respectively image-wise and object-wise training and validation splits.

cs.LG↗

arXiv2017

From Gap-ETH to FPT-Inapproximability: Clique, Dominating Set, and More

We consider questions that arise from the intersection between the areas of polynomial-time approximation algorithms, subexponential-time algorithms, and fixed-parameter tractable algorithms. The questions, which have been asked several times (e.g., [Marx08, FGMS12, DF13]), are whether there is a non-trivial FPT-approximation algorithm for the Maximum Clique (Clique) and Minimum Dominating Set (DomSet) problems parameterized by the size of the optimal solution. In particular, letting $\text{OPT}$ be the optimum and $N$ be the size of the input, is there an algorithm that runs in $t(\text{OPT})\text{poly}(N)$ time and outputs a solution of size $f(\text{OPT})$, for any functions $t$ and $f$ that are independent of $N$ (for Clique, we want $f(\text{OPT})=ω(1)$)? In this paper, we show that both Clique and DomSet admit no non-trivial FPT-approximation algorithm, i.e., there is no $o(\text{OPT})$-FPT-approximation algorithm for Clique and no $f(\text{OPT})$-FPT-approximation algorithm for DomSet, for any function $f$ (e.g., this holds even if $f$ is the Ackermann function). In fact, our results imply something even stronger: The best way to solve Clique and DomSet, even approximately, is to essentially enumerate all possibilities. Our results hold under the Gap Exponential Time Hypothesis (Gap-ETH) [Dinur16, MR16], which states that no $2^{o(n)}$-time algorithm can distinguish between a satisfiable 3SAT formula and one which is not even $(1 - ε)$-satisfiable for some constant $ε> 0$. Besides Clique and DomSet, we also rule out non-trivial FPT-approximation for Maximum Balanced Biclique, Maximum Subgraphs with Hereditary Properties, and Maximum Induced Matching in bipartite graphs. Additionally, we rule out $k^{o(1)}$-FPT-approximation algorithm for Densest $k$-Subgraph although this ratio does not yet match the trivial $O(k)$-approximation algorithm.

cs.CC↗

arXiv2017

Improving TSP tours using dynamic programming over tree decomposition

Given a traveling salesman problem (TSP) tour $H$ in graph $G$ a $k$-move is an operation which removes $k$ edges from $H$, and adds $k$ edges of $G$ so that a new tour $H'$ is formed. The popular $k$-OPT heuristics for TSP finds a local optimum by starting from an arbitrary tour $H$ and then improving it by a sequence of $k$-moves. Until 2016, the only known algorithm to find an improving $k$-move for a given tour was the naive solution in time $O(n^k)$. At ICALP'16 de Berg, Buchin, Jansen and Woeginger showed an $O(n^{\lfloor 2/3k \rfloor+1})$-time algorithm. We show an algorithm which runs in $O(n^{(1/4+ε_k)k})$ time, where $\lim ε_k = 0$. We are able to show that it improves over the state of the art for every $k=5,\ldots,10$. For the most practically relevant case $k=5$ we provide a slightly refined algorithm running in $O(n^{3.4})$ time. We also show that for the $k=4$ case, improving over the $O(n^3)$-time algorithm of de Berg et al. would be a major breakthrough: an $O(n^{3-ε})$-time algorithm for any $ε>0$ would imply an $O(n^{3-δ})$-time algorithm for the ALL PAIRS SHORTEST PATHS problem, for some $δ>0$.

cs.DS↗

arXiv2016

Approximation and Parameterized Complexity of Minimax Approval Voting

We present three results on the complexity of Minimax Approval Voting. First, we study Minimax Approval Voting parameterized by the Hamming distance $d$ from the solution to the votes. We show Minimax Approval Voting admits no algorithm running in time $\mathcal{O}^\star(2^{o(d\log d)})$, unless the Exponential Time Hypothesis (ETH) fails. This means that the $\mathcal{O}^\star(d^{2d})$ algorithm of Misra et al. [AAMAS 2015] is essentially optimal. Motivated by this, we then show a parameterized approximation scheme, running in time $\mathcal{O}^\star(\left({3}/ε\right)^{2d})$, which is essentially tight assuming ETH. Finally, we get a new polynomial-time randomized approximation scheme for Minimax Approval Voting, which runs in time $n^{\mathcal{O}(1/ε^2 \cdot \log(1/ε))} \cdot \mathrm{poly}(m)$, almost matching the running time of the fastest known PTAS for Closest String due to Ma and Sun [SIAM J. Comp. 2009].

cs.DS↗

arXiv2016

Designing FPT algorithms for cut problems using randomized contractions

We introduce a new technique for designing fixed-parameter algorithms for cut problems, namely randomized contractions. We apply our framework to obtain the first FPT algorithm for the Unique Label Cover problem and new FPT algorithms with exponential speed up for the Steiner Cut and Node Multiway Cut-Uncut problems. More precisely, we show the following: - We prove that the parameterized version of the Unique Label Cover problem, which is the base of the Unique Games Conjecture, can be solved in 2^{O(k^2\log |Σ|)}n^4\log n deterministic time (even in the stronger, vertex-deletion variant) where k is the number of unsatisfied edges and |Σ| is the size of the alphabet. As a consequence, we show that one can in polynomial time solve instances of Unique Games where the number of edges allowed not to be satisfied is upper bounded by O(\sqrt{\log n}) to optimality, which improves over the trivial O(1) upper bound. - We prove that the Steiner Cut problem can be solved in 2^{O(k^2\log k)}n^4\log n deterministic time and \tilde{O}(2^{O(k^2\log k)}n^2) randomized time where k is the size of the cutset. This result improves the double exponential running time of the recent work of Kawarabayashi and Thorup (FOCS'11). - We show how to combine considering `cut' and `uncut' constraints at the same time. More precisely, we define a robust problem Node Multiway Cut-Uncut that can serve as an abstraction of introducing uncut constraints, and show that it admits an algorithm running in 2^{O(k^2\log k)}n^4\log n deterministic time where k is the size of the cutset. To the best of our knowledge, the only known way of tackling uncut constraints was via the approach of Marx, O'Sullivan and Razgon (STACS'10), which yields algorithms with double exponential running time. An interesting aspect of our technique is that, unlike important separators, it can handle real weights.

cs.DS↗