Searcharxiv⌕ Search

arXiv subjects

Rustem Takhanov

Publications and source records attributed to Rustem Takhanov.

34 records · Page 2Linked to original sources
arXiv2021

How many moments does MMD compare?

We present a new way of study of Mercer kernels, by corresponding to a special kernel $K$ a pseudo-differential operator $p({\mathbf x}, D)$ such that $\mathcal{F} p({\mathbf x}, D)^†p({\mathbf x}, D) \mathcal{F}^{-1}$ acts on smooth functions in the same way as an integral operator associated with $K$ (where $\mathcal{F}$ is the Fourier transform). We show that kernels defined by pseudo-differential operators are able to approximate uniformly any continuous Mercer kernel on a compact set. The symbol $p({\mathbf x}, {\mathbf y})$ encapsulates a lot of useful information about the structure of the Maximum Mean Discrepancy distance defined by the kernel $K$. We approximate $p({\mathbf x}, {\mathbf y})$ with the sum of the first $r$ terms of the Singular Value Decomposition of $p$, denoted by $p_r({\mathbf x}, {\mathbf y})$. If ordered singular values of the integral operator associated with $p({\mathbf x}, {\mathbf y})$ die down rapidly, the MMD distance defined by the new symbol $p_r$ differs from the initial one only slightly. Moreover, the new MMD distance can be interpreted as an aggregated result of comparing $r$ local moments of two probability distributions. The latter results holds under the condition that right singular vectors of the integral operator associated with $p$ are uniformly bounded. But even if this is not satisfied we can still hold that the Hilbert-Schmidt distance between $p$ and $p_r$ vanishes. Thus, we report an interesting phenomenon: the MMD distance measures the difference of two probability distributions with respect to a certain number of local moments, $r^\ast$, and this number $r^\ast$ depends on the speed with which singular values of $p$ die down.

cs.LG↗
arXiv2019

Context Vectors are Reflections of Word Vectors in Half the Dimensions

This paper takes a step towards theoretical analysis of the relationship between word embeddings and context embeddings in models such as word2vec. We start from basic probabilistic assumptions on the nature of word vectors, context vectors, and text generation. These assumptions are well supported either empirically or theoretically by the existing literature. Next, we show that under these assumptions the widely-used word-word PMI matrix is approximately a random symmetric Gaussian ensemble. This, in turn, implies that context vectors are reflections of word vectors in approximately half the dimensions. As a direct application of our result, we suggest a theoretically grounded way of tying weights in the SGNS model.

stat.ML↗
arXiv2019

Fourier Neural Networks: A Comparative Study

We review neural network architectures which were motivated by Fourier series and integrals and which are referred to as Fourier neural networks. These networks are empirically evaluated in synthetic and real-world tasks. Neither of them outperforms the standard neural network with sigmoid activation function in the real-world tasks. All neural networks, both Fourier and the standard one, empirically demonstrate lower approximation error than the truncated Fourier series when it comes to an approximation of a known function of multiple variables.

cs.NE↗
arXiv2018

Fourier analysis perspective for sufficient dimension reduction problem

A theory of sufficient dimension reduction (SDR) is developed from an optimizational perspective. In our formulation of the problem, instead of dealing with raw data, we assume that our ground truth includes a mapping ${\mathbf f}: {\mathbb R}^n\rightarrow {\mathbb R}^m$ and a probability distribution function $p$ over ${\mathbb R}^n$, both given analytically. We formulate SDR as a problem of finding a function ${\mathbf g}: {\mathbb R}^k\rightarrow {\mathbb R}^m$ and a matrix $P\in {\mathbb R}^{k\times n}$ such that ${\mathbb E}_{{\mathbf x}\sim p({\mathbf x})} \left|{\mathbf f}({\mathbf x}) - {\mathbf g}(P{\mathbf x})\right|^2$ is minimal. It turns out that the latter problem allows a reformulation in the dual space, i.e. instead of searching for ${\mathbf g}(P{\mathbf x})$ we suggest searching for its Fourier transform. First, we characterize all tempered distributions that can serve as the Fourier transform of such functions. The reformulation in the dual space can be interpreted as a problem of finding a $k$-dimensional linear subspace $S$ and a tempered distribution ${\mathbf t}$ supported in $S$ such that ${\mathbf t}$ is "close" in a certain sense to the Fourier transform of ${\mathbf f}$. Instead of optimizing over generalized functions with a $k$-dimensional support, we suggest minimizing over ordinary functions but with an additional term $R$ that penalizes a strong distortion of the support from any $k$-dimensional linear subspace. For a specific case of $R$, we develop an algorithm that can be formulated for functions given in the initial form as well as for their Fourier transforms. Eventually, we report results of numerical experiments with a discretized version of the latter algorithm.

cs.LG↗
arXiv2018

Reusing Weights in Subword-aware Neural Language Models

We propose several ways of reusing subword embeddings and other weights in subword-aware neural language models. The proposed techniques do not benefit a competitive character-aware model, but some of them improve the performance of syllable- and morpheme-aware models while showing significant reductions in model sizes. We discover a simple hands-on principle: in a multi-layer input embedding model, layers should be tied consecutively bottom-up if reused at output. Our best morpheme-aware model with properly reused weights beats the competitive word-level model by a large margin across multiple languages and has 20%-87% fewer parameters.

cs.CL↗
arXiv2017

Patterns versus Characters in Subword-aware Neural Language Modeling

Words in some natural languages can have a composite structure. Elements of this structure include the root (that could also be composite), prefixes and suffixes with which various nuances and relations to other words can be expressed. Thus, in order to build a proper word representation one must take into account its internal structure. From a corpus of texts we extract a set of frequent subwords and from the latter set we select patterns, i.e. subwords which encapsulate information on character $n$-gram regularities. The selection is made using the pattern-based Conditional Random Field model with $l_1$ regularization. Further, for every word we construct a new sequence over an alphabet of patterns. The new alphabet's symbols confine a local statistical context stronger than the characters, therefore they allow better representations in ${\mathbb{R}}^n$ and are better building blocks for word representation. In the task of subword-aware language modeling, pattern-based models outperform character-based analogues by 2-20 perplexity points. Also, a recurrent neural network in which a word is represented as a sum of embeddings of its patterns is on par with a competitive and significantly more sophisticated character-based convolutional architecture.

cs.CL↗
arXiv2017

Hybrid VCSPs with crisp and conservative valued templates

A constraint satisfaction problem (CSP) is a problem of computing a homomorphism ${\bf R} \rightarrow {\bf Γ}$ between two relational structures. Analyzing its complexity has been a very fruitful research direction, especially for fixed template CSPs, denoted $CSP({\bf Γ})$, in which the right side structure ${\bf Γ}$ is fixed and the left side structure ${\bf R}$ is unconstrained. Recently, the hybrid setting, written $CSP_{\mathcal{H}}({\bf Γ})$, where both sides are restricted simultaneously, attracted some attention. It assumes that ${\bf R}$ is taken from a class of relational structures $\mathcal{H}$ that additionally is closed under inverse homomorphisms. The last property allows to exploit algebraic tools that have been developed for fixed template CSPs. The key concept that connects hybrid CSPs with fixed-template CSPs is the so called "lifted language". Namely, this is a constraint language ${\bf Γ}_{\bf R}$ that can be constructed from an input ${\bf R}$. The tractability of that language for any input ${\bf R}\in\mathcal{H}$ is a necessary condition for the tractability of the hybrid problem. In the first part we investigate templates ${\bf Γ}$ for which the latter condition is not only necessary, but also is sufficient. We call such templates ${\bf Γ}$ widely tractable. For this purpose, we construct from ${\bf Γ}$ a new finite relational structure ${\bf Γ}'$ and define $\mathcal{H}_0$ as a class of structures homomorphic to ${\bf Γ}'$. We prove that wide tractability is equivalent to the tractability of $CSP_{\mathcal{H}_0}({\bf Γ})$. Our proof is based on the key observation that ${\bf R}$ is homomorphic to ${\bf Γ}'$ if and only if the core of ${\bf Γ}_{\bf R}$ is preserved by a Siggers polymorphism. Analogous result is shown for valued conservative CSPs.

cs.CC↗
arXiv2017

Syllable-aware Neural Language Models: A Failure to Beat Character-aware Ones

Syllabification does not seem to improve word-level RNN language modeling quality when compared to character-based segmentation. However, our best syllable-aware language model, achieving performance comparable to the competitive character-aware model, has 18%-33% fewer parameters and is trained 1.2-2.2 times faster.

cs.CL↗
arXiv2017

Inference algorithms for pattern-based CRFs on sequence data

We consider Conditional Random Fields (CRFs) with pattern-based potentials defined on a chain. In this model the energy of a string (labeling) $x_1...x_n$ is the sum of terms over intervals $[i,j]$ where each term is non-zero only if the substring $x_i...x_j$ equals a prespecified pattern $α$. Such CRFs can be naturally applied to many sequence tagging problems. We present efficient algorithms for the three standard inference tasks in a CRF, namely computing (i) the partition function, (ii) marginals, and (iii) computing the MAP. Their complexities are respectively $O(n L)$, $O(n L \ell_{max})$ and $O(n L \min\{|D|,\log (\ell_{max}+1)\})$ where $L$ is the combined length of input patterns, $\ell_{max}$ is the maximum length of a pattern, and $D$ is the input alphabet. This improves on the previous algorithms of (Ye et al., 2009) whose complexities are respectively $O(n L |D|)$, $O(n |Γ| L^2 \ell_{max}^2)$ and $O(n L |D|)$, where $|Γ|$ is the number of input patterns. In addition, we give an efficient algorithm for sampling. Finally, we consider the case of non-positive weights. (Komodakis & Paragios, 2009) gave an $O(n L)$ algorithm for computing the MAP. We present a modification that has the same worst-case complexity but can beat it in the best case.

cs.LG↗
arXiv2015

Effectiveness of Structural Restrictions for Hybrid CSPs

Constraint Satisfaction Problem (CSP) is a fundamental algorithmic problem that appears in many areas of Computer Science. It can be equivalently stated as computing a homomorphism $\mbox{$\bR \rightarrow \bGamma$}$ between two relational structures, e.g.\ between two directed graphs. Analyzing its complexity has been a prominent research direction, especially for {\em fixed template CSPs} in which the right side $\bGamma$ is fixed and the left side $\bR$ is unconstrained. Far fewer results are known for the {\em hybrid} setting that restricts both sides simultaneously. It assumes that $\bR$ belongs to a certain class of relational structures (called a {\em structural restriction} in this paper). We study which structural restrictions are {\em effective}, i.e.\ there exists a fixed template $\bGamma$ (from a certain class of languages) for which the problem is tractable when $\bR$ is restricted, and NP-hard otherwise. We provide a characterization for structural restrictions that are {\em closed under inverse homomorphisms}. The criterion is based on the {\em chromatic number} of a relational structure defined in this paper; it generalizes the standard chromatic number of a graph. As our main tool, we use the algebraic machinery developed for fixed template CSPs. To apply it to our case, we introduce a new construction called a "lifted language." We also give a characterization for structural restrictions corresponding to minor-closed families of graphs, extend results to certain Valued CSPs (namely conservative valued languages), and state implications for CSPs with ordered variables, (valued) CSPs on structures with large girth, and for the maximum weight independent set problem on some restricted families of graphs including graphs with large girth.

cs.CC↗
arXiv2014

Combining pattern-based CRFs and weighted context-free grammars

We consider two models for the sequence labeling (tagging) problem. The first one is a {\em Pattern-Based Conditional Random Field }(\PB), in which the energy of a string (chain labeling) $x=x_1\ldots x_n\in D^n$ is a sum of terms over intervals $[i,j]$ where each term is non-zero only if the substring $x_i\ldots x_j$ equals a prespecified word $w\in Λ$. The second model is a {\em Weighted Context-Free Grammar }(\WCFG) frequently used for natural language processing. \PB and \WCFG encode local and non-local interactions respectively, and thus can be viewed as complementary. We propose a {\em Grammatical Pattern-Based CRF model }(\GPB) that combines the two in a natural way. We argue that it has certain advantages over existing approaches such as the {\em Hybrid model} of Bened{í} and Sanchez that combines {\em $\mbox{$N$-grams}$} and \WCFGs. The focus of this paper is to analyze the complexity of inference tasks in a \GPB such as computing MAP. We present a polynomial-time algorithm for general \GPBs and a faster version for a special case that we call {\em Interaction Grammars}.

cs.FL↗
arXiv2012

Extensions of the Minimum Cost Homomorphism Problem

Assume $D$ is a finite set and $R$ is a finite set of functions from $D$ to the natural numbers. An instance of the minimum $R$-cost homomorphism problem ($MinHom_R$) is a set of variables $V$ subject to specified constraints together with a positive weight $c_{vr}$ for each combination of $v \in V$ and $r \in R$. The aim is to find a function $f:V \rightarrow D$ such that $f$ satisfies all constraints and $\sum_{v \in V} \sum_{r \in R} c_{vr}r(f(v))$ is minimized. This problem unifies well-known optimization problems such as the minimum cost homomorphism problem and the maximum solution problem, and this makes it a computationally interesting fragment of the valued CSP framework for optimization problems. We parameterize $MinHom_R\left(Γ\right)$ by {\em constraint languages}, i.e. sets $Γ$ of relations that are allowed in constraints. A constraint language is called {\em conservative} if every unary relation is a member of it; such constraint languages play an important role in understanding the structure of constraint problems. The dichotomy conjecture for $MinHom_R$ is the following statement: if $Γ$ is a constraint language, then $MinHom_R\left(Γ\right)$ is either polynomial-time solvable or NP-complete. For $MinHom$ the dichotomy result has been recently obtained [Takhanov, STACS, 2010] and the goal of this paper is to expand this result to the case of $MinHom_R$ with conservative constraint language. For arbitrary $R$ this problem is still open, but assuming certain restrictions on $R$ we prove a dichotomy. As a consequence of this result we obtain a dichotomy for the conservative maximum solution problem.

cs.CC↗
arXiv2010

A Dichotomy Theorem for General Minimum Cost Homomorphism Problem

In the constraint satisfaction problem ($CSP$), the aim is to find an assignment of values to a set of variables subject to specified constraints. In the minimum cost homomorphism problem ($MinHom$), one is additionally given weights $c_{va}$ for every variable $v$ and value $a$, and the aim is to find an assignment $f$ to the variables that minimizes $\sum_{v} c_{vf(v)}$. Let $MinHom(Γ)$ denote the $MinHom$ problem parameterized by the set of predicates allowed for constraints. $MinHom(Γ)$ is related to many well-studied combinatorial optimization problems, and concrete applications can be found in, for instance, defence logistics and machine learning. We show that $MinHom(Γ)$ can be studied by using algebraic methods similar to those used for CSPs. With the aid of algebraic techniques, we classify the computational complexity of $MinHom(Γ)$ for all choices of $Γ$. Our result settles a general dichotomy conjecture previously resolved only for certain classes of directed graphs, [Gutin, Hell, Rafiey, Yeo, European J. of Combinatorics, 2008].

cs.LG↗
arXiv2007

On the monotonization of the training set

We consider the problem of minimal correction of the training set to make it consistent with monotonic constraints. This problem arises during analysis of data sets via techniques that require monotone data. We show that this problem is NP-hard in general and is equivalent to finding a maximal independent set in special orgraphs. Practically important cases of that problem considered in detail. These are the cases when a partial order given on the replies set is a total order or has a dimension 2. We show that the second case can be reduced to maximization of a quadratic convex function on a convex set. For this case we construct an approximate polynomial algorithm based on convex optimization.

cs.LG↗