arXiv subjects

Richard Nock

Publications and source records attributed to Richard Nock.

At least 73 records · Page 4Linked to original sources

arXiv2016

Fast Learning from Distributed Datasets without Entity Matching

Consider the following data fusion scenario: two datasets/peers contain the same real-world entities described using partially shared features, e.g. banking and insurance company records of the same customer base. Our goal is to learn a classifier in the cross product space of the two domains, in the hard case in which no shared ID is available -- e.g. due to anonymization. Traditionally, the problem is approached by first addressing entity matching and subsequently learning the classifier in a standard manner. We present an end-to-end solution which bypasses matching entities, based on the recently introduced concept of Rademacher observations (rados). Informally, we replace the minimisation of a loss over examples, which requires to solve entity resolution, by the equivalent minimisation of a (different) loss over rados. Among others, key properties we show are (i) a potentially huge subset of these rados does not require to perform entity matching, and (ii) the algorithm that provably minimizes the rado loss over these rados has time and space complexities smaller than the algorithm minimizing the equivalent example loss. Last, we relax a key assumption of the model, that the data is vertically partitioned among peers --- in this case, we would not even know the existence of a solution to entity resolution. In this more general setting, experiments validate the possibility of significantly beating even the optimal peer in hindsight.

cs.LG↗

arXiv2016

Learning Games and Rademacher Observations Losses

It has recently been shown that supervised learning with the popular logistic loss is equivalent to optimizing the exponential loss over sufficient statistics about the class: Rademacher observations (rados). We first show that this unexpected equivalence can actually be generalized to other example / rado losses, with necessary and sufficient conditions for the equivalence, exemplified on four losses that bear popular names in various fields: exponential (boosting), mean-variance (finance), Linear Hinge (on-line learning), ReLU (deep learning), and unhinged (statistics). Second, we show that the generalization unveils a surprising new connection to regularized learning, and in particular a sufficient condition under which regularizing the loss over examples is equivalent to regularizing the rados (with Minkowski sums) in the equivalent rado loss. This brings simple and powerful rado-based learning algorithms for sparsity-controlling regularization, that we exemplify on a boosting algorithm for the regularized exponential rado-loss, which formally boosts over four types of regularization, including the popular ridge and lasso, and the recently coined slope --- we obtain the first proven boosting algorithm for this last regularization. Through our first contribution on the equivalence of rado and example-based losses, Omega-R.AdaBoost~appears to be an efficient proxy to boost the regularized logistic loss over examples using whichever of the four regularizers. Experiments display that regularization consistently improves performances of rado-based learning, and may challenge or beat the state of the art of example-based learning even when learning over small sets of rados. Finally, we connect regularization to differential privacy, and display how tiny budgets can be afforded on big domains while beating (protected) example-based learning.

cs.LG↗

arXiv2016

k-variates++: more pluses in the k-means++

k-means++ seeding has become a de facto standard for hard clustering algorithms. In this paper, our first contribution is a two-way generalisation of this seeding, k-variates++, that includes the sampling of general densities rather than just a discrete set of Dirac densities anchored at the point locations, and a generalisation of the well known Arthur-Vassilvitskii (AV) approximation guarantee, in the form of a bias+variance approximation bound of the global optimum. This approximation exhibits a reduced dependency on the "noise" component with respect to the optimal potential --- actually approaching the statistical lower bound. We show that k-variates++ reduces to efficient (biased seeding) clustering algorithms tailored to specific frameworks; these include distributed, streaming and on-line clustering, with direct approximation results for these algorithms. Finally, we present a novel application of k-variates++ to differential privacy. For either the specific frameworks considered here, or for the differential privacy setting, there is little to no prior results on the direct application of k-means++ and its approximation bounds --- state of the art contenders appear to be significantly more complex and / or display less favorable (approximation) properties. We stress that our algorithms can still be run in cases where there is \textit{no} closed form solution for the population minimizer. We demonstrate the applicability of our analysis via experimental evaluation on several domains and settings, displaying competitive performances vs state of the art.

cs.LG↗

arXiv2016

Loss factorization, weakly supervised learning and label noise robustness

We prove that the empirical risk of most well-known loss functions factors into a linear term aggregating all labels with a term that is label free, and can further be expressed by sums of the loss. This holds true even for non-smooth, non-convex losses and in any RKHS. The first term is a (kernel) mean operator --the focal quantity of this work-- which we characterize as the sufficient statistic for the labels. The result tightens known generalization bounds and sheds new light on their interpretation. Factorization has a direct application on weakly supervised learning. In particular, we demonstrate that algorithms like SGD and proximal methods can be adapted with minimal effort to handle weak supervision, once the mean operator has been estimated. We apply this idea to learning with asymmetric noisy labels, connecting and extending prior work. Furthermore, we show that most losses enjoy a data-dependent (by the mean operator) form of noise robustness, in contrast with known negative results.

cs.LG↗

arXiv2015

On conformal divergences and their population minimizers

Total Bregman divergences are a recent tweak of ordinary Bregman divergences originally motivated by applications that required invariance by rotations. They have displayed superior results compared to ordinary Bregman divergences on several clustering, computer vision, medical imaging and machine learning tasks. These preliminary results raise two important problems : First, report a complete characterization of the left and right population minimizers for this class of total Bregman divergences. Second, characterize a principled superset of total and ordinary Bregman divergences with good clustering properties, from which one could tailor the choice of a divergence to a particular application. In this paper, we provide and study one such superset with interesting geometric features, that we call conformal divergences, and focus on their left and right population minimizers. Our results are obtained in a recently coined $(u, v)$-geometric structure that is a generalization of the dually flat affine connections in information geometry. We characterize both analytically and geometrically the population minimizers. We prove that conformal divergences (resp. total Bregman divergences) are essentially exhaustive for their left (resp. right) population minimizers. We further report new results and extend previous results on the robustness to outliers of the left and right population minimizers, and discuss the role of the $(u, v)$-geometric structure in clustering. Additional results are also given.

cs.IT↗

arXiv2015

Rademacher Observations, Private Data, and Boosting

The minimization of the logistic loss is a popular approach to batch supervised learning. Our paper starts from the surprising observation that, when fitting linear (or kernelized) classifiers, the minimization of the logistic loss is \textit{equivalent} to the minimization of an exponential \textit{rado}-loss computed (i) over transformed data that we call Rademacher observations (rados), and (ii) over the \textit{same} classifier as the one of the logistic loss. Thus, a classifier learnt from rados can be \textit{directly} used to classify \textit{observations}. We provide a learning algorithm over rados with boosting-compliant convergence rates on the \textit{logistic loss} (computed over examples). Experiments on domains with up to millions of examples, backed up by theoretical arguments, display that learning over a small set of random rados can challenge the state of the art that learns over the \textit{complete} set of examples. We show that rados comply with various privacy requirements that make them good candidates for machine learning in a privacy framework. We give several algebraic, geometric and computational hardness results on reconstructing examples from rados. We also show how it is possible to craft, and efficiently learn from, rados in a differential privacy framework. Tests reveal that learning from differentially private rados can compete with learning from random rados, and hence with batch learning from examples, achieving non-trivial privacy vs accuracy tradeoffs.

cs.LG↗

arXiv2014

Further results on the hyperbolic Voronoi diagrams

In Euclidean geometry, it is well-known that the $k$-order Voronoi diagram in $\mathbb{R}^d$ can be computed from the vertical projection of the $k$-level of an arrangement of hyperplanes tangent to a convex potential function in $\mathbb{R}^{d+1}$: the paraboloid. Similarly, we report for the Klein ball model of hyperbolic geometry such a {\em concave} potential function: the northern hemisphere. Furthermore, we also show how to build the hyperbolic $k$-order diagrams as equivalent clipped power diagrams in $\mathbb{R}^d$. We investigate the hyperbolic Voronoi diagram in the hyperboloid model and show how it reduces to a Klein-type model using central projections.

cs.CG↗

arXiv2014

Further heuristics for $k$-means: The merge-and-split heuristic and the $(k,l)$-means

Finding the optimal $k$-means clustering is NP-hard in general and many heuristics have been designed for minimizing monotonically the $k$-means objective. We first show how to extend Lloyd's batched relocation heuristic and Hartigan's single-point relocation heuristic to take into account empty-cluster and single-point cluster events, respectively. Those events tend to increasingly occur when $k$ or $d$ increases, or when performing several restarts. First, we show that those special events are a blessing because they allow to partially re-seed some cluster centers while further minimizing the $k$-means objective function. Second, we describe a novel heuristic, merge-and-split $k$-means, that consists in merging two clusters and splitting this merged cluster again with two new centers provided it improves the $k$-means objective. This novel heuristic can improve Hartigan's $k$-means when it has converged to a local minimum. We show empirically that this merge-and-split $k$-means improves over the Hartigan's heuristic which is the {\em de facto} method of choice. Finally, we propose the $(k,l)$-means objective that generalizes the $k$-means objective by associating the data points to their $l$ closest cluster centers, and show how to either directly convert or iteratively relax the $(k,l)$-means into a $k$-means in order to reach better local minima.

cs.LG↗

arXiv2014

Optimal interval clustering: Application to Bregman clustering and statistical mixture learning

We present a generic dynamic programming method to compute the optimal clustering of $n$ scalar elements into $k$ pairwise disjoint intervals. This case includes 1D Euclidean $k$-means, $k$-medoids, $k$-medians, $k$-centers, etc. We extend the method to incorporate cluster size constraints and show how to choose the appropriate $k$ by model selection. Finally, we illustrate and refine the method on two case studies: Bregman clustering and statistical mixture learning maximizing the complete likelihood.

cs.IT↗

arXiv2013

Total Jensen divergences: Definition, Properties and k-Means++ Clustering

We present a novel class of divergences induced by a smooth convex function called total Jensen divergences. Those total Jensen divergences are invariant by construction to rotations, a feature yielding regularization of ordinary Jensen divergences by a conformal factor. We analyze the relationships between this novel class of total Jensen divergences and the recently introduced total Bregman divergences. We then proceed by defining the total Jensen centroids as average distortion minimizers, and study their robustness performance to outliers. Finally, we prove that the k-means++ initialization that bypasses explicit centroid computations is good enough in practice to guarantee probabilistically a constant approximation factor to the optimal k-means clustering.

cs.IT↗

arXiv2013

On the Chi square and higher-order Chi distances for approximating f-divergences

We report closed-form formula for calculating the Chi square and higher-order Chi distances between statistical distributions belonging to the same exponential family with affine natural space, and instantiate those formula for the Poisson and isotropic Gaussian families. We then describe an analytic formula for the $f$-divergences based on Taylor expansions and relying on an extended class of Chi-type distances.

cs.IT↗

arXiv2013

Combining Feature and Prototype Pruning by Uncertainty Minimization

We focus in this paper on dataset reduction techniques for use in k-nearest neighbor classification. In such a context, feature and prototype selections have always been independently treated by the standard storage reduction algorithms. While this certifying is theoretically justified by the fact that each subproblem is NP-hard, we assume in this paper that a joint storage reduction is in fact more intuitive and can in practice provide better results than two independent processes. Moreover, it avoids a lot of distance calculations by progressively removing useless instances during the feature pruning. While standard selection algorithms often optimize the accuracy to discriminate the set of solutions, we use in this paper a criterion based on an uncertainty measure within a nearest-neighbor graph. This choice comes from recent results that have proven that accuracy is not always the suitable criterion to optimize. In our approach, a feature or an instance is removed if its deletion improves information of the graph. Numerous experiments are presented in this paper and a statistical analysis shows the relevance of our approach, and its tolerance in the presence of noise.

cs.LG↗

arXiv2011

On Rényi and Tsallis entropies and divergences for exponential families

Many common probability distributions in statistics like the Gaussian, multinomial, Beta or Gamma distributions can be studied under the unified framework of exponential families. In this paper, we prove that both Rényi and Tsallis divergences of distributions belonging to the same exponential family admit a generic closed form expression. Furthermore, we show that Rényi and Tsallis entropies can also be calculated in closed-form for sub-families including the Gaussian or exponential distributions, among others.

cs.IT↗

arXiv2010

Boosting k-NN for categorization of natural scenes

The k-nearest neighbors (k-NN) classification rule has proven extremely successful in countless many computer vision applications. For example, image categorization often relies on uniform voting among the nearest prototypes in the space of descriptors. In spite of its good properties, the classic k-NN rule suffers from high variance when dealing with sparse prototype datasets in high dimensions. A few techniques have been proposed to improve k-NN classification, which rely on either deforming the nearest neighborhood relationship or modifying the input space. In this paper, we propose a novel boosting algorithm, called UNN (Universal Nearest Neighbors), which induces leveraged k-NN, thus generalizing the classic k-NN rule. We redefine the voting rule as a strong classifier that linearly combines predictions from the k closest prototypes. Weak classifiers are learned by UNN so as to minimize a surrogate risk. A major feature of UNN is the ability to learn which prototypes are the most relevant for a given class, thus allowing one for effective data reduction. Experimental results on the synthetic two-class dataset of Ripley show that such a filtering strategy is able to reject "noisy" prototypes. We carried out image categorization experiments on a database containing eight classes of natural scenes. We show that our method outperforms significantly the classic k-NN classification, while enabling significant reduction of the computational cost by means of data filtering.

cs.CV↗

arXiv2009

Hyperbolic Voronoi diagrams made easy

We present a simple framework to compute hyperbolic Voronoi diagrams of finite point sets as affine diagrams. We prove that bisectors in Klein's non-conformal disk model are hyperplanes that can be interpreted as power bisectors of Euclidean balls. Therefore our method simply consists in computing an equivalent clipped power diagram followed by a mapping transformation depending on the selected representation of the hyperbolic space (e.g., Poincaré conformal disk or upper-plane representations). We discuss on extensions of this approach to weighted and $k$-order diagrams, and describe their dual triangulations. Finally, we consider two useful primitives on the hyperbolic Voronoi diagrams for designing tailored user interfaces of an image catalog browsing application in the hyperbolic disk: (1) finding nearest neighbors, and (2) computing smallest enclosing balls.

cs.CG↗

arXiv2009

Information geometries and Microeconomic Theories

More than thirty years ago, Charnes, Cooper and Schinnar (1976) established an enlightening contact between economic production functions (EPFs) -- a cornerstone of neoclassical economics -- and information theory, showing how a generalization of the Cobb-Douglas production function encodes homogeneous functions. As expected by Charnes \textit{et al.}, the contact turns out to be much broader: we show how information geometry as pioneered by Amari and others underpins static and dynamic descriptions of microeconomic cornerstones. We show that the most popular EPFs are fundamentally grounded in a very weak axiomatization of economic transition costs between inputs. The strength of this characterization is surprising, as it geometrically bonds altogether a wealth of collateral economic notions -- advocating for applications in various economic fields --: among all, it characterizes (i) Marshallian and Hicksian demands and their geometric duality, (ii) Slutsky-type properties for the transformation paths, (iii) Roy-type properties for their elementary variations.

q-fin.GN↗

arXiv2008

Analyse spectrale des textes: détection automatique des frontières de langue et de discours

We propose a theoretical framework within which information on the vocabulary of a given corpus can be inferred on the basis of statistical information gathered on that corpus. Inferences can be made on the categories of the words in the vocabulary, and on their syntactical properties within particular languages. Based on the same statistical data, it is possible to build matrices of syntagmatic similarity (bigram transition matrices) or paradigmatic similarity (probability for any pair of words to share common contexts). When clustered with respect to their syntagmatic similarity, words tend to group into sublanguage vocabularies, and when clustered with respect to their paradigmatic similarity, into syntactic or semantic classes. Experiments have explored the first of these two possibilities. Their results are interpreted in the frame of a Markov chain modelling of the corpus' generative processe(s): we show that the results of a spectral analysis of the transition matrix can be interpreted as probability distributions of words within clusters. This method yields a soft clustering of the vocabulary into sublanguages which contribute to the generation of heterogeneous corpora. As an application, we show how multilingual texts can be visually segmented into linguistically homogeneous segments. Our method is specifically useful in the case of related languages which happened to be mixed in corpora.

cs.CL↗

arXiv2008

Soft Uncoupling of Markov Chains for Permeable Language Distinction: A New Algorithm

Without prior knowledge, distinguishing different languages may be a hard task, especially when their borders are permeable. We develop an extension of spectral clustering -- a powerful unsupervised classification toolbox -- that is shown to resolve accurately the task of soft language distinction. At the heart of our approach, we replace the usual hard membership assignment of spectral clustering by a soft, probabilistic assignment, which also presents the advantage to bypass a well-known complexity bottleneck of the method. Furthermore, our approach relies on a novel, convenient construction of a Markov chain out of a corpus. Extensive experiments with a readily available system clearly display the potential of the method, which brings a visually appealing soft distinction of languages that may define altogether a whole corpus.

cs.CL↗