arXiv subjects

Jacek Tabor

Publications and source records attributed to Jacek Tabor.

At least 127 records · Page 7Linked to original sources

arXiv2017

Efficient mixture model for clustering of sparse high dimensional binary data

In this paper we propose a mixture model, SparseMix, for clustering of sparse high dimensional binary data, which connects model-based with centroid-based clustering. Every group is described by a representative and a probability distribution modeling dispersion from this representative. In contrast to classical mixture models based on EM algorithm, SparseMix: -is especially designed for the processing of sparse data, -can be efficiently realized by an on-line Hartigan optimization algorithm, -is able to automatically reduce unnecessary clusters. We perform extensive experimental studies on various types of data, which confirm that SparseMix builds partitions with higher compatibility with reference grouping than related methods. Moreover, constructed representatives often better reveal the internal structure of data.

cs.LG↗

arXiv2017

Semi-supervised model-based clustering with controlled clusters leakage

In this paper, we focus on finding clusters in partially categorized data sets. We propose a semi-supervised version of Gaussian mixture model, called C3L, which retrieves natural subgroups of given categories. In contrast to other semi-supervised models, C3L is parametrized by user-defined leakage level, which controls maximal inconsistency between initial categorization and resulting clustering. Our method can be implemented as a module in practical expert systems to detect clusters, which combine expert knowledge with true distribution of data. Moreover, it can be used for improving the results of less flexible clustering techniques, such as projection pursuit clustering. The paper presents extensive theoretical analysis of the model and fast algorithm for its efficient optimization. Experimental results show that C3L finds high quality clustering model, which can be applied in discovering meaningful groups in partially classified data.

cs.LG↗

arXiv2017

Spherical Wards clustering and generalized Voronoi diagrams

Gaussian mixture model is very useful in many practical problems. Nevertheless, it cannot be directly generalized to non Euclidean spaces. To overcome this problem we present a spherical Gaussian-based clustering approach for partitioning data sets with respect to arbitrary dissimilarity measure. The proposed method is a combination of spherical Cross-Entropy Clustering with a generalized Wards approach. The algorithm finds the optimal number of clusters by automatically removing groups which carry no information. Moreover, it is scale invariant and allows for forming of spherically-shaped clusters of arbitrary sizes. In order to graphically represent and interpret the results the notion of Voronoi diagram was generalized to non Euclidean spaces and applied for introduced clustering method.

cs.LG↗

arXiv2017

Generalized RBF kernel for incomplete data

We construct $\bf genRBF$ kernel, which generalizes the classical Gaussian RBF kernel to the case of incomplete data. We model the uncertainty contained in missing attributes making use of data distribution and associate every point with a conditional probability density function. This allows to embed incomplete data into the function space and to define a kernel between two missing data points based on scalar product in $L_2$. Experiments show that introduced kernel applied to SVM classifier gives better results than other state-of-the-art methods, especially in the case when large number of features is missing. Moreover, it is easy to implement and can be used together with any kernel approaches with no additional modifications.

cs.LG↗

arXiv2017

Pointed subspace approach to incomplete data

Incomplete data are often represented as vectors with filled missing attributes joined with flag vectors indicating missing components. In this paper we generalize this approach and represent incomplete data as pointed affine subspaces. This allows to perform various affine transformations of data, as whitening or dimensionality reduction. We embed such generalized missing data into a vector space by mapping pointed affine subspace (generalized missing data point) to a vector containing imputed values joined with a corresponding projection matrix. Such an operation preserves the scalar product of the embedding defined for flag vectors and allows to input transformed incomplete data to typical classification methods.

cs.LG↗

arXiv2017

ICA based on the data asymmetry

Independent Component Analysis (ICA) - one of the basic tools in data analysis - aims to find a coordinate system in which the components of the data are independent. Most of existing methods are based on the minimization of the function of fourth-order moment (kurtosis). Skewness (third-order moment) has received much less attention. In this paper we present a competitive approach to ICA based on the Split Gaussian distribution, which is well adapted to asymmetric data. Consequently, we obtain a method which works better than the classical approaches, especially in the case when the underlying density is not symmetric, which is a typical situation in the color distribution in images.

math.ST↗

arXiv2015

Introduction to Cross-Entropy Clustering The R Package CEC

The R Package CEC performs clustering based on the cross-entropy clustering (CEC) method, which was recently developed with the use of information theory. The main advantage of CEC is that it combines the speed and simplicity of $k$-means with the ability to use various Gaussian mixture models and reduce unnecessary clusters. In this work we present a practical tutorial to CEC based on the R Package CEC. Functions are provided to encompass the whole process of clustering.

cs.LG↗

arXiv2015

On rigorous estimates of eigenspaces and eigenvalues of a matrix

We present a method of cones for rigorous estimations of eigenvectors, eigenspaces and eigenvalues of a matrix. The key notion is the cone-domination and is inspired by ideas from hyperbolic dynamical systems. We present theorems which allow to rigorously locate the spectrum of the matrix and the eigenspaces, also multidimensional ones in case of eigenvalues of multiplicity greater than one or clusters of close eigenvalues. In case of isolated eigenvalue we show that the our method give the same or better estimates than ones known in literature.

math.DS↗

arXiv2015

Maximum Entropy Linear Manifold for Learning Discriminative Low-dimensional Representation

Representation learning is currently a very hot topic in modern machine learning, mostly due to the great success of the deep learning methods. In particular low-dimensional representation which discriminates classes can not only enhance the classification procedure, but also make it faster, while contrary to the high-dimensional embeddings can be efficiently used for visual based exploratory data analysis. In this paper we propose Maximum Entropy Linear Manifold (MELM), a multidimensional generalization of Multithreshold Entropy Linear Classifier model which is able to find a low-dimensional linear data projection maximizing discriminativeness of projected classes. As a result we obtain a linear embedding which can be used for classification, class aware dimensionality reduction and data visualization. MELM provides highly discriminative 2D projections of the data which can be used as a method for constructing robust classifiers. We provide both empirical evaluation as well as some interesting theoretical properties of our objective function such us scale and affine transformation invariance, connections with PCA and bounding of the expected balanced accuracy error.

cs.LG↗

arXiv2015

Extreme Entropy Machines: Robust information theoretic classification

Most of the existing classification methods are aimed at minimization of empirical risk (through some simple point-based error measured with loss function) with added regularization. We propose to approach this problem in a more information theoretic way by investigating applicability of entropy measures as a classification model objective function. We focus on quadratic Renyi's entropy and connected Cauchy-Schwarz Divergence which leads to the construction of Extreme Entropy Machines (EEM). The main contribution of this paper is proposing a model based on the information theoretic concepts which on the one hand shows new, entropic perspective on known linear classifiers and on the other leads to a construction of very robust method competetitive with the state of the art non-information theoretic ones (including Support Vector Machines and Extreme Learning Machines). Evaluation on numerous problems spanning from small, simple ones from UCI repository to the large (hundreads of thousands of samples) extremely unbalanced (up to 100:1 classes' ratios) datasets shows wide applicability of the EEM in real life problems and that it scales well.

cs.LG↗

arXiv2014

Cluster based RBF Kernel for Support Vector Machines

In the classical Gaussian SVM classification we use the feature space projection transforming points to normal distributions with fixed covariance matrices (identity in the standard RBF and the covariance of the whole dataset in Mahalanobis RBF). In this paper we add additional information to Gaussian SVM by considering local geometry-dependent feature space projection. We emphasize that our approach is in fact an algorithm for a construction of the new Gaussian-type kernel. We show that better (compared to standard RBF and Mahalanobis RBF) classification results are obtained in the simple case when the space is preliminary divided by k-means into two sets and points are represented as normal distributions with a covariances calculated according to the dataset partitioning. We call the constructed method C$_k$RBF, where $k$ stands for the amount of clusters used in k-means. We show empirically on nine datasets from UCI repository that C$_2$RBF increases the stability of the grid search (measured as the probability of finding good parameters).

cs.LG↗

arXiv2014

Multithreshold Entropy Linear Classifier

Linear classifiers separate the data with a hyperplane. In this paper we focus on the novel method of construction of multithreshold linear classifier, which separates the data with multiple parallel hyperplanes. Proposed model is based on the information theory concepts -- namely Renyi's quadratic entropy and Cauchy-Schwarz divergence. We begin with some general properties, including data scale invariance. Then we prove that our method is a multithreshold large margin classifier, which shows the analogy to the SVM, while in the same time works with much broader class of hypotheses. What is also interesting, proposed method is aimed at the maximization of the balanced quality measure (such as Matthew's Correlation Coefficient) as opposed to very common maximization of the accuracy. This feature comes directly from the optimization problem statement and is further confirmed by the experiments on the UCI datasets. It appears, that our Multithreshold Entropy Linear Classifier (MELC) obtaines similar or higher scores than the ones given by SVM on both synthetic and real data. We show how proposed approach can be benefitial for the cheminformatics in the task of ligands activity prediction, where despite better classification results, MELC gives some additional insight into the data structure (classes of underrepresented chemical compunds).

cs.LG↗

arXiv2013

Optimal Rescaling and the Mahalanobis Distance

One of the basic problems in data analysis lies in choosing the optimal rescaling (change of coordinate system) to study properties of a given data-set $Y$. The classical Mahalanobis approach has its basis in the classical normalization/rescaling formula $Y \ni y \to Σ_Y^{-1/2} \cdot (y-\mathrm{m}_Y)$, where $\mathrm{m}_Y$ denotes the mean of $Y$ and $Σ_Y$ the covariance matrix . Based on the cross-entropy we generalize this approach and define the parameter which measures the fit of a given affine rescaling of $Y$ compared to the Mahalanobis one. This allows in particular to find an optimal change of coordinate system which satisfies some additional conditions. In particular we show that in the case when we put origin of coordinate system in $ \mathrm{m} $ the optimal choice is given by the transformation $Y \ni y \to Σ_Y^{-1/2} \cdot (y-\mathrm{m}_Y)$, where $$ Σ=Σ_Y(Σ_Y-\frac{(\mathrm{m}-\mathrm{m}_Y)(\mathrm{m}-\mathrm{m}_Y)^T}{1+\|\mathrm{m}-\mathrm{m}_Y\|_{Σ_Y}^2})^{-1}Σ_Y. $$

stat.ME↗

arXiv2012

Cross-Entropy Clustering

We construct a cross-entropy clustering (CEC) theory which finds the optimal number of clusters by automatically removing groups which carry no information. Moreover, our theory gives simple and efficient criterion to verify cluster validity. Although CEC can be build on an arbitrary family of densities, in the most important case of Gaussian CEC: {\em -- the division into clusters is affine invariant; -- the clustering will have the tendency to divide the data into ellipsoid-type shapes; -- the approach is computationally efficient as we can apply Hartigan approach.} We study also with particular attention clustering based on the Spherical Gaussian densities and that of Gaussian densities with covariance $s \I$. In the letter case we show that with $s$ converging to zero we obtain the classical k-means clustering.

cs.IT↗

arXiv2012

Detection of elliptical shapes via cross-entropy clustering

The problem of finding elliptical shapes in an image will be considered. We discuss the solution which uses cross-entropy clustering. The proposed method allows the search for ellipses with predefined sizes and position in the space. Moreover, it works well for search of ellipsoids in higher dimensions.

cs.CV↗

arXiv2012

Strict localization of eigenvectors and eigenvalues

In this article we show and implement a simple and effcient method to strictly locate eigenvectors and eigenvalues of a given matrix, based on the modified cone condition. As a consequence we can also effectively localize zeros of complex polynomials.

math.NA↗

arXiv2012

Weighted Approach to Rényi Entropy

Rényi entropy of order αis a general measure of entropy. In this paper we derive estimations for the Rényi entropy of the mixture of sources in terms of the entropy of the single sources. These relations allow to compute the Rényi entropy dimension of arbitrary order of a mixture of measures. The key for obtaining these results is our new definition of the weighted Rényi entropy. It is shown that weighted entropy is equal to the classical Rényi entropy.

cs.IT↗

arXiv2012

The memory centre

Let $x \in \R$ be given. As we know the, amount of bits needed to binary code $x$ with given accuracy ($h \in \R$) is approximately $ \m_{h}(x) \approx \log_{2}(\max {1, |\frac{x}{h}|}). $ We consider the problem where we should translate the origin $a$ so that the mean amount of bits needed to code randomly chosen element from a realization of a random variable $X$ is minimal. In other words, we want to find $a \in \R$ such that $$ \R \ni a \to \mathrm{E} (\m_{h} (X-a)) $$ attains minimum.

cs.IT↗