arXiv subjects

Jüri Lember

Publications and source records attributed to Jüri Lember.

At least 19 recordsLinked to original sources

arXiv2025

Strong consistency of pseudo-likelihood parameter estimator for univariate Gaussian mixture models

We consider a new method for estimating the parameters of univariate Gaussian mixture models. The method relies on a nonparametric density estimator $\hat{f}_n$ (typically a kernel estimator). For every set of Gaussian mixture components, $\hat{f}_n$ is used to find the best set of mixture weights. That set is obtained by minimizing the $L_2$ distance between $\hat{f}_n$ and the Gaussian mixture density with the given component parameters. The densities together with the obtained weights are then plugged in to the likelihood function, resulting in the so-called pseudo-likelihood function. The final parameter estimators are the parameter values that maximize the pseudo-likelihood function together with the corresponding weights. The advantages of the pseudo-likelihood over the full likelihood are: 1) its arguments are the means and variances only, mixture weights are also functions of the means and variances; 2) unlike the likelihood function, it is always bounded above. Thus, the maximizer of the pseudo-likelihood function -- referred to as the pseudo-likelihood estimator -- always exists. In this article, we prove that the pseudo-likelihood estimator is strongly consistent.

math.ST↗

arXiv2025

Branch-and-bound method for calculating Viterbi path in triplet Markov models

We consider a bivariate, possibly non-homogeneous, finite-state Markov chain $(X,U)=\{(X_t,U_t)\}_{t=1}^n$. We are interested in the marginal process $X$, which typically is not a Markov chain. The goal is to find a realization (path) $x=(x_1,\ldots,x_n)$ with maximal probability $P(X=x)$. If $X$ is Markov chain, then such path can be efficiently found using the celebrated Viterbi algorithm. However, when $X$ is not Markovian, identifying the most probable path -- hereafter referred to as the Viterbi path -- becomes computationally expensive. In this paper, we explore the branch-and-bound method for finding Viterbi paths. The method is based on the lower and upper bounds on maximum probability $\max_x P(X=x)$, and the objective of the paper is to exploit the joint Markov property of $(X,Y)$ to calculate possibly good bounds in possibly cheap way. This research is motivated by decoding or segmentation problem in triplet Markov models. A triplet Markov model is trivariate homogeneous Markov process $(X,U,Y)$. In decoding, a realization of one marginal process $Y$ is observed (representing the data), while $X$ and $U$ are latent processes. The process $U$ serves as a nuisance variable, whereas $X$ is the process of primary interest. Decoding refers to estimating the hidden sequence $X$ based solely on the observation $Y$. Conditional on $Y$, the latent processes $(X, U)$ form a non-homogeneous Markov chain. In this context, the Viterbi path corresponds to the maximum a posteriori (MAP) estimate of $X$, making it a natural choice for signal reconstruction.

stat.CO↗

arXiv2024

Equal Requests are Asymptotically Hardest for Data Recovery

In a distributed storage system serving hot data, the data recovery performance becomes important, captured e.g. by the service rate. We give partial evidence for it being hardest to serve a sequence of equal user requests (as in PIR coding regime) both for concrete and random user requests and server contents. We prove that a constant request sequence is locally hardest to serve: If enough copies of each vector are stored in servers, then if a request sequence with all requests equal can be served then we can still serve it if a few requests are changed. For random iid server contents, with number of data symbols constant (for simplicity) and the number of servers growing, we show that the maximum number of user requests we can serve divided by the number of servers we need approaches a limit almost surely. For uniform server contents, we show this limit is 1/2, both for sequences of copies of a fixed request and of any requests, so it is at least as hard to serve equal requests as any requests. For iid requests independent from the uniform server contents the limit is at least 1/2 and equal to 1/2 if requests are all equal to a fixed request almost surely, confirming the same. As a building block, we deduce from a 1952 result of Marshall Hall, Jr. on abelian groups, that any collection of half as many requests as coded symbols in the doubled binary simplex code can be served by this code. This implies the fractional version of the Functional Batch Code Conjecture that allows half-servers.

cs.IT↗

arXiv2022

Hybrid classifiers of pairwise Markov models

The article studies segmentation problem (also known as classification problem) with pairwise Markov models (PMMs). A PMM is a process where the observation process and underlying state sequence form a two-dimensional Markov chain, it is a natural generalization of a hidden Markov model. To demonstrate the richness of the class of PMMs, we examine closer a few examples of rather different types of PMMs: a model for two related Markov chains, a model that allows to model an inhomogeneous Markov chain as a homogeneous one and a semi-Markov model. The segmentation problem assumes that one of the marginal processes is observed and the other one is not, the problem is to estimate the unobserved state path given the observations. The standard state path estimators often used are the so-called Viterbi path (a sequence with maximum state path probability given the observations) or the pointwise maximum a posteriori (PMAP) path (a sequence that maximizes the conditional state probability for given observations pointwise). Both these estimators have their limitations, therefore we derive formulas for calculating the so-called hybrid path estimators which interpolate between the PMAP and Viterbi path. We apply the introduced algorithms to the studied models in order to demonstrate the properties of different segmentation methods, and to illustrate large variation in behaviour of different segmentation methods in different PMMs. The studied examples show that a segmentation method should always be chosen with care by taking into account the particular model of interest.

stat.ME↗

arXiv2021

Regenerativity of Viterbi process for pairwise Markov models

For hidden Markov models one of the most popular estimates of the hidden chain is the Viterbi path -- the path maximising the posterior probability. We consider a more general setting, called the pairwise Markov model (PMM), where the joint process consisting of finite-state hidden process and observation process is assumed to be a Markov chain. It has been recently proven that under some conditions the Viterbi path of the PMM can almost surely be extended to infinity, thereby defining the infinite Viterbi decoding of the observation sequence, called the Viterbi process. This was done by constructing a block of observations, called a barrier, which ensures that the Viterbi path goes trough a given state whenever this block occurs in the observation sequence. In this paper we prove that the joint process consisting of Viterbi process and PMM is regenerative. The proof involves a delicate construction of regeneration times which coincide with the occurrences of barriers. As one possible application of our theory, some results on the asymptotics of the Viterbi training algorithm are derived.

cs.IT↗

arXiv2021

Exponential forgetting of smoothing distributions for pairwise Markov models

We consider a bivariate Markov chain $Z=\{Z_k\}_{k \geq 1}=\{(X_k,Y_k)\}_{k \geq 1}$ taking values on product space ${\cal Z}={\cal X} \times{ \cal Y}$, where ${\cal X}$ is possibly uncountable space and ${\cal Y}=\{1,\ldots, |{\cal Y}|\}$ is a finite state-space. The purpose of the paper is to find sufficient conditions that guarantee the exponential convergence of smoothing, filtering and predictive probabilities: $$\sup_{n\geq t}\|P(Y_{t:\infty}\in \cdot|X_{l:n})-P(Y_{t:\infty}\in \cdot|X_{s:n}) \|_{\rm TV} \leq K_s α^{t}, \quad \mbox{a.s.}$$ Here $t\geq s\geq l\geq 1$, $K_s$ is $σ(X_{s:\infty})$-measurable finite random variable and $α\in (0,1)$ is fixed. In the second part of the paper, we establish two-sided versions of the above-mentioned convergence. We show that the desired convergences hold under fairly general conditions. A special case of above-mentioned very general model is popular hidden Markov model (HMM). We prove that in HMM-case, our assumptions are more general than all similar mixing-type of conditions encountered in practice, yet relatively easy to verify.

math.PR↗

arXiv2020

An evolutionary model that satisfies detailed balance

We propose a class of evolutionary models that involves an arbitrary exchangeable process as the breeding process and different selection schemes. In those models, a new genome is born according to the breeding process, and then a genome is removed according to the selection scheme that involves fitness. Thus the population size remains constant. The process evolves according to a Markov chain, and, unlike in many other existing models, the stationary distribution -- so called mutation-selection equilibrium -- can be easily found and studied. The behaviour of the stationary distribution when the population size increases is our main object of interest. Several phase-transition theorems are proved.

cs.NE↗

arXiv2020

Estimating the logarithm of characteristic function and stability parameter for symmetric stable laws

Let $X_1,\ldots,X_n$ be an i.i.d. sample from symmetric stable distribution with stability parameter $α$ and scale parameter $γ$. Let $φ_n$ be the empirical characteristic function. We prove an uniform large deviation inequality: given preciseness $ε>0$ and probability $p\in (0,1)$, there exists universal (depending on $ε$ and $p$ but not depending on $α$ and $γ$) constant $\bar{r}>0$ so that $$P\big(\sup_{u>0:r(u)\leq \bar{r}}|r(u)-\hat{r}(u)|\geq ε\big)\leq p,$$ where $r(u)=(uγ)^α$ and $\hat{r}(u)=-\ln|φ_n(u)|$. As an applications of the result, we show how it can be used in estimation unknown stability parameter $α$.

math.ST↗

arXiv2020

MAP segmentation in Bayesian hidden Markov models: a case study

We consider the problem of estimating the maximum posterior probability (MAP) state sequence for a finite state and finite emission alphabet hidden Markov model (HMM) in the Bayesian setup, where both emission and transition matrices have Dirichlet priors. We study a training set consisting of thousands of protein alignment pairs. The training data is used to set the prior hyperparameters for Bayesian MAP segmentation. Since the Viterbi algorithm is not applicable any more, there is no simple procedure to find the MAP path, and several iterative algorithms are considered and compared. The main goal of the paper is to test the Bayesian setup against the frequentist one, where the parameters of HMM are estimated using the training data.

stat.ML↗

arXiv2019

Estimation of Viterbi path in Bayesian hidden Markov models

The article studies different methods for estimating the Viterbi path in the Bayesian framework. The Viterbi path is an estimate of the underlying state path in hidden Markov models (HMMs), which has a maximum posterior probability (MAP). For an HMM with given parameters, the Viterbi path can be easily found with the Viterbi algorithm. In the Bayesian framework the Viterbi algorithm is not applicable and several iterative methods can be used instead. We introduce a new EM-type algorithm for finding the MAP path and compare it with various other methods for finding the MAP path, including the variational Bayes approach and MCMC methods. Examples with simulated data are used to compare the performance of the methods. The main focus is on non-stochastic iterative methods and our results show that the best of those methods work as well or better than the best MCMC methods. Our results demonstrate that when the primary goal is segmentation, then it is more reasonable to perform segmentation directly by considering the transition and emission parameters as nuisance parameters.

stat.CO↗

arXiv2017

Quantifying the Estimation Error of Principal Components

Principal component analysis is an important pattern recognition and dimensionality reduction tool in many applications. Principal components are computed as eigenvectors of a maximum likelihood covariance $\widehatΣ$ that approximates a population covariance $Σ$, and these eigenvectors are often used to extract structural information about the variables (or attributes) of the studied population. Since PCA is based on the eigendecomposition of the proxy covariance $\widehatΣ$ rather than the ground-truth $Σ$, it is important to understand the approximation error in each individual eigenvector as a function of the number of available samples. The recent results of Kolchinskii and Lounici yield such bounds. In the present paper we sharpen these bounds and show that eigenvectors can often be reconstructed to a required accuracy from a sample of strictly smaller size order.

math.ST↗

arXiv2017

Existence of infinite Viterbi path for pairwise Markov models

For hidden Markov models one of the most popular estimates of the hidden chain is the Viterbi path -- the path maximising the posterior probability. We consider a more general setting, called the pairwise Markov model, where the joint process consisting of finite-state hidden regime and observation process is assumed to be a Markov chain. We prove that under some conditions it is possible to extend the Viterbi path to infinity for almost every observation sequence which in turn enables to define an infinite Viterbi decoding of the observation process, called the Viterbi process. This is done by constructing a block of observations, called a barrier, which ensures that the Viterbi path goes trough a given state whenever this block occurs in the observation sequence.

math.ST↗

arXiv2017

Comparison of hidden Markov chain models and hidden Markov random field models in estimation of computed tomography images

There is an interest to replace computed tomography (CT) images with magnetic resonance (MR) images for a number of diagnostic and therapeutic workflows. In this article, predicting CT images from a number of magnetic resonance imaging (MRI) sequences using regression approach is explored. Two principal areas of application for estimated CT images are dose calculations in MRI-based radiotherapy treatment planning and attenuation correction for positron emission tomography (PET)/MRI. The main purpose of this work is to investigate the performance of hidden Markov (chain) models (HMMs) in comparison to hidden Markov random field (HMRF) models when predicting CT images of head. Our study shows that HMMs have clear advantages over HMRF models in this particular application. Obtained results suggest that HMMs deserve a further study for investigating their potential in modelling applications where the most natural theoretical choice would be the class of HMRF models.

stat.AP↗

arXiv2016

Lower Bounds on the Generalized Central Moments of the Optimal Alignments Score of Random Sequences

We present a general approach to the problem of determining tight asymptotic lower bounds for generalized central moments of the optimal alignment score of two independent sequences of i.i.d. random variables. At first, these are obtained under a main assumption for which sufficient conditions are provided. When the main assumption fails, we nevertheless develop a "uniform approximation" method leading to asymptotic lower bounds. Our general results are then applied to the length of the longest common subsequence of binary strings, in which case asymptotic lower bounds are obtained for the moments and the exponential moments of the optimal score. As a byproduct, a local upper bound on the rate function associated with the length of the longest common subsequences of two binary strings is also obtained.

math.PR↗

arXiv2016

Lower bounds for moments of global scores of pairwise Markov chains

Let $X_1,X_2,\ldots$ and $Y_1,Y_2,\ldots$ be two random sequences so that every random variable takes values in a finite set $\mathbb{A}$. We consider a global similarity score $L_n:=L(X_1,\ldots,X_n;Y_1,\ldots,Y_n)$ that measures the homology (relatedness) of words $(X_1,\ldots,X_n)$ and $(Y_1,\ldots,Y_n)$. A typical example of such score is the length of the longest common subsequence. We study the order of central absolute moment $E|L_n-EL_n|^r$ in the case where two-dimensional process $(X_1,Y_1),(X_2,Y_2),\ldots$ is a Markov chain on $\mathbb{A}\times \mathbb{A}$. This is a very general model involving independent Markov chains, hidden Markov models, Markov switching models and many more. Our main result establishes a general condition that guarantees that $E|L_n-EL_n|^r\asymp n^{r\over 2}$. We also perform simulations indicating the validity of the condition.

math.PR↗

arXiv2015

New Bounds for Permutation Codes in Ulam Metric

New bounds on the cardinality of permutation codes equipped with the Ulam distance are presented. First, an integer-programming upper bound is derived, which improves on the Singleton-type upper bound in the literature for some lengths. Second, several probabilistic lower bounds are developed, which improve on the known lower bounds for large minimum distances. The results of a computer search for permutation codes are also presented.

cs.IT↗

arXiv2014

Optimal alignments of longest common subsequences and their path properties

We investigate the behavior of optimal alignment paths for homologous (related) and independent random sequences. An alignment between two finite sequences is optimal if it corresponds to the longest common subsequence (LCS). We prove the existence of lowest and highest optimal alignments and study their differences. High differences between the extremal alignments imply the high variety of all optimal alignments. We present several simulations indicating that the homologous (having the same common ancestor) sequences have typically the distance between the extremal alignments of much smaller size than independent sequences. In particular, the simulations suggest that for the homologous sequences, the growth of the distance between the extremal alignments is logarithmical. The main theoretical results of the paper prove that (under some assumptions) this is the case, indeed. The paper suggests that the properties of the optimal alignment paths characterize the relatedness of the sequences.

math.ST↗

arXiv2013

On the accuracy of the Viterbi alignment

In a hidden Markov model, the underlying Markov chain is usually hidden. Often, the maximum likelihood alignment (Viterbi alignment) is used as its estimate. Although having the biggest likelihood, the Viterbi alignment can behave very untypically by passing states that are at most unexpected. To avoid such situations, the Viterbi alignment can be modified by forcing it not to pass these states. In this article, an iterative procedure for improving the Viterbi alignment is proposed and studied. The iterative approach is compared with a simple bunch approach where a number of states with low probability are all replaced at the same time. It can be seen that the iterative way of adjusting the Viterbi alignment is more efficient and it has several advantages over the bunch approach. The same iterative algorithm for improving the Viterbi alignment can be used in the case of peeping, that is when it is possible to reveal hidden states. In addition, lower bounds for classification probabilities of the Viterbi alignment under different conditions on the model parameters are studied.

stat.ME↗