Searcharxiv⌕ Search

arXiv subjects

Jarek Duda

Publications and source records attributed to Jarek Duda.

At least 37 records · Page 2Linked to original sources

Time delay multi-feature correlation analysis to extract subtle dependencies from EEG signals

Electroencephalography (EEG) signals are resultants of extremely complex brain activity. Some details of this hidden dynamics might be accessible through e.g. joint distributions $ρ_{Δt}$ of signals of pairs of electrodes shifted by various time delays (lag $Δt$). A standard approach is monitoring a single evaluation of such joint distributions, like Pearson correlation (or mutual information), which turns out relatively uninteresting - as expected, there is usually a small peak for zero delay and nearly symmetric drop with delay. In contrast, such a complex signal might be composed of multiple types of statistical dependencies - this article proposes approach to automatically decompose and extract them. Specifically, we model such joint distributions as polynomials, estimated separately for all considered lag dependencies, then with PCA dimensionality reduction we find the dominant joint density distortion directions $f_v$. This way we get a few lag dependent features $a_i(Δt)$ describing separate dominating statistical dependencies of known contributions: $ρ_{Δt}(y,z)\approx \sum_{i=1}^r a_i(Δt)\, f_{v_i}(y,z)$. Such features complement Pearson correlation, extracting hidden more complex behavior, e.g. with asymmetry which might be related with direction of information transfer, extrema suggesting characteristic delays, or oscillatory behavior suggesting some periodicity. There is also discussed extension of Granger causality to such multi-feature joint density analysis, suggesting e.g. two separate causality waves. While this early article is initial fundamental research, in future it might help e.g. with understanding of cortex hidden dynamics, diagnosis of pathologies like epilepsy, determination of precise electrode position, or building brain-computer interface.

eess.SP↗

Improving SGD convergence by online linear regression of gradients in multiple statistically relevant directions

Deep neural networks are usually trained with stochastic gradient descent (SGD), which minimizes objective function using very rough approximations of gradient, only averaging to the real gradient. Standard approaches like momentum or ADAM only consider a single direction, and do not try to model distance from extremum - neglecting valuable information from calculated sequence of gradients, often stagnating in some suboptimal plateau. Second order methods could exploit these missed opportunities, however, beside suffering from very large cost and numerical instabilities, many of them attract to suboptimal points like saddles due to negligence of signs of curvatures (as eigenvalues of Hessian). Saddle-free Newton method is a rare example of addressing this issue - changes saddle attraction into repulsion, and was shown to provide essential improvement for final value this way. However, it neglects noise while modelling second order behavior, focuses on Krylov subspace for numerical reasons, and requires costly eigendecomposion. Maintaining SFN advantages, there are proposed inexpensive ways for exploiting these opportunities. Second order behavior is linear dependence of first derivative - we can optimally estimate it from sequence of noisy gradients with least square linear regression, in online setting here: with weakening weights of old gradients. Statistically relevant subspace is suggested by PCA of recent noisy gradients - in online setting it can be made by slowly rotating considered directions toward new gradients, gradually replacing old directions with recent statistically relevant. Eigendecomposition can be also performed online: with regularly performed step of QR method to maintain diagonal Hessian. Outside the second order modeled subspace we can simultaneously perform gradient descent.

cs.LG↗

P?=NP as minimization of degree 4 polynomial, integration or Grassmann number problem, and new graph isomorphism problem approaches

While the P vs NP problem is mainly approached form the point of view of discrete mathematics, this paper proposes reformulations into the field of abstract algebra, geometry, fourier analysis and of continuous global optimization - which advanced tools might bring new perspectives and approaches for this question. The first one is equivalence of satisfaction of 3-SAT problem with the question of reaching zero of a nonnegative degree 4 multivariate polynomial (sum of squares), what could be tested from the perspective of algebra by using discriminant. It could be also approached as a continuous global optimization problem inside $[0,1]^n$, for example in physical realizations like adiabatic quantum computers. However, the number of local minima usually grows exponentially. Reducing to degree 2 polynomial plus constraints of being in $\{0,1\}^n$, we get geometric formulations as the question if plane or sphere intersects with $\{0,1\}^n$. There will be also presented some non-standard perspectives for the Subset-Sum, like through convergence of a series, or zeroing of $\int_0^{2π} \prod_i \cos(φk_i) dφ$ fourier-type integral for some natural $k_i$. The last discussed approach is using anti-commuting Grassmann numbers $θ_i$, making $(A \cdot \textrm{diag}(θ_i))^n$ nonzero only if $A$ has a Hamilton cycle. Hence, the P$\ne$NP assumption implies exponential growth of matrix representation of Grassmann numbers. There will be also discussed a looking promising algebraic/geometric approach to the graph isomorphism problem -- tested to successfully distinguish strongly regular graphs with up to 29 vertices.

cs.CC↗

Predicting probability distributions for cancer therapy drug selection optimization

Large variability between cell lines brings a difficult optimization problem of drug selection for cancer therapy. Standard approaches use prediction of value for this purpose, corresponding e.g. to expected value of their distribution. This article shows superiority of working on, predicting the entire probability distributions - proposing basic tools for this purpose. We are mostly interested in the best drug in their batch to be tested - proper optimization of their selection for extreme statistics requires knowledge of the entire probability distributions, which for distributions of drug properties among cell lines often turn out binomial, e.g. depending on corresponding gene. Hence for basic prediction mechanism there is proposed mixture of two Gaussians, trying to predict its weight based on additional information.

q-bio.QM↗

Compression Optimality of Asymmetric Numeral Systems

Compression also known as entropy coding has a rich and long history. However, a recent explosion of multimedia Internet applications (such as teleconferencing and video streaming for instance) renews an interest in fast compression that also squeezes out as much redundancy as possible. In 2009 Jarek Duda invented his asymmetric numeral system (ANS). Apart from a beautiful mathematical structure, it is very efficient and offers compression with a very low residual redundancy. ANS works well for any symbol source statistics. Besides, ANS has become a preferred compression algorithm in the IT industry. However, designing ANS instance requires a random selection of its symbol spread function. Consequently, each ANS instance offers compression with a slightly different compression rate. The paper investigates compression optimality of ANS. It shows that ANS is optimal (i.e. the entropies of encoding and source are equal) for any symbol sources whose probability distribution is described by natural powers of 1/2. We use Markov chains to calculate ANS state probabilities. This allows us to determine ANS compression rate precisely. We present two algorithms for finding ANS instances with high compression rates. The first explores state probability approximations in order to choose ANS instances with better compression rates. The second algorithm is a probabilistic one. It finds ANS instances, whose compression rate can be made as close to the best rate as required. This is done at the expense of the number $θ$ of internal random ``coin'' tosses. The algorithm complexity is ${\cal O}(θL^3)$, where $L$ is the number of ANS states. The complexity can be reduced to ${\cal O}(θL\log{L})$ if we use a fast matrix inversion. If the algorithm is implemented on quantum computer, its complexity becomes ${\cal O}(θ(\log{L})^3)$.

cs.IT↗

Low cost prediction of probability distributions of molecular properties for early virtual screening

While there is a general focus on predictions of values, mathematically more appropriate is prediction of probability distributions: with additional possibilities like prediction of uncertainty, higher moments and quantiles. For the purpose of the computer-aided drug design field, this article applies Hierarchical Correlation Reconstruction approach, previously applied in the analysis of demographic, financial and astronomical data. Instead of a single linear regression to predict values, it uses multiple linear regressions to independently predict multiple moments, finally combining them into predicted probability distribution, here of several ADMET properties based on substructural fingerprint developed by Klekota\&Roth. Discussed application example is inexpensive selection of a percentage of molecules with properties nearly certain to be in a predicted or chosen range during virtual screening. Such an approach can facilitate the interpretation of the results as the predictions characterized by high rate of uncertainty are automatically detected. In addition, for each of the investigated predictive problems, we detected crucial structural features, which should be carefully considered when optimizing compounds towards particular property. The whole methodology developed in the study constitutes therefore a great support for medicinal chemists, as it enable fast rejection of compounds with the lowest potential of desired physicochemical/ADMET characteristic and guides the compound optimization process.

q-bio.BM↗

Encoding of probability distributions for Asymmetric Numeral Systems

Many data compressors regularly encode probability distributions for entropy coding - requiring minimal description length type of optimizations. Canonical prefix/Huffman coding usually just writes lengths of bit sequences, this way approximating probabilities with powers-of-2. Operating on more accurate probabilities usually allows for better compression ratios, and is possible e.g. using arithmetic coding and Asymmetric Numeral Systems family. Especially the multiplication-free tabled variant of the latter (tANS) builds automaton often replacing Huffman coding due to better compression at similar computational cost - e.g. in popular Facebook Zstandard and Apple LZFSE compressors. There is discussed encoding of probability distributions for such applications, especially using Pyramid Vector Quantizer(PVQ)-based approach with deformation, bucket approximation, prefix trees, improving accuracy with additional bits, also tuned symbol spread for tANS.

cs.IT↗

Predicting conditional probability distributions of redshifts of Active Galactic Nuclei using Hierarchical Correlation Reconstruction

While there is a general focus on prediction of values, real data often only allows to predict conditional probability distributions, with capabilities bounded by conditional entropy $H(Y|X)$. If additionally estimating uncertainty, we can treat a predicted value as the center of Gaussian of Laplace distribution - idealization which can be far from complex conditional distributions of real data. This article applies Hierarchical Correlation Reconstruction (HCR) approach to inexpensively predict quite complex conditional probability distributions (e.g. multimodal): by independent MSE estimation of multiple moment-like parameters, which allow to reconstruct the conditional distribution. Using linear regression for this purpose, we get interpretable models: with coefficients describing contributions of features to conditional moments. This article extends on the original approach especially by using Canonical Correlation Analysis (CCA) for feature optimization and l1 "lasso" regularization, focusing on practical problem of prediction of redshift of Active Galactic Nuclei (AGN) based on Fourth Fermi-LAT Data Release 2 (4LAC) dataset.

cs.LG↗

Context binning, model clustering and adaptivity for data compression of genetic data

Rapid growth of genetic databases means huge savings from improvements in their data compression, what requires better inexpensive statistical models. This article proposes automatized optimizations e.g. of Markov-like models, especially context binning and model clustering. While it is popular to just remove low bits of the context, proposed context binning automatically optimizes such reduction as tabled: state=bin[context] determining probability distribution, this way extracting nearly all useful information also from very large contexts, into a relatively small number of states. The second proposed approach: model clustering uses k-means clustering in space of general statistical models, allowing to optimize a few models (as cluster centroids) to be chosen e.g. separately for each read. There are also briefly discussed some adaptivity techniques to include data non-stationarity.

cs.IT↗

Fast optimization of common basis for matrix set through Common Singular Value Decomposition

SVD (singular value decomposition) is one of the basic tools of machine learning, allowing to optimize basis for a given matrix. However, sometimes we have a set of matrices $\{A_k\}_k$ instead, and would like to optimize a single common basis for them: find orthogonal matrices $U$, $V$, such that $\{U^T A_k V\}$ set of matrices is somehow simpler. For example DCT-II is orthonormal basis of functions commonly used in image/video compression - as discussed here, this kind of basis can be quickly automatically optimized for a given dataset. While also discussed gradient descent optimization might be computationally costly, there is proposed CSVD (common SVD): fast general approach based on SVD. Specifically, we choose $U$ as built of eigenvectors of $\sum_i (w_k)^q (A_k A_k^T)^p$ and $V$ of $\sum_k (w_k)^q (A_k^T A_k)^p$, where $w_k$ are their weights, $p,q>0$ are some chosen powers e.g. 1/2, optionally with normalization e.g. $A \to A - rc^T$ where $r_i=\sum_j A_{ij}, c_j =\sum_i A_{ij}$.

cs.LG↗

Diffusion models for atomic scale electron currents in semiconductor, p-n junction

While semiconductor electronics is at heart of modern world, and now uses 5nm or smaller processes of single atoms, it seems there are missing models of actual electron currents in these scales - which could help with more conscious design of future electronics. This article proposes such practical methodology allowing to model approximated electron flows in semiconductor, nonlinear Ohm law in p-n junction, and hopefully more complex systems e.g. built of transistors. It assumes electron hopping between atoms using Maximal Entropy Random Walk based diffusion - chosen accordingly to (Jaynes) maximal entropy principle, this way leading to the same stationary probability density as quantum models. Due to Anderson-like localization in nonhomogeneous lattice of semiconductor, electrons are imprisoned in entopic wells, e.g. requiring to exceed a potential barrier for conductance.

cond-mat.stat-mech↗

Nearly accurate solutions for Ising-like models using Maximal Entropy Random Walk

While one-dimensional Markov processes are well understood, going to higher dimensions there are only a few analytically solved Ising-like models, in practice requiring to use relatively costly, uncontrollable and inaccurate Monte-Carlo methods. There is discussed analytical approach for e.g. $width\times \infty$ approximation of lattice, also exploiting Hammersley-Clifford theorem to generate random Gibbs/Markov field through scanning line-by-line using local statistical model as in lossless image compression. While its conditional distributions could be found with Monte-Carlo methods, there is discussed use of Maximal Entropy Random Walk (MERW) to calculate them from approximation of lattice as infinite in one direction and finite in the remaining. Specifically, in the finite directions there is built alphabet of all patterns, then transition matrix containing energy for all pairs of such patterns is built, from its dominant eigenvector getting probability distribution of pairs of patterns in Boltzmann distribution of their infinite sequences, which can be translated into local statistical model for line-by-line scan. Such inexpensive models, requiring seconds on a laptop for attached implementation and directly providing probability distributions of patterns, were tested for mean entropy and energy per node, getting maximal $\approx 0.02$ error from analytical solution near critical point, which quickly improves to extremely accurate e.g. $\approx 10^{-10}$ error for $J\approx 0.2$.

cs.IT↗

Improving distribution and flexible quantization for DCT coefficients

While it is a common knowledge that AC coefficients of Fourier-related transforms, like DCT-II of JPEG image compression, are from Laplace distribution, there was tested more general EPD (exponential power distribution) $ρ\sim \exp(-(|x-μ|/σ)^κ)$ family, leading to maximum likelihood estimated (MLE) $κ\approx 0.5$ instead of Laplace distribution $κ=1$ - such replacement gives $\approx 0.1$ bits/value mean savings (per pixel for grayscale, up to $3\times$ for RGB). There is also discussed predicting distributions (as $μ, σ, κ$ parameters) for DCT coefficients from already decoded coefficients in the current and neighboring DCT blocks. Predicting values $(μ)$ from neighboring blocks allows to reduce blocking artifacts, also improve compression ratio - for which prediction of uncertainty/width $σ$ alone provides much larger $\approx 0.5$ bits/value mean savings opportunity (often neglected). Especially for such continuous distributions, there is also discussed quantization approach through optimized continuous \emph{quantization density function} $q$, which inverse CDF (cumulative distribution function) $Q$ on regular lattice $\{Q^{-1}((i-1/2)/N):i=1\ldots N\}$ gives quantization nodes - allowing for flexible inexpensive choice of optimized (non-uniform) quantization - of varying size $N$, with rate-distortion control. Optimizing $q$ for distortion alone leads to significant improvement, however, at cost of increased entropy due to more uniform distribution. Optimizing both turns out leading to nearly uniform quantization here, with automatized tail handling.

cs.IT↗

Exploiting context dependence for image compression with upsampling

Image compression with upsampling encodes information to succeedingly increase image resolution, for example by encoding differences in FUIF and JPEG XL. It is useful for progressive decoding, also often can improve compression ratio - both for lossless compression and e.g. DC coefficients of lossy. However, the currently used solutions rather do not exploit context dependence for encoding of such upscaling information. This article discusses simple inexpensive general techniques for this purpose, which allowed to save on average $0.645$ bits/difference (between $0.138$ and $1.489$) for the last upscaling for 48 standard $512\times 512$ grayscale 8 bit images - compared to assumption of fixed Laplace distribution. Using least squares linear regression of context to predict center of Laplace distribution gave on average $0.393$ bits/difference savings. The remaining savings were obtained by additionally predicting width of this Laplace distribution, also using just the least squares linear regression. For RGB images, optimization of color transform alone gave mean $\approx 4.6\%$ size reduction comparing to standard YCrCb if using fixed transform, $\approx 6.3\%$ if optimizing transform individually for each image. Then further mean $\approx 10\%$ reduction was obtained if predicting Laplace parameters based on context. The presented simple inexpensive general methodology can be also used for different types of data like DCT coefficients in lossy image compression.

eess.IV↗

Adaptive exponential power distribution with moving estimator for nonstationary time series

While standard estimation assumes that all datapoints are from probability distribution of the same fixed parameters $θ$, we will focus on maximum likelihood (ML) adaptive estimation for nonstationary time series: separately estimating parameters $θ_T$ for each time $T$ based on the earlier values $(x_t)_{t<T}$ using (exponential) moving ML estimator $θ_T=\arg\max_θl_T$ for $l_T=\sum_{t<T} η^{T-t} \ln(ρ_θ(x_t))$ and some $η\in(0,1]$. Computational cost of such moving estimator is generally much higher as we need to optimize log-likelihood multiple times, however, in many cases it can be made inexpensive thanks to dependencies. We focus on such example: $ρ(x)\propto \exp(-|(x-μ)/σ|^κ/κ)$ exponential power distribution (EPD) family, which covers wide range of tail behavior like Gaussian ($κ=2$) or Laplace ($κ=1$) distribution. It is also convenient for such adaptive estimation of scale parameter $σ$ as its standard ML estimation is $σ^κ$ being average $\|x-μ\|^κ$. By just replacing average with exponential moving average: $(σ_{T+1})^κ=η(σ_T)^κ+(1-η)|x_T-μ|^κ$ we can inexpensively make it adaptive. It is tested on daily log-return series for DJIA companies, leading to essentially better log-likelihoods than standard (static) estimation, with optimal $κ$ tails types varying between companies. Presented general alternative estimation philosophy provides tools which might be useful for building better models for analysis of nonstationary time-series.

stat.ML↗

SGD momentum optimizer with step estimation by online parabola model

In stochastic gradient descent, especially for neural network training, there are currently dominating first order methods: not modeling local distance to minimum. This information required for optimal step size is provided by second order methods, however, they have many difficulties, starting with full Hessian having square of dimension number of coefficients. This article proposes a minimal step from successful first order momentum method toward second order: online parabola modelling in just a single direction: normalized $\hat{v}$ from momentum method. It is done by estimating linear trend of gradients $\vec{g}=\nabla F(\vecθ)$ in $\hat{v}$ direction: such that $g(\vecθ_\bot+θ\hat{v})\approx λ(θ-p)$ for $θ= \vecθ\cdot \hat{v}$, $g= \vec{g}\cdot \hat{v}$, $\vecθ_\bot=\vecθ-θ\hat{v}$. Using linear regression, $λ$, $p$ are MSE estimated by just updating four averages (of $g$, $θ$, $gθ$, $θ^2$) in the considered direction. Exponential moving averages allow here for inexpensive online estimation, weakening contribution of the old gradients. Controlling sign of curvature $λ$, we can repel from saddles in contrast to attraction in standard Newton method. In the remaining directions: not considered in second order model, we can simultaneously perform e.g. gradient descent. There is also discussed its learning rate approximation as $μ=σ_θ/ σ_g$, allowing e.g. for adaptive SGD - with learning rate separately optimized (2nd order) for each parameter.

cs.LG↗

Parametric context adaptive Laplace distribution for multimedia compression

Data compression often subtracts prediction and encodes the difference (residue) e.g. assuming Laplace distribution, for example for images, videos, audio, or numerical data. Its performance is strongly dependent on the proper choice of width (scale parameter) of this parametric distribution, can be improved if optimizing it based on local situation like context. For example in popular LOCO-I \cite{loco} (JPEG-LS) lossless image compressor there is used 3 dimensional context quantized into 365 discrete possibilities treated independently. This article discusses inexpensive approaches for exploiting their dependencies with autoregressive ARCH-like context dependent models for parameters of parametric distribution for residue, also evolving in time for adaptive case. For example tested such 4 or 11 parameter models turned out to provide similar performance as 365 parameter LOCO-I model for 48 tested images. Beside smaller headers, such reduction of number of parameters can lead to better generalization. In contrast to context quantization approaches, parameterized models also allow to directly use higher dimensional contexts, for example using information from all 3 color channels, further pixels, some additional region classifiers, or from interleaving multi-scale scanning - for which there is proposed Haar upscale scan combining advantages of Haar wavelets with possibility of scanning exploiting local contexts.

eess.IV↗

Credibility evaluation of income data with hierarchical correlation reconstruction

In situations like tax declarations or analyzes of household budgets we would like to automatically evaluate credibility of exogenous variable (declared income) based on some available (endogenous) variables - we want to build a model and train it on provided data sample to predict (conditional) probability distribution of exogenous variable based on values of endogenous variables. Using Polish household budget survey data there will be discussed simple and systematic adaptation of hierarchical correlation reconstruction (HCR) technique for this purpose, which allows to combine interpretability of statistics with modelling of complex densities like in machine learning. For credibility evaluation we normalize marginal distribution of predicted variable to $ρ\approx 1$ uniform distribution on $[0,1]$ using empirical distribution function $(x=EDF(y)\in[0,1])$, then model density of its conditional distribution $(\textrm{Pr}(x_0|x_1 x_2\ldots))$ as a linear combination of orthonormal polynomials using coefficients modelled as linear combinations of features of the remaining variables. These coefficients can be calculated independently, have similar interpretation as cumulants, additionally allowing to directly reconstruct probability distribution. Values corresponding to high predicted density can be considered as credible, while low density suggests disagreement with statistics of data sample, for example to mark for manual verification a chosen percentage of data points evaluated as the least credible.

cs.LG↗