arXiv subjects

Václav Šmídl

Publications and source records attributed to Václav Šmídl.

At least 19 recordsLinked to original sources

arXiv2026

Exploiting Gradients in Bayesian Inference of Expensive Simulators

Simulators based on differential equations are ubiquitous in science and engineering. They are often used in simulation-based inference to evaluate the posterior distribution of the input parameters based on real-world observations of the simulator outputs. However, inference becomes challenging when individual simulator evaluations are computationally expensive. In such cases, a Bayesian optimization-based active learning approach with Gaussian process surrogate models has been used to maximize the information obtained from a limited simulation budget. Recently, gradients of simulator outputs with respect to input parameters have become increasingly available, yet they are rarely exploited for inference. Even though we only need to learn the simulator input-output relationship, gradient information can provide an additional valuable signal to guide the active learning procedure. This is of particular interest in the case of expensive simulators, when sample efficiency is crucial. In this paper, we demonstrate how incorporating gradient information into the Gaussian process surrogate accelerates Bayesian optimization-based inference under a limited simulation budget. Our results show significant improvement in convergence speed from using gradient information. For reverse-mode differentiation, the inference efficiency gains are maintained when accounting for the additional computational cost. In contrast, for forward-mode differentiation, the inference speed-up does not outweigh the computational costs. These results indicate that gradient-enhanced surrogates are beneficial primarily in problems where the number of parameters exceeds the output dimensionality, where reverse-mode differentiation is efficient.

cs.LG↗

arXiv2026

GEMSS: A Variational Method for Discovering Multiple Sparse Solutions in Classification and Regression Problems

In underdetermined regression and classification problems, multiple feature subsets often yield equivalent predictive performance. In applied settings, especially with $n \ll p$, high dimension or collinearities, it is valuable to provide a domain expert with a menu of statistically plausible explanations, rather than one arbitrary solution. This creates the need for appropriate methods. We present Gaussian Ensemble for Multiple Sparse Solutions (GEMSS), a method that uses a single variational mixture to approximate the corresponding multimodal posterior. Its evidence lower bound contains a built-in repulsion between the mixture's components, enabling the model to simultaneously produce several distinct sparse solutions. We evaluate GEMSS on a novel, reusable benchmark. The ground-truth solution set and its structure are known by construction and set-level recovery metrics are evaluated. GEMSS consistently outperforms dedicated multiplicity methods (Enumeration LASSO, ALFESE), two strong sampling baselines that approximate the same posterior (Randomized-LASSO ensemble, BB-SSL), and naive iterative masking. As solutions' overlap increases, the gap widens and additional ensemble restarts cannot close it. Only ALFESE proves competitive. Further, GEMSS is validated on real-world datasets, producing multiple distinct and highly predictive solutions: the practical goal that existing methods struggle to meet. The open-source Python package 'gemss' is available (github.com/kat-er-ina/gemss) and democratized through a free online application at huggingface.co/spaces/kat-er-ina/gemss.

cs.LG↗

arXiv2025

Sparse Probabilistic Graph Circuits

Deep generative models (DGMs) for graphs achieve impressively high expressive power thanks to very efficient and scalable neural networks. However, these networks contain non-linearities that prevent analytical computation of many standard probabilistic inference queries, i.e., these DGMs are considered \emph{intractable}. While recently proposed Probabilistic Graph Circuits (PGCs) address this issue by enabling \emph{tractable} probabilistic inference, they operate on dense graph representations with $\mathcal{O}(n^2)$ complexity for graphs with $n$ nodes and \emph{$m$ edges}. To address this scalability issue, we introduce Sparse PGCs, a new class of tractable generative models that operate directly on sparse graph representation, reducing the complexity to $\mathcal{O}(n + m)$, which is particularly beneficial for $m \ll n^2$. In the context of de novo drug design, we empirically demonstrate that SPGCs retain exact inference capabilities, improve memory efficiency and inference speed, and match the performance of intractable DGMs in key metrics.

cs.LG↗

arXiv2025

Probabilistic Graph Circuits: Deep Generative Models for Tractable Probabilistic Inference over Graphs

Deep generative models (DGMs) have recently demonstrated remarkable success in capturing complex probability distributions over graphs. Although their excellent performance is attributed to powerful and scalable deep neural networks, it is, at the same time, exactly the presence of these highly non-linear transformations that makes DGMs intractable. Indeed, despite representing probability distributions, intractable DGMs deny probabilistic foundations by their inability to answer even the most basic inference queries without approximations or design choices specific to a very narrow range of queries. To address this limitation, we propose probabilistic graph circuits (PGCs), a framework of tractable DGMs that provide exact and efficient probabilistic inference over (arbitrary parts of) graphs. Nonetheless, achieving both exactness and efficiency is challenging in the permutation-invariant setting of graphs. We design PGCs that are inherently invariant and satisfy these two requirements, yet at the cost of low expressive power. Therefore, we investigate two alternative strategies to achieve the invariance: the first sacrifices the efficiency, and the second sacrifices the exactness. We demonstrate that ignoring the permutation invariance can have severe consequences in anomaly detection, and that the latter approach is competitive with, and sometimes better than, existing intractable DGMs in the context of molecular graph generation.

cs.LG↗

arXiv2024

Sum-Product-Set Networks: Deep Tractable Models for Tree-Structured Graphs

Daily internet communication relies heavily on tree-structured graphs, embodied by popular data formats such as XML and JSON. However, many recent generative (probabilistic) models utilize neural networks to learn a probability distribution over undirected cyclic graphs. This assumption of a generic graph structure brings various computational challenges, and, more importantly, the presence of non-linearities in neural networks does not permit tractable probabilistic inference. We address these problems by proposing sum-product-set networks, an extension of probabilistic circuits from unstructured tensor data to tree-structured graph data. To this end, we use random finite sets to reflect a variable number of nodes and edges in the graph and to allow for exact and efficient inference. We demonstrate that our tractable model performs comparably to various intractable models based on neural networks.

cs.LG↗

arXiv2024

GraphSPNs: Sum-Product Networks Benefit From Canonical Orderings

Deep generative models have recently made a remarkable progress in capturing complex probability distributions over graphs. However, they are intractable and thus unable to answer even the most basic probabilistic inference queries without resorting to approximations. Therefore, we propose graph sum-product networks (GraphSPNs), a tractable deep generative model which provides exact and efficient inference over (arbitrary parts of) graphs. We investigate different principles to make SPNs permutation invariant. We demonstrate that GraphSPNs are able to (conditionally) generate novel and chemically valid molecular graphs, being competitive to, and sometimes even better than, existing intractable models. We find out that (Graph)SPNs benefit from ensuring the permutation invariance via canonical ordering.

cs.LG↗

arXiv2023

Is AUC the best measure for practical comparison of anomaly detectors?

The area under receiver operating characteristics (AUC) is the standard measure for comparison of anomaly detectors. Its advantage is in providing a scalar number that allows a natural ordering and is independent on a threshold, which allows to postpone the choice. In this work, we question whether AUC is a good metric for anomaly detection, or if it gives a false sense of comfort, due to relying on assumptions which are unlikely to hold in practice. Our investigation shows that variations of AUC emphasizing accuracy at low false positive rate seem to be better correlated with the needs of practitioners, but also that we can compare anomaly detectors only in the case when we have representative examples of anomalous samples. This last result is disturbing, as it suggests that in many cases, we should do active or few-show learning instead of pure anomaly detection.

cs.LG↗

arXiv2023

Nonlinear classifiers for ranking problems based on kernelized SVM

Many classification problems focus on maximizing the performance only on the samples with the highest relevance instead of all samples. As an example, we can mention ranking problems, accuracy at the top or search engines where only the top few queries matter. In our previous work, we derived a general framework including several classes of these linear classification problems. In this paper, we extend the framework to nonlinear classifiers. Utilizing a similarity to SVM, we dualize the problems, add kernels and propose a componentwise dual ascent method.

cs.LG↗

arXiv2023

DeepTopPush: Simple and Scalable Method for Accuracy at the Top

Accuracy at the top is a special class of binary classification problems where the performance is evaluated only on a small number of relevant (top) samples. Applications include information retrieval systems or processes with manual (expensive) postprocessing. This leads to minimizing the number of irrelevant samples above a threshold. We consider classifiers in the form of an arbitrary (deep) network and propose a new method DeepTopPush for minimizing the loss function at the top. Since the threshold depends on all samples, the problem is non-decomposable. We modify the stochastic gradient descent to handle the non-decomposability in an end-to-end training manner and propose a way to estimate the threshold only from values on the current minibatch and one delayed value. We demonstrate the excellent performance of DeepTopPush on visual recognition datasets and two real-world applications. The first one selects a small number of molecules for further drug testing. The second one uses real malware data, where we detected 46\% malware at an extremely low false alarm rate of $10^{-5}$.

cs.LG↗

arXiv2021

Fitting large mixture models using stochastic component selection

Traditional methods for unsupervised learning of finite mixture models require to evaluate the likelihood of all components of the mixture. This becomes computationally prohibitive when the number of components is large, as it is, for example, in the sum-product (transform) networks. Therefore, we propose to apply a combination of the expectation maximization and the Metropolis-Hastings algorithm to evaluate only a small number of, stochastically sampled, components, thus substantially reducing the computational cost. The Markov chain of component assignments is sequentially generated across the algorithm's iterations, having a non-stationary target distribution whose parameters vary via a gradient-descent scheme. We put emphasis on generality of our method, equipping it with the ability to train both shallow and deep mixture models which involve complex, and possibly nonlinear, transformations. The performance of our method is illustrated in a variety of synthetic and real-data contexts, considering deep models, such as mixtures of normalizing flows and sum-product (transform) networks.

cs.LG↗

arXiv2021

Comparison of Anomaly Detectors: Context Matters

Deep generative models are challenging the classical methods in the field of anomaly detection nowadays. Every new method provides evidence of outperforming its predecessors, often with contradictory results. The objective of this comparison is twofold: to compare anomaly detection methods of various paradigms with focus on deep generative models, and identification of sources of variability that can yield different results. The methods were compared on popular tabular and image datasets. We identified the main sources of variability to be experimental conditions: i) the type data set (tabular or image) and the nature of anomalies (statistical or semantic), and ii) strategy of selection of hyperparameters, especially the number of available anomalies in the validation set. Different methods perform the best in different contexts, i.e. combination of experimental conditions together with computational time. This explains the variability of the previous results and highlights the importance of careful specification of the context in the publication of a new method. All our code and results are available for download.

cs.LG↗

arXiv2020

Neural Power Units

Conventional Neural Networks can approximate simple arithmetic operations, but fail to generalize beyond the range of numbers that were seen during training. Neural Arithmetic Units aim to overcome this difficulty, but current arithmetic units are either limited to operate on positive numbers or can only represent a subset of arithmetic operations. We introduce the Neural Power Unit (NPU) that operates on the full domain of real numbers and is capable of learning arbitrary power functions in a single layer. The NPU thus fixes the shortcomings of existing arithmetic units and extends their expressivity. We achieve this by using complex arithmetic without requiring a conversion of the network to complex numbers. A simplification of the unit to the RealNPU yields a highly transparent model. We show that the NPUs outperform their competitors in terms of accuracy and sparsity on artificial arithmetic datasets, and that the RealNPU can discover the governing equations of a dynamical system only from data.

cs.LG↗

arXiv2020

Solvability of the Power Flow Problem in DC Overhead Wire Circuit

Proper traffic simulation of electric vehicles, which draw energy from overhead wires, requires adequate modeling of traction infrastructure. Such vehicles include trains, trams or trolleybuses. Since the requested power demands depend on a traffic situation, the overhead wire DC electrical circuit is associated with a non-linear power flow problem. Although the Newton-Raphson method is well-known and widely accepted for seeking its solution, the existence of such a solution is not guaranteed. Particularly in situations where the vehicle power demands are too high (during acceleration), the solution of the studied problem may not exist. To deal with such cases, we introduce a numerical method which seeks maximal suppliable power demands for which the solution exists. This corresponds to introducing a scaling parameter to reduce the demanded power. The interpretation of the scaling parameter is the amount of energy which is absent in the system, and which needs to be provided by external sources such as on-board batteries. We propose an efficient two-stage algorithm to find the optimal scaling parameter and the resulting potentials in the overhead wire network. We perform a comparison with a naive approach and present a real-world simulation of part of the Pilsen city in the Czech Republic. These simulations are performed in the traffic micro-simulator SUMO, a popular open-source traffic simulation platform.

math.OC↗

arXiv2020

Expectation-Maximization Algorithm for Identification of Mesh-based Compartment Thermal Model of Power Modules

Accurate knowledge of temperatures in power semiconductor modules is crucial for proper thermal management of such devices. Precise prediction of temperatures allows to operate the system at the physical limit of the device avoiding undesirable over-temperatures and thus improve reliability of the module. Commonly used thermal models can be based on detailed expert knowledge of the device's physical structure or on precise and complete temperature distribution measurements. The latter approach is more often used in the industry. Recently, we have proposed a linear time invariant state-space thermal model based on a compartment representation and its identification procedure that is based on the Expectation-Maximization algorithm from incomplete temperature data. However, the model still requires to measure temperatures of all active elements. In this contribution, we aim to relax the need for all measurements. Therefore, we replace the previous dark gray-box approach with a structured compartment model. The structure of the model is designed by a mesh-based discretization of the physical layout of the module. All compartments are assumed to share parameters that are identified from the data of the measured elements. Temperatures of the unmeasured elements are predicted using the shared parameters. Identification of the parameters is possible only with suitable regularization due to limited amount of the data. In particular, the model tightening is accomplished by sharing parameters among compartments and by constraining the process covariance matrix of the model in this contribution. Applicability of the proposed identification procedure is discussed in terms of growing state-space and therefore speeding up of the identification algorithm is suggested. Performance of the proposed approach is tested and demonstrated on simulated data.

eess.SP↗

arXiv2020

Rodent: Relevance determination in differential equations

We aim to identify the generating, ordinary differential equation (ODE) from a set of trajectories of a partially observed system. Our approach does not need prescribed basis functions to learn the ODE model, but only a rich set of Neural Arithmetic Units. For maximal explainability of the learnt model, we minimise the state size of the ODE as well as the number of non-zero parameters that are needed to solve the problem. This sparsification is realized through a combination of the Variational Auto-Encoder (VAE) and Automatic Relevance Determination (ARD). We show that it is possible to learn not only one specific model for a single process, but a manifold of models representing harmonic signals as well as a manifold of Lotka-Volterra systems.

stat.ML↗

arXiv2020

General Framework for Binary Classification on Top Samples

Many binary classification problems minimize misclassification above (or below) a threshold. We show that instances of ranking problems, accuracy at the top or hypothesis testing may be written in this form. We propose a general framework to handle these classes of problems and show which known methods (both known and newly proposed) fall into this framework. We provide a theoretical analysis of this framework and mention selected possible pitfalls the methods may encounter. We suggest several numerical improvements including the implicit derivative and stochastic gradient descent. We provide an extensive numerical study. Based both on the theoretical properties and numerical experiments, we conclude the paper by suggesting which method should be used in which situation.

cs.LG↗

arXiv2019

Anomaly scores for generative models

Reconstruction error is a prevalent score used to identify anomalous samples when data are modeled by generative models, such as (variational) auto-encoders or generative adversarial networks. This score relies on the assumption that normal samples are located on a manifold and all anomalous samples are located outside. Since the manifold can be learned only where the training data lie, there are no guarantees how the reconstruction error behaves elsewhere and the score, therefore, seems to be ill-defined. This work defines an anomaly score that is theoretically compatible with generative models, and very natural for (variational) auto-encoders as they seem to be prevalent. The new score can be also used to select hyper-parameters and models. Finally, we explain why reconstruction error delivers good experimental results despite weak theoretical justification.

stat.ML↗

arXiv2018

Are generative deep models for novelty detection truly better?

Many deep models have been recently proposed for anomaly detection. This paper presents comparison of selected generative deep models and classical anomaly detection methods on an extensive number of non--image benchmark datasets. We provide statistical comparison of the selected models, in many configurations, architectures and hyperparamaters. We arrive to conclusion that performance of the generative models is determined by the process of selection of their hyperparameters. Specifically, performance of the deep generative models deteriorates with decreasing amount of anomalous samples used in hyperparameter selection. In practical scenarios of anomaly detection, none of the deep generative models systematically outperforms the kNN.

cs.LG↗