arXiv subjects

Andrew Polar

Publications and source records attributed to Andrew Polar.

6 recordsLinked to original sources

arXiv2025

Concurrent training methods for Kolmogorov-Arnold networks: Disjoint datasets and FPGA implementation

The present paper introduces concurrency-driven enhancements to the training algorithm for the Kolmogorov-Arnold networks (KANs) that is based on the Newton-Kaczmarz (NK) method. Prior research shows that KANs trained using the NK-based approach outperform classical neural networks (multilayer perceptrons - MLPs) both in terms of accuracy and training time. Up to now, the fundamental limitation of the algorithm has been the sequential computation of the updates - each update depends on the results of the previous step, obstructing parallelisation; even though parallelisation of some parts of the algorithm, such as the evaluation of the basis functions, has already been proposed and tested. However, substantial acceleration is achievable. Three complementary concurrency-driven novel strategies are proposed in the present paper: (i) a pre-training procedure tailored to the NK updates' structure, (ii) training on disjoint subsets of data together with models' merging, and (iii) a division-free customisation of the algorithm for field-programmable gate arrays (FPGAs), which is implemented and tested directly on the device. Computational experiments are used to assess the scalability and to compare the authors' KAN implementation with popular MLP packages. All presented experimental results are fully reproducible, with the complete source codes available online.

cs.LG↗

arXiv2023

Construction of the Kolmogorov-Arnold representation using the Newton-Kaczmarz method

It is known that any continuous multivariate function can be represented exactly by a composition functions of a single variable - the so-called Kolmogorov-Arnold representation. It can be a convenient tool for tasks where it is required to obtain a predictive model that maps some vector input of a black box system into a scalar output. In this case, the representation may not be exact, and it is more correct to refer to such structure as the Kolmogorov-Arnold model (or, as more recently popularised, 'network'). Construction of such model based on the recorded input-output data is a challenging task. In the present paper, it is suggested to decompose the underlying functions of the representation into continuous basis functions and parameters. It is then proposed to find the parameters using the Newton-Kaczmarz method for solving systems of non-linear equations. The algorithm is then modified to support parallelisation. The paper demonstrates that such approach is also an excellent tool for data-driven solution of partial differential equations. Numerical examples show that for the considered model, the Newton-Kaczmarz method for parameter estimation is efficient and more robust with respect to the section of the initial guess than the straightforward application of the Gauss-Newton method. Finally, the Kolmogorov-Arnold model is compared to the MATLAB's built-in neural networks on a relatively large-scale problem (25 inputs, datasets of 10 million records), significantly outperforming the multilayer perceptrons (MLPs) in this particular problem (4-10 minutes vs. 4-8 hours of training time, as well as higher accuracy, lower CPU usage, and smaller memory footprint).

math.NA↗

arXiv2021

Probabilistic Kolmogorov-Arnold Network

The Kolmogorov-Arnold network (KAN) is a regression model that is based on a representation of an arbitrary continuous multivariate function by a composition of functions of a single variable. Experimentally-obtained datasets for regression models typically include uncertainties, which in some cases, cannot be neglected. The conventional way to account for the latter is to model confidence intervals of the systems' outputs in addition to the expected values of the outputs. However, such information may be insufficient, and in some cases, researchers aim to obtain probability distributions of the outputs. The present paper proposes a method for estimating probability distributions of the outputs in the case of aleatoric uncertainty (i.e. for systems that produce different outputs each time an experiment is executed with the same inputs). The suggested approach covers input-dependent probability distributions of the outputs and is capable of capturing the multi-modality, as well as the variation of the distribution type with the inputs. Although the method is applicable to any regression model, the present paper combines it with KANs, since the specific structure of KANs leads to computationally-efficient models' construction. The source code is available online.

cs.LG↗

arXiv2020

A deep machine learning algorithm for construction of the Kolmogorov-Arnold representation

The Kolmogorov-Arnold representation is a proven adequate replacement of a continuous multivariate function by an hierarchical structure of multiple functions of one variable. The proven existence of such representation inspired many researchers to search for a practical way of its construction, since such model answers the needs of machine learning. This article shows that the Kolmogorov-Arnold representation is not only a composition of functions but also a particular case of a tree of the discrete Urysohn operators. The article introduces new, quick and computationally stable algorithm for constructing of such Urysohn trees. Besides continuous multivariate functions, the suggested algorithm covers the cases with quantised inputs and combination of quantised and continuous inputs. The article also contains multiple results of testing of the suggested algorithm on publicly available datasets, used also by other researchers for benchmarking.

math.OC↗

arXiv2019

Canonical block-oriented model

The block-oriented models are usually based on linear dynamic and non-linear static blocks that are connected in various sequential/parallel ways. Some particular configurations of the involved blocks result in the well-known Hammerstein, Wiener, Hammerstein-Wiener and generalised Hammerstein models. The Urysohn model is a lesser-known model; it is represented by a single non-linear dynamic block and can be approximated by a number of parallel Hammerstein blocks. In this paper, it is shown that any block-oriented model can be adequately replaced by a single Urysohn block followed by a single static non-linear block. Furthermore, a method of the so-called non-parametric identification of such object is introduced.

math.OC↗

arXiv2018

Modelling non-linear control systems using the discrete Urysohn operator

This paper introduces a multiple-input discrete Urysohn operator for modelling non-linear control systems and a technique of its identification by processing the observed input and output signals. It is shown that, due to the nature of the discrete Urysohn operator, the identification problem always has an infinity of solutions, which exactly convert the inputs to the output. The suggested iterative identification procedure, however, leads to a unique solution with the minimum norm, requires only few arithmetic operations with the parameter values and is applicable to a real-time identification, running concurrently with the data reading. The efficiency of the proposed modelling and identification approaches is demonstrated using an example of a non-linear mechanical system, which is represented by a differential equation, and an example of a complex real-world dynamic object.

math.OC↗