arXiv subjects

Robert Simon Fong

Publications and source records attributed to Robert Simon Fong.

8 recordsLinked to original sources

arXiv2026

Symplectic Representation of Legendre Dynamics

Modern learning systems act on internal representations of data, yet how these representations encode underlying physical or statistical structure is often left implicit. In physics, symplecticity keeps Hamiltonian systems faithful to their phase-space geometry. Recent learning methods impose such geometric structure either in the dynamics or through training losses. Here we ask a different question: what would it mean for the representation itself to obey a symplectic conservation law? We pose this representation-level constraint through Legendre duality: the relation $p = dψ(q)$ between primal and dual coordinates, which in exponential family models is the information-geometric pairing of natural and expectation parameters. We formalize Legendre dynamics as stochastic processes whose trajectories remain on Legendre graphs, where the evolving primal-dual parameters stay Legendre dual. We show that this class includes linear time-invariant Gaussian process regression and Ornstein-Uhlenbeck dynamics. Geometrically, we characterize the symplectomorphisms of cotangent bundles that preserve all Legendre graphs. We show that these maps are exactly cotangent lifts of base diffeomorphisms followed by exact fibre translations. This gives an explicit normal form for Legendre-preserving representation updates. Dynamically, we prove that the normal form is realized by Hamiltonians that are at most linear in the momentum. This realization principle is used to construct linear and nonlinear Hamiltonian Symplectic Reservoirs (SR) whose recurrent updates preserve Legendre graphs by construction. This is the only normal form that preserves Legendre duality, so the architecture follows from the invariant. Numerical experiments confirm the normal-form identities and distinguish Legendre preserving Hamiltonian SRs from generic symplectic and standard reservoir baselines.

cs.LG↗

arXiv2025

Structuring Multiple Simple Cycle Reservoirs with Particle Swarm Optimization

Reservoir Computing (RC) is a time-efficient computational paradigm derived from Recurrent Neural Networks (RNNs). The Simple Cycle Reservoir (SCR) is an RC model that stands out for its minimalistic design, offering extremely low construction complexity and proven capability of universally approximating time-invariant causal fading memory filters, even in the linear dynamics regime. This paper introduces Multiple Simple Cycle Reservoirs (MSCRs), a multi-reservoir framework that extends Echo State Networks (ESNs) by replacing a single large reservoir with multiple interconnected SCRs. We demonstrate that optimizing MSCR using Particle Swarm Optimization (PSO) outperforms existing multi-reservoir models, achieving competitive predictive performance with a lower-dimensional state space. By modeling interconnections as a weighted Directed Acyclic Graph (DAG), our approach enables flexible, task-specific network topology adaptation. Numerical simulations on three benchmark time-series prediction tasks confirm these advantages over rival algorithms. These findings highlight the potential of MSCR-PSO as a promising framework for optimizing multi-reservoir systems, providing a foundation for further advancements and applications of interconnected SCRs for developing efficient AI devices.

cs.NE↗

arXiv2024

Linear Simple Cycle Reservoirs at the edge of stability perform Fourier decomposition of the input driving signals

This paper explores the representational structure of linear Simple Cycle Reservoirs (SCR) operating at the edge of stability. We view SCR as providing in their state space feature representations of the input-driving time series. By endowing the state space with the canonical dot-product, we ``reverse engineer" the corresponding kernel (inner product) operating in the original time series space. The action of this time-series kernel is fully characterized by the eigenspace of the corresponding metric tensor. We demonstrate that when linear SCRs are constructed at the edge of stability, the eigenvectors of the time-series kernel align with the Fourier basis. This theoretical insight is supported by numerical experiments.

cs.NE↗

arXiv2024

Efficient Digital Quadratic Unconstrained Binary Optimization Solvers for SAT Problems

Boolean satisfiability (SAT) is a propositional logic problem of determining whether an assignment of variables satisfies a Boolean formula. Many combinatorial optimization problems can be formulated in Boolean SAT logic -- either as k-SAT decision problems or Max k-SAT optimization problems, with conflict-driven (CDCL) solvers being the most prominent. Despite their ability to handle large instances, CDCL-based solvers have fundamental scalability limitations. In light of this, we propose recently-developed quadratic unconstrained binary optimization (QUBO) solvers as an alternative platform for 3-SAT problems. To utilize them, we implement a 2-step [3-SAT]-[Max 2-SAT]-[QUBO] conversion procedure and present a rigorous proof to explicitly calculate the number of both satisfied and violated clauses of the original 3-SAT instance from the transformed Max 2-SAT formulation. We then demonstrate, through numerical simulations on several benchmark instances, that digital QUBO solvers can achieve state-of-the-art accuracy on 78-variable 3-SAT benchmark problems. Our work facilitates the broader use of quantum annealers on noisy intermediate-scale quantum (NISQ) devices, as well as their quantum-inspired digital counterparts, for solving 3-SAT problems.

math.OC↗

arXiv2024

Universality of Real Minimal Complexity Reservoir

Reservoir Computing (RC) models, a subclass of recurrent neural networks, are distinguished by their fixed, non-trainable input layer and dynamically coupled reservoir, with only the static readout layer being trained. This design circumvents the issues associated with backpropagating error signals through time, thereby enhancing both stability and training efficiency. RC models have been successfully applied across a broad range of application domains. Crucially, they have been demonstrated to be universal approximators of time-invariant dynamic filters with fading memory, under various settings of approximation norms and input driving sources. Simple Cycle Reservoirs (SCR) represent a specialized class of RC models with a highly constrained reservoir architecture, characterized by uniform ring connectivity and binary input-to-reservoir weights with an aperiodic sign pattern. For linear reservoirs, given the reservoir size, the reservoir construction has only one degree of freedom -- the reservoir cycle weight. Such architectures are particularly amenable to hardware implementations without significant performance degradation in many practical tasks. In this study we endow these observations with solid theoretical foundations by proving that SCRs operating in real domain are universal approximators of time-invariant dynamic filters with fading memory. Our results supplement recent research showing that SCRs in the complex domain can approximate, to arbitrary precision, any unrestricted linear reservoir with a non-linear readout. We furthermore introduce a novel method to drastically reduce the number of SCR units, making such highly constrained architectures natural candidates for low-complexity hardware implementations. Our findings are supported by empirical studies on real-world time series datasets.

cs.LG↗

arXiv2024

Simple Cycle Reservoirs are Universal

Reservoir computation models form a subclass of recurrent neural networks with fixed non-trainable input and dynamic coupling weights. Only the static readout from the state space (reservoir) is trainable, thus avoiding the known problems with propagation of gradient information backwards through time. Reservoir models have been successfully applied in a variety of tasks and were shown to be universal approximators of time-invariant fading memory dynamic filters under various settings. Simple cycle reservoirs (SCR) have been suggested as severely restricted reservoir architecture, with equal weight ring connectivity of the reservoir units and input-to-reservoir weights of binary nature with the same absolute value. Such architectures are well suited for hardware implementations without performance degradation in many practical tasks. In this contribution, we rigorously study the expressive power of SCR in the complex domain and show that they are capable of universal approximation of any unrestricted linear reservoir system (with continuous readout) and hence any time-invariant fading memory filter over uniformly bounded input streams.

cs.NE↗

arXiv2024

Predictive Modeling in the Reservoir Kernel Motif Space

This work proposes a time series prediction method based on the kernel view of linear reservoirs. In particular, the time series motifs of the reservoir kernel are used as representational basis on which general readouts are constructed. We provide a geometric interpretation of our approach shedding light on how our approach is related to the core reservoir models and in what way the two approaches differ. Empirical experiments then compare predictive performances of our suggested model with those of recent state-of-art transformer based models, as well as the established recurrent network model - LSTM. The experiments are performed on both univariate and multivariate time series and with a variety of prediction horizons. Rather surprisingly we show that even when linear readout is employed, our method has the capacity to outperform transformer models on univariate time series and attain competitive results on multivariate benchmark datasets. We conclude that simple models with easily controllable capacity but capturing enough memory and subsequence structure can outperform potentially over-complicated deep learning models. This does not mean that reservoir motif based models are preferable to other more complex alternatives - rather, when introducing a new complex time series model one should employ as a sanity check simple, but potentially powerful alternatives/baselines such as reservoir models or the models introduced here.

cs.LG↗

arXiv2020

A Framework for Population-Based Stochastic Optimization on Abstract Riemannian Manifolds

We present Extended Riemannian Stochastic Derivative-Free Optimization (Extended RSDFO), a novel population-based stochastic optimization algorithm on Riemannian manifolds that addresses the locality and implicit assumptions of manifold optimization in the literature. We begin by investigating the Information Geometrical structure of statistical model over Riemannian manifolds. This establishes a geometrical framework of Extended RSDFO using both the statistical geometry of the decision space and the Riemannian geometry of the search space. We construct locally inherited probability distribution via an orientation-preserving diffeomorphic bundle morphism, and then extend the information geometrical structure to mixture densities over totally bounded subsets of manifolds. The former relates the information geometry of the decision space and the local point estimations on the search space manifold. The latter overcomes the locality of parametric probability distributions on Riemannian manifolds. We then construct Extended RSDFO and study its structure and properties from a geometrical perspective. We show that Extended RSDFO's expected fitness improves monotonically and it's global eventual convergence in finitely many steps on connected compact Riemannian manifolds. Extended RSDFO is compared to state-of-the-art manifold optimization algorithms on multi-modal optimization problems over a variety of manifolds. In particular, we perform a novel synthetic experiment on Jacob's ladder to motivate and necessitate manifold optimization. Jacob's ladder is a non-compact manifold of countably infinite genus, which cannot be expressed as polynomial constraints and does not have a global representation in an ambient Euclidean space. Optimization problems on Jacob's ladder thus cannot be addressed by traditional (constraint) optimization methods on Euclidean spaces.

math.OC↗