arXiv subjects

Thiparat Chotibut

Publications and source records attributed to Thiparat Chotibut.

33 records · Page 2Linked to original sources

arXiv2022

Explainable Natural Language Processing with Matrix Product States

Despite empirical successes of recurrent neural networks (RNNs) in natural language processing (NLP), theoretical understanding of RNNs is still limited due to intrinsically complex non-linear computations. We systematically analyze RNNs' behaviors in a ubiquitous NLP task, the sentiment analysis of movie reviews, via the mapping between a class of RNNs called recurrent arithmetic circuits (RACs) and a matrix product state (MPS). Using the von-Neumann entanglement entropy (EE) as a proxy for information propagation, we show that single-layer RACs possess a maximum information propagation capacity, reflected by the saturation of the EE. Enlarging the bond dimension beyond the EE saturation threshold does not increase model prediction accuracies, so a minimal model that best estimates the data statistics can be inferred. Although the saturated EE is smaller than the maximum EE allowed by the area law, our minimal model still achieves ~99% training accuracies in realistic sentiment analysis data sets. Thus, low EE is not a warrant against the adoption of single-layer RACs for NLP. Contrary to a common belief that long-range information propagation is the main source of RNNs' successes, we show that single-layer RACs harness high expressiveness from the subtle interplay between the information propagation and the word vector embeddings. Our work sheds light on the phenomenology of learning in RACs, and more generally on the explainability of RNNs for NLP, using tools from many-body quantum physics.

cond-mat.dis-nn↗

arXiv2022

Resistance distance distribution in large sparse random graphs

We consider an Erdos-Renyi random graph consisting of N vertices connected by randomly and independently drawing an edge between every pair of them with probability c/N so that at N->infinity one obtains a graph of finite mean degree c. In this regime, we study the distribution of resistance distances between the vertices of this graph and develop an auxiliary field representation for this quantity in the spirit of statistical field theory. Using this representation, a saddle point evaluation of the resistance distance distribution is possible at N->infinity in terms of an 1/c expansion. The leading order of this expansion captures the results of numerical simulations very well down to rather small values of c; for example, it recovers the empirical distribution at c=4 or 6 with an overlap of around 90%. At large values of c, the distribution tends to a Gaussian of mean 2/c and standard deviation sqrt{2/c^3}. At small values of c, the distribution is skewed toward larger values, as captured by our saddle point analysis, and many fine features appear in addition to the main peak, including subleading peaks that can be traced back to resistance distances between vertices of specific low degrees and the rest of the graph. We develop a more refined saddle point scheme that extracts the corresponding degree-differentiated resistance distance distributions. We then use this approach to recover analytically the most apparent of the subleading peaks that originates from vertices of degree 1. Rather intuitively, this subleading peak turns out to be a copy of the main peak, shifted by one unit of resistance distance and scaled down by the probability for a vertex to have degree 1. We comment on a possible lack of smoothness in the true N->infinity distribution suggested by the numerics.

cond-mat.dis-nn↗

arXiv2022

Unpredictable dynamics in congestion games: memory loss can prevent chaos

We study the dynamics of simple congestion games with two resources where a continuum of agents behaves according to a version of Experience-Weighted Attraction (EWA) algorithm. The dynamics is characterized by two parameters: the (population) intensity of choice $a>0$ capturing the economic rationality of the total population of agents and a discount factor $σ\in [0,1]$ capturing a type of memory loss where past outcomes matter exponentially less than the recent ones. Finally, our system adds a third parameter $b \in (0,1)$, which captures the asymmetry of the cost functions of the two resources. It is the proportion of the agents using the first resource at Nash equilibrium, with $b=1/2$ capturing a symmetric network. Within this simple framework, we show a plethora of bifurcation phenomena where behavioral dynamics destabilize from global convergence to equilibrium, to limit cycles or even (formally proven) chaos as a function of the parameters $a$, $b$ and $σ$. Specifically, we show that for any discount factor $σ$ the system will be destabilized for a sufficiently large intensity of choice $a$. Although for discount factor $σ=0$ almost always (i.e., $b \neq 1/2$) the system will become chaotic, as $σ$ increases the chaotic regime will give place to the attracting periodic orbit of period 2. Therefore, memory loss can simplify game dynamics and make the system predictable. We complement our theoretical analysis with simulations and several bifurcation diagrams that showcase the unyielding complexity of the population dynamics (e.g., attracting periodic orbits of different lengths) even in the simplest possible potential games.

cs.GT↗

arXiv2022

Hybrid Quantum-Classical Algorithms for Loan Collection Optimization with Loan Loss Provisions

Banks are required to set aside funds in their income statement, known as a loan loss provision (LLP), to account for potential loan defaults and expenses. By treating the LLP as a global constraint, we propose a hybrid quantum-classical algorithm to solve Quadratic Constrained Binary Optimization (QCBO) models for loan collection optimization. The objective is to find a set of optimal loan collection actions that maximizes the expected net profit presented to the bank as well as the financial welfare in the financial network of loanees, while keeping the LLP at its minimum. Our algorithm consists of three parts: a classical divide-and-conquer algorithm to enable a large-scale optimization, a quantum alternating operator ansatz (QAOA) algorithm to maximize the objective function, and a classical sampling algorithm to handle the LLP. We apply the algorithm to a real-world data set with 600 loanees and 5 possible collection actions. The QAOA is performed using up to 35 qubits on a classical computer. We show that the presence of the QAOA can improve the expected net profit by approximately $70\%$, compared to when the QAOA is absent from the hybrid algorithm. Our work illustrates the use of near-term quantum devices to tackle real-world optimization problems.

quant-ph↗

arXiv2021

The birth of geometry in exponential random graphs

Inspired by the prospect of having discretized spaces emerge from random graphs, we construct a collection of simple and explicit exponential random graph models that enjoy, in an appropriate parameter regime, a roughly constant vertex degree and form very large numbers of simple polygons (triangles or squares). The models avoid the collapse phenomena that plague naive graph Hamiltonians based on triangle or square counts. More than that, statistically significant numbers of other geometric primitives (small pieces of regular lattices, cubes) emerge in our ensemble, even though they are not in any way explicitly pre-programmed into the formulation of the graph Hamiltonian, which only depends on properties of paths of length 2. While much of our motivation comes from hopes to construct a graph-based theory of random geometry (Euclidean quantum gravity), our presentation is completely self-contained within the context of exponential random graph theory, and the range of potential applications is considerably more broad.

cond-mat.dis-nn↗

arXiv2021

Quantum diffusion map for nonlinear dimensionality reduction

Inspired by random walk on graphs, diffusion map (DM) is a class of unsupervised machine learning that offers automatic identification of low-dimensional data structure hidden in a high-dimensional dataset. In recent years, among its many applications, DM has been successfully applied to discover relevant order parameters in many-body systems, enabling automatic classification of quantum phases of matter. However, classical DM algorithm is computationally prohibitive for a large dataset, and any reduction of the time complexity would be desirable. With a quantum computational speedup in mind, we propose a quantum algorithm for DM, termed quantum diffusion map (qDM). Our qDM takes as an input $N$ classical data vectors, performs an eigen-decomposition of the Markov transition matrix in time $O(\log^3 N)$, and classically constructs the diffusion map via the readout (tomography) of the eigenvectors, giving a total expected runtime proportional to $N^2 \text{polylog}\, N$. Lastly, quantum subroutines in qDM for constructing a Markov transition matrix, and for analyzing its spectral properties can also be useful for other random walk-based algorithms.

quant-ph↗

arXiv2021

Follow-the-Regularized-Leader Routes to Chaos in Routing Games

We study the emergence of chaotic behavior of Follow-the-Regularized Leader (FoReL) dynamics in games. We focus on the effects of increasing the population size or the scale of costs in congestion games, and generalize recent results on unstable, chaotic behaviors in the Multiplicative Weights Update dynamics to a much larger class of FoReL dynamics. We establish that, even in simple linear non-atomic congestion games with two parallel links and any fixed learning rate, unless the game is fully symmetric, increasing the population size or the scale of costs causes learning dynamics to become unstable and eventually chaotic, in the sense of Li-Yorke and positive topological entropy. Furthermore, we show the existence of novel non-standard phenomena such as the coexistence of stable Nash equilibria and chaos in the same game. We also observe the simultaneous creation of a chaotic attractor as another chaotic attractor gets destroyed. Lastly, although FoReL dynamics can be strange and non-equilibrating, we prove that the time average still converges to an exact equilibrium for any choice of learning rate and any scale of costs.

cs.GT↗

arXiv2020

Broken Detailed Balance and Non-Equilibrium Dynamics in Noisy Social Learning Models

We propose new Degroot-type social learning models with feedback in a continuous time, to investigate the effect of a noisy information source on consensus formation in a social network. Unlike the standard Degroot framework, noisy information models destroy consensus formation. On the other hand, the noisy opinion dynamics converge to the equilibrium distribution that encapsulates correlations among agents' opinions. Interestingly, such an equilibrium distribution is also a non-equilibrium steady state (NESS) with a non-zero probabilistic current loop. Thus, noisy information source leads to a NESS at long times that encodes persistent correlated opinion dynamics of learning agents. Our model provides a simple realization of NESS in the context of social learning. Other phenomena such as synchronization of opinions when agents are subject to a common noise are also studied.

physics.soc-ph↗

arXiv2019

Biologically Plausible Sequence Learning with Spiking Neural Networks

Motivated by the celebrated discrete-time model of nervous activity outlined by McCulloch and Pitts in 1943, we propose a novel continuous-time model, the McCulloch-Pitts network (MPN), for sequence learning in spiking neural networks. Our model has a local learning rule, such that the synaptic weight updates depend only on the information directly accessible by the synapse. By exploiting asymmetry in the connections between binary neurons, we show that MPN can be trained to robustly memorize multiple spatiotemporal patterns of binary vectors, generalizing the ability of the symmetric Hopfield network to memorize static spatial patterns. In addition, we demonstrate that the model can efficiently learn sequences of binary pictures as well as generative models for experimental neural spike-train data. Our learning rule is consistent with spike-timing-dependent plasticity (STDP), thus providing a theoretical ground for the systematic design of biologically inspired networks with large and robust long-range sequence storage capacity.

cond-mat.dis-nn↗

arXiv2019

The route to chaos in routing games: When is Price of Anarchy too optimistic?

Routing games are amongst the most studied classes of games. Their two most well-known properties are that learning dynamics converge to equilibria and that all equilibria are approximately optimal. In this work, we perform a stress test for these classic results by studying the ubiquitous dynamics, Multiplicative Weights Update, in different classes of congestion games, uncovering intricate non-equilibrium phenomena. As the system demand increases, the learning dynamics go through period-doubling bifurcations, leading to instabilities, chaos and large inefficiencies even in the simplest case of non-atomic routing games with two paths of linear cost where the Price of Anarchy is equal to one. Starting with this simple class, we show that every system has a carrying capacity, above which it becomes unstable. If the equilibrium flow is a symmetric $50-50\%$ split, the system exhibits one period-doubling bifurcation. A single periodic attractor of period two replaces the attracting fixed point. Although the Price of Anarchy is equal to one, in the large population limit the time-average social cost for all but a zero measure set of initial conditions converges to its worst possible value. For asymmetric equilibrium flows, increasing the demand eventually forces the system into Li-Yorke chaos with positive topological entropy and periodic orbits of all possible periods. Remarkably, in all non-equilibrating regimes, the time-average flows on the paths converge exactly to the equilibrium flows, a property akin to no-regret learning in zero-sum games. These results are robust. We extend them to routing games with arbitrarily many strategies, polynomial cost functions, non-atomic as well as atomic routing games and heteregenous users. Our results are also applicable to any sequence of shrinking learning rates, e.g., $1/\sqrt{T}$, by allowing for a dynamically increasing population size.

cs.GT↗

arXiv2019

Mean Field Theory of Activation Functions in Deep Neural Networks

We present a Statistical Mechanics (SM) model of deep neural networks, connecting the energy-based and the feed forward networks (FFN) approach. We infer that FFN can be understood as performing three basic steps: encoding, representation validation and propagation. From the meanfield solution of the model, we obtain a set of natural activations -- such as Sigmoid, $\tanh$ and ReLu -- together with the state-of-the-art, Swish; this represents the expected information propagating through the network and tends to ReLu in the limit of zero noise.We study the spectrum of the Hessian on an associated classification task, showing that Swish allows for more consistent performances over a wider range of network architectures.

cs.LG↗

arXiv2018

Family of chaotic maps from game theory

From a two-agent, two-strategy congestion game where both agents apply the multiplicative weights update algorithm, we obtain a two-parameter family of maps of the unit square to itself. Interesting dynamics arise on the invariant diagonal, on which a two-parameter family of bimodal interval maps exhibits periodic orbits and chaos. While the fixed point $b$ corresponding to a Nash equilibrium of such map $f$ is usually repelling, it is globally Cesaro attracting on the diagonal, that is, \[ \lim_{n\to\infty}\frac1n\sum_{k=0}^{n-1}f^k(x)=b \] for every $x$ in the minimal invariant interval. This solves a known open question whether there exists a nontrivial smooth map other than $x\mapsto axe^{-x}$ with centers of mass of all periodic orbits coinciding. We also study the dependence of the dynamics on the two parameters.

math.DS↗

arXiv2016

Population Genetics with Fluctuating Population Sizes

Standard neutral population genetics theory with a strictly fixed population size has important limitations. An alternative model that allows independently fluctuating population sizes and reproduces the standard neutral evolution is reviewed. We then study a situation such that the competing species are neutral at the equilibrium population size but population size fluctuations nevertheless favor fixation of one species over the other. In this case, a separation of timescales emerges naturally and allows adiabatic elimination of a fast population size variable to deduce the fluctuations-induced selection dynamics near the equilibrium population size. The results highlight the incompleteness of the standard population genetics with a strictly fixed population size.

q-bio.PE↗

arXiv2016

Evolutionary Dynamics with Fluctuating Population Sizes and Strong Mutualism

Game theory ideas provide a useful framework for studying evolutionary dynamics in a well-mixed environment. This approach, however, typically enforces a strictly fixed overall population size, deemphasizing natural growth processes. We study a competitive Lotka-Volterra model, with number fluctuations, that accounts for natural population growth and encompasses interaction scenarios typical of evolutionary games. We show that, in an appropriate limit, the model describes standard evolutionary games with both genetic drift and overall population size fluctuations. However, there are also regimes where a varying population size can strongly influence the evolutionary dynamics. We focus on the strong mutualism scenario and demonstrate that standard evolutionary game theory fails to describe our simulation results. We then analytically and numerically determine fixation probabilities as well as mean fixation times using matched asymptotic expansions, taking into account the population size degree of freedom. These results elucidate the interplay between population dynamics and evolutionary dynamics in well-mixed systems.

q-bio.PE↗

arXiv2016

Striated Populations in Disordered Environments with Advection

Growth in static and controlled environments such as a Petri dish can be used to study the spatial population dynamics of microorganisms. However, natural populations such as marine microbes experience fluid advection and often grow up in heterogeneous environments. We investigate a generalized Fisher-Kolmogorov-Petrovsky-Piscounov (FKPP) equation describing single species population subject to a constant flow field and quenched random spatially inhomogeneous growth rates with a fertile overall growth condition. We analytically and numerically demonstrate that the non-equilibrium steady-state population density develops a flow-driven striation pattern. The striations are highly asymmetric with a longitudinal correlation length that diverges linearly with the flow speed and a transverse correlation length that approaches a finite velocity-independent value. Linear response theory is developed to study the statistics of the steady states. Theoretical predictions show excellent agreement with the numerical steady states of the generalized FKPP equation obtained from Lattice Boltzmann simulations. These findings suggest that, although the growth disorder can be spatially uncorrelated, correlated population structures with striations emerge naturally at sufficiently strong advection.

q-bio.PE↗