arXiv subjects

Wenlian Lu

Publications and source records attributed to Wenlian Lu.

At least 37 records · Page 2Linked to original sources

arXiv2024

Toward stochastic neural computing

The highly irregular spiking activity of cortical neurons and behavioral variability suggest that the brain could operate in a fundamentally probabilistic way. Mimicking how the brain implements and learns probabilistic computation could be a key to developing machine intelligence that can think more like humans. In this work, we propose a theory of stochastic neural computing (SNC) in which streams of noisy inputs are transformed and processed through populations of nonlinearly coupled spiking neurons. To account for the propagation of correlated neural variability, we derive from first principles a moment embedding for spiking neural network (SNN). This leads to a new class of deep learning model called the moment neural network (MNN) which naturally generalizes rate-based neural networks to second order. As the MNN faithfully captures the stationary statistics of spiking neural activity, it can serve as a powerful proxy for training SNN with zero free parameters. Through joint manipulation of mean firing rate and noise correlations in a task-driven way, the model is able to learn inference tasks while simultaneously minimizing prediction uncertainty, resulting in enhanced inference speed. We further demonstrate the application of our method to Intel's Loihi neuromorphic hardware. The proposed theory of SNC may open up new opportunities for developing machine intelligence capable of computing uncertainty and for designing unconventional computing architectures.

cs.NE↗

arXiv2024

Towards free-response paradigm: a theory on decision-making in spiking neural networks

The energy-efficient and brain-like information processing abilities of Spiking Neural Networks (SNNs) have attracted considerable attention, establishing them as a crucial element of brain-inspired computing. One prevalent challenge encountered by SNNs is the trade-off between inference speed and accuracy, which requires sufficient time to achieve the desired level of performance. Drawing inspiration from animal behavior experiments that demonstrate a connection between decision-making reaction times, task complexity, and confidence levels, this study seeks to apply these insights to SNNs. The focus is on understanding how SNNs make inferences, with a particular emphasis on untangling the interplay between signal and noise in decision-making processes. The proposed theoretical framework introduces a new optimization objective for SNN training, highlighting the importance of not only the accuracy of decisions but also the development of predictive confidence through learning from past experiences. Experimental results demonstrate that SNNs trained according to this framework exhibit improved confidence expression, leading to better decision-making outcomes. In addition, a strategy is introduced for efficient decision-making during inference, which allows SNNs to complete tasks more quickly and can use stopping times as indicators of decision confidence. By integrating neuroscience insights with neuromorphic computing, this study opens up new possibilities to explore the capabilities of SNNs and advance their application in complex decision-making scenarios.

cs.NE↗

arXiv2024

Let LLMs Take on the Latest Challenges! A Chinese Dynamic Question Answering Benchmark

How to better evaluate the capabilities of Large Language Models (LLMs) is the focal point and hot topic in current LLMs research. Previous work has noted that due to the extremely high cost of iterative updates of LLMs, they are often unable to answer the latest dynamic questions well. To promote the improvement of Chinese LLMs' ability to answer dynamic questions, in this paper, we introduce CDQA, a Chinese Dynamic QA benchmark containing question-answer pairs related to the latest news on the Chinese Internet. We obtain high-quality data through a pipeline that combines humans and models, and carefully classify the samples according to the frequency of answer changes to facilitate a more fine-grained observation of LLMs' capabilities. We have also evaluated and analyzed mainstream and advanced Chinese LLMs on CDQA. Extensive experiments and valuable insights suggest that our proposed CDQA is challenging and worthy of more further study. We believe that the benchmark we provide will become one of the key data resources for improving LLMs' Chinese question-answering ability in the future.

cs.CL↗

arXiv2024

On Time-Varying Delayed Stochastic Differential Systems with Non-Markovian Switching Parameters

This paper focuses on time-varying delayed stochastic differential systems with stochastically switching parameters formulated by a unified switching behavior combining a discrete adapted process and a Cox process. Unlike prior studies limited to stationary and ergodic switching scenarios, our research emphasizes non-Markovian, non-stationary, and non-ergodic cases. It arrives at more general results regarding stability analysis with a more rigorous methodology. The theoretical results are validated through numerical examples.

math.DS↗

arXiv2024

Probabilistic computation and uncertainty quantification with emerging covariance

Building robust, interpretable, and secure AI system requires quantifying and representing uncertainty under a probabilistic perspective to mimic human cognitive abilities. However, probabilistic computation presents significant challenges for most conventional artificial neural network, as they are essentially implemented in a deterministic manner. In this paper, we develop an efficient probabilistic computation framework by truncating the probabilistic representation of neural activation up to its mean and covariance and construct a moment neural network that encapsulates the nonlinear coupling between the mean and covariance of the underlying stochastic network. We reveal that when only the mean but not the covariance is supervised during gradient-based learning, the unsupervised covariance spontaneously emerges from its nonlinear coupling with the mean and faithfully captures the uncertainty associated with model predictions. Our findings highlight the inherent simplicity of probabilistic computation by seamlessly incorporating uncertainty into model prediction, paving the way for integrating it into large-scale AI systems.

cs.LG↗

arXiv2024

Learn to integrate parts for whole through correlated neural variability

Sensory perception originates from the responses of sensory neurons, which react to a collection of sensory signals linked to various physical attributes of a singular perceptual object. Unraveling how the brain extracts perceptual information from these neuronal responses is a pivotal challenge in both computational neuroscience and machine learning. Here we introduce a statistical mechanical theory, where perceptual information is first encoded in the correlated variability of sensory neurons and then reformatted into the firing rates of downstream neurons. Applying this theory, we illustrate the encoding of motion direction using neural covariance and demonstrate high-fidelity direction recovery by spiking neural networks. Networks trained under this theory also show enhanced performance in classifying natural images, achieving higher accuracy and faster inference speed. Our results challenge the traditional view of neural covariance as a secondary factor in neural coding, highlighting its potential influence on brain function.

q-bio.NC↗

arXiv2023

Self-organization of nonlinearly coupled neural fluctuations into synergistic population codes

Neural activity in the brain exhibits correlated fluctuations that may strongly influence the properties of neural population coding. However, how such correlated neural fluctuations may arise from the intrinsic neural circuit dynamics and subsequently affect the computational properties of neural population activity remains poorly understood. The main difficulty lies in resolving the nonlinear coupling between correlated fluctuations with the overall dynamics of the system. In this study, we investigate the emergence of synergistic neural population codes from the intrinsic dynamics of correlated neural fluctuations in a neural circuit model capturing realistic nonlinear noise coupling of spiking neurons. We show that a rich repertoire of spatial correlation patterns naturally emerges in a bump attractor network and further reveals the dynamical regime under which the interplay between differential and noise correlations leads to synergistic codes. Moreover, we find that negative correlations may induce stable bound states between two bumps, a phenomenon previously unobserved in firing rate models. These noise-induced effects of bump attractors lead to a number of computational advantages including enhanced working memory capacity and efficient spatiotemporal multiplexing and can account for a range of cognitive and behavioral phenomena related to working memory. This study offers a dynamical approach to investigating realistic correlated neural fluctuations and insights to their roles in cortical computations.

physics.bio-ph↗

arXiv2023

A General Description of Criticality in Neural Network Models

Recent experimental observations have supported the hypothesis that the cerebral cortex operates in a dynamical regime near criticality, where the neuronal network exhibits a mixture of ordered and disordered patterns. However, A comprehensive study of how criticality emerges and how to reproduce it is still lacking. In this study, we investigate coupled networks with conductance-based neurons and illustrate the co-existence of different spiking patterns, including asynchronous irregular (AI) firing and synchronous regular (SR) state, along with a scale-invariant neuronal avalanche phenomenon (criticality). We show that fast-acting synaptic coupling can evoke neuronal avalanches in the mean-dominated regime but has little effect in the fluctuation-dominated regime. In a narrow region of parameter space, the network exhibits avalanche dynamics with power-law avalanche size and duration distributions. We conclude that three stages which may be responsible for reproducing the synchronized bursting: mean-dominated subthreshold dynamics, fast-initiating a spike event, and time-delayed inhibitory cancellation. Remarkably, we illustrate the mechanisms underlying critical avalanches in the presence of noise, which can be explained as a stochastic crossing state around the Hopf bifurcation under the mean-dominated regime. Moreover, we apply the ensemble Kalman filter to determine and track effective connections for the neuronal network. The method is validated on noisy synthetic BOLD signals and could exactly reproduce the corresponding critical network activity. Our results provide a special perspective to understand and model the criticality, which can be useful for large-scale modeling and computation of brain dynamics.

q-bio.NC↗

arXiv2023

Digital Twin Brain: a simulation and assimilation platform for whole human brain

In this work, we present a computing platform named digital twin brain (DTB) that can simulate spiking neuronal networks of the whole human brain scale and more importantly, a personalized biological brain structure. In comparison to most brain simulations with a homogeneous global structure, we highlight that the sparseness, couplingness and heterogeneity in the sMRI, DTI and PET data of the brain has an essential impact on the efficiency of brain simulation, which is proved from the scaling experiments that the DTB of human brain simulation is communication-intensive and memory-access intensive computing systems rather than computation-intensive. We utilize a number of optimization techniques to balance and integrate the computation loads and communication traffics from the heterogeneous biological structure to the general GPU-based HPC and achieve leading simulation performance for the whole human brain-scaled spiking neuronal networks. On the other hand, the biological structure, equipped with a mesoscopic data assimilation, enables the DTB to investigate brain cognitive function by a reverse-engineering method, which is demonstrated by a digital experiment of visual evaluation on the DTB. Furthermore, we believe that the developing DTB will be a promising powerful platform for a large of research orients including brain-inspiredintelligence, rain disease medicine and brain-machine interface.

cs.NE↗

arXiv2023

Modify Training Directions in Function Space to Reduce Generalization Error

We propose theoretical analyses of a modified natural gradient descent method in the neural network function space based on the eigendecompositions of neural tangent kernel and Fisher information matrix. We firstly present analytical expression for the function learned by this modified natural gradient under the assumptions of Gaussian distribution and infinite width limit. Thus, we explicitly derive the generalization error of the learned neural network function using theoretical methods from eigendecomposition and statistics theory. By decomposing of the total generalization error attributed to different eigenspace of the kernel in function space, we propose a criterion for balancing the errors stemming from training set and the distribution discrepancy between the training set and the true data. Through this approach, we establish that modifying the training direction of the neural network in function space leads to a reduction in the total generalization error. Furthermore, We demonstrate that this theoretical framework is capable to explain many existing results of generalization enhancing methods. These theoretical results are also illustrated by numerical examples on synthetic data.

stat.ML↗

arXiv2023

Synchronization analysis of high order layered complex networks

In this letter, we propose high order layered complex networks. The synchronization is discussed in detail. The relations of synchronization, individual coupling matrices and the intrinsic function of the uncoupled system are given. As special cases, synchronization of monolayer networks and multiplex networks discussed in literature can be obtained.

nlin.CD↗

arXiv2022

On a framework of data assimilation for neuronal networks

When handling real-world data modeled by a complex network dynamical system, the number of the parameters is always even much more than the size of the data. Therefore, in many cases, it is impossible to estimate these parameters and however, the exact value of each parameter is frequently less interesting than the distribution of the parameters may contain important information towards understanding the system and data. In this paper, we propose this question arising by employing a data assimilation approach to estimate the distribution of the parameters in the leakage-integrate-fire (LIF) neuronal network model from the experimental data, for example, the blood-oxygen-level-dependent (BOLD) signal. Herein, we assume that the parameters of the neurons and synapses are inhomogeneous but independently identical distributed following certain distribution with unknown hyperparameters. Thus, we estimate these hyperparameters of the distributions of the parameters, instead of estimating the parameters themselves. We formulate this problem under the framework of data assimilation and hierarchical Bayesian method, and present an efficient method named Hierarchical Data Assimilation (HDA) to conduct the statistical inference on the neuronal network model with the BOLD signal data simulated by the hemodynamic model. We consider the LIF neuronal networks with four synapses and show that the proposed algorithm can estimate the BOLD signals and the hyperparameters with good preciseness. In addition, we discuss the influence on the performance of the algorithm configuration and the LIF network model setup.

math.DS↗

arXiv2022

FedDKD: Federated Learning with Decentralized Knowledge Distillation

The performance of federated learning in neural networks is generally influenced by the heterogeneity of the data distribution. For a well-performing global model, taking a weighted average of the local models, as done by most existing federated learning algorithms, may not guarantee consistency with local models in the space of neural network maps. In this paper, we propose a novel framework of federated learning equipped with the process of decentralized knowledge distillation (FedDKD) (i.e., without data on the server). The FedDKD introduces a module of decentralized knowledge distillation (DKD) to distill the knowledge of the local models to train the global model by approaching the neural network map average based on the metric of divergence defined in the loss function, other than only averaging parameters as done in literature. Numeric experiments on various heterogeneous datasets reveal that FedDKD outperforms the state-of-the-art methods with more efficient communication and training in a few DKD steps, especially on some extremely heterogeneous datasets.

cs.LG↗

arXiv2022

Analytic Investigation for Spatio-temporal Patterns Propagation in Spiking Neural Networks

Based upon the moment closure approach, a Gaussian random field is constructed to quantitatively and analytically characterize the dynamics of a random point field. The approach provides us with a theoretical tool to investigate synchronized spike propagation in a feedforward or recurrent spiking neural network. We show that the balance between the excitation and inhibition postsynaptic potentials is required for the occurrence of synfire chains. In particular, with a balanced network, the critical packet size of invasion and annihilation is observed. We also derive a sufficient analytic condition for the synchronization propagation in an asynchronous environment, which further allows us to disclose the possibility of spatial synaptic structure to sustain a stable synfire chain. Our findings are in good agreement with simulations and help us understand the propagation of spatio-temporal patterns in a random point flied.

math.DS↗

arXiv2021

Simpler, Faster, Stronger: Breaking The log-K Curse On Contrastive Learners With FlatNCE

InfoNCE-based contrastive representation learners, such as SimCLR, have been tremendously successful in recent years. However, these contrastive schemes are notoriously resource demanding, as their effectiveness breaks down with small-batch training (i.e., the log-K curse, whereas K is the batch-size). In this work, we reveal mathematically why contrastive learners fail in the small-batch-size regime, and present a novel simple, non-trivial contrastive objective named FlatNCE, which fixes this issue. Unlike InfoNCE, our FlatNCE no longer explicitly appeals to a discriminative classification goal for contrastive learning. Theoretically, we show FlatNCE is the mathematical dual formulation of InfoNCE, thus bridging the classical literature on energy modeling; and empirically, we demonstrate that, with minimal modification of code, FlatNCE enables immediate performance boost independent of the subject-matter engineering efforts. The significance of this work is furthered by the powerful generalization of contrastive learning techniques, and the introduction of new tools to monitor and diagnose contrastive training. We substantiate our claims with empirical evidence on CIFAR10, ImageNet, and other datasets, where FlatNCE consistently outperforms InfoNCE.

stat.ML↗

arXiv2021

Finite time Convergence of Pinning Synchronization with a single Nonlinear Controller

In this paper, we discuss distributive synchronization of complex networks in finite time, with a single nonlinear pinning controller. The results apply to heterogeneous dynamic networks, too. Different from many models, which assume the coupling matrix being symmetric (or the connecting graph is undirected), here, the coupling matrix is asymmetric (or the connecting graph is directed

nlin.AO↗

arXiv2021

An Event-based Parameter Switching Method for Controlling Cybersecurity Dynamics

This paper proposes a new event-based parameter switching method for the control tasks of cybersecurity in the context of preventive and reactive cyber defense dynamics. Our parameter switching method helps avoid excessive control costs as well as guarantees the dynamics to converge as our desired speed. Meanwhile, it can be proved that this approach is Zeno-free. A new estimation method with adaptive time windows is used to bridge the gap between the probability state and the sampling state. With the new estimation method, several practical experiments are given afterwards.

cs.CR↗

arXiv2021

Finite time Convergence of Pinning Synchronization with Linear and Nonlinear Controllers

In this paper, we propose several models, which can realize synchronization of complex networks in finite time effectively. The results apply to heterogeneous dynamic networks, too. The mechanism of finite time convergence is revealed. Different from many models, which assume the coupling matrix being symmetric (or the connecting graph is undirected), here, the coupling matrix is asymmetric (or the connecting graph is directed).

nlin.AO↗