arXiv subjects

Daniel Soudry

Publications and source records attributed to Daniel Soudry.

85 records · Page 5Linked to original sources

arXiv2016

No bad local minima: Data independent training error guarantees for multilayer neural networks

We use smoothed analysis techniques to provide guarantees on the training loss of Multilayer Neural Networks (MNNs) at differentiable local minima. Specifically, we examine MNNs with piecewise linear activation functions, quadratic loss and a single output, under mild over-parametrization. We prove that for a MNN with one hidden layer, the training error is zero at every differentiable local minimum, for almost every dataset and dropout-like noise realization. We then extend these results to the case of more than one hidden layer. Our theoretical guarantees assume essentially nothing on the training data, and are verified numerically. These results suggest why the highly non-convex loss of such MNNs can be easily optimized using local updates (e.g., stochastic gradient descent), as observed empirically.

stat.ML↗

arXiv2016

Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1

We introduce a method to train Binarized Neural Networks (BNNs) - neural networks with binary weights and activations at run-time. At training-time the binary weights and activations are used for computing the parameters gradients. During the forward pass, BNNs drastically reduce memory size and accesses, and replace most arithmetic operations with bit-wise operations, which is expected to substantially improve power-efficiency. To validate the effectiveness of BNNs we conduct two sets of experiments on the Torch7 and Theano frameworks. On both, BNNs achieved nearly state-of-the-art results over the MNIST, CIFAR-10 and SVHN datasets. Last but not least, we wrote a binary matrix multiplication GPU kernel with which it is possible to run our MNIST BNN 7 times faster than with an unoptimized GPU kernel, without suffering any loss in classification accuracy. The code for training and running our BNNs is available on-line.

cs.LG↗

arXiv2016

Binarized Neural Networks

We introduce a method to train Binarized Neural Networks (BNNs) - neural networks with binary weights and activations at run-time and when computing the parameters' gradient at train-time. We conduct two sets of experiments, each based on a different framework, namely Torch7 and Theano, where we train BNNs on MNIST, CIFAR-10 and SVHN, and achieve nearly state-of-the-art results. During the forward pass, BNNs drastically reduce memory size and accesses, and replace most arithmetic operations with bit-wise operations, which might lead to a great increase in power-efficiency. Last but not least, we wrote a binary matrix multiplication GPU kernel with which it is possible to run our MNIST BNN 7 times faster than with an unoptimized GPU kernel, without suffering any loss in classification accuracy. The code for training and running our BNNs is available.

cs.LG↗

arXiv2015

Training Binary Multilayer Neural Networks for Image Classification using Expectation Backpropagation

Compared to Multilayer Neural Networks with real weights, Binary Multilayer Neural Networks (BMNNs) can be implemented more efficiently on dedicated hardware. BMNNs have been demonstrated to be effective on binary classification tasks with Expectation BackPropagation (EBP) algorithm on high dimensional text datasets. In this paper, we investigate the capability of BMNNs using the EBP algorithm on multiclass image classification tasks. The performances of binary neural networks with multiple hidden layers and different numbers of hidden units are examined on MNIST. We also explore the effectiveness of image spatial filters and the dropout technique in BMNNs. Experimental results on MNIST dataset show that EBP can obtain 2.12% test error with binary weights and 1.66% test error with real weights, which is comparable to the results of standard BackPropagation algorithm on fully connected MNNs.

cs.NE↗

arXiv2014

A shotgun sampling solution for the common input problem in neural connectivity inference

Inferring connectivity in neuronal networks remains a key challenge in statistical neuroscience. The `common input' problem presents the major roadblock: it is difficult to reliably distinguish causal connections between pairs of observed neurons from correlations induced by common input from unobserved neurons. Since available recording techniques allow us to sample from only a small fraction of large networks simultaneously with sufficient temporal resolution, naive connectivity estimators that neglect these common input effects are highly biased. This work proposes a `shotgun' experimental design, in which we observe multiple sub-networks briefly, in a serial manner. Thus, while the full network cannot be observed simultaneously at any given time, we may be able to observe most of it during the entire experiment. Using a generalized linear model for a spiking recurrent neural network, we develop scalable approximate Bayesian methods to perform network inference given this type of data, in which only a small fraction of the network is observed in each time bin. We demonstrate in simulation that, using this method: (1) The shotgun experimental design can eliminate the biases induced by common input effects. (2) Networks with thousands of neurons, in which only a small fraction of the neurons is observed in each time bin, could be quickly and accurately estimated. (3) Performance can be improved if we exploit prior information about the probability of having a connection between two neurons, its dependence on neuronal cell types (e.g., Dale's law), or its dependence on the distance between neurons.

q-bio.NC↗

arXiv2014

A structured matrix factorization framework for large scale calcium imaging data analysis

We present a structured matrix factorization approach to analyzing calcium imaging recordings of large neuronal ensembles. Our goal is to simultaneously identify the locations of the neurons, demix spatially overlapping components, and denoise and deconvolve the spiking activity of each neuron from the slow dynamics of the calcium indicator. The matrix factorization approach relies on the observation that the spatiotemporal fluorescence activity can be expressed as a product of two matrices: a spatial matrix that encodes the location of each neuron in the optical field and a temporal matrix that characterizes the calcium concentration of each neuron over time. We present a simple approach for estimating the dynamics of the calcium indicator as well as the observation noise statistics from the observed data. These parameters are then used to set up the matrix factorization problem in a constrained form that requires no further parameter tuning. We discuss initialization and post-processing techniques that enhance the performance of our method, along with efficient and largely parallelizable algorithms. We apply our method to {\it in vivo} large scale multi-neuronal imaging data and also demonstrate how similar methods can be used for the analysis of {\it in vivo} dendritic imaging data.

q-bio.NC↗

arXiv2014

Spiking input-output relation for general biophysical neuron models

Cortical neurons include many sub-cellular processes, operating at multiple timescales, which may affect their response to stimulation through non-linear and stochastic interaction with ion channels and ionic concentrations. Since new processes are constantly being discovered, biophysical neuron models increasingly become "too complex to be useful" yet "too simple to be realistic". A fundamental open question in theoretical neuroscience pertains to how this deadlock may be resolved. In order to tackle this problem, we first define the notion of a "excitable neuron model". Then we analytically derive the input-output relation of such neuronal models, relating input spike trains to output spikes based on known biophysical properties. Thus we obtain closed-form expressions for the mean firing rates, all second order statistics (input-state-output correlation and spectra) and construct optimal linear estimators for the neuronal response and internal state. These results are guaranteed to hold, given a few generic assumptions, for any stochastic biophysical neuron model (with an arbitrary number of slow kinetic processes) under general sparse stimulation. This solution suggests that the common simplifying approach that ignores much of the complexity of the neuron might actually be unnecessary and even deleterious in some cases. Specifically, the stochasticity of ion channels and the temporal sparseness of inputs is exactly what rendered our analysis tractable, allowing us to incorporate slow kinetics.

q-bio.NC↗

arXiv2014

The neuron's response at extended timescales

Many systems are modulated by unknown slow processes. This hinders analysis in highly non-linear systems, such as excitable systems. We show that for such systems, if the input matches the sparse `spiky' nature of the output, the spiking input-output relation can be derived. We use this relation to reproduce and interpret the irregular and complex 1/f response observed in isolated neurons stimulated over days. We decompose the neuronal response into contributions from its long history of internal noise and its short (few minutes) history of inputs, quantifying memory, noise and stability.

q-bio.NC↗

arXiv2013

Mean Field Bayes Backpropagation: scalable training of multilayer neural networks with binary weights

Significant success has been reported recently using deep neural networks for classification. Such large networks can be computationally intensive, even after training is over. Implementing these trained networks in hardware chips with a limited precision of synaptic weights may improve their speed and energy efficiency by several orders of magnitude, thus enabling their integration into small and low-power electronic devices. With this motivation, we develop a computationally efficient learning algorithm for multilayer neural networks with binary weights, assuming all the hidden neurons have a fan-out of one. This algorithm, derived within a Bayesian probabilistic online setting, is shown to work well for both synthetic and real-world problems, performing comparably to algorithms with real-valued weights, while retaining computational tractability.

stat.ML↗

arXiv2013

Slow dynamics of neuronal excitability under pulse stimulation

Neurons fire irregularly on multiple timescales when stimulated with a periodic pulse train. This raises two questions: Does this irregularity imply significant intrinsic stochasticity? Can existing neuron models be readily extended to describe behavior at long timescales? We show here that for commonly studied neuronal models, dynamics is not chaotic and can only produce stable and periodic firing patterns. This is done by transforming the neuron model to an analytically tractable piecewise linear discrete map. Thus we answer "yes" and "no" to the above questions, respectively.

q-bio.CB↗

arXiv2012

An exact reduction of the master equation to a strictly stable system with an explicit expression for the stationary distribution

The evolution of a continuous time Markov process with a finite number of states is usually calculated by the Master equation - a linear differential equations with a singular generator matrix. We derive a general method for reducing the dimensionality of the Master equation by one by using the probability normalization constraint, thus obtaining a affine differential equation with a (non-singular) stable generator matrix. Additionally, the reduced form yields a simple explicit expression for the stationary probability distribution, which is usually derived implicitly. Finally, we discuss the application of this method to stochastic differential equations.

q-bio.QM↗

arXiv2011

Simple, Fast and Accurate Implementation of the Diffusion Approximation Algorithm for Stochastic Ion Channels with Multiple States

The phenomena that emerge from the interaction of the stochastic opening and closing of ion channels (channel noise) with the non-linear neural dynamics are essential to our understanding of the operation of the nervous system. The effects that channel noise can have on neural dynamics are generally studied using numerical simulations of stochastic models. Algorithms based on discrete Markov Chains (MC) seem to be the most reliable and trustworthy, but even optimized algorithms come with a non-negligible computational cost. Diffusion Approximation (DA) methods use Stochastic Differential Equations (SDE) to approximate the behavior of a number of MCs, considerably speeding up simulation times. However, model comparisons have suggested that DA methods did not lead to the same results as in MC modeling in terms of channel noise statistics and effects on excitability. Recently, it was shown that the difference arose because MCs were modeled with coupled activation subunits, while the DA was modeled using uncoupled activation subunits. Implementations of DA with coupled subunits, in the context of a specific kinetic scheme, yielded similar results to MC. However, it remained unclear how to generalize these implementations to different kinetic schemes, or whether they were faster than MC algorithms. Additionally, a steady state approximation was used for the stochastic terms, which, as we show here, can introduce significant inaccuracies. We derived the SDE explicitly for any given ion channel kinetic scheme. The resulting generic equations were surprisingly simple and interpretable - allowing an easy and efficient DA implementation. The algorithm was tested in a voltage clamp simulation and in two different current clamp simulations, yielding the same results as MC modeling. Also, the simulation efficiency of this DA method demonstrated considerable superiority over MC methods.

q-bio.NC↗

arXiv2010

History dependent dynamics in a generic model of ion channels - an analytic study

Recent experiments have demonstrated that the timescale of adaptation of single neurons and ion channel populations to stimuli slows down as the length of stimulation increases; in fact, no upper bound on temporal time-scales seems to exist in such systems. Furthermore, patch clamp experiments on single ion channels have hinted at the existence of large, mostly unobservable, inactivation state spaces within a single ion channel. This raises the question of the relation between this multitude of inactivation states and the observed behavior. In this work we propose a minimal model for ion channel dynamics which does not assume any specific structure of the inactivation state space. The model is simple enough to render an analytical study possible. This leads to a clear and concise explanation of the experimentally observed exponential history-dependent relaxation in sodium channels in a voltage clamp setting, and shows that their recovery rate from slow inactivation must be voltage dependent. Furthermore, we predict that history-dependent relaxation cannot be created by overly sparse spiking activity. While the model was created with ion channel populations in mind, its simplicity and genericalness render it a good starting point for modeling similar effects in other systems, and for scaling up to higher levels such as single neurons which are also known to exhibit multiple time scales.

q-bio.SC↗