arXiv subjects

Marc Bocquet

Publications and source records attributed to Marc Bocquet.

At least 55 records · Page 3Linked to original sources

arXiv2020

Low Power In-Memory Implementation of Ternary Neural Networks with Resistive RAM-Based Synapse

The design of systems implementing low precision neural networks with emerging memories such as resistive random access memory (RRAM) is a major lead for reducing the energy consumption of artificial intelligence (AI). Multiple works have for example proposed in-memory architectures to implement low power binarized neural networks. These simple neural networks, where synaptic weights and neuronal activations assume binary values, can indeed approach state-of-the-art performance on vision tasks. In this work, we revisit one of these architectures where synapses are implemented in a differential fashion to reduce bit errors, and synaptic weights are read using precharge sense amplifiers. Based on experimental measurements on a hybrid 130 nm CMOS/RRAM chip and on circuit simulation, we show that the same memory array architecture can be used to implement ternary weights instead of binary weights, and that this technique is particularly appropriate if the sense amplifier is operated in near-threshold regime. We also show based on neural network simulation on the CIFAR-10 image recognition task that going from binary to ternary neural networks significantly increases neural network performance. These results highlight that AI circuits function may sometimes be revisited when operated in low power regimes.

cs.ET↗

arXiv2020

Bayesian inference of chaotic dynamics by merging data assimilation, machine learning and expectation-maximization

The reconstruction from observations of high-dimensional chaotic dynamics such as geophysical flows is hampered by (i) the partial and noisy observations that can realistically be obtained, (ii) the need to learn from long time series of data, and (iii) the unstable nature of the dynamics. To achieve such inference from the observations over long time series, it has been suggested to combine data assimilation and machine learning in several ways. We show how to unify these approaches from a Bayesian perspective using expectation-maximization and coordinate descents. In doing so, the model, the state trajectory and model error statistics are estimated all together. Implementations and approximations of these methods are discussed. Finally, we numerically and successfully test the approach on two relevant low-order chaotic models with distinct identifiability.

stat.ML↗

arXiv2020

On temporal scale separation in coupled data assimilation with the ensemble Kalman filter

Coupled data assimilation (CDA) distinctively appears as a main concern in numerical weather and climate prediction with major efforts put forward worldwide. The core issue is the scale separation acting as a barrier that hampers the propagation of the information across model components. We provide a brief survey of CDA, and then focus on CDA using the ensemble Kalman filter (EnKF). We consider first coupled equations with temporal scale difference and deduce that: (i) cross components effects are strong from the slow to the fast scale, but, (ii) intra-component effects are much stronger in the fast scale. While observing the slow scale is desirable and benefits the fast, the latter must be observed with high frequency otherwise the error will affect the slow scale. Experiments are performed using the atmosphere-ocean model, MAOOAM. Six configurations are considered, differing for the strength of the atmosphere-ocean coupling and/or the number of model modes. A comprehensive dynamical characterisation of the model configurations is provided by examining the Lyapunov spectrum, Kolmogorov entropy and Kaplan-Yorke attractor dimension. We also compute the covariant Lyapunov vectors and use them to explain how model instabilities act on different model's modes according to the coupling strength. The experiments confirm the importance of observing the fast scale, but show also that, despite its slow temporal scale, frequent observations in the ocean are beneficial. The relation between the ensemble size and the unstable subspace dimension has been studied. Results largely ratify what known for uncoupled system: the condition N>n0 is necessary for the EnKF to converge. But the quasi-degeneracy of the Lyapunov spectrum of MAOOAM, with many near-zero exponents, is potentially the cause of the smooth gradual reduction of the analysis error observed for some model configurations, even when N>n0.

physics.ao-ph↗

arXiv2019

Digital Biologically Plausible Implementation of Binarized Neural Networks with Differential Hafnium Oxide Resistive Memory Arrays

The brain performs intelligent tasks with extremely low energy consumption. This work takes inspiration from two strategies used by the brain to achieve this energy efficiency: the absence of separation between computing and memory functions, and the reliance on low precision computation. The emergence of resistive memory technologies indeed provides an opportunity to co-integrate tightly logic and memory in hardware. In parallel, the recently proposed concept of Binarized Neural Network, where multiplications are replaced by exclusive NOR (XNOR) logic gates, offers a way to implement artificial intelligence using very low precision computation. In this work, we therefore propose a strategy to implement low energy Binarized Neural Networks, which employs brain-inspired concepts, while retaining energy benefits from digital electronics. We design, fabricate and test a memory array, including periphery and sensing circuits, optimized for this in-memory computing scheme. Our circuit employs hafnium oxide resistive memory integrated in the back end of line of a 130 nanometer CMOS process, in a two transistors - two resistors cell, which allows performing the exclusive NOR operations of the neural network directly within the sense amplifiers. We show, based on extensive electrical measurements, that our design allows reducing the amount of bit errors on the synaptic weights, without the use of formal error correcting codes. We design a whole system using this memory array. We show on standard machine learning tasks (MNIST, CIFAR-10, ImageNet and an ECG task) that the system has an inherent resilience to bit errors. We evidence that its energy consumption is attractive compared to more standard approaches, and that it can use the memory devices in regimes where they exhibit particularly low programming energy and high endurance.

cs.ET↗

arXiv2019

Implementing Binarized Neural Networks with Magnetoresistive RAM without Error Correction

One of the most exciting applications of Spin Torque Magnetoresistive Random Access Memory (ST-MRAM) is the in-memory implementation of deep neural networks, which could allow improving the energy efficiency of Artificial Intelligence by orders of magnitude with regards to its implementation on computers and graphics cards. In particular, ST-MRAM could be ideal for implementing Binarized Neural Networks (BNNs), a type of deep neural networks discovered in 2016, which can achieve state-of-the-art performance with a highly reduced memory footprint with regards to conventional artificial intelligence approaches. The challenge of ST-MRAM, however, is that it is prone to write errors and usually requires the use of error correction. In this work, we show that these bit errors can be tolerated by BNNs to an outstanding level, based on examples of image recognition tasks (MNIST, CIFAR-10 and ImageNet): bit error rates of ST-MRAM up to 0.1% have little impact on recognition accuracy. The requirements for ST-MRAM are therefore considerably relaxed for BNNs with regards to traditional applications. By consequence, we show that for BNNs, ST-MRAMs can be programmed with weak (low-energy) programming conditions, without error correcting codes. We show that this result can allow the use of low energy and low area ST-MRAM cells, and show that the energy savings at the system level can reach a factor two.

cs.ET↗

arXiv2019

Stochastic Computing for Hardware Implementation of Binarized Neural Networks

Binarized Neural Networks, a recently discovered class of neural networks with minimal memory requirements and no reliance on multiplication, are a fantastic opportunity for the realization of compact and energy efficient inference hardware. However, such neural networks are generally not entirely binarized: their first layer remains with fixed point input. In this work, we propose a stochastic computing version of Binarized Neural Networks, where the input is also binarized. Simulations on the example of the Fashion-MNIST and CIFAR-10 datasets show that such networks can approach the performance of conventional Binarized Neural Networks. We evidence that the training procedure should be adapted for use with stochastic computing. Finally, the ASIC implementation of our scheme is investigated, in a system that closely associates logic and memory, implemented by Spin Torque Magnetoresistive Random Access Memory. This analysis shows that the stochastic computing approach can allow considerable savings with regards to conventional Binarized Neural networks in terms of area (62% area reduction on the Fashion-MNIST task). It can also allow important savings in terms of energy consumption, if we accept reasonable reduction of accuracy: for example a factor 2.1 can be saved, with the cost of 1.4% in Fashion-MNIST test accuracy. These results highlight the high potential of Binarized Neural Networks for hardware implementation, and that adapting them to hardware constrains can provide important benefits.

cs.ET↗

arXiv2019

Outstanding Bit Error Tolerance of Resistive RAM-Based Binarized Neural Networks

Resistive random access memories (RRAM) are novel nonvolatile memory technologies, which can be embedded at the core of CMOS, and which could be ideal for the in-memory implementation of deep neural networks. A particularly exciting vision is using them for implementing Binarized Neural Networks (BNNs), a class of deep neural networks with a highly reduced memory footprint. The challenge of resistive memory, however, is that they are prone to device variation, which can lead to bit errors. In this work we show that BNNs can tolerate these bit errors to an outstanding level, through simulations of networks on the MNIST and CIFAR10 tasks. If a standard BNN is used, up to 10^-4 bit error rate can be tolerated with little impact on recognition performance on both MNIST and CIFAR10. We then show that by adapting the training procedure to the fact that the BNN will be operated on error-prone hardware, this tolerance can be extended to a bit error rate of 4x10^-2. The requirements for RRAM are therefore a lot less stringent for BNNs than more traditional applications. We show, based on experimental measurements on a RRAM HfO2 technology, that this result can allow reduce RRAM programming energy by a factor 30.

cs.ET↗

arXiv2019

In-Memory and Error-Immune Differential RRAM Implementation of Binarized Deep Neural Networks

RRAM-based in-Memory Computing is an exciting road for implementing highly energy efficient neural networks. This vision is however challenged by RRAM variability, as the efficient implementation of in-memory computing does not allow error correction. In this work, we fabricated and tested a differential HfO2-based memory structure and its associated sense circuitry, which are ideal for in-memory computing. For the first time, we show that our approach achieves the same reliability benefits as error correction, but without any CMOS overhead. We show, also for the first time, that it can naturally implement Binarized Deep Neural Networks, a very recent development of Artificial Intelligence, with extreme energy efficiency, and that the system is fully satisfactory for image recognition applications. Finally, we evidence how the extra reliability provided by the differential memory allows programming the devices in low voltage conditions, where they feature high endurance of billions of cycles.

cs.ET↗

arXiv2019

Adaptive covariance inflation in the ensemble Kalman filter by Gaussian scale mixtures

This paper studies multiplicative inflation: the complementary scaling of the state covariance in the ensemble Kalman filter (EnKF). Firstly, error sources in the EnKF are catalogued and discussed in relation to inflation; nonlinearity is given particular attention as a source of sampling error. In response, the "finite-size" refinement known as the EnKF-N is re-derived via a Gaussian scale mixture, again demonstrating how it yields adaptive inflation. Existing methods for adaptive inflation estimation are reviewed, and several insights are gained from a comparative analysis. One such adaptive inflation method is selected to complement the EnKF-N to make a hybrid that is suitable for contexts where model error is present and imperfectly parameterized. Benchmarks are obtained from experiments with the two-scale Lorenz model and its slow-scale truncation. The proposed hybrid EnKF-N method of adaptive inflation is found to yield systematic accuracy improvements in comparison with the existing methods, albeit to a moderate degree.

physics.data-an↗

arXiv2018

Estimating model evidence using ensemble-based data assimilation with localization - The model selection problem

IIn recent years, there has been a growing interest in applying data assimilation (DA) methods, originally designed for state estimation, to the model selection problem. In this setting, Carrassi et al. (2017) introduced the contextual formulation of model evidence (CME) and showed that CME can be efficiently computed using a hierarchy of ensemble-based DA procedures. Although Carrassi et al. (2017) analyzed the DA methods most commonly used for operational atmospheric and oceanic prediction worldwide, they did not study these methods in conjunction with localization to a specific domain. Yet any application of ensemble DA methods to realistic geophysical models requires the implementation of some form of localization. The present study extends the theory for estimating CME to ensemble DA methods with domain localization. The domain-localized CME (DL-CME) developed herein is tested for model selection with two models: (i) the Lorenz 40-variable mid-latitude atmospheric dynamics model (L95); and (ii) the simplified global atmospheric SPEEDY model. The CME is compared to the root-mean-square-error (RMSE) as a metric for model selection. The experiments show that CME improves systematically over the RMSE, and that this skill improvement is further enhanced by applying localization in the estimate of the CME, using the DL-CME. The potential use and range of applications of the CME and DL-CME as a model selection metric are also discussed.

stat.ME↗

arXiv2018

Asymptotic forecast uncertainty and the unstable subspace in the presence of additive model error

It is well understood that dynamic instability is among the primary drivers of forecast uncertainty in chaotic, physical systems. Data assimilation techniques have been designed to exploit this phenomena, reducing the effective dimension of the data assimilation problem to the directions of rapidly growing errors. Recent mathematical work has, moreover, provided formal proofs of the central hypothesis of the Assimilation in the Unstable Subspace methodology of Anna Trevisan and her collaborators: for filters and smoothers in perfect, linear, Gaussian models, the distribution of forecast errors asymptotically conforms to the unstable-neutral subspace. Specifically, the column span of the forecast and posterior error covariances asymptotically align with the span of backward Lyapunov vectors with non-negative exponents. Earlier mathematical studies have focused on perfect models, and this current work now explores the relationship between dynamical instability, the precision of observations and the evolution of forecast error in linear models with additive model error. We prove bounds for the asymptotic uncertainty, explicitly relating the rate of dynamical expansion, model precision and observational accuracy. Formalizing this relationship, we provide a novel, necessary criterion for the boundedness of forecast errors. Furthermore, we numerically explore the relationship between observational design, dynamical instability and filter boundedness. Additionally, we include a detailed introduction to the Multiplicative Ergodic Theorem and to the theory and construction of Lyapunov vectors.

math.OC↗

arXiv2018

Data Assimilation in the Geosciences - An overview on methods, issues and perspectives

We commonly refer to state-estimation theory in geosciences as data assimilation. This term encompasses the entire sequence of operations that, starting from the observations of a system, and from additional statistical and dynamical information (such as a dynamical evolution model), provides an estimate of its state. Data assimilation is standard practice in numerical weather prediction, but its application is becoming widespread in many other areas of climate, atmosphere, ocean and environment modeling; in all circumstances where one intends to estimate the state of a large dynamical system based on limited information. While the complexity of data assimilation, and of the methods thereof, stands on its interdisciplinary nature across statistics, dynamical systems and numerical optimization, when applied to geosciences an additional difficulty arises by the continually increasing sophistication of the environmental models. Thus, in spite of data assimilation being nowadays ubiquitous in geosciences, it has so far remained a topic mostly reserved to experts. We aim this overview article at geoscientists with a background in mathematical and physical modeling, who are interested in the rapid development of data assimilation and its growing domains of application in environmental science, but so far have not delved into its conceptual and methodological complexities.

physics.ao-ph↗

arXiv2017

An iterative ensemble Kalman filter in presence of additive model error

The iterative ensemble Kalman filter (IEnKF) in a deterministic framework was introduced in Sakov et al. (2012) to extend the ensemble Kalman filter (EnKF) and improve its performance in mildly up to strongly nonlinear cases. However, the IEnKF assumes that the model is perfect. This assumption simplified the update of the system at a time different from the observation time, which made it natural to apply the IEnKF for smoothing. In this study, we generalise the IEnKF to the case of imperfect model with additive model error. The new method called IEnKF-Q conducts a Gauss-Newton minimisation in ensemble space. It combines the propagated analysed ensemble anomalies from the previous cycle and model noise ensemble anomalies into a single ensemble of anomalies, and by doing so takes an algebraic form similar to that of the IEnKF. The performance of the IEnKF-Q is tested in a number of experiments with the Lorenz-96 model, which show that the method consistently outperforms both the EnKF and the IEnKF naively modified to accommodate additive model noise.

physics.ao-ph↗

arXiv2017

Stochastic parameterization identification using ensemble Kalman filtering combined with expectation-maximization and Newton-Raphson maximum likelihood methods

For modelling geophysical systems, large-scale processes are described through a set of coarse-grained dynamical equations while small-scale processes are represented via parameterizations. This work proposes a method for identifying the best possible stochastic parameterization from noisy data. State-the-art sequential estimation methods such as Kalman and particle filters do not achieve this goal succesfully because both suffer from the collapse of the parameter posterior distribution. To overcome this intrinsic limitation, we propose two statistical learning methods. They are based on the combination of two methodologies: the maximization of the likelihood via Expectation-Maximization (EM) and Newton-Raphson (NR) algorithms which are mainly applied in the statistic and machine learning communities, and the ensemble Kalman filter (EnKF). The methods are derived using a Bayesian approach for a hidden Markov model. They are applied to infer deterministic and stochastic physical parameters from noisy observations in coarse-grained dynamical models. Numerical experiments are conducted using the Lorenz-96 dynamical system with one and two scales as a proof-of-concept. The imperfect coarse-grained model is modelled through a one-scale Lorenz-96 system in which a stochastic parameterization is incorpored to represent the small-scale dynamics. The algorithms are able to identify an optimal stochastic parameterization with a good accuracy under moderate observational noise. The proposed EnKF-EM and EnKF-NR are promising statistical learning methods for developing stochastic parameterizations in high-dimensional geophysical models.

physics.ao-ph↗

arXiv2017

Degenerate Kalman filter error covariances and their convergence onto the unstable subspace

The characteristics of the model dynamics are critical in the performance of (ensemble) Kalman filters. In particular, as emphasized in the seminal work of Anna Trevisan and co-authors, the error covariance matrix is asymptotically supported by the unstable-neutral subspace only, i.e., it is spanned by the backward Lyapunov vectors with non-negative exponents. This behavior is at the core of algorithms known as Assimilation in the Unstable Subspace, although a formal proof was still missing. This paper provides the analytical proof of the convergence of the Kalman filter covariance matrix onto the unstable-neutral subspace when the dynamics and the observation operator are linear and when the dynamical model is error-free, for any, possibly rank-deficient, initial error covariance matrix. The rate of convergence is provided as well. The derivation is based on an expression that explicitly relates the error covariances at an arbitrary time to the initial ones. It is also shown that if the unstable and neutral directions of the model are sufficiently observed and if the column space of the initial covariance matrix has a non-zero projection onto all of the forward Lyapunov vectors associated with the unstable and neutral directions of the dynamics, the covariance matrix of the Kalman filter collapses onto an asymptotic sequence which is independent of the initial covariances. Numerical results are also shown to illustrate and support the theoretical findings.

math.OC↗

arXiv2016

Estimating model evidence using data assimilation

We review the field of data assimilation (DA) from a Bayesian perspective and show that, in addition to its by now common application to state estimation, DA may be used for model selection. An important special case of the latter is the discrimination between a factual model --- which corresponds, to the best of the modeler's knowledge, to the situation in the actual world in which a sequence of events has occurred --- and a counterfactual model, in which a particular forcing or process might be absent or just quantitatively different from the actual world. Three different ensemble-DA methods are reviewed for this purpose: the ensemble Kalman filter (EnKF), the ensemble four-dimensional variational smoother (En-4D-Var), and the iterative ensemble Kalman smoother (IEnKS). An original contextual formulation of model evidence (CME) is introduced. It is shown how to apply these three methods to compute CME, using the approximated time-dependent probability distribution functions (pdfs) each of them provide in the process of state estimation. The theoretical formulae so derived are applied to two simplified nonlinear and chaotic models: (i) the Lorenz three-variable convection (L63) model, and (ii) the Lorenz 40-variable mid-latitude atmospheric dynamics model (L95). The numerical results of these three DA-based methods and those of an integration based on importance sampling are compared. It is found that better CME estimates are obtained by using DA, and the IEnKS method appears to be best among the DA methods. Differences among the performance of the three DA-based methods are discussed as a function of model properties. Finally, the methodology is implemented for parameter estimation and for event attribution.

stat.AP↗

arXiv2015

DADA: Data Assimilation for the Detection and Attribution of Weather- and Climate-related Events

We describe a new approach allowing for systematic causal attribution of weather and climate-related events, in near-real time. The method is purposely designed to facilitate its implementation at meteorological centers by relying on data treatments that are routinely performed when numerically forecasting the weather. Namely, we show that causal attribution can be obtained as a by-product of so-called data assimilation procedures that are run on a daily basis to update the meteorological model with new atmospheric observations; hence, the proposed methodology can take advantage of the powerful computational and observational capacity of weather forecasting centers. We explain the theoretical rationale of this approach and sketch the most prominent features of a "data assimilation-based detection and attribution" (DADA) procedure. The proposal is illustrated in the context of the classical three-variable Lorenz model with additional forcing. Several theoretical and practical research questions that need to be addressed to make the proposal readily operational within weather forecasting centers are finally laid out.

stat.AP↗

arXiv2014

Local ensemble transform Kalman filter, a fast non-stationary control law for adaptive optics on ELTs: theoretical aspects and first simulation results

We propose a new algorithm for an adaptive optics system control law, based on the Linear Quadratic Gaussian approach and a Kalman Filter adaptation with localizations. It allows to handle non-stationary behaviors, to obtain performance close to the optimality defined with the residual phase variance minimization criterion, and to reduce the computational burden with an intrinsically parallel implementation on the Extremely Large Telescopes (ELTs).

astro-ph.IM↗