arXiv subjects

Osvaldo Simeone

Publications and source records attributed to Osvaldo Simeone.

At least 235 records · Page 13Linked to original sources

arXiv2020

BiSNN: Training Spiking Neural Networks with Binary Weights via Bayesian Learning

Artificial Neural Network (ANN)-based inference on battery-powered devices can be made more energy-efficient by restricting the synaptic weights to be binary, hence eliminating the need to perform multiplications. An alternative, emerging, approach relies on the use of Spiking Neural Networks (SNNs), biologically inspired, dynamic, event-driven models that enhance energy efficiency via the use of binary, sparse, activations. In this paper, an SNN model is introduced that combines the benefits of temporally sparse binary activations and of binary weights. Two learning rules are derived, the first based on the combination of straight-through and surrogate gradient techniques, and the second based on a Bayesian paradigm. Experiments validate the performance loss with respect to full-precision implementations, and demonstrate the advantage of the Bayesian paradigm in terms of accuracy and calibration.

cs.LG↗

arXiv2020

Spiking Neural Networks -- Part I: Detecting Spatial Patterns

Spiking Neural Networks (SNNs) are biologically inspired machine learning models that build on dynamic neuronal models processing binary and sparse spiking signals in an event-driven, online, fashion. SNNs can be implemented on neuromorphic computing platforms that are emerging as energy-efficient co-processors for learning and inference. This is the first of a series of three papers that introduce SNNs to an audience of engineers by focusing on models, algorithms, and applications. In this first paper, we first cover neural models used for conventional Artificial Neural Networks (ANNs) and SNNs. Then, we review learning algorithms and applications for SNNs that aim at mimicking the functionality of ANNs by detecting or generating spatial patterns in rate-encoded spiking signals. We specifically discuss ANN-to-SNN conversion and neural sampling. Finally, we validate the capabilities of SNNs for detecting and generating spatial patterns through experiments.

cs.NE↗

arXiv2020

Spiking Neural Networks -- Part III: Neuromorphic Communications

Synergies between wireless communications and artificial intelligence are increasingly motivating research at the intersection of the two fields. On the one hand, the presence of more and more wirelessly connected devices, each with its own data, is driving efforts to export advances in machine learning (ML) from high performance computing facilities, where information is stored and processed in a single location, to distributed, privacy-minded, processing at the end user. On the other hand, ML can address algorithm and model deficits in the optimization of communication protocols. However, implementing ML models for learning and inference on battery-powered devices that are connected via bandwidth-constrained channels remains challenging. This paper explores two ways in which Spiking Neural Networks (SNNs) can help address these open problems. First, we discuss federated learning for the distributed training of SNNs, and then describe the integration of neuromorphic sensing, SNNs, and impulse radio technologies for low-power remote inference.

cs.NE↗

arXiv2020

Learning to Demodulate from Few Pilots via Offline and Online Meta-Learning

This paper considers an Internet-of-Things (IoT) scenario in which devices sporadically transmit short packets with few pilot symbols over a fading channel. Devices are characterized by unique transmission non-idealities, such as I/Q imbalance. The number of pilots is generally insufficient to obtain an accurate estimate of the end-to-end channel, which includes the effects of fading and of the transmission-side distortion. This paper proposes to tackle this problem by using meta-learning. Accordingly, pilots from previous IoT transmissions are used as meta-training data in order to train a demodulator that is able to quickly adapt to new end-to-end channel conditions from few pilots. Various state-of-the-art meta-learning schemes are adapted to the problem at hand and evaluated, including Model-Agnostic Meta-Learning (MAML), First-Order MAML (FOMAML), REPTILE, and fast Context Adaptation VIA meta-learning (CAVIA). Both offline and online solutions are developed. In the latter case, an integrated online meta-learning and adaptive pilot number selection scheme is proposed. Numerical results validate the advantages of meta-learning as compared to training schemes that either do not leverage prior transmissions or apply a standard joint learning algorithms on previously received data.

eess.SP↗

arXiv2020

Adaptive Coding and Channel Shaping Through Reconfigurable Intelligent Surfaces: An Information-Theoretic Analysis

A communication link aided by a reconfigurable intelligent surface (RIS) is studied in which the transmitter can control the state of the RIS via a finite-rate control link. Channel state information (CSI) is acquired at the receiver based on pilot-assisted channel estimation, and it may or may not be shared with the transmitter. Considering quasi-static fading channels with imperfect CSI, capacity-achieving signalling is shown to implement joint encoding of the transmitted signal and of the response of the RIS. This demonstrates the information-theoretic optimality of RIS-based modulation, or "single-RF MIMO" systems. In addition, a novel signalling strategy based on separate layered encoding that enables practical successive cancellation-type decoding at the receiver is proposed. Numerical experiments show that the conventional scheme that fixes the reflection pattern of the RIS, irrespective of the transmitted information, as to maximize the achievable rate is strictly suboptimal, and is outperformed by the proposed adaptive coding strategies at all practical signal-to-noise ratio (SNR) levels.

cs.IT↗

arXiv2020

Free Energy Minimization: A Unified Framework for Modelling, Inference, Learning,and Optimization

The goal of these lecture notes is to review the problem of free energy minimization as a unified framework underlying the definition of maximum entropy modelling, generalized Bayesian inference, learning with latent variables, statistical learning analysis of generalization,and local optimization. Free energy minimization is first introduced, here and historically, as a thermodynamic principle. Then, it is described mathematically in the context of Fenchel duality. Finally, the mentioned applications to modelling, inference, learning, and optimization are covered starting from basic principles.

eess.SP↗

arXiv2020

Computation capacities of a broad class of signaling networks are higher than their communication capacities

Due to structural and functional abnormalities or genetic variations and mutations, there may be dysfunctional molecules within an intracellular signaling network that do not allow the network to correctly regulate its output molecules, such as transcription factors. This disruption in signaling interrupts normal cellular functions and may eventually develop some pathological conditions. In this paper, computation capacity of signaling networks is introduced as a fundamental limit on signaling capability and performance of such networks. The computation capacity measures the maximum number of computable inputs, that is, the maximum number of input values for which the correct functional output values can be recovered from the erroneous network outputs, when the network contains some dysfunctional molecules. This contrasts with the conventional communication capacity that measures instead the maximum number of input values that can be correctly distinguished based on the erroneous network outputs. The computation capacity is higher than the communication capacity, if the network response function is not a one-to-one function of the input signals. By explicitly incorporating the effect of signaling errors that result in the network dysfunction, the computation capacity provides more information about the network and its malfunction. Two examples of signaling networks are studied here, one regulating caspase3 and another regulating NFkB, for which computation and communication capacities are analyzed. Higher computation capacities are observed for both networks. One biological implication of this finding is that signaling networks may have more capacity than that specified by the conventional communication capacity metric. The effect of feedback is also studied. In summary, this paper reports findings on a new fundamental feature of the signaling capability of cell signaling networks.

q-bio.MN↗

arXiv2020

Transfer Meta-Learning: Information-Theoretic Bounds and Information Meta-Risk Minimization

Meta-learning automatically infers an inductive bias by observing data from a number of related tasks. The inductive bias is encoded by hyperparameters that determine aspects of the model class or training algorithm, such as initialization or learning rate. Meta-learning assumes that the learning tasks belong to a task environment, and that tasks are drawn from the same task environment both during meta-training and meta-testing. This, however, may not hold true in practice. In this paper, we introduce the problem of transfer meta-learning, in which tasks are drawn from a target task environment during meta-testing that may differ from the source task environment observed during meta-training. Novel information-theoretic upper bounds are obtained on the transfer meta-generalization gap, which measures the difference between the meta-training loss, available at the meta-learner, and the average loss on meta-test data from a new, randomly selected, task in the target task environment. The first bound, on the average transfer meta-generalization gap, captures the meta-environment shift between source and target task environments via the KL divergence between source and target data distributions. The second, PAC-Bayesian bound, and the third, single-draw bound, account for this shift via the log-likelihood ratio between source and target task distributions. Furthermore, two transfer meta-learning solutions are introduced. For the first, termed Empirical Meta-Risk Minimization (EMRM), we derive bounds on the average optimality gap. The second, referred to as Information Meta-Risk Minimization (IMRM), is obtained by minimizing the PAC-Bayesian bound. IMRM is shown via experiments to potentially outperform EMRM.

cs.LG↗

arXiv2020

Adaptive Offline and Online Similarity-Based Caching

With similarity-based content delivery, the request for a content can be satisfied by delivering a related content under a dissimilarity cost. This letter addresses the joint optimization of caching and similarity-based delivery decisions across a network so as to minimize the weighted sum of average delay and dissimilarity cost. A convergent alternate gradient descent ascent algorithm is first introduced for an offline scenario with prior knowledge of the request rates, and then extended to an online setting. Numerical results validate the advantages of the approach with respect to standard per-cache solutions.

cs.NI↗

arXiv2020

End-to-End Learning of Neuromorphic Wireless Systems for Low-Power Edge Artificial Intelligence

This paper introduces a novel "all-spike" low-power solution for remote wireless inference that is based on neuromorphic sensing, Impulse Radio (IR), and Spiking Neural Networks (SNNs). In the proposed system, event-driven neuromorphic sensors produce asynchronous time-encoded data streams that are encoded by an SNN, whose output spiking signals are pulse modulated via IR and transmitted over general frequence-selective channels; while the receiver's inputs are obtained via hard detection of the received signals and fed to an SNN for classification. We introduce an end-to-end training procedure that treats the cascade of encoder, channel, and decoder as a probabilistic SNN-based autoencoder that implements Joint Source-Channel Coding (JSCC). The proposed system, termed NeuroJSCC, is compared to conventional synchronous frame-based and uncoded transmissions in terms of latency and accuracy. The experiments confirm that the proposed end-to-end neuromorphic edge architecture provides a promising framework for efficient and low-latency remote sensing, communication, and inference.

cs.NE↗

arXiv2020

SpinAPS: A High-Performance Spintronic Accelerator for Probabilistic Spiking Neural Networks

We discuss a high-performance and high-throughput hardware accelerator for probabilistic Spiking Neural Networks (SNNs) based on Generalized Linear Model (GLM) neurons, that uses binary STT-RAM devices as synapses and digital CMOS logic for neurons. The inference accelerator, termed "SpinAPS" for Spintronic Accelerator for Probabilistic SNNs, implements a principled direct learning rule for first-to-spike decoding without the need for conversion from pre-trained ANNs. The proposed solution is shown to achieve comparable performance with an equivalent ANN on handwritten digit and human activity recognition benchmarks. The inference engine, SpinAPS, is shown through software emulation tools to achieve 4x performance improvement in terms of GSOPS/W/mm2 when compared to an equivalent SRAM-based design. The architecture leverages probabilistic spiking neural networks that employ first-to-spike decoding rule to make inference decisions at low latencies, achieving 75% of the test performance in as few as 4 algorithmic time steps on the handwritten digit benchmark. The accelerator also exhibits competitive performance with other memristor-based DNN/SNN accelerators and state-of-the-art GPUs.

cs.NE↗

arXiv2020

Memristors -- from In-memory computing, Deep Learning Acceleration, Spiking Neural Networks, to the Future of Neuromorphic and Bio-inspired Computing

Machine learning, particularly in the form of deep learning, has driven most of the recent fundamental developments in artificial intelligence. Deep learning is based on computational models that are, to a certain extent, bio-inspired, as they rely on networks of connected simple computing units operating in parallel. Deep learning has been successfully applied in areas such as object/pattern recognition, speech and natural language processing, self-driving vehicles, intelligent self-diagnostics tools, autonomous robots, knowledgeable personal assistants, and monitoring. These successes have been mostly supported by three factors: availability of vast amounts of data, continuous growth in computing power, and algorithmic innovations. The approaching demise of Moore's law, and the consequent expected modest improvements in computing power that can be achieved by scaling, raise the question of whether the described progress will be slowed or halted due to hardware limitations. This paper reviews the case for a novel beyond CMOS hardware technology, memristors, as a potential solution for the implementation of power-efficient in-memory computing, deep learning accelerators, and spiking neural networks. Central themes are the reliance on non-von-Neumann computing architectures and the need for developing tailored learning and inference algorithms. To argue that lessons from biology can be useful in providing directions for further progress in artificial intelligence, we briefly discuss an example based reservoir computing. We conclude the review by speculating on the big picture view of future neuromorphic and brain-inspired computing systems.

cs.ET↗

arXiv2020

Address-Event Variable-Length Compression for Time-Encoded Data

Time-encoded signals, such as social network update logs and spiking traces in neuromorphic processors, are defined by multiple traces carrying information in the timing of events, or spikes. When time-encoded data is processed at a remote site with respect to the location it is produced, the occurrence of events needs to be encoded and transmitted in a timely fashion. The standard Address-Event Representation (AER) protocol for neuromorphic chips encodes the indices of the "spiking" traces in the payload of a packet produced at the same time the events are recorded, hence implicitly encoding the events' timing in the timing of the packet. This paper investigates the potential bandwidth saving that can be obtained by carrying out variable-length compression of packets' payloads. Compression leverages both intra-trace and inter-trace correlations over time that are typical in applications such as social networks or neuromorphic computing. The approach is based on discrete-time Hawkes processes and entropy coding with conditional codebooks. Results from an experiment based on a real-world retweet dataset are also provided.

cs.IT↗

arXiv2020

Fog-Based Detection for Random-Access IoT Networks with Per-Measurement Preambles

Internet of Things (IoT) systems may be deployed to monitor spatially distributed quantities of interests (QoIs), such as noise or pollution levels. This paper considers a fog-based IoT network, in which active IoT devices transmit measurements of the monitored QoIs to the local edge node (EN), while the ENs are connected to a cloud processor via limited-capacity fronthaul links. While the conventional approach uses preambles as metadata for reserving communication resources, here we consider assigning preambles directly to measurement levels across all devices. The resulting Type-Based Multiple Access (TBMA) protocol enables the efficient remote detection of the QoIs, rather than of the individual payloads. The performance of both edge and cloud-based detection or hypothesis testing is evaluated in terms of error exponents. Cloud-based hypothesis testing is shown theoretically and via numerical results to be advantageous when the inter-cell interference power and the fronthaul capacity are sufficiently large.

cs.IT↗

arXiv2020

Optimizing Over-the-Air Computation in IRS-Aided C-RAN Systems

Over-the-air computation (AirComp) is an efficient solution to enable federated learning on wireless channels. AirComp assumes that the wireless channels from different devices can be controlled, e.g., via transmitter-side phase compensation, in order to ensure coherent on-air combining. Intelligent reflecting surfaces (IRSs) can provide an alternative, or additional, means of controlling channel propagation conditions. This work studies the advantages of deploying IRSs for AirComp systems in a large-scale cloud radio access network (C-RAN). In this system, worker devices upload locally updated models to a parameter server (PS) through distributed access points (APs) that communicate with the PS on finite-capacity fronthaul links. The problem of jointly optimizing the IRSs' reflecting phases and a linear detector at the PS is tackled with the goal of minimizing the mean squared error (MSE) of a parameter estimated at the PS. Numerical results validate the advantages of deploying IRSs with optimized phases for AirComp in C-RAN systems.

eess.SP↗

arXiv2020

VOWEL: A Local Online Learning Rule for Recurrent Networks of Probabilistic Spiking Winner-Take-All Circuits

Networks of spiking neurons and Winner-Take-All spiking circuits (WTA-SNNs) can detect information encoded in spatio-temporal multi-valued events. These are described by the timing of events of interest, e.g., clicks, as well as by categorical numerical values assigned to each event, e.g., like or dislike. Other use cases include object recognition from data collected by neuromorphic cameras, which produce, for each pixel, signed bits at the times of sufficiently large brightness variations. Existing schemes for training WTA-SNNs are limited to rate-encoding solutions, and are hence able to detect only spatial patterns. Developing more general training algorithms for arbitrary WTA-SNNs inherits the challenges of training (binary) Spiking Neural Networks (SNNs). These amount, most notably, to the non-differentiability of threshold functions, to the recurrent behavior of spiking neural models, and to the difficulty of implementing backpropagation in neuromorphic hardware. In this paper, we develop a variational online local training rule for WTA-SNNs, referred to as VOWEL, that leverages only local pre- and post-synaptic information for visible circuits, and an additional common reward signal for hidden circuits. The method is based on probabilistic generalized linear neural models, control variates, and variational regularization. Experimental results on real-world neuromorphic datasets with multi-valued events demonstrate the advantages of WTA-SNNs over conventional binary SNNs trained with state-of-the-art methods, especially in the presence of limited computing resources.

cs.LG↗

arXiv2020

LAGC: Lazily Aggregated Gradient Coding for Straggler-Tolerant and Communication-Efficient Distributed Learning

Gradient-based distributed learning in Parameter Server (PS) computing architectures is subject to random delays due to straggling worker nodes, as well as to possible communication bottlenecks between PS and workers. Solutions have been recently proposed to separately address these impairments based on the ideas of gradient coding, worker grouping, and adaptive worker selection. This paper provides a unified analysis of these techniques in terms of wall-clock time, communication, and computation complexity measures. Furthermore, in order to combine the benefits of gradient coding and grouping in terms of robustness to stragglers with the communication and computation load gains of adaptive selection, novel strategies, named Lazily Aggregated Gradient Coding (LAGC) and Grouped-LAG (G-LAG), are introduced. Analysis and results show that G-LAG provides the best wall-clock time and communication performance, while maintaining a low computational cost, for two representative distributions of the computing times of the worker nodes.

cs.IT↗

arXiv2020

End-to-End Fast Training of Communication Links Without a Channel Model via Online Meta-Learning

When a channel model is not available, the end-to-end training of encoder and decoder on a fading noisy channel generally requires the repeated use of the channel and of a feedback link. An important limitation of the approach is that training should be generally carried out from scratch for each new channel. To cope with this problem, prior works considered joint training over multiple channels with the aim of finding a single pair of encoder and decoder that works well on a class of channels. In this paper, we propose to obviate the limitations of joint training via meta-learning. The proposed approach is based on a meta-training phase in which the online gradient-based meta-learning of the decoder is coupled with the joint training of the encoder via the transmission of pilots and the use of a feedback link. Accounting for channel variations during the meta-training phase, this work demonstrates the advantages of meta-learning in terms of number of pilots as compared to conventional methods when the feedback link is only available for meta-training and not at run time.

eess.SP↗