arXiv subjects

Anatoly Khina

Publications and source records attributed to Anatoly Khina.

At least 19 recordsLinked to original sources

arXiv2026

Markovian Continuity of the MMSE

Minimum mean square error (MMSE) estimation is widely used in signal processing, information theory, and related fields. Despite its practical robustness, the MMSE can be discontinuous under standard notions of stochastic convergence. To bridge this gap, we review classical counterexamples to the continuity of the MMSE and observe that they share a common pathology: along the approximating sequence, the observation is strictly more informative about the limit estimand than the limit observation is. Motivated by practical acquisition mechanisms, we study MMSE continuity under two natural constraints: (1) continuity of the second moment, and (2) a degradedness (Markov) restriction ensuring that each approximating observation is no more informative than the limit observation is about the limit estimand. Under these conditions, we establish continuity of the MMSE and of the MMSE estimator. We provide complementary semicontinuity results and continuity guarantees in related settings and establish continuity under linear estimation. We further extend the analysis to the families of Bregman divergences and continuous metric cost functions, including the Kullback-Leibler and Jensen-Shannon divergences as special cases.

eess.SP↗

arXiv2026

Beyond Fixed Points: Superpolynomial Capacity of Asymmetric Hopfield Networks

Classical Hopfield networks are limited to static patterns due to symmetric weights, whereas asymmetric networks can encode temporal sequences via limit-cycle attractors. Achieving high-capacity storage of long sequences in classical synchronous asymmetric networks, however, has remained a challenge. We present a simple and robust construction within the classical asymmetric Hopfield model with binary neurons and synchronous updates, that allows $n$ neurons to support $\exp\!\big(Ω(n/(\log n)^2)\big)$ distinct limit-cycle attractors, each with period $\exp\!\big(Ω(\sqrt n/\log n)\big)$ and robust to random noise with flip probability up to $\frac12-o(1)$, yielding superpolynomial capacity in both the number and length of stored sequences. This is the first demonstration of such capacity for asymmetric Hopfield networks, which we obtain by combining results from combinatorics, number theory and the analysis of opinion dynamics. Our findings show that synchronous asymmetric Hopfield networks possess a sequence-memory capacity which is larger and more robust than previously recognized, demonstrating that, in both biological and artificial neural systems, robust sequence representation can be achieved through coarse architectural motifs rather than complex nonlinearities.

cs.LG↗

arXiv2024

Batch Normalization Decomposed

\emph{Batch normalization} is a successful building block of neural network architectures. Yet, it is not well understood. A neural network layer with batch normalization comprises three components that affect the representation induced by the network: \emph{recentering} the mean of the representation to zero, \emph{rescaling} the variance of the representation to one, and finally applying a \emph{non-linearity}. Our work follows the work of Hadi Daneshmand, Amir Joudaki, Francis Bach [NeurIPS~'21], which studied deep \emph{linear} neural networks with only the rescaling stage between layers at initialization. In our work, we present an analysis of the other two key components of networks with batch normalization, namely, the recentering and the non-linearity. When these two components are present, we observe a curious behavior at initialization. Through the layers, the representation of the batch converges to a single cluster except for an odd data point that breaks far away from the cluster in an orthogonal direction. We shed light on this behavior from two perspectives: (1) we analyze the geometrical evolution of a simplified indicative model; (2) we prove a stability result for the aforementioned~configuration.

cs.LG↗

arXiv2024

Modulation and Estimation with a Helper

The problem of transmitting a parameter value over an additive white Gaussian noise (AWGN) channel is considered, where, in addition to the transmitter and the receiver, there is a helper that observes the noise non-causally and provides a description of limited rate $R_h$ to the transmitter and/or the receiver. We derive upper and lower bounds on the optimal achievable $α$-th moment of the estimation error and show that they coincide for small values of $α$ and for high values of $R_h$. The upper bound relies on a recently proposed channel-coding scheme that effectively conveys $R_h$ bits essentially error-free and the rest of the rate - over the same AWGN channel without help, with the error-free bits being allocated to the most significant bits of the quantized parameter. We then concentrate on the setting with a total transmit energy constraint, for which we derive achievability results for both channel coding and parameter modulation for several scenarios: when the helper assists only the transmitter or only the receiver and knows the noise, and when the helper assists the transmitter and/or the receiver and knows both the noise and the message. In particular, for the message-informed helper that assists both the receiver and the transmitter, it is shown that the error probability in the channel-coding task decays doubly exponentially. Finally, we translate these results to those for continuous-time power-limited AWGN channels with unconstrained bandwidth. As a byproduct, we show that the capacity with a message-informed helper that is available only at the transmitter can exceed the sum of the capacity without help and the help rate $R_h$, when the helper knows only the noise but not the message.

cs.IT↗

arXiv2023

Information Velocity of Cascaded Gaussian Channels with Feedback

We consider a line network of nodes, connected by additive white Gaussian noise channels, equipped with local feedback. We study the velocity at which information spreads over this network. For transmission of a data packet, we give an explicit positive lower bound on the velocity, for any packet size. Furthermore, we consider streaming, that is, transmission of data packets generated at a given average arrival rate. We show that a positive velocity exists as long as the arrival rate is below the individual Gaussian channel capacity, and provide an explicit lower bound. Our analysis involves applying pulse-amplitude modulation to the data (successively in the streaming case), and using linear mean-squared error estimation at the network nodes. Due to the analog linear nature of the scheme, the results extend to any additive noise. For general noise, we derive exponential error-probability bounds. Moreover, for (sub-)Gaussian noise we show a doubly-exponential behavior, which reduces to the celebrated Schalkwijk-Kailath scheme when considering a single node. Viewing the constellation as an "analog source", we also provide bounds on the exponential decay of the mean-squared error of source transmission over the network.

cs.IT↗

arXiv2023

Universal Joint Source-Channel Coding Under an Input Energy Constraint

We consider the problem of transmitting a source over an infinite-bandwidth additive white Gaussian noise channel with unknown noise level under an input energy constraint. We construct a universal scheme that uses modulo-lattice modulation with multiple layers; for each layer, we employ either analog linear modulation or analog pulse position modulation (PPM). We show that the designed scheme with linear layers requires less energy compared to existing solutions to achieve the same quadratically increasing distortion profile with the noise level; replacing the linear layers with PPM layers offers an additional improvement.

cs.IT↗

arXiv2022

A Johnson--Lindenstrauss Framework for Randomly Initialized CNNs

How does the geometric representation of a dataset change after the application of each randomly initialized layer of a neural network? The celebrated Johnson--Lindenstrauss lemma answers this question for linear fully-connected neural networks (FNNs), stating that the geometry is essentially preserved. For FNNs with the ReLU activation, the angle between two inputs contracts according to a known mapping. The question for non-linear convolutional neural networks (CNNs) becomes much more intricate. To answer this question, we introduce a geometric framework. For linear CNNs, we show that the Johnson--Lindenstrauss lemma continues to hold, namely, that the angle between two inputs is preserved. For CNNs with ReLU activation, on the other hand, the behavior is richer: The angle between the outputs contracts, where the level of contraction depends on the nature of the inputs. In particular, after one layer, the geometry of natural images is essentially preserved, whereas for Gaussian correlated inputs, CNNs exhibit the same contracting behavior as FNNs with ReLU activation.

cs.LG↗

arXiv2022

The Information Velocity of Packet-Erasure Links

We consider the problem of in-order packet transmission over a cascade of packet-erasure links with acknowledgment (ACK) signals, interconnected by relays. We treat first the case of transmitting a single packet, in which ACKs are unnecessary, over links with independent identically distributed erasures. For this case, we derive tight upper and lower bounds on the probability of arrive failure within an allowed end-to-end communication delay over a given number of links. When the number of links is commensurate with the allowed delay, we determine the maximal ratio between the two -- coined information velocity -- for which the arrive-failure probability decays to zero; we further derive bounds on the arrive-failure probability when the ratio is below the information velocity, determine the exponential arrive-failure decay rate, and extend the treatment to links with different erasure probabilities. We then elevate all these results for a stream of packets with independent geometrically distributed interarrival times, and prove that the information velocity and the exponential decay rate remain the same for any stationary ergodic arrival process and for deterministic interarrival times. We demonstrate the significance of the derived fundamental limits -- the information velocity and the arrive-failure exponential decay rate -- by comparing them to simulation results.

cs.IT↗

arXiv2021

Energy-limited Joint Source--Channel Coding via Analog Pulse Position Modulation

We study the problem of transmitting a source sample with minimum distortion over an infinite-bandwidth additive white Gaussian noise channel under an energy constraint. To that end, we construct a joint source--channel coding scheme using analog pulse position modulation (PPM) and bound its quadratic distortion. We show that this scheme outperforms existing techniques since its quadratic distortion attains both the exponential and polynomial decay orders of Burnashev's outer bound. We supplement our theoretical results with numerical simulations and comparisons to existing schemes.

cs.IT↗

arXiv2020

Causal Posterior Matching and its Applications

We consider the problem of communication over the binary symmetric channel with feedback, where the information sequence is made available in a causal, possibly random, fashion. We develop a real-time variant of the renowned Horstein scheme and provide analytical guarantees for its error-probability exponential decay rate. We further use the scheme to stabilize an unstable control plant over a binary symmetric channel and compare the analytical guarantees with its empirical performance as well as with those of anytime-reliable codes.

cs.IT↗

arXiv2020

Monotonicity of the Trace-Inverse of Covariance Submatrices and Two-Sided Prediction

It is common to assess the "memory strength" of a stationary process looking at how fast the normalized log-determinant of its covariance submatrices (i.e., entropy rate) decreases. In this work, we propose an alternative characterization in terms of the normalized trace-inverse of the covariance submatrices. We show that this sequence is monotonically non-decreasing and is constant if and only if the process is white. Furthermore, while the entropy rate is associated with one-sided prediction errors (present from past), the new measure is associated with two-sided prediction errors (present from past and future). This measure can be used as an alternative to Burg's maximum-entropy principle for spectral estimation. We also propose a counterpart for non-stationary processes, by looking at the average trace-inverse of subsets.

eess.SP↗

arXiv2020

Real-Time Variable-to-Fixed Lossless Source Coding of Randomly Arriving Symbols

We address the recently suggested problem of causal lossless coding of a randomly arriving source samples. We construct variable-to-fixed coding schemes and show that they outperform the previously considered fixed-to-variable schemes when traffic is high both in terms of delay and Age of Information by appealing to tools from queueing theory. We supplement our theoretical bounds with numerical simulations.

cs.IT↗

arXiv2020

Gauss-Markov Source Tracking with Side Information: Lower Bounds

We consider the problem of causal source coding and causal decoding of a Gauss--Markov source, where the decoder has causal access to a side-information signal. We define the information causal rate-distortion function with causal decoder side information and prove that it bounds from below its operational counterpart. We further explain how to adapt the result to the setting of control over communication channels.

cs.IT↗

arXiv2020

Learning-based attacks in cyber-physical systems

We introduce the problem of learning-based attacks in a simple abstraction of cyber-physical systems---the case of a discrete-time, linear, time-invariant plant that may be subject to an attack that overrides the sensor readings and the controller actions. The attacker attempts to learn the dynamics of the plant and subsequently override the controller's actuation signal, to destroy the plant without being detected. The attacker can feed fictitious sensor readings to the controller using its estimate of the plant dynamics and mimic the legitimate plant operation. The controller, on the other hand, is constantly on the lookout for an attack; once the controller detects an attack, it immediately shuts the plant off. In the case of scalar plants, we derive an upper bound on the attacker's deception probability for any measurable control policy when the attacker uses an arbitrary learning algorithm to estimate the system dynamics. We then derive lower bounds for the attacker's deception probability for both scalar and vector plants by assuming a specific authentication test that inspects the empirical variance of the system disturbance. We also show how the controller can improve the security of the system by superimposing a carefully crafted privacy-enhancing signal on top of the "nominal control policy." Finally, for nonlinear scalar dynamics that belong to the Reproducing Kernel Hilbert Space (RKHS), we investigate the performance of attacks based on nonlinear Gaussian-processes (GP) learning algorithms.

eess.SY↗

arXiv2020

Schemes for LQG Control over Gaussian Channels with Side Information

We consider the problem of controlling an unstable scalar linear plant over a power-constrained additive white Gaussian noise (AWGN) channel, where the controller/receiver has access to an additional noisy measurement of the state of the control system. To that end, we view the noisy measurement as side information and recast the problem to that of joint source-channel coding with side information at the receiver. We argue that judicious modulo-based schemes improve over their linear counterparts and allow to avoid a large increase in the transmit power due to the ignorance of the side information at the sensor/transmitter. We demonstrate the usefulness of our technique for the settings where i) the sensor is oblivious of the control objectives, control actions and previous controller state estimates, ii) the system output tracks a desired reference signal that is available only at the controller via integral control action.

eess.SY↗

arXiv2018

Algorithms for Optimal Control with Fixed-Rate Feedback

We consider a discrete-time linear quadratic Gaussian networked control setting where the (full information) observer and controller are separated by a fixed-rate noiseless channel. The minimal rate required to stabilize such a system has been well studied. However, for a given fixed rate, how to quantize the states so as to optimize performance is an open question of great theoretical and practical significance. We concentrate on minimizing the control cost for first-order scalar systems. To that end, we use the Lloyd-Max algorithm and leverage properties of logarithmically-concave functions and sequential Bayesian filtering to construct the optimal quantizer that greedily minimizes the cost at every time instant. By connecting the globally optimal scheme to the problem of scalar successive refinement, we argue that its gain over the proposed greedy algorithm is negligible. This is significant since the globally optimal scheme is often computationally intractable. All the results are proven for the more general case of disturbances with logarithmically-concave distributions and rate-limited time-varying noiseless channels. We further extend the framework to event-triggered control by allowing to convey information via an additional "silent symbol", i.e., by avoiding transmitting bits; by constraining the minimal probability of silence we attain a tradeoff between the transmission rate and the control cost for rates below one bit per sample.

eess.SY↗

arXiv2018

Tracking and Control of Gauss-Markov Processes over Packet-Drop Channels with Acknowledgments

We consider the problem of tracking the state of Gauss-Markov processes over rate-limited erasure-prone links. We concentrate first on the scenario in which several independent processes are seen by a single observer. The observer maps the processes into finite-rate packets that are sent over the erasure-prone links to a state estimator, and are acknowledged upon packet arrivals. The aim of the state estimator is to track the processes with zero delay and with minimum mean square error (MMSE). We show that, in the limit of many processes, greedy quantization with respect to the squared error distortion is optimal. That is, there is no tension between optimizing the MMSE of the process in the current time instant and that of future times. For the case of packet erasures with delayed acknowledgments, we connect the problem to that of compression with side information that is known at the observer and may be known at the state estimator - where the most recent packets serve as side information that may have been erased, and demonstrate that the loss due to a delay by one time unit is rather small. For the scenario where only one process is tracked by the observer-state estimator system, we further show that variable-length coding techniques are within a small gap of the many-process outer bound. We demonstrate the usefulness of the proposed approach for the simple setting of discrete-time scalar linear quadratic Gaussian control with a limited data-rate feedback that is susceptible to packet erasures.

cs.IT↗

arXiv2017

The Dirty MIMO Multiple-Access Channel

In the scalar dirty multiple-access channel, in addition to Gaussian noise, two additive interference signals are present, each known non-causally to a single transmitter. It was shown by Philosof et al. that for strong interferences, an i.i.d. ensemble of codes does not achieve the capacity region. Rather, a structured-codes approach was presented, that was shown to be optimal in the limit of high signal-to-noise ratios, where the sum-capacity is dictated by the minimal ("bottleneck") channel gain. In this paper, we consider the multiple-input multiple-output (MIMO) variant of this setting. In order to incorporate structured codes in this case, one can utilize matrix decompositions that transform the channel into effective parallel scalar dirty multiple-access channels. This approach however suffers from a "bottleneck" effect for each effective scalar channel and therefore the achievable rates strongly depend on the chosen decomposition. It is shown that a recently proposed decomposition, where the diagonals of the effective channel matrices are equal up to a scaling factor, is optimal at high signal-to-noise ratios, under an equal rank assumption. This approach is then extended to any number of transmitters. Finally, an application to physical-layer network coding for the MIMO two-way relay channel is presented.

cs.IT↗