arXiv subjects

Ather Gattami

Publications and source records attributed to Ather Gattami.

At least 19 recordsLinked to original sources

arXiv2024

Deep Reinforcement Learning: A Convex Optimization Approach

In this paper, we consider reinforcement learning of nonlinear systems with continuous state and action spaces. We present an episodic learning algorithm, where we for each episode use convex optimization to find a two-layer neural network approximation of the optimal $Q$-function. The convex optimization approach guarantees that the weights calculated at each episode are optimal, with respect to the given sampled states and actions of the current episode. For stable nonlinear systems, we show that the algorithm converges and that the converging parameters of the trained neural network can be made arbitrarily close to the optimal neural network parameters. In particular, if the regularization parameter in the training phase is given by $ρ$, then the parameters of the trained neural network converge to $w$, where the distance between $w$ and the optimal parameters $w^\star$ is bounded by $\mathcal{O}(ρ)$. That is, when the number of episodes goes to infinity, there exists a constant $C$ such that \[ \|w-w^\star\| \le Cρ. \] In particular, our algorithm converges arbitrarily close to the optimal neural network parameters as the regularization parameter goes to zero. As a consequence, our algorithm converges fast due to the polynomial-time convergence of convex optimization algorithms.

math.OC↗

arXiv2023

Decentralized Online Bandit Optimization on Directed Graphs with Regret Bounds

We consider a decentralized multiplayer game, played over $T$ rounds, with a leader-follower hierarchy described by a directed acyclic graph. For each round, the graph structure dictates the order of the players and how players observe the actions of one another. By the end of each round, all players receive a joint bandit-reward based on their joint action that is used to update the player strategies towards the goal of minimizing the joint pseudo-regret. We present a learning algorithm inspired by the single-player multi-armed bandit problem and show that it achieves sub-linear joint pseudo-regret in the number of rounds for both adversarial and stochastic bandit rewards. Furthermore, we quantify the cost incurred due to the decentralized nature of our problem compared to the centralized setting.

cs.LG↗

arXiv2022

Learning Team Decisions

In this paper, we treat linear quadratic team decision problems, where a team of agents minimizes a convex quadratic cost function over $T$ time steps subject to possibly distinct linear measurements of the state of nature. We assume that the state of nature is a Gaussian random variable and that the agents do not know the cost function nor the linear functions mapping the state of nature to their measurements. We present a gradient-descent based algorithm with an expected regret of $O(\log(T))$ for full information gradient feedback and $O(\sqrt(T))$ for bandit feedback. In the case of bandit feedback, the expected regret has an additional multiplicative term $O(d)$ where $d$ reflects the number of learned parameters.

math.OC↗

arXiv2022

Provably Efficient Model-Free Algorithm for MDPs with Peak Constraints

In the optimization of dynamic systems, the variables typically have constraints. Such problems can be modeled as a Constrained Markov Decision Process (CMDP). This paper considers the peak Constrained Markov Decision Process (PCMDP), where the agent chooses the policy to maximize total reward in the finite horizon as well as satisfy constraints at each epoch with probability 1. We propose a model-free algorithm that converts PCMDP problem to an unconstrained problem and a Q-learning based approach is applied. We define the concept of probably approximately correct (PAC) to the proposed PCMDP problem. The proposed algorithm is proved to achieve an $(ε,p)$-PAC policy when the episode $K\geqΩ(\frac{I^2H^6SA\ell}{ε^2})$, where $S$ and $A$ are the number of states and actions, respectively. $H$ is the number of epochs per episode. $I$ is the number of constraint functions, and $\ell=\log(\frac{SAT}{p})$. We note that this is the first result on PAC kind of analysis for PCMDP with peak constraints, where the transition dynamics are not known apriori. We demonstrate the proposed algorithm on an energy harvesting problem and a single machine scheduling problem, where it performs close to the theoretical upper bound of the studied optimization problem.

math.OC↗

arXiv2021

Conditional Mutual information-based Contrastive Loss for Financial Time Series Forecasting

We present a representation learning framework for financial time series forecasting. One challenge of using deep learning models for finance forecasting is the shortage of available training data when using small datasets. Direct trend classification using deep neural networks trained on small datasets is susceptible to the overfitting problem. In this paper, we propose to first learn compact representations from time series data, then use the learned representations to train a simpler model for predicting time series movements. We consider a class-conditioned latent variable model. We train an encoder network to maximize the mutual information between the latent variables and the trend information conditioned on the encoded observed variables. We show that conditional mutual information maximization can be approximated by a contrastive loss. Then, the problem is transformed into a classification task of determining whether two encoded representations are sampled from the same class or not. This is equivalent to performing pairwise comparisons of the training datapoints, and thus, improves the generalization ability of the encoder network. We use deep autoregressive models as our encoder to capture long-term dependencies of the sequence data. Empirical experiments indicate that our proposed method has the potential to advance state-of-the-art performance.

cs.LG↗

arXiv2021

Reinforcement Learning for Multi-Objective and Constrained Markov Decision Processes

In this paper, we consider the problem of optimization and learning for constrained and multi-objective Markov decision processes, for both discounted rewards and expected average rewards. We formulate the problems as zero-sum games where one player (the agent) solves a Markov decision problem and its opponent solves a bandit optimization problem, which we here call Markov-Bandit games. We extend Q-learning to solve Markov-Bandit games and show that our new Q-learning algorithms converge to the optimal solutions of the zero-sum Markov-Bandit games, and hence converge to the optimal solutions of the constrained and multi-objective Markov decision problems. We provide a numerical example where we calculate the optimal policies and show by simulations that the algorithm converges to the calculated optimal policies. To the best of our knowledge, this is the first time learning algorithms guarantee convergence to optimal stationary policies for the constrained MDP problem with discounted and expected average rewards, respectively.

math.OC↗

arXiv2021

Model-Free Algorithm and Regret Analysis for MDPs with Long-Term Constraints

In the optimization of dynamical systems, the variables typically have constraints. Such problems can be modeled as a constrained Markov Decision Process (CMDP). This paper considers a model-free approach to the problem, where the transition probabilities are not known. In the presence of long-term (or average) constraints, the agent has to choose a policy that maximizes the long-term average reward as well as satisfy the average constraints in each episode. The key challenge with the long-term constraints is that the optimal policy is not deterministic in general, and thus standard Q-learning approaches cannot be directly used. This paper uses concepts from constrained optimization and Q-learning to propose an algorithm for CMDP with long-term constraints. For any $γ\in(0,\frac{1}{2})$, the proposed algorithm is shown to achieve $O(T^{1/2+γ})$ regret bound for the obtained reward and $O(T^{1-γ/2})$ regret bound for the constraint violation, where $T$ is the total number of steps. We note that these are the first results on regret analysis for MDP with long-term constraints, where the transition probabilities are not known apriori.

cs.LG↗

arXiv2019

Reinforcement Learning of Markov Decision Processes with Peak Constraints

In this paper, we consider reinforcement learning of Markov Decision Processes (MDP) with peak constraints, where an agent chooses a policy to optimize an objective and at the same time satisfy additional constraints. The agent has to take actions based on the observed states, reward outputs, and constraint-outputs, without any knowledge about the dynamics, reward functions, and/or the knowledge of the constraint-functions. We introduce a game theoretic approach to construct reinforcement learning algorithms where the agent maximizes an unconstrained objective that depends on the simulated action of the minimizing opponent which acts on a finite set of actions and the output data of the constraint functions (rewards). We show that the policies obtained from maximin Q-learning converge to the optimal policies. To the best of our knowledge, this is the first time learning algorithms guarantee convergence to optimal stationary policies for the MDP problem with peak constraints for both discounted and expected average rewards.

math.OC↗

arXiv2019

Feedback Capacity of Gaussian Channels Revisited

In this paper, we revisit the problem of finding the average capacity of the Gaussian feedback channel. First, we consider the problem of finding the average capacity of the analog Gaussian noise channel where the noise has an arbitrary spectral density. We introduce a new approach to the problem where we solve the problem over a finite number of transmissions and then consider the limit of an infinite number of transmissions. Further, we consider the important special case of stationary Gaussian noise with finite memory. We show that the channel capacity at stationarity can be found by solving a semi-definite program, and hence computationally tractable. We also give new proofs and results of the non stationary solution which bridges the gap between results in the literature for the stationary and non stationary feedback channel capacities. It's shown that a linear communication feedback strategy is optimal. Similar to the solution of the stationary problem, it's shown that the optimal linear strategy is to transmit a linear combination of the information symbols to be communicated and the innovations for the estimation error of the state of the noise process.

cs.IT↗

arXiv2016

Communicating One Bit over a Delay Constrained Gaussian MIMO Channel with Feedback

The energy-optimal scheme is found for communicating one bit over a memoryless Gaussian channel with an ideal feedback channel. It is assumed that the channel is allowed to be used at most N times before decoding. The optimal coding/decoding strategy is derived by dynamic programming. It is found that feedback gives a significant performance gain and that the optimal strategies are discontinuous. It is also shown that most of the performance increase can be obtained even with a one-bit feedback channel. The optimal scheme is compared with the strategy by Kailath-Schalkwijk and is found to be significantly more effective. For the case of a diagonal MIMO channel where measurement noise variances are equal along the sub channels we also show that the problem can be reduced to the previous case of transmitting one bit over a scalar feedback channel.

cs.IT↗

arXiv2015

Time Localization and Capacity of Faster-Than-Nyquist Signaling

In this paper, we consider communication over the bandwidth limited analog white Gaussian noise channel using non-orthogonal pulses. In particular, we consider non-orthogonal transmission by signaling samples at a rate higher than the Nyquist rate. Using the faster-than-Nyquist (FTN) framework, Mazo showed that one may transmit symbols carried by sinc pulses at a higher rate than that dictated by Nyquist without loosing bit error rate. However, as we will show in this paper, such pulses are not necessarily well localized in time. In fact, assuming that signals in the FTN framework are well localized in time, one can construct a signaling scheme that violates the Shannon capacity bound. We also show directly that FTN signals are in general not well localized in time. Therefore, the results of Mazo do not imply that one can transmit more data per time unit without degrading performance in terms of error probability. We also consider FTN signaling in the case of pulses that are different from the sinc pulses. We show that one can use a precoding scheme of low complexity to remove the inter-symbol interference. This leads to the possibility of increasing the number of transmitted samples per time unit and compensate for spectral inefficiency due to signaling at the Nyquist rate of the non sinc pulses. We demonstrate the power of the precoding scheme by simulations.

cs.IT↗

arXiv2015

Team Decision Problems with Convex Quadratic Constraints

In this paper, we consider linear quadratic team problems with an arbitrary number of quadratic constraints in both stochastic and deterministic settings. The team consists of players with different measurements about the state of nature. The objective of the team is to minimize a quadratic cost subject to additional finite number of quadratic constraints. We first consider the problem of countably infinite number of players in the team for a bounded state of nature with a Gaussian distribution and show that linear decisions are optimal. Then, we consider the problem of team decision problems with additional convex quadratic constraints and show that linear decisions are optimal for both the finite and infinite number of players in the team. For the finite player case, the optimal linear decisions can be found by solving a semidefinite program. Finally, we consider the problem of minimizing a quadratic objective for the worst case scenario, subject to an arbitrary number of deterministic quadratic constraints. We show that linear decisions are optimal and can be found by solving a semidefinite program. Finally, we apply the developed theory on dynamic team decision problems in linear quadratic settings.

math.OC↗

arXiv2015

Optimal Communication of States of Dynamical Systems over Gaussian Channels with Noisy Feedback: The Scalar Case

We consider the problem of communicating the state of a dynamical system via a Shannon Gaussian channel. The receiver, which acts as both a decoder and estimator, observes the noisy measurement of the channel output and makes an optimal estimate of the state of the dynamical system in the minimum mean square sense. Noisy feedback from the receiver to the transmitter is present. The transmitter observes the noise-corrupted feedback message from the receiver together with a possibly noisy measurement of the state the dynamical system. These measurements are then used to encode the message to be transmitted over a noisy Gaussian channel, where a per symbol power constraint is imposed on the transmitted message. Thus, we get a mixed problem of Shannon's source-channel coding problem and a sort of Kalman filtering problem. In particular, we consider two feedback instances, one being feedback of receiver measurements and the second being the receiver's state estimates. We show that optimal encoders and decoders are linear filters with a finite memory and we give explicitly the state space realizations of the optimal filters. For the case where the transmitter has access to noisy measurements of the state, we derive a separation principle for the optimal communication scheme. Furthermore, we investigate the presence of noiseless feedback or no feedback from the receiver to the transmitter. Necessary and sufficient conditions for the existence of a stationary solution are also given for the feedback cases considered.

cs.IT↗

arXiv2015

Kalman meets Shannon

We consider the problem of communicating the state of a dynamical system via a Shannon Gaussian channel. The receiver, which acts as both a decoder and estimator, observes the noisy measurement of the channel output and makes an optimal estimate of the state of the dynamical system in the minimum mean square sense. The transmitter observes a possibly noisy measurement of the state of the dynamical system. These measurements are then used to encode the message to be transmitted over a noisy Gaussian channel, where a per sample power constraint is imposed on the transmitted message. Thus, we get a mixed problem of Shannon's source-channel coding problem and a sort of Kalman filtering problem. We first consider the problem of communication with full state measurements at the transmitter and show that optimal linear encoders don't need to have memory and the optimal linear decoders have an order of at most that of the state dimension. We also give explicitly the structure of the optimal linear filters. For the case where the transmitter has access to noisy measurements of the state, we derive a separation principle for the optimal communication scheme, where the transmitter needs a filter with an order of at most the dimension of the state of the dynamical system. The results are derived for first order linear dynamical systems, but may be extended to MIMO systems with arbitrary order.

cs.IT↗

arXiv2015

Optimal Data and Training Symbol Ratio for Communication over Uncertain Channels

We consider the problem of determining the power ratio between the training symbols and data symbols in order to maximize the channel capacity for transmission over uncertain channels with a channel estimate available at both the transmitter and receiver. The receiver makes an estimate of the channel by using a known sequence of training symbols. This channel estimate is then transmitted back to the transmitter. The capacity that the transceiver maximizes is the worst case capacity, in the sense that given a noise covariance, the transceiver maximizes the minimal capacity over all distributions of the measurement noise under a fixed covariance matrix known at both the transmitter and receiver. We give an exact expression of the channel capacity as a function of the channel covariance matrix, and the number of training symbols used during a coherence time interval. This expression determines the number of training symbols that need to be used by finding the optimal integer number of training symbols that maximize the channel capacity. As a bi-product, we show that linear filters are optimal at both the transmitter and receiver.

cs.IT↗

arXiv2015

Primal robustness and semidefinite cones

This paper reformulates and streamlines the core tools of robust stability and performance for LTI systems using now-standard methods in convex optimization. In particular, robustness analysis can be formulated directly as a primal convex (semidefinite program or SDP) optimization problem using sets of gramians whose closure is a semidefinite cone. This allows various constraints such as structured uncertainty to be included directly, and worst-case disturbances and perturbations constructed directly from the primal variables. Well known results such as the KYP lemma and various scaled small gain tests can also be obtained directly through standard SDP duality. To readers familiar with robustness and SDPs, the framework should appear obvious, if only in retrospect. But this is also part of its appeal and should enhance pedagogy, and we hope suggest new research. There is a key lemma proving closure of a grammian that is also obvious but our current proof appears unnecessarily cumbersome, and a final aim of this paper is to enlist the help of experts in robust control and convex optimization in finding simpler alternatives.

eess.SY↗

arXiv2014

H infinity Analysis Revisited

This paper proposes a direct, and simple approach to the H infinity norm calculation in more general settings. In contrast to the method based on the Kalman-Yakubovich-Popov lemma, our approach does not require a controllability assumption, and returns a sinusoidal input that achieves the H infinity norm of the system including its frequency. In addition, using a semidefinite programming duality, we present a new proof of the Kalman- Yakubovich-Popov lemma, and make a connection between strong duality and controllability. Finally, we generalize our approach towards the generalized Kalman-Yakubovich-Popov lemma, which considers input signals within a finite spectrum.

math.OC↗

arXiv2014

Multi-Objective Optimal Control with Arbitrary Additive and Multiplicative Noise

In this paper, we consider the problem of multi-objective optimal control of a dynamical system with additive and multiplicative noises with given second moments and arbitrary probability distributions. The objectives are given by quadratic constraints in the state and controller, where the quadratic forms maybe indefinite and thus not necessarily convex. We show that the problem can be transformed to a semidefinite program and hence convex. The optimization problem is to be optimized with respect to a certain variable serving as the covariance matrix of the state and the controller. We show that affine controllers are optimal and depend on the optimal covariance matrix. Furthermore, we show that optimal controllers are linear if all the quadratic forms are convex in the control variable. The solutions are presented for both the finite and infinite horizon cases. We give a necessary and sufficient condition for mean square stabilizability of the dynamical system with additive and multiplicative noises. The condition is a Lyapunov-like condition whose solution is again given by the covariance matrix of the state and the control variable. The results are illustrated with an example.

math.OC↗