arXiv subjects

Arnob Ghosh

Publications and source records attributed to Arnob Ghosh.

At least 37 records · Page 2Linked to original sources

arXiv2023

Achieving Fairness in Multi-Agent Markov Decision Processes Using Reinforcement Learning

Fairness plays a crucial role in various multi-agent systems (e.g., communication networks, financial markets, etc.). Many multi-agent dynamical interactions can be cast as Markov Decision Processes (MDPs). While existing research has focused on studying fairness in known environments, the exploration of fairness in such systems for unknown environments remains open. In this paper, we propose a Reinforcement Learning (RL) approach to achieve fairness in multi-agent finite-horizon episodic MDPs. Instead of maximizing the sum of individual agents' value functions, we introduce a fairness function that ensures equitable rewards across agents. Since the classical Bellman's equation does not hold when the sum of individual value functions is not maximized, we cannot use traditional approaches. Instead, in order to explore, we maintain a confidence bound of the unknown environment and then propose an online convex optimization based approach to obtain a policy constrained to this confidence region. We show that such an approach achieves sub-linear regret in terms of the number of episodes. Additionally, we provide a probably approximately correct (PAC) guarantee based on the obtained regret bound. We also propose an offline RL algorithm and bound the optimality gap with respect to the optimal fair solution. To mitigate computational complexity, we introduce a policy-gradient type method for the fair objective. Simulation experiments also demonstrate the efficacy of our approach.

cs.LG↗

arXiv2023

Provably Efficient Model-Free Algorithms for Non-stationary CMDPs

We study model-free reinforcement learning (RL) algorithms in episodic non-stationary constrained Markov Decision Processes (CMDPs), in which an agent aims to maximize the expected cumulative reward subject to a cumulative constraint on the expected utility (cost). In the non-stationary environment, reward, utility functions, and transition kernels can vary arbitrarily over time as long as the cumulative variations do not exceed certain variation budgets. We propose the first model-free, simulator-free RL algorithms with sublinear regret and zero constraint violation for non-stationary CMDPs in both tabular and linear function approximation settings with provable performance guarantees. Our results on regret bound and constraint violation for the tabular case match the corresponding best results for stationary CMDPs when the total budget is known. Additionally, we present a general framework for addressing the well-known challenges associated with analyzing non-stationary CMDPs, without requiring prior knowledge of the variation budget. We apply the approach for both tabular and linear approximation settings.

cs.LG↗

arXiv2023

Pricing in Ride-sharing Markets : Effects of network competition and autonomous vehicles

Autonomous vehicles will be an integral part of ride-sharing services in the future. This setting differs from traditional ride-sharing marketplaces because of the absence of the supply side (drivers). However, it has far-reaching consequences because in addition to pricing, players now have to make decisions on how to distribute fleets across network locations and re-balance vehicles in order to serve future demand. In this paper, we explore a duopoly setting in the ride-sharing marketplace where the players have fully autonomous fleets. Each ride-service provider (RSP)'s prices depend on the prices and the supply of the other player. We formulate their decision-making problems using a game-theoretic setup where each player seeks to find the optimal prices and supplies at each node while considering the decisions of the other player. This leads to a scenario where the players' optimization problems are coupled and it is challenging to find the equilibrium. We characterize the types of demand functions (e.g.: linear) for which this game admits an exact potential function and can be solved efficiently. For other types of demand functions, we propose an iterative algorithm to compute the equilibrium. We conclude by providing numerical insights into how different kinds of equilibria would play out in the market when the players are asymmetric. Our numerical evaluations also provide insights into how the regulator needs to consider network effects while deciding regulation in order to avoid unfavorable outcomes.

cs.GT↗

arXiv2023

Provably Efficient Model-Free Constrained RL with Linear Function Approximation

We study the constrained reinforcement learning problem, in which an agent aims to maximize the expected cumulative reward subject to a constraint on the expected total value of a utility function. In contrast to existing model-based approaches or model-free methods accompanied with a `simulator', we aim to develop the first model-free, simulator-free algorithm that achieves a sublinear regret and a sublinear constraint violation even in large-scale systems. To this end, we consider the episodic constrained Markov decision processes with linear function approximation, where the transition dynamics and the reward function can be represented as a linear function of some known feature mapping. We show that $\tilde{\mathcal{O}}(\sqrt{d^3H^3T})$ regret and $\tilde{\mathcal{O}}(\sqrt{d^3H^3T})$ constraint violation bounds can be achieved, where $d$ is the dimension of the feature mapping, $H$ is the length of the episode, and $T$ is the total number of steps. Our bounds are attained without explicitly estimating the unknown transition model or requiring a simulator, and they depend on the state space only through the dimension of the feature mapping. Hence our bounds hold even when the number of states goes to infinity. Our main results are achieved via novel adaptations of the standard LSVI-UCB algorithms. In particular, we first introduce primal-dual optimization into the LSVI-UCB algorithm to balance between regret and constraint violation. More importantly, we replace the standard greedy selection with respect to the state-action function in LSVI-UCB with a soft-max policy. This turns out to be key in establishing uniform concentration for the constrained case via its approximation-smoothness trade-off. We also show that one can achieve an even zero constraint violation while still maintaining the same order with respect to $T$.

cs.LG↗

arXiv2022

Provably Efficient Model-free RL in Leader-Follower MDP with Linear Function Approximation

We consider a multi-agent episodic MDP setup where an agent (leader) takes action at each step of the episode followed by another agent (follower). The state evolution and rewards depend on the joint action pair of the leader and the follower. Such type of interactions can find applications in many domains such as smart grids, mechanism design, security, and policymaking. We are interested in how to learn policies for both the players with provable performance guarantee under a bandit feedback setting. We focus on a setup where both the leader and followers are {\em non-myopic}, i.e., they both seek to maximize their rewards over the entire episode and consider a linear MDP which can model continuous state-space which is very common in many RL applications. We propose a {\em model-free} RL algorithm and show that $\tilde{\mathcal{O}}(\sqrt{d^3H^3T})$ regret bounds can be achieved for both the leader and the follower, where $d$ is the dimension of the feature mapping, $H$ is the length of the episode, and $T$ is the total number of steps under the bandit feedback information setup. Thus, our result holds even when the number of states becomes infinite. The algorithm relies on {\em novel} adaptation of the LSVI-UCB algorithm. Specifically, we replace the standard greedy policy (as the best response) with the soft-max policy for both the leader and the follower. This turns out to be key in establishing uniform concentration bound for the value functions. To the best of our knowledge, this is the first sub-linear regret bound guarantee for the Markov games with non-myopic followers with function approximation.

cs.LG↗

arXiv2022

Interference Constrained Beam Alignment for Time-Varying Channels via Kernelized Bandits

To fully utilize the abundant spectrum resources in millimeter wave (mmWave), Beam Alignment (BA) is necessary for large antenna arrays to achieve large array gains. In practical dynamic wireless environments, channel modeling is challenging due to time-varying and multipath effects. In this paper, we formulate the beam alignment problem as a non-stationary online learning problem with the objective to maximize the received signal strength under interference constraint. In particular, we employ the non-stationary kernelized bandit to leverage the correlation among beams and model the complex beamforming and multipath channel functions. Furthermore, to mitigate interference to other user equipment, we leverage the primal-dual method to design a constrained UCB-type kernelized bandit algorithm. Our theoretical analysis indicates that the proposed algorithm can adaptively adjust the beam in time-varying environments, such that both the cumulative regret of the received signal and constraint violations have sublinear bounds with respect to time. This result is of independent interest for applications such as adaptive pricing and news ranking. In addition, the algorithm assumes the channel is a black-box function and does not require any prior knowledge for dynamic channel modeling, and thus is applicable in a variety of scenarios. We further show that if the information about the channel variation is known, the algorithm will have better theoretical guarantees and performance. Finally, we conduct simulations to highlight the effectiveness of the proposed algorithm.

cs.NI↗

arXiv2021

Discriminatory Price Mechanism for Smart Grid

We consider a scenario where a retailer can set different prices for different consumers in a smart grid. The retailer's objective is to maximize the revenue, minimize the operating cost, and maximize the consumer's welfare. The retailer wants to optimize a convex combination of the above objectives using price signals specific to each consumer. However, variability in unit prices across consumers is bounded by a parameter $η$, hence limiting the discrimination. We formulate the pricing problem as a Stackelberg game where the retailer is the leader and consumers are followers. Since the retailer's optimization problem turns out to be non-convex, we convexify it via relaxations. We provide performance guarantees for the relaxations in the asymptotic sense (when number of consumers tends to $\infty$). Further, we show that despite the variability in pricing, the pricing scheme proposed by our model is fair as higher prices are charged to consumers who have higher willingness for demand. We extend our analysis to the scenario where consumers can feed energy back to the grid via net-metering. We show that our pricing policy promotes fairness even in this scenario as prosumers who contribute more to the grid, are given large cuts on buying rates. The policy is also found to incentivize more prosumers to invest in renewable energy, thus encouraging sustainability.

math.OC↗

arXiv2021

Traffic Control in a Mixed Autonomy Scenario at Urban Intersections: An Optimization-based Framework

We consider an intersection zone where autonomous vehicles (AVs) and human-driven vehicles (HDVs) can be present. As a new vehicle arrives, the traffic controller needs to decide and impose an optimal sequence of the vehicles that will exit the intersection zone. The traffic controller can send information regarding the time at which an AV can cross the intersection; however, the traffic controller can not communicate with the HDVs, rather the HDVs can only be controlled using the traffic lights. We formulate the problem as an integer constrained non-linear optimization problem where the traffic-intersection controller only communicates with a subset of the AVs. Since the number of possible combinations increases exponentially with the number of vehicles in the system, we relax the original problem and proposes an algorithm that gives the optimal solution of the relaxed problem and yet only scales linearly with the number of vehicles in the system. The numerical evaluation shows that our algorithm outperforms the First-In-First-Out (FIFO) algorithm.

eess.SY↗

arXiv2021

Control of a Mixed Autonomy Signalised Urban Intersection: An Action-Delayed Reinforcement Learning Approach

We consider a mixed autonomy scenario where the traffic intersection controller decides whether the traffic light will be green or red at each lane for multiple traffic-light blocks. The objective of the traffic intersection controller is to minimize the queue length at each lane and maximize the outflow of vehicles over each block. We consider that the traffic intersection controller informs the autonomous vehicle (AV) whether the traffic light will be green or red for the future traffic-light block. Thus, the AV can adapt its dynamics by solving an optimal control problem. We model the decision process of the traffic intersection controller as a deterministic delay Markov decision process owing to the delayed action by the traffic controller. We propose Reinforcement-learning based algorithm to obtain the optimal policy. We show - empirically - that our algorithm converges and reduces the energy costs of AVs drastically as the traffic controller communicates with the AVs.

eess.SY↗

arXiv2021

Design of Incentive Mechanisms Using Prospect Theory to Promote Better Sell-back Behavior among Prosumers

Users can now give back energies to the grid using distributed resources. Proper incentive mechanisms are required for such users, also known as prosumers, in order to maximize the sell-back amount while maintaining the retailer's profit. However, all the existing literature considers expected utility theory (EUT) where they assume that prosumers maximize their expected payoff. We consider prospect theory (PT) which models the behavior of humans in the face of uncertainty in a better manner. We show that in a day-ahead contract pricing mechanism, the actual optimal value of contract and the sell-back amount may be smaller compared to the one computed by the EUT. We also propose a lottery-based mechanism and show that such a mechanism can increase the sell-back amount while increasing the retailer's savings compared to day-ahead contract pricing.

math.OC↗

arXiv2021

Competition among Ride Service Providers with Autonomous Vehicles

Autonomous vehicles (AVs) are attractive for ride service providers (RSPs) in part because they eliminate the need to compete for human drivers. We investigate a scenario where two RSPs with AVs compete for customers. We model the problem as a game where the RSPs select prices for each origin-destination pair over multiple time periods in an underlying graph representing the customers' desired trips. Each RSP also decides the number of AVs to be stationed at each node at each time period to serve the customers' demands. The number of customers who avail of the service of an RSP depends on the price selected by the RSP and its competitor. Since the strategy choices available to an RSP depends on its competitor, we seek to compute a Generalized Nash equilibrium (GNE). We show that there may be multiple GNEs. However, when an RSP selects prices in order to deter its competitor when it is not serving a source-destination pair, the game has a potential function and admits a unique GNE. We also compare the competitive prices with a monopoly price where only one RSP is in the market. Numerically, we show that if a network consists of two equal-size spatial clusters of demand where the demand between clusters is low, the RSPs may partition the market, i.e, one cluster is served by only one RSP. Hence, the competitive price may become close to the monopoly price.

cs.GT↗

arXiv2020

Model Free Reinforcement Learning Algorithm for Stationary Mean field Equilibrium for Multiple Types of Agents

We consider a multi-agent Markov strategic interaction over an infinite horizon where agents can be of multiple types. We model the strategic interaction as a mean-field game in the asymptotic limit when the number of agents of each type becomes infinite. Each agent has a private state; the state evolves depending on the distribution of the state of the agents of different types and the action of the agent. Each agent wants to maximize the discounted sum of rewards over the infinite horizon which depends on the state of the agent and the distribution of the state of the leaders and followers. We seek to characterize and compute a stationary multi-type Mean field equilibrium (MMFE) in the above game. We characterize the conditions under which a stationary MMFE exists. Finally, we propose Reinforcement learning (RL) based algorithm using policy gradient approach to find the stationary MMFE when the agents are unaware of the dynamics. We, numerically, evaluate how such kind of interaction can model the cyber attacks among defenders and adversaries, and show how RL based algorithm can converge to an equilibrium.

cs.MA↗

arXiv2019

Reinforcement Learning for Mean Field Game

Stochastic games provide a framework for interactions among multiple agents and enable a myriad of applications. In these games, agents decide on actions simultaneously, the state of every agent moves to the next state, and each agent receives a reward. However, finding an equilibrium (if exists) in this game is often difficult when the number of agents becomes large. This paper focuses on finding a mean-field equilibrium (MFE) in an action coupled stochastic game setting in an episodic framework. It is assumed that the impact of the other agents' can be assumed by the empirical distribution of the mean of the actions. All agents know the action distribution and employ lower-myopic best response dynamics to choose the optimal oblivious strategy. This paper proposes a posterior sampling based approach for reinforcement learning in the mean-field game, where each agent samples a transition probability from the previous transitions. We show that the policy and action distributions converge to the optimal oblivious strategy and the limiting distribution, respectively, which constitute an MFE.

cs.LG↗

arXiv2019

DeepPool: Distributed Model-free Algorithm for Ride-sharing using Deep Reinforcement Learning

The success of modern ride-sharing platforms crucially depends on the profit of the ride-sharing fleet operating companies, and how efficiently the resources are managed. Further, ride-sharing allows sharing costs and, hence, reduces the congestion and emission by making better use of vehicle capacities. In this work, we develop a distributed model-free, DeepPool, that uses deep Q-network (DQN) techniques to learn optimal dispatch policies by interacting with the environment. Further, DeepPool efficiently incorporates travel demand statistics and deep learning models to manage dispatching vehicles for improved ride sharing services. Using real-world dataset of taxi trip records in New York City, DeepPool performs better than other strategies, proposed in the literature, that do not consider ride sharing or do not dispatch the vehicles to regions where the future demand is anticipated. Finally, DeepPool can adapt rapidly to dynamic environments since it is implemented in a distributed manner in which each vehicle solves its own DQN individually without coordination.

cs.NE↗

arXiv2019

Competition with Three-Tier Spectrum Access and Spectrum Monitoring

The Citizens Broadband Radio Service (CBRS) recently adopted in the U.S. enables two tiers of commercial users to share spectrum with a third tier of incumbent users. This sharing can be further assisted by Environmental Sensing Capability operators (ESCs), that monitor the spectrum occupancy to determine when use of the spectrum will not harm incumbents. Two key aspects of this framework that impact how firms may compete are the differences in information provided by different ESCs and the different tiers in which a firm may access the spectrum. We develop a game theoretic model that captures both of these features and analyze it to gain insight into their impact. Specifically, we consider a priority access (PA) tier firm has access to the both licensed band and unlicensed band, and a general authorized access (GAA) tier firm has access only to the unlicensed band. The PA tier and GAA tier firms compete for users. Our analysis reveals that the amount of unlicensed and licensed bandwidth in the CBRS must be chosen judiciously in order to maximize the social welfare. We also show that a limited amount of unlicensed access of PA tier firm is beneficial to the user's surplus as well as to the social welfare.

cs.GT↗

arXiv2018

Optimized Portfolio Contracts for Bidding the Cloud

Amazon EC2 provides two most popular pricing schemes--i) the {\em costly} on-demand instance where the job is guaranteed to be completed, and ii) the {\em cheap} spot instance where a job may be interrupted. We consider a user can select a combination of on-demand and spot instances to finish a task. Thus he needs to find the optimal bidding price for the spot-instance, and the portion of the job to be run on the on-demand instance. We formulate the problem as an optimization problem and seek to find the optimal solution. We consider three bidding strategies: one-time requests with expected guarantee and one-time requests with penalty for incomplete job and violating the deadline, and persistent requests. Even without a penalty on incomplete jobs, the optimization problem turns out to be non-convex. Nevertheless, we show that the portion of the job to be run on the on-demand instance is at most half. If the job has a higher execution time or smaller deadline, the bidding price is higher and vice versa. Additionally, the user never selects the on-demand instance if the execution time is smaller than the deadline. The numerical results illustrate the sensitivity of the effective portfolio to several of the parameters involved in the model. Our empirical analysis on the Amazon EC2 data shows that our strategies can be employed on the real instances, where the expected total cost of the proposed scheme decreases over 45\% compared to the baseline strategy.

cs.DC↗

arXiv2018

A Robust Algorithm for Tile-based 360-degree Video Streaming with Uncertain FoV Estimation

We propose a robust scheme for streaming 360-degree immersive videos to maximize the quality of experience (QoE). Our streaming approach introduces a holistic analytical framework built upon the formal method of stochastic optimization. We propose a robust algorithm which provides a streaming rate such that the video quality degrades below that rate with very low probability even in presence of uncertain head movement, and bandwidth. It assumes the knowledge of the viewing probability of different portions (tiles) of a panoramic scene. Such probabilities can be easily derived from crowdsourced measurements performed by 360 video content providers. We then propose efficient methods to solve the problem at runtime while achieving a bounded optimality gap (in terms of the QoE). We implemented our proposed approaches using emulation. Using real users' head movement traces and real cellular bandwidth traces, we show that our algorithms significantly outperform the baseline algorithms by at least in $30\%$ in the QoE metric. Our algorithm gives a streaming rate which is $50\%$ higher compared to the baseline algorithms when the prediction error is high.

cs.MM↗

arXiv2018

Exchange of Renewable Energy among Prosumers using Blockchain with Dynamic Pricing

We consider users which may have renewable energy harvesting devices, or distributed generators. Such users can behave as consumer or producer (hence, we denote them as prosumers) at different time instances. A prosumer may sell the energy to other prosumers in exchange of money. We consider a demand response model, where the price of conventional energy depends on the total demand of all the prosumers at a certain time. A prosumer depending on its own utility has to select the amount of energy it wants to buy either from the grid or from other prosumers, or the amount of excess energy it wants to sell to other prosumers. However, the strategy, and the payoff of a prosumer inherently depends on the strategy of other prosumers as a prosumer can only buy if the other prosumers are willing to sell. We formulate the problem as a coupled constrained game, and seek to obtain the generalized Nash equilibrium. We show that the game is a concave potential game and show that there exists a unique generalized Nash equilibrium. We consider that a platform will set the price for distributed interchange of energy among the prosumers in order to minimize the consumption of the conventional energy. We propose a distributed algorithm where the platform sets a price to each prosumer, and then each prosumer at a certain time only optimizes its own payoff. The prosumer then updates the price depending on the supply and demand for each prosumer. We show that the algorithm converges to an optimal generalized Nash equilibrium. The distributed algorithm also provides an optimal price for the exchange market.

cs.GT↗