arXiv subjects

Kailash Gogineni

Publications and source records attributed to Kailash Gogineni.

8 recordsLinked to original sources

arXiv2024

SwiftRL: Towards Efficient Reinforcement Learning on Real Processing-In-Memory Systems

Reinforcement Learning (RL) trains agents to learn optimal behavior by maximizing reward signals from experience datasets. However, RL training often faces memory limitations, leading to execution latencies and prolonged training times. To overcome this, SwiftRL explores Processing-In-Memory (PIM) architectures to accelerate RL workloads. We achieve near-linear performance scaling by implementing RL algorithms like Tabular Q-learning and SARSA on UPMEM PIM systems and optimizing for hardware. Our experiments on OpenAI GYM environments using UPMEM hardware demonstrate superior performance compared to CPU and GPU implementations.

cs.LG↗

arXiv2023

AccMER: Accelerating Multi-Agent Experience Replay with Cache Locality-aware Prioritization

Multi-Agent Experience Replay (MER) is a key component of off-policy reinforcement learning~(RL) algorithms. By remembering and reusing experiences from the past, experience replay significantly improves the stability of RL algorithms and their learning efficiency. In many scenarios, multiple agents interact in a shared environment during online training under centralized training and decentralized execution~(CTDE) paradigm. Current multi-agent reinforcement learning~(MARL) algorithms consider experience replay with uniform sampling or based on priority weights to improve transition data sample efficiency in the sampling phase. However, moving transition data histories for each agent through the processor memory hierarchy is a performance limiter. Also, as the agents' transitions continuously renew every iteration, the finite cache capacity results in increased cache misses. To this end, we propose \name, that repeatedly reuses the transitions~(experiences) for a window of $n$ steps in order to improve the cache locality and minimize the transition data movement, instead of sampling new transitions at each step. Specifically, our optimization uses priority weights to select the transitions so that only high-priority transitions will be reused frequently, thereby improving the cache performance. Our experimental results on the Predator-Prey environment demonstrate the effectiveness of reusing the essential transitions based on the priority weights, where we observe an end-to-end training time reduction of $25.4\%$~(for $32$ agents) compared to existing prioritized MER algorithms without notable degradation in the mean reward.

cs.MA↗

arXiv2023

Towards Efficient Multi-Agent Learning Systems

Multi-Agent Reinforcement Learning (MARL) is an increasingly important research field that can model and control multiple large-scale autonomous systems. Despite its achievements, existing multi-agent learning methods typically involve expensive computations in terms of training time and power arising from large observation-action space and a huge number of training steps. Therefore, a key challenge is understanding and characterizing the computationally intensive functions in several popular classes of MARL algorithms during their training phases. Our preliminary experiments reveal new insights into the key modules of MARL algorithms that limit the adoption of MARL in real-world systems. We explore neighbor sampling strategy to improve cache locality and observe performance improvement ranging from 26.66% (3 agents) to 27.39% (12 agents) during the computationally intensive mini-batch sampling phase. Additionally, we demonstrate that improving the locality leads to an end-to-end training time reduction of 10.2% (for 12 agents) compared to existing multi-agent algorithms without significant degradation in the mean reward.

cs.MA↗

arXiv2023

Scalability Bottlenecks in Multi-Agent Reinforcement Learning Systems

Multi-Agent Reinforcement Learning (MARL) is a promising area of research that can model and control multiple, autonomous decision-making agents. During online training, MARL algorithms involve performance-intensive computations such as exploration and exploitation phases originating from large observation-action space belonging to multiple agents. In this article, we seek to characterize the scalability bottlenecks in several popular classes of MARL algorithms during their training phases. Our experimental results reveal new insights into the key modules of MARL algorithms that limit the scalability, and outline potential strategies that may help address these performance issues.

cs.MA↗

arXiv2022

A Framework for Server Authentication using Communication Protocol Dialects

In today's world, computer networks have become vulnerable to numerous attacks. In both wireless and wired networks, one of the most common attacks is man-in-the-middle attacks, within which session hijacking, context confusion attacks have been the most attempted. A potential attacker may have enough time to launch an attack targeting these vulnerabilities (such as rerouting the target request to a malicious server or hijacking the traffic). A viable strategy to solve this problem is, by dynamically changing the system properties, configurations and create unique fingerprints to identify the source. However, the existing work of fingerprinting mainly focuses on lower-level properties (e.g IP address), and only these types of properties are restricted for mutation. We develop a novel system, called Verify-Pro, to provide server authentication using communication protocol dialects, that uses a client-server architecture based on network protocols for customizing the communication transactions. For each session, a particular sequence of handshakes will be used as dialects. So, given the context, with the establishment of a one-time username and password, we use the dialects as an authentication mechanism for each request (e.g get filename in FTP) throughout the session, which enforces continuous authentication. Specifically, we leverage a machine learning approach on both client and server machines to trigger a specific dialect that dynamically changes for each request. We implement a prototype of Verify-Pro and evaluate its practicality on standard communication protocols FTP, HTTP & internet of things protocol MQTT. Our experimental results show that by sending misleading information through message packets from an attacker at the application layer, it is possible for the recipient to identify if the sender is genuine or a spoofed one, with a negligible overhead of 0.536%.

cs.CR↗

arXiv2021

MPD: Moving Target Defense through Communication Protocol Dialects

Communication protocol security is among the most significant challenges of the Internet of Things (IoT) due to the wide variety of hardware and software technologies involved. Moving target defense (MTD) has been adopted as an innovative strategy to solve this problem by dynamically changing target system properties and configurations to obfuscate the attack surface. Nevertheless, the existing work of MTD primarily focuses on lower-level properties (e.g., IP addresses or port numbers), and only a limited number of variations can be generated based on these properties. In this paper, we propose a new approach of MTD through communication protocol dialects (MPD) - which dynamically customizes a communication protocol into various protocol dialects and leverages them to create a moving target defense. Specifically, MPD harnesses a dialect generating function to create protocol dialects and then a mapping function to select one specific dialect for each packet during communication. To keep different network entities in synchronization, we also design a self-synchronization mechanism utilizing a pseudo-random number generator with the input of a pre-shared secret key and previously sent packets. We implement a prototype of MPD and evaluate its feasibility on standard network protocol (i.e., File Transfer Protocol) and internet of things protocol (i.e., Message Queuing Telemetry Transport). The results indicate that MPD can create a moving target defense with protocol dialects to effectively address various attacks - including the denial of service attack and malicious packet modifications - with negligible overhead.

cs.CR↗

arXiv2021

Integrated Reasoning Engine for Pointer-related Code Clone Detection

Detecting similar code fragments, usually referred to as code clones, is an important task. In particular, code clone detection can have significant uses in the context of vulnerability discovery, refactoring and plagiarism detection. However, false positives are inevitable and always require manual reviews. In this paper, we propose Twin-Finder+, a novel closed-loop approach for pointer-related code clone detection that integrates machine learning and symbolic execution techniques to achieve precision. Twin-Finder+ introduces a formal verification mechanism to automate such manual reviews process. Our experimental results show Twin-Finder+ that can remove 91.69% false positives in average. We further conduct security analysis for memory safety using real-world applications, Links version 2.14 and libreOffice-6.0.0.1. Twin-Finder+ is able to find 6 unreported bugs in Links version 2.14 and one public patched bug in libreOffice-6.0.0.1.

cs.SE↗

arXiv2020

Twin-Finder: Integrated Reasoning Engine for Pointer-related Code Clone Detection

Detecting code clones is crucial in various software engineering tasks. In particular, code clone detection can have significant uses in the context of analyzing and fixing bugs in large scale applications. However, prior works, such as machine learning-based clone detection, may cause a considerable amount of false positives. In this paper, we propose Twin-Finder, a novel, closed-loop approach for pointer-related code clone detection that integrates machine learning and symbolic execution techniques to achieve precision. Twin-Finder introduces a clone verification mechanism to formally verify if two clone samples are indeed clones and a feedback loop to automatically generated formal rules to tune machine learning algorithm and further reduce the false positives. Our experimental results show that Twin-Finder can swiftly identify up 9X more code clones comparing to a tree-based clone detector, Deckard and remove an average 91.69% false positives.

cs.SE↗