arXiv subjects

Imad Ahmad

Publications and source records attributed to Imad Ahmad.

8 recordsLinked to original sources

arXiv2024

AI-assisted Code Authoring at Scale: Fine-tuning, deploying, and mixed methods evaluation

Generative LLMs have been shown to effectively power AI-based code authoring tools that can suggest entire statements or blocks of code during code authoring. In this paper we present CodeCompose, an AI-assisted code authoring tool developed and deployed at Meta internally. CodeCompose is based on the InCoder LLM that merges generative capabilities with bi-directionality. We have scaled up CodeCompose to serve tens of thousands of developers at Meta, across 9 programming languages and several coding surfaces. We present our experience in making design decisions about the model and system architecture for CodeCompose that addresses these challenges. To release a LLM model at this scale, we needed to first ensure that it is sufficiently accurate. In a random sample of 20K source code files, depending on the language, we are able to reproduce hidden lines between 40% and 58% of the time, an improvement of 1.4x and 4.1x over a model trained only on public data. We gradually rolled CodeCompose out to developers. At the time of this writing, 16K developers have used it with 8% of their code coming directly from CodeCompose. To triangulate our numerical findings, we conduct a thematic analysis on the feedback from 70 developers. We find that 91.5% of the feedback is positive, with the most common themes being discovering APIs, dealing with boilerplate code, and accelerating coding. Meta continues to integrate this feedback into CodeCompose.

cs.SE↗

arXiv2024

Multi-line AI-assisted Code Authoring

CodeCompose is an AI-assisted code authoring tool powered by large language models (LLMs) that provides inline suggestions to 10's of thousands of developers at Meta. In this paper, we present how we scaled the product from displaying single-line suggestions to multi-line suggestions. This evolution required us to overcome several unique challenges in improving the usability of these suggestions for developers. First, we discuss how multi-line suggestions can have a 'jarring' effect, as the LLM's suggestions constantly move around the developer's existing code, which would otherwise result in decreased productivity and satisfaction. Second, multi-line suggestions take significantly longer to generate; hence we present several innovative investments we made to reduce the perceived latency for users. These model-hosting optimizations sped up multi-line suggestion latency by 2.5x. Finally, we conduct experiments on 10's of thousands of engineers to understand how multi-line suggestions impact the user experience and contrast this with single-line suggestions. Our experiments reveal that (i) multi-line suggestions account for 42% of total characters accepted (despite only accounting for 16% for displayed suggestions) (ii) multi-line suggestions almost doubled the percentage of keystrokes saved for users from 9% to 17%. Multi-line CodeCompose has been rolled out to all engineers at Meta, and less than 1% of engineers have opted out of multi-line suggestions.

cs.SE↗

arXiv2023

Learning to Learn to Predict Performance Regressions in Production at Meta

Catching and attributing code change-induced performance regressions in production is hard; predicting them beforehand, even harder. A primer on automatically learning to predict performance regressions in software, this article gives an account of the experiences we gained when researching and deploying an ML-based regression prediction pipeline at Meta. In this paper, we report on a comparative study with four ML models of increasing complexity, from (1) code-opaque, over (2) Bag of Words, (3) off-the-shelve Transformer-based, to (4) a bespoke Transformer-based model, coined SuperPerforator. Our investigation shows the inherent difficulty of the performance prediction problem, which is characterized by a large imbalance of benign onto regressing changes. Our results also call into question the general applicability of Transformer-based architectures for performance prediction: an off-the-shelve CodeBERT-based approach had surprisingly poor performance; our highly customized SuperPerforator architecture initially achieved prediction performance that was just on par with simpler Bag of Words models, and only outperformed them for down-stream use cases. This ability of SuperPerforator to transfer to an application with few learning examples afforded an opportunity to deploy it in practice at Meta: it can act as a pre-filter to sort out changes that are unlikely to introduce a regression, truncating the space of changes to search a regression in by up to 43%, a 45x improvement over a random baseline. To gain further insight into SuperPerforator, we explored it via a series of experiments computing counterfactual explanations. These highlight which parts of a code change the model deems important, thereby validating the learned black-box model.

cs.SE↗

arXiv2018

Flexible Fractional Repetition Codes for Distributed Storage Networks

Consider the following fundamental question of distributed storage networks: Given any arbitrary $(n,k,d)$ values, whether there exists an intelligent helper selection scheme (assuming unlimited memory and computing power) that can strictly improve the storage-bandwidth (S-B) tradeoff. Ahmad et al. 18' answered this question by proving that for a subset of $(n,k,d)$ values, no helper selection scheme can ever improve the S-B tradeoff, and for the $(n,k,d)$ not in that subset, a new scheme called family helper selection (FHS) can strictly improve the S-B tradeoff over a blind helper selection scheme. Nonetheless, the analysis of FHS is done by a min-cut analysis with no actual code construction. This work fills this gap between pure min-cut analysis and actual code construction by pairing FHS with a new, generalized version of the existing fractional repetition (FR) codes. Specifically, existing FR codes are exact-repair codes that admit the highly-desirable repair-by-transfer property, but its unique construction limits the application to a restricted set of $(n,k,d)$ values. In contrast, our new construction, termed flexible fractional repetition codes, can be applied to arbitrary $(n,k,d)$ while retaining most of the practical benefits of FR codes, i.e., admitting small repair bandwidth, being exact-repair, and being almost repairable-by-transfer.

cs.IT↗

arXiv2016

Locally Repairable Regenerating Codes: Node Unavailability and the Insufficiency of Stationary Local Repair

Locally repairable codes (LRCs) are ingeniously designed distributed storage codes with a (usually small) fixed set of helper nodes participating in repair. Since most existing LRCs assume exact repair and allow full exchange of the stored data ($β=α$) from the helper nodes, they can be viewed as a generalization of the traditional erasure codes (ECs) with a much desired feature of local repairability via predetermined sets of helpers. However, it also means that they lack the features of (i) functional repair, and (ii) partial information-exchange ($β<α$) in the original regenerating codes (RCs), which could further reduce the repair bandwidth. Motivated by the significant bandwidth reduction of RCs over ECs, existing works by Ahmad et al and by Hollmann studied the concept of "locally repairable regenerating codes (LRRCs)" that successfully combine functional repair and partial information exchange of regenerating codes with the much-desired local repairability feature of LRC. The resulting LRRCs demonstrate significant bandwidth reduction. One important issue that needs to be addressed by any local repair schemes (including both LRCs and LRRCs) is that sometimes designated helper nodes may be temporarily unavailable, the result of multiple failures, degraded reads, or other network dynamics. Under the setting of LRRCs with temporary node unavailability, this work studies the impact of different helper selection methods. It proves that with node unavailability, all existing methods of helper selection, including those used in RCs and LRCs, can be insufficient in terms of achieving the optimal repair-bandwidth. For some scenarios, it is necessary to combine LRRCs with a new helper selection method, termed dynamic helper selection, to achieve optimal repair bandwidth. This work also compares the performance of different helper selection methods and answers...

cs.IT↗

arXiv2016

Locally Repairable Regenerating Code Constructions

In this work, we give locally repairable regenerating code (LRRC) [1]-[3], [5], [6] constructions that can protect the file size promised by the graph analysis of the modified family helper selection (MFHS) scheme [1] at the minimum-bandwidth-regenerating (MBR) point.

cs.IT↗

arXiv2016

When Can Helper Node Selection Improve Regenerating Codes? Part I: Graph-Based Analysis

Regenerating codes (RCs) can significantly reduce the repair-bandwidth of distributed storage networks. Initially, the analysis of RCs was based on the assumption that during the repair process, the newcomer does not distinguish (among all surviving nodes) which nodes to access, i.e., the newcomer is oblivious to the set of helpers being used. Such a scheme is termed the blind helper selection (BHS) scheme. Nonetheless, it is intuitive in practice that the newcomer should choose to access only those "good" helpers. In this two-part paper, a new characterization of the effect of choosing the helper nodes in terms of the storage-bandwidth tradeoff is given. Specifically, the answer to the following fundamental question is provided: Under what condition does proactively choosing the helper nodes improve the storage-bandwidth tradeoff? Through a graph-based analysis, this Part I paper answers this question by providing a necessary and sufficient condition under which optimally choosing good helpers strictly improves the storage-bandwidth tradeoff. A low-complexity helper selection solution, termed the family helper selection (FHS) scheme, is proposed and the corresponding storage/repair-bandwidth curve is characterized. This Part I paper also proves that under some design parameters, the FHS scheme is indeed optimal among all helper selection schemes. In the Part II paper, an explicit construction of an exact-repair code is proposed that achieves the minimum-bandwidth-regenerating (MBR) point of the FHS scheme. The new exact-repair code can be viewed as a generalization of the existing fractional repetition code.

cs.IT↗

arXiv2014

When and By How Much Can Helper Node Selection Improve Regenerating Codes?

Regenerating codes (RCs) can significantly reduce the repair-bandwidth of distributed storage networks. Initially, the analysis of RCs was based on the assumption that during the repair process, the newcomer does not distinguish (among all surviving nodes) which nodes to access, i.e., the newcomer is oblivious to the set of helpers being used. Such a scheme is termed the blind repair (BR) scheme. Nonetheless, it is intuitive in practice that the newcomer should choose to access only those "good" helpers. In this paper, a new characterization of the effect of choosing the helper nodes in terms of the storage-bandwidth tradeoff is given. Specifically, answers to the following fundamental questions are given: Under what conditions does proactively choosing the helper nodes improve the storage-bandwidth tradeoff? Can this improvement be analytically quantified? This paper answers the former question by providing a necessary and sufficient condition under which optimally choosing good helpers strictly improves the storage-bandwidth tradeoff. To answer the latter question, a low-complexity helper selection solution, termed the family repair (FR) scheme, is proposed and the corresponding storage/repair-bandwidth curve is characterized. For example, consider a distributed storage network with 60 total number of nodes and the network is resilient against 50 node failures. If the number of helper nodes is 10, then the FR scheme and its variant demonstrate 27% reduction in the repair-bandwidth when compared to the BR solution. This paper also proves that under some design parameters, the FR scheme is indeed optimal among all helper selection schemes. An explicit construction of an exact-repair code is also proposed that can achieve the minimum-bandwidth-regenerating point of the FR scheme. The new exact-repair code can be viewed as a generalization of the existing fractional repetition code.

cs.IT↗