arXiv subjects

Colin B. Clement

Publications and source records attributed to Colin B. Clement.

14 recordsLinked to original sources

arXiv2024

Inverse Melting of Polar Order in Chemically Substituted BaTiO3

In many condensed matter systems, long range order emerges at low temperatures as thermal fluctuations subside. In the presence of competing interactions or quenched disorder, however, some systems can show unusual configurations that become more disordered at low temperature, a rare phenomenon known as "inverse melting". Here, we discover an inverse melting of the polar order in a ferroelectric oxide with quenched chemical disorder (BaTi1-xZrxO3) through direct atomicscale visualization using in situ scanning transmission electron microscopy. In contrast to the clean BaTiO3 parent system in which long range order tracks lower temperatures, we observe in the doped system BaTi1-xZrxO3 that thermally driven fluctuations at high temperature give way to a more ordered state and then to a re-entrant disordered configuration at even lower temperature. Such an inverse melting of the polar order is likely linked to the random field generated by Zr dopants, which modulates the energy landscape arising from the competition between thermal fluctuations and random field pinning potential. These visualizations highlight a rich landscape of order and disorder in materials with quenched disorder, which may be key to understanding their advanced functionalities.

cond-mat.dis-nn↗

arXiv2022

DeepPERF: A Deep Learning-Based Approach For Improving Software Performance

Improving software performance is an important yet challenging part of the software development cycle. Today, the majority of performance inefficiencies are identified and patched by performance experts. Recent advancements in deep learning approaches and the wide-spread availability of open source data creates a great opportunity to automate the identification and patching of performance problems. In this paper, we present DeepPERF, a transformer-based approach to suggest performance improvements for C# applications. We pretrain DeepPERF on English and Source code corpora and followed by finetuning for the task of generating performance improvement patches for C# applications. Our evaluation shows that our model can generate the same performance improvement suggestion as the developer fix in ~53% of the cases, getting ~34% of them verbatim in our expert-verified dataset of performance changes made by C# developers. Additionally, we evaluate DeepPERF on 50 open source C# repositories on GitHub using both benchmark and unit tests and find that our model is able to suggest valid performance improvements that can improve both CPU usage and Memory allocations. So far we've submitted 19 pull-requests with 28 different performance optimizations and 11 of these PRs have been approved by the project owners.

cs.SE↗

arXiv2022

Generating Examples From CLI Usage: Can Transformers Help?

Continuous evolution in modern software often causes documentation, tutorials, and examples to be out of sync with changing interfaces and frameworks. Relying on outdated documentation and examples can lead programs to fail or be less efficient or even less secure. In response, programmers need to regularly turn to other resources on the web such as StackOverflow for examples to guide them in writing software. We recognize that this inconvenient, error-prone, and expensive process can be improved by using machine learning applied to software usage data. In this paper, we present our practical system which uses machine learning on large-scale telemetry data and documentation corpora, generating appropriate and complex examples that can be used to improve documentation. We discuss both feature-based and transformer-based machine learning approaches and demonstrate that our system achieves 100% coverage for the used functionalities in the product, providing up-to-date examples upon every release and reduces the numbers of PRs submitted by software owners writing and editing documentation by >68%. We also share valuable lessons learnt during the 3 years that our production quality system has been deployed for Azure Cloud Command Line Interface (Azure CLI).

cs.SE↗

arXiv2022

Training and Evaluating a Jupyter Notebook Data Science Assistant

We study the feasibility of a Data Science assistant powered by a sequence-to-sequence transformer by training a new model JuPyT5 on all publicly available Jupyter Notebook GitHub repositories and developing a new metric: Data Science Problems (DSP). DSP is a collection of 1119 problems curated from 306 pedagogical notebooks with 92 dataset dependencies, natural language and Markdown problem descriptions, and assert-based unit tests. These notebooks were designed to test university students' mastery of various Python implementations of Math and Data Science, and we now leverage them to study the ability of JuPyT5 to understand and pass the tests. We analyze the content of DSP, validate its quality, and we find that given 100 sampling attempts JuPyT5 is able to solve 77.5\% of the DSP problems. We further present various ablation and statistical analyses and compare DSP to other recent natural language to code benchmarks.

cs.LG↗

arXiv2021

Long-Range Modeling of Source Code Files with eWASH: Extended Window Access by Syntax Hierarchy

Statistical language modeling and translation with transformers have found many successful applications in program understanding and generation tasks, setting high benchmarks for tools in modern software development environments. The finite context window of these neural models means, however, that they will be unable to leverage the entire relevant context of large files and packages for any given task. While there are many efforts to extend the context window, we introduce an architecture-independent approach for leveraging the syntactic hierarchies of source code for incorporating entire file-level context into a fixed-length window. Using concrete syntax trees of each source file we extract syntactic hierarchies and integrate them into context window by selectively removing from view more specific, less relevant scopes for a given task. We evaluate this approach on code generation tasks and joint translation of natural language and source code in Python programming language, achieving a new state-of-the-art in code completion and summarization for Python in the CodeXGLUE benchmark. We also introduce new CodeXGLUE benchmarks for user-experience-motivated tasks: code completion with normalized literals, method body completion/code summarization conditioned on file-level context.

cs.LG↗

arXiv2021

Distilling Transformers for Neural Cross-Domain Search

Pre-trained transformers have recently clinched top spots in the gamut of natural language tasks and pioneered solutions to software engineering tasks. Even information retrieval has not been immune to the charm of the transformer, though their large size and cost is generally a barrier to deployment. While there has been much work in streamlining, caching, and modifying transformer architectures for production, here we explore a new direction: distilling a large pre-trained translation model into a lightweight bi-encoder which can be efficiently cached and queried. We argue from a probabilistic perspective that sequence-to-sequence models are a conceptually ideal---albeit highly impractical---retriever. We derive a new distillation objective, implementing it as a data augmentation scheme. Using natural language source code search as a case study for cross-domain search, we demonstrate the validity of this idea by significantly improving upon the current leader of the CodeSearchNet challenge, a recent natural language code search benchmark.

cs.IR↗

arXiv2021

Reconstruction of Current Densities from Magnetic Images by Bayesian Inference

Electronic transport is at the heart of many phenomena in condensed matter physics and material science. Magnetic imaging is a non-invasive tool for detecting electric current in materials and devices. A two-dimensional current density can be reconstructed from an image of a single component of the magnetic field produced by the current. In this work, we approach the reconstruction problem in the framework of Bayesian inference, i.e. we solve for the most likely current density given an image obtained by a magnetic probe. To enforce a sensible current density priors are used to associate a cost with unphysical features such as pixel-to-pixel oscillations or current outside the device boundary. Beyond previous work, our approach does not require analytically tractable priors and therefore creates flexibility to use priors that have not been explored in the context of current reconstruction. Here, we implement several such priors that have desirable properties. A challenging aspect of imposing a prior is choosing the optimal strength. We describe an empirical way to determine the appropriate strength of the prior. We test our approach on numerically generated examples. Our code is released in an open-source \texttt{python} package called \texttt{pysquid}.

physics.data-an↗

arXiv2021

DeepDebug: Fixing Python Bugs Using Stack Traces, Backtranslation, and Code Skeletons

The joint task of bug localization and program repair is an integral part of the software development process. In this work we present DeepDebug, an approach to automated debugging using large, pretrained transformers. We begin by training a bug-creation model on reversed commit data for the purpose of generating synthetic bugs. We apply these synthetic bugs toward two ends. First, we directly train a backtranslation model on all functions from 200K repositories. Next, we focus on 10K repositories for which we can execute tests, and create buggy versions of all functions in those repositories that are covered by passing tests. This provides us with rich debugging information such as stack traces and print statements, which we use to finetune our model which was pretrained on raw source code. Finally, we strengthen all our models by expanding the context window beyond the buggy function itself, and adding a skeleton consisting of that function's parent class, imports, signatures, docstrings, and method bodies, in order of priority. On the QuixBugs benchmark, we increase the total number of fixes found by over 50%, while also decreasing the false positive rate from 35% to 5% and decreasing the timeout from six hours to one minute. On our own benchmark of executable tests, our model fixes 68% of all bugs on its first attempt without using traces, and after adding traces it fixes 75% on first attempt. We will open-source our framework and validation set for evaluating on executable tests.

cs.SE↗

arXiv2020

PyMT5: multi-mode translation of natural language and Python code with transformers

Simultaneously modeling source code and natural language has many exciting applications in automated software development and understanding. Pursuant to achieving such technology, we introduce PyMT5, the Python method text-to-text transfer transformer, which is trained to translate between all pairs of Python method feature combinations: a single model that can both predict whole methods from natural language documentation strings (docstrings) and summarize code into docstrings of any common style. We present an analysis and modeling effort of a large-scale parallel corpus of 26 million Python methods and 7.7 million method-docstring pairs, demonstrating that for docstring and method generation, PyMT5 outperforms similarly-sized auto-regressive language models (GPT2) which were English pre-trained or randomly initialized. On the CodeSearchNet test set, our best model predicts 92.1% syntactically correct method bodies, achieved a BLEU score of 8.59 for method generation and 16.3 for docstring generation (summarization), and achieved a ROUGE-L F-score of 24.8 for method generation and 36.7 for docstring generation.

cs.LG↗

arXiv2020

Visualizing probabilistic models in Minkowski space with intensive symmetrized Kullback-Leibler embedding

We show that the predicted probability distributions for any $N$-parameter statistical model taking the form of an exponential family can be explicitly and analytically embedded isometrically in a $N{+}N$-dimensional Minkowski space. That is, the model predictions can be visualized as control parameters are varied, preserving the natural distance between probability distributions. All pairwise distances between model instances are given by the symmetrized Kullback-Leibler divergence. We give formulas for these intensive symmetrized Kullback Leibler (isKL) coordinate embeddings, and illustrate the resulting visualizations with the Bernoulli (coin toss) problem, the ideal gas, $n$ sided die, the nonlinear least squares fit, and the Gaussian fit. We highlight how isKL can be used to determine the minimum number of parameters needed to describe probabilistic data, and conclude by visualizing the prediction space of the two-dimensional Ising model, where we examine the manifold behavior near its critical point.

cond-mat.stat-mech↗

arXiv2019

Visualizing probabilistic models: Intensive Principal Component Analysis

Unsupervised learning makes manifest the underlying structure of data without curated training and specific problem definitions. However, the inference of relationships between data points is frustrated by the `curse of dimensionality' in high-dimensions. Inspired by replica theory from statistical mechanics, we consider replicas of the system to tune the dimensionality and take the limit as the number of replicas goes to zero. The result is the intensive embedding, which is not only isometric (preserving local distances) but allows global structure to be more transparently visualized. We develop the Intensive Principal Component Analysis (InPCA) and demonstrate clear improvements in visualizations of the Ising model of magnetic spins, a neural network, and the dark energy cold dark matter (ΛCDM) model as applied to the Cosmic Microwave Background.

cond-mat.stat-mech↗

arXiv2019

On the Use of ArXiv as a Dataset

The arXiv has collected 1.5 million pre-print articles over 28 years, hosting literature from scientific fields including Physics, Mathematics, and Computer Science. Each pre-print features text, figures, authors, citations, categories, and other metadata. These rich, multi-modal features, combined with the natural graph structure---created by citation, affiliation, and co-authorship---makes the arXiv an exciting candidate for benchmarking next-generation models. Here we take the first necessary steps toward this goal, by providing a pipeline which standardizes and simplifies access to the arXiv's publicly available data. We use this pipeline to extract and analyze a 6.7 million edge citation graph, with an 11 billion word corpus of full-text research articles. We present some baseline classification results, and motivate application of more exciting generative graph models.

cs.IR↗

arXiv2019

Normal form for renormalization groups

The results of the renormalization group are commonly advertised as the existence of power law singularities near critical points. The classic predictions are often violated and logarithmic and exponential corrections are treated on a case-by-case basis. We use the mathematics of normal form theory to systematically group these into universality families of seemingly unrelated systems united by common scaling variables. We recover and explain the existing literature and predict the nonlinear generalization for the universal homogeneous scaling functions. We show that this procedure leads to a better handling of the singularity even in classic cases and elaborate our framework using several examples.

cond-mat.stat-mech↗

arXiv2019

Image registration and super resolution from first principles

Image registration is the inference of transformations relating noisy and distorted images. It is fundamental in computer vision, experimental physics, and medical imaging. Many algorithms and analyses exist for inferring shift, rotation, and nonlinear transformations between image coordinates. Even in the simplest case of translation, however, all known algorithms are biased and none have achieved the precision limit of the Cramer Rao bound (CRB). Following Bayesian inference, we prove that the standard method of shifting one image to match another cannot reach the CRB. We show that the bias can be cured and the CRB reached if, instead, we use Super Registration: learning an optimal model for the underlying image and shifting that to match the data. Our theory shows that coarse-graining oversampled images can improve registration precision of the standard method. For oversampled data, our method does not yield striking improvements as measured by eye. In these cases, however, we show our new registration method can lead to dramatic improvements in extractable information, for example, inferring $10\times$ more precise particle positions.

physics.data-an↗