Searcharxiv⌕ Search

arXiv subjects

Payel Das

Publications and source records attributed to Payel Das.

At least 55 records · Page 3Linked to original sources
arXiv2024

ProtIR: Iterative Refinement between Retrievers and Predictors for Protein Function Annotation

Protein function annotation is an important yet challenging task in biology. Recent deep learning advancements show significant potential for accurate function prediction by learning from protein sequences and structures. Nevertheless, these predictor-based methods often overlook the modeling of protein similarity, an idea commonly employed in traditional approaches using sequence or structure retrieval tools. To fill this gap, we first study the effect of inter-protein similarity modeling by benchmarking retriever-based methods against predictors on protein function annotation tasks. Our results show that retrievers can match or outperform predictors without large-scale pre-training. Building on these insights, we introduce a novel variational pseudo-likelihood framework, ProtIR, designed to improve function predictors by incorporating inter-protein similarity modeling. This framework iteratively refines knowledge between a function predictor and retriever, thereby combining the strengths of both predictors and retrievers. ProtIR showcases around 10% improvement over vanilla predictor-based methods. Besides, it achieves performance on par with protein language model-based methods, yet without the need for massive pre-training, highlighting the efficacy of our framework. Code will be released upon acceptance.

q-bio.BM↗
arXiv2024

Structure-Informed Protein Language Model

Protein language models are a powerful tool for learning protein representations through pre-training on vast protein sequence datasets. However, traditional protein language models lack explicit structural supervision, despite its relevance to protein function. To address this issue, we introduce the integration of remote homology detection to distill structural information into protein language models without requiring explicit protein structures as input. We evaluate the impact of this structure-informed training on downstream protein function prediction tasks. Experimental results reveal consistent improvements in function annotation accuracy for EC number and GO term prediction. Performance on mutant datasets, however, varies based on the relationship between targeted properties and protein structures. This underscores the importance of considering this relationship when applying structure-aware training to protein function prediction tasks. Code and model weights are available at https://github.com/DeepGraphLearning/esm-s.

q-bio.BM↗
arXiv2024

Assembling a high-precision abundance catalogue of solar twins in GALAH for phylogenetic studies

Stellar chemical abundances have proved themselves a key source of information for understanding the evolution of the Milky Way, and the scale of major stellar surveys such as GALAH have massively increased the amount of chemical data available. However, progress is hampered by the level of precision in chemical abundance data as well as the visualization methods for comparing the multidimensional outputs of chemical evolution models to stellar abundance data. Machine learning methods have greatly improved the former; while the application of tree-building or phylogenetic methods borrowed from biology are beginning to show promise with the latter. Here we analyse a sample of GALAH solar twins to address these issues. We apply The Cannon algorithm to generate a catalogue of about 40,000 solar twins with 14 high precision abundances which we use to perform a phylogenetic analysis on a selection of stars that have two different ranges of eccentricities. From our analyses we are able to find a group with mostly stars on circular orbits and some old stars with eccentric orbits whose age-[Y/Mg] relation agrees remarkably well with the chemical clocks published by previous high precision abundance studies. Our results show the power of combining survey data with machine learning and phylogenetics to reconstruct the history of the Milky Way.

astro-ph.GA↗
arXiv2023

Physics-enhanced deep surrogates for partial differential equations

Many physics and engineering applications demand Partial Differential Equations (PDE) property evaluations that are traditionally computed with resource-intensive high-fidelity numerical solvers. Data-driven surrogate models provide an efficient alternative but come with a significant cost of training. Emerging applications would benefit from surrogates with an improved accuracy-cost tradeoff, while studied at scale. Here we present a "physics-enhanced deep-surrogate" ("PEDS") approach towards developing fast surrogate models for complex physical systems, which is described by PDEs. Specifically, a combination of a low-fidelity, explainable physics simulator and a neural network generator is proposed, which is trained end-to-end to globally match the output of an expensive high-fidelity numerical solver. Experiments on three exemplar testcases, diffusion, reaction-diffusion, and electromagnetic scattering models, show that a PEDS surrogate can be up to 3$\times$ more accurate than an ensemble of feedforward neural networks with limited data ($\approx 10^3$ training points), and reduces the training data need by at least a factor of 100 to achieve a target error of 5%. Experiments reveal that PEDS provides a general, data-driven strategy to bridge the gap between a vast array of simplified physical models with corresponding brute-force numerical solvers modeling complex systems, offering accuracy, speed, data efficiency, as well as physical insights into the process.

cs.LG↗
arXiv2023

AlphaFold Distillation for Protein Design

Inverse protein folding, the process of designing sequences that fold into a specific 3D structure, is crucial in bio-engineering and drug discovery. Traditional methods rely on experimentally resolved structures, but these cover only a small fraction of protein sequences. Forward folding models like AlphaFold offer a potential solution by accurately predicting structures from sequences. However, these models are too slow for integration into the optimization loop of inverse folding models during training. To address this, we propose using knowledge distillation on folding model confidence metrics, such as pTM or pLDDT scores, to create a faster and end-to-end differentiable distilled model. This model can then be used as a structure consistency regularizer in training the inverse folding model. Our technique is versatile and can be applied to other design tasks, such as sequence-based protein infilling. Experimental results show that our method outperforms non-regularized baselines, yielding up to 3% improvement in sequence recovery and up to 45% improvement in protein diversity while maintaining structural consistency in generated sequences. Code is available at https://github.com/IBM/AFDistill

q-bio.BM↗
arXiv2023

The Impact of Positional Encoding on Length Generalization in Transformers

Length generalization, the ability to generalize from small training context sizes to larger ones, is a critical challenge in the development of Transformer-based language models. Positional encoding (PE) has been identified as a major factor influencing length generalization, but the exact impact of different PE schemes on extrapolation in downstream tasks remains unclear. In this paper, we conduct a systematic empirical study comparing the length generalization performance of decoder-only Transformers with five different position encoding approaches including Absolute Position Embedding (APE), T5's Relative PE, ALiBi, and Rotary, in addition to Transformers without positional encoding (NoPE). Our evaluation encompasses a battery of reasoning and mathematical tasks. Our findings reveal that the most commonly used positional encoding methods, such as ALiBi, Rotary, and APE, are not well suited for length generalization in downstream tasks. More importantly, NoPE outperforms other explicit positional encoding methods while requiring no additional computation. We theoretically demonstrate that NoPE can represent both absolute and relative PEs, but when trained with SGD, it mostly resembles T5's relative PE attention patterns. Finally, we find that scratchpad is not always helpful to solve length generalization and its format highly impacts the model's performance. Overall, our work suggests that explicit position embeddings are not essential for decoder-only Transformers to generalize well to longer sequences.

cs.CL↗
arXiv2023

The impact of binary stars on the dust and metal evolution of galaxies

We present detailed implementations of (a) binary stellar evolution (using binary_c) and (b) dust production and destruction into the cosmological semi-analytic galaxy evolution simulation, L-Galaxies. This new version of L-Galaxies is compared to a version assuming only single stars and to global and spatially-resolved observational data across a range of redshifts ($z$). We find that binaries have a negligible impact on the stellar masses, gas masses, and star formation rates of galaxies only if the total mass ejected by massive stars is unchanged. This is because massive stars determine the strength of supernova (SN) feedback, which in turn regulates galaxy growth. Binary effects, such as common envelope ejection and novae, affect carbon and nitrogen enrichment in galaxies, however heavier alpha elements are more affected by the choice of SN and wind yields. Unlike many other simulations, the new L-Galaxies reproduces observed dust-to-metal (DTM) and dust-to-gas (DTG) ratios at $z\sim{}0-4$. This is mainly due to shorter dust accretion timescales in dust-rich environments. However, dust masses are under-predicted at $z>4$, highlighting the need for enhanced dust production at early times in simulations, possibly accompanied by increased star formation. On sub-galactic scales, there is very good agreement between L-Galaxies and observed dust and metal radial profiles at $z=0$. A drop in DTM ratio is also found in diffuse, low-metallicity regions, contradicting the assumption of a universal value. We hope that this work serves as a useful template for binary stellar evolution implementations in other cosmological simulations in future.

astro-ph.GA↗
arXiv2023

A Systematic Study of Joint Representation Learning on Protein Sequences and Structures

Learning effective protein representations is critical in a variety of tasks in biology such as predicting protein functions. Recent sequence representation learning methods based on Protein Language Models (PLMs) excel in sequence-based tasks, but their direct adaptation to tasks involving protein structures remains a challenge. In contrast, structure-based methods leverage 3D structural information with graph neural networks and geometric pre-training methods show potential in function prediction tasks, but still suffers from the limited number of available structures. To bridge this gap, our study undertakes a comprehensive exploration of joint protein representation learning by integrating a state-of-the-art PLM (ESM-2) with distinct structure encoders (GVP, GearNet, CDConv). We introduce three representation fusion strategies and explore different pre-training techniques. Our method achieves significant improvements over existing sequence- and structure-based methods, setting new state-of-the-art for function annotation. This study underscores several important design choices for fusing protein sequence and structure information. Our implementation is available at https://github.com/DeepGraphLearning/ESM-GearNet.

q-bio.QM↗
arXiv2023

On the evolutionary history of a simulated disc galaxy as seen by phylogenetic trees

Phylogenetic methods have long been used in biology, and more recently have been extended to other fields - for example, linguistics and technology - to study evolutionary histories. Galaxies also have an evolutionary history, and fall within this broad phylogenetic framework. Under the hypothesis that chemical abundances can be used as a proxy for interstellar medium's DNA, phylogenetic methods allow us to reconstruct hierarchical similarities and differences among stars - essentially a tree of evolutionary relationships and thus history. In this work, we apply phylogenetic methods to a simulated disc galaxy obtained with a chemo-dynamical code to test the approach. We found that at least 100 stellar particles are required to reliably portray the evolutionary history of a selected stellar population in this simulation, and that the overall evolutionary history is reliably preserved when the typical uncertainties in the chemical abundances are smaller than 0.08 dex. The results show that the shape of the trees are strongly affected by the age-metallicity relation, as well as the star formation history of the galaxy. We found that regions with low star formation rates produce shorter trees than regions with high star formation rates. Our analysis demonstrates that phylogenetic methods can shed light on the process of galaxy evolution.

astro-ph.GA↗
arXiv2023

Efficient Equivariant Transfer Learning from Pretrained Models

Efficient transfer learning algorithms are key to the success of foundation models on diverse downstream tasks even with limited data. Recent works of Basu et al. (2023) and Kaba et al. (2022) propose group averaging (equitune) and optimization-based methods, respectively, over features from group-transformed inputs to obtain equivariant outputs from non-equivariant neural networks. While Kaba et al. (2022) are only concerned with training from scratch, we find that equitune performs poorly on equivariant zero-shot tasks despite good finetuning results. We hypothesize that this is because pretrained models provide better quality features for certain transformations than others and simply averaging them is deleterious. Hence, we propose λ-equitune that averages the features using importance weights, λs. These weights are learned directly from the data using a small neural network, leading to excellent zero-shot and finetuned results that outperform equitune. Further, we prove that λ-equitune is equivariant and a universal approximator of equivariant functions. Additionally, we show that the method of Kaba et al. (2022) used with appropriate loss functions, which we call equizero, also gives excellent zero-shot and finetuned performance. Both equitune and equizero are special cases of λ-equitune. To show the simplicity and generality of our method, we validate on a wide range of diverse applications and models such as 1) image classification using CLIP, 2) deep Q-learning, 3) fairness in natural language generation (NLG), 4) compositional generalization in languages, and 5) image classification using pretrained CNNs such as Resnet and Alexnet.

cs.LG↗
arXiv2023

Hierarchical Grammar-Induced Geometry for Data-Efficient Molecular Property Prediction

The prediction of molecular properties is a crucial task in the field of material and drug discovery. The potential benefits of using deep learning techniques are reflected in the wealth of recent literature. Still, these techniques are faced with a common challenge in practice: Labeled data are limited by the cost of manual extraction from literature and laborious experimentation. In this work, we propose a data-efficient property predictor by utilizing a learnable hierarchical molecular grammar that can generate molecules from grammar production rules. Such a grammar induces an explicit geometry of the space of molecular graphs, which provides an informative prior on molecular structural similarity. The property prediction is performed using graph neural diffusion over the grammar-induced geometry. On both small and large datasets, our evaluation shows that this approach outperforms a wide spectrum of baselines, including supervised and pre-trained graph neural networks. We include a detailed ablation study and further analysis of our solution, showing its effectiveness in cases with extremely limited data. Code is available at https://github.com/gmh14/Geo-DEG.

cs.LG↗
arXiv2023

Action-based dynamical models of M31-like galaxies

In this work, we present an action-based dynamical equilibrium model to constrain the phase-space distribution of stars in the stellar halo, present-day dark matter distribution, and the total mass distribution in M31-like galaxies. The model comprises a three-component gravitational potential (stellar bulge, stellar disk, and a dark matter halo), and a double-power law distribution function (DF), $f(\mathbf{J})$, which is a function of actions. A Bayesian model-fitting algorithm was implemented that enabled both parameters of the potential and DF to be explored. After testing the model-fitting algorithm on mock data drawn from the model itself, it was applied to a set of three M31-like haloes from the Auriga simulations (Auriga 21, Auriga 23, Auriga 24). Furthermore, we tested the equilibrium assumption and the ability of a double-power law distribution function to represent the stellar halo stars. The model incurs an error in the total enclosed mass of around 10 percent out to 100 kpc, thus justifying the equilibrium assumption. Furthermore, the double-power law DF used proves to be an appropriate description of the investigated M31-like halos. The anisotropy profiles of the halos were also investigated and discussed from a merger history point of view.

astro-ph.GA↗
arXiv2023

Pre-Training Protein Encoder via Siamese Sequence-Structure Diffusion Trajectory Prediction

Self-supervised pre-training methods on proteins have recently gained attention, with most approaches focusing on either protein sequences or structures, neglecting the exploration of their joint distribution, which is crucial for a comprehensive understanding of protein functions by integrating co-evolutionary information and structural characteristics. In this work, inspired by the success of denoising diffusion models in generative tasks, we propose the DiffPreT approach to pre-train a protein encoder by sequence-structure joint diffusion modeling. DiffPreT guides the encoder to recover the native protein sequences and structures from the perturbed ones along the joint diffusion trajectory, which acquires the joint distribution of sequences and structures. Considering the essential protein conformational variations, we enhance DiffPreT by a method called Siamese Diffusion Trajectory Prediction (SiamDiff) to capture the correlation between different conformers of a protein. SiamDiff attains this goal by maximizing the mutual information between representations of diffusion trajectories of structurally-correlated conformers. We study the effectiveness of DiffPreT and SiamDiff on both atom- and residue-level structure-based protein understanding tasks. Experimental results show that the performance of DiffPreT is consistently competitive on all tasks, and SiamDiff achieves new state-of-the-art performance, considering the mean ranks on all tasks. Our implementation is available at https://github.com/DeepGraphLearning/SiamDiff.

cs.LG↗
arXiv2023

Reprogramming Pretrained Language Models for Antibody Sequence Infilling

Antibodies comprise the most versatile class of binding molecules, with numerous applications in biomedicine. Computational design of antibodies involves generating novel and diverse sequences, while maintaining structural consistency. Unique to antibodies, designing the complementarity-determining region (CDR), which determines the antigen binding affinity and specificity, creates its own unique challenges. Recent deep learning models have shown impressive results, however the limited number of known antibody sequence/structure pairs frequently leads to degraded performance, particularly lacking diversity in the generated sequences. In our work we address this challenge by leveraging Model Reprogramming (MR), which repurposes pretrained models on a source language to adapt to the tasks that are in a different language and have scarce data - where it may be difficult to train a high-performing model from scratch or effectively fine-tune an existing pre-trained model on the specific task. Specifically, we introduce ReprogBert in which a pretrained English language model is repurposed for protein sequence infilling - thus considers cross-language adaptation using less data. Results on antibody design benchmarks show that our model on low-resourced antibody sequence dataset provides highly diverse CDR sequences, up to more than a two-fold increase of diversity over the baselines, without losing structural integrity and naturalness. The generated sequences also demonstrate enhanced antigen binding specificity and virus neutralization ability. Code is available at https://github.com/IBM/ReprogBERT

q-bio.BM↗
arXiv2023

Biological Sequence Design with GFlowNets

Design of de novo biological sequences with desired properties, like protein and DNA sequences, often involves an active loop with several rounds of molecule ideation and expensive wet-lab evaluations. These experiments can consist of multiple stages, with increasing levels of precision and cost of evaluation, where candidates are filtered. This makes the diversity of proposed candidates a key consideration in the ideation phase. In this work, we propose an active learning algorithm leveraging epistemic uncertainty estimation and the recently proposed GFlowNets as a generator of diverse candidate solutions, with the objective to obtain a diverse batch of useful (as defined by some utility function, for example, the predicted anti-microbial activity of a peptide) and informative candidates after each round. We also propose a scheme to incorporate existing labeled datasets of candidates, in addition to a reward function, to speed up learning in GFlowNets. We present empirical results on several biological sequence design tasks, and we find that our method generates more diverse and novel batches with high scoring candidates compared to existing approaches.

q-bio.BM↗
arXiv2023

The outer low-$α$ disc of the Milky Way -- I: evidence for the first pericentric passage of Sagittarius?

Phase-space data, chemistry, and ages together reveal a complex structure in the outer low-$α$ disc of the Milky Way. The age-vertical velocity dispersion profiles beyond the Solar Neighbourhood show a significant jump at 6 Gyr for stars beyond the Galactic plane. Stars older than 6 Gyr are significantly hotter than younger stars. The chemistry and age histograms reveal a bump at [Fe/H] = -0.5, [$α$/Fe] = 0.1, and an age of 7.2 Gyr in the outer disc. Finally, viewing the stars beyond 13.5 kpc in the age-metallicity plane reveals a faint streak just below this bump, towards lower metallicities at the same age. Given the uncertainty in age, we believe these features are linked and suggest a pericentric passage of a massive satellite 6 Gyr ago that heated pre-existing stars, led to a starburst in existing gas. New stars also formed from the metal-poorer infalling gas. The impulse approximation was used to characterise the interaction with a satellite, finding a mass of ~1e11 M$_{\odot}$, and a pericentric position between 12 and 16 kpc. The evidence points to an interaction with the Sagittarius dwarf galaxy, likely its first pericentric passage.

astro-ph.GA↗
arXiv2023

Equi-Tuning: Group Equivariant Fine-Tuning of Pretrained Models

We introduce equi-tuning, a novel fine-tuning method that transforms (potentially non-equivariant) pretrained models into group equivariant models while incurring minimum $L_2$ loss between the feature representations of the pretrained and the equivariant models. Large pretrained models can be equi-tuned for different groups to satisfy the needs of various downstream tasks. Equi-tuned models benefit from both group equivariance as an inductive bias and semantic priors from pretrained models. We provide applications of equi-tuning on three different tasks: image classification, compositional generalization in language, and fairness in natural language generation (NLG). We also provide a novel group-theoretic definition for fairness in NLG. The effectiveness of this definition is shown by testing it against a standard empirical method of fairness in NLG. We provide experimental results for equi-tuning using a variety of pretrained models: Alexnet, Resnet, VGG, and Densenet for image classification; RNNs, GRUs, and LSTMs for compositional generalization; and GPT2 for fairness in NLG. We test these models on benchmark datasets across all considered tasks to show the generality and effectiveness of the proposed method.

cs.LG↗
arXiv2023

Accelerating Material Design with the Generative Toolkit for Scientific Discovery

With the growing availability of data within various scientific domains, generative models hold enormous potential to accelerate scientific discovery. They harness powerful representations learned from datasets to speed up the formulation of novel hypotheses with the potential to impact material discovery broadly. We present the Generative Toolkit for Scientific Discovery (GT4SD). This extensible open-source library enables scientists, developers, and researchers to train and use state-of-the-art generative models to accelerate scientific discovery focused on material design.

cs.LG↗