arXiv subjects

Payel Das

Publications and source records attributed to Payel Das.

At least 37 records · Page 2Linked to original sources

arXiv2025

Fundamental Safety-Capability Trade-offs in Fine-tuning Large Language Models

Fine-tuning Large Language Models (LLMs) on some task-specific datasets has been a primary use of LLMs. However, it has been empirically observed that this approach to enhancing capability inevitably compromises safety, a phenomenon also known as the safety-capability trade-off in LLM fine-tuning. This paper presents a theoretical framework for understanding the interplay between safety and capability in two primary safety-aware LLM fine-tuning strategies, providing new insights into the effects of data similarity, context overlap, and alignment loss landscape. Our theoretical results characterize the fundamental limits of the safety-capability trade-off in LLM fine-tuning, which are also validated by numerical experiments.

stat.ML↗

arXiv2025

Can Memory-Augmented Language Models Generalize on Reasoning-in-a-Haystack Tasks?

Large language models often expose their brittleness in reasoning tasks, especially while executing long chains of reasoning over context. We propose MemReasoner, a new and simple memory-augmented LLM architecture, in which the memory learns the relative order of facts in context, and enables hopping over them, while the decoder selectively attends to the memory. MemReasoner is trained end-to-end, with optional supporting fact supervision of varying degrees. We train MemReasoner, along with existing memory-augmented transformer models and a state-space model, on two distinct synthetic multi-hop reasoning tasks. Experiments performed under a variety of challenging scenarios, including the presence of long distractor text or target answer changes in test set, show strong generalization of MemReasoner on both single- and two-hop tasks. This generalization of MemReasoner is achieved using none-to-weak supporting fact supervision (using none and 1\% of supporting facts for one- and two-hop tasks, respectively). In contrast, baseline models overall struggle to generalize and benefit far less from using full supporting fact supervision. The results highlight the importance of explicit memory mechanisms, combined with additional weak supervision, for improving large language model's context processing ability toward reasoning tasks.

cs.CL↗

arXiv2025

EpMAN: Episodic Memory AttentioN for Generalizing to Longer Contexts

Recent advances in Large Language Models (LLMs) have yielded impressive successes on many language tasks. However, efficient processing of long contexts using LLMs remains a significant challenge. We introduce \textbf{EpMAN} -- a method for processing long contexts in an \textit{episodic memory} module while \textit{holistically attending to} semantically relevant context chunks. The output of \textit{episodic attention} is then used to reweigh the decoder's self-attention to the stored KV cache of the context during training and generation. When an LLM decoder is trained using \textbf{EpMAN}, its performance on multiple challenging single-hop long-context recall and question-answering benchmarks is found to be stronger and more robust across the range from 16k to 256k tokens than baseline decoders trained with self-attention, and popular retrieval-augmented generation frameworks.

cs.CL↗

arXiv2024

Multi-Scale Representation Learning for Protein Fitness Prediction

Designing novel functional proteins crucially depends on accurately modeling their fitness landscape. Given the limited availability of functional annotations from wet-lab experiments, previous methods have primarily relied on self-supervised models trained on vast, unlabeled protein sequence or structure datasets. While initial protein representation learning studies solely focused on either sequence or structural features, recent hybrid architectures have sought to merge these modalities to harness their respective strengths. However, these sequence-structure models have so far achieved only incremental improvements when compared to the leading sequence-only approaches, highlighting unresolved challenges effectively leveraging these modalities together. Moreover, the function of certain proteins is highly dependent on the granular aspects of their surface topology, which have been overlooked by prior models. To address these limitations, we introduce the Sequence-Structure-Surface Fitness (S3F) model - a novel multimodal representation learning framework that integrates protein features across several scales. Our approach combines sequence representations from a protein language model with Geometric Vector Perceptron networks encoding protein backbone and detailed surface topology. The proposed method achieves state-of-the-art fitness prediction on the ProteinGym benchmark encompassing 217 substitution deep mutational scanning assays, and provides insights into the determinants of protein function. Our code is at https://github.com/DeepGraphLearning/S3F.

cs.LG↗

arXiv2024

SEAL: Safety-enhanced Aligned LLM Fine-tuning via Bilevel Data Selection

Fine-tuning on task-specific data to boost downstream performance is a crucial step for leveraging Large Language Models (LLMs). However, previous studies have demonstrated that fine-tuning the models on several adversarial samples or even benign data can greatly comprise the model's pre-equipped alignment and safety capabilities. In this work, we propose SEAL, a novel framework to enhance safety in LLM fine-tuning. SEAL learns a data ranker based on the bilevel optimization to up rank the safe and high-quality fine-tuning data and down rank the unsafe or low-quality ones. Models trained with SEAL demonstrate superior quality over multiple baselines, with 8.5% and 9.7% win rate increase compared to random selection respectively on Llama-3-8b-Instruct and Merlinite-7b models. Our code is available on github https://github.com/hanshen95/SEAL.

cs.LG↗

arXiv2024

Large Language Models can be Strong Self-Detoxifiers

Reducing the likelihood of generating harmful and toxic output is an essential task when aligning large language models (LLMs). Existing methods mainly rely on training an external reward model (i.e., another language model) or fine-tuning the LLM using self-generated data to influence the outcome. In this paper, we show that LLMs have the capability of self-detoxification without the use of an additional reward model or re-training. We propose \textit{Self-disciplined Autoregressive Sampling (SASA)}, a lightweight controlled decoding algorithm for toxicity reduction of LLMs. SASA leverages the contextual representations from an LLM to learn linear subspaces characterizing toxic v.s. non-toxic output in analytical forms. When auto-completing a response token-by-token, SASA dynamically tracks the margin of the current output to steer the generation away from the toxic subspace, by adjusting the autoregressive sampling strategy. Evaluated on LLMs of different scale and nature, namely Llama-3.1-Instruct (8B), Llama-2 (7B), and GPT2-L models with the RealToxicityPrompts, BOLD, and AttaQ benchmarks, SASA markedly enhances the quality of the generated sentences relative to the original models and attains comparable performance to state-of-the-art detoxification techniques, significantly reducing the toxicity level by only using the LLM's internal representations.

cs.LG↗

arXiv2024

Keeping Up with the Language Models: Systematic Benchmark Extension for Bias Auditing

Bias auditing of language models (LMs) has received considerable attention as LMs are becoming widespread. As such, several benchmarks for bias auditing have been proposed. At the same time, the rapid evolution of LMs can make these benchmarks irrelevant in no time. Bias auditing is further complicated by LM brittleness: when a presumably biased outcome is observed, is it due to model bias or model brittleness? We propose enlisting the models themselves to help construct bias auditing datasets that remain challenging, and introduce bias measures that distinguish between different types of model errors. First, we extend an existing bias benchmark for NLI (BBNLI) using a combination of LM-generated lexical variations, adversarial filtering, and human validation. We demonstrate that the newly created dataset BBNLI-next is more challenging than BBNLI: on average, BBNLI-next reduces the accuracy of state-of-the-art NLI models from 95.3%, as observed by BBNLI, to a strikingly low 57.5%. Second, we employ BBNLI-next to showcase the interplay between robustness and bias: we point out shortcomings in current bias scores and propose bias measures that take into account both bias and model brittleness. Third, despite the fact that BBNLI-next was designed with non-generative models in mind, we show that the new dataset is also able to uncover bias in state-of-the-art open-source generative LMs. Note: All datasets included in this work are in English and they address US-centered social biases. In the spirit of efficient NLP research, no model training or fine-tuning was performed to conduct this research. Warning: This paper contains offensive text examples.

cs.CL↗

arXiv2024

Attribute Graphs Underlying Molecular Generative Models: Path to Learning with Limited Data

Training generative models that capture rich semantics of the data and interpreting the latent representations encoded by such models are very important problems in un-/self-supervised learning. In this work, we provide a simple algorithm that relies on perturbation experiments on latent codes of a pre-trained generative autoencoder to uncover an attribute graph that is implied by the generative model. We perform perturbation experiments to check for influence of a given latent variable on a subset of attributes. Given this, we show that one can fit an effective graphical model that models a structural equation model between latent codes taken as exogenous variables and attributes taken as observed variables. One interesting aspect is that a single latent variable controls multiple overlapping subsets of attributes unlike conventional approaches that try to impose full independence. Using a pre-trained generative autoencoder trained on a large dataset of small molecules, we demonstrate that the graphical model between various molecular attributes and latent codes learned by our algorithm can be used to predict a specific property for molecules which are drawn from a different distribution. We compare prediction models trained on various feature subsets chosen by simple baselines, as well as existing causal discovery and sparse learning/feature selection methods, with the ones in the derived Markov blanket from our method. Results show empirically that the predictor that relies on our Markov blanket attributes is robust to distribution shifts when transferred or fine-tuned with a few samples from the new distribution, especially when training data is limited.

cs.LG↗

arXiv2024

EDGE: Predictable Scatter in the Stellar Mass--Halo Mass Relation of Dwarf Galaxies

The stellar-mass--halo-mass (SMHM) relation is central to our understanding of galaxy formation and the nature of dark matter. However, its normalisation, slope, and scatter are highly uncertain at dwarf galaxy scales. In this paper, we present DarkLight, a new semi-empirical dwarf galaxy formation model designed to robustly predict the SMHM relation for the smallest galaxies. DarkLight harnesses a correlation between the mean star formation rate of dwarfs and their peak rotation speed -- the $\langle$SFR$\rangle$-$v_{\rm max}$ relation -- that we derive from simulations and observations. Given the sparsity of data for isolated dwarfs with $v_{\rm max} \lesssim 20$ km/s, we fit the $\langle$SFR$\rangle$-$v_{\rm max}$ relation to observational data for dwarfs above this velocity scale and to the high-resolution EDGE cosmological simulations below. Reionisation quenching is implemented via distinct $\langle$SFR$\rangle$-$v_{\rm max}$ relations before and after reionisation. We find that the SMHM scatter is small at reionisation, $\sim$0.2 dex, but rises to $\sim$0.5 dex ($1σ$) at a halo mass of $\sim$10$^9$ M$_\odot$ as star formation is quenched by reionisation but dark matter halo masses continue to grow. While we do not find a significant break in the slope of the SMHM relation, one can be introduced if reionisation occurs early ($z_{\rm quench} \gtrsim 5$). Finally, we find that dwarfs can be star forming today down to a halo mass of $\sim$2 $\times 10^9$ M$_\odot$. We predict that the lowest mass star forming dwarf irregulars in the nearby universe are the tip of the iceberg of a much larger population of quiescent isolated dwarfs.

astro-ph.GA↗

arXiv2024

Larimar: Large Language Models with Episodic Memory Control

Efficient and accurate updating of knowledge stored in Large Language Models (LLMs) is one of the most pressing research challenges today. This paper presents Larimar - a novel, brain-inspired architecture for enhancing LLMs with a distributed episodic memory. Larimar's memory allows for dynamic, one-shot updates of knowledge without the need for computationally expensive re-training or fine-tuning. Experimental results on multiple fact editing benchmarks demonstrate that Larimar attains accuracy comparable to most competitive baselines, even in the challenging sequential editing setup, but also excels in speed - yielding speed-ups of 8-10x depending on the base LLM - as well as flexibility due to the proposed architecture being simple, LLM-agnostic, and hence general. We further provide mechanisms for selective fact forgetting, information leakage prevention, and input context length generalization with Larimar and show their effectiveness. Our code is available at https://github.com/IBM/larimar

cs.LG↗

arXiv2024

The Chemical Diversity of the Metal-Poor Milky Way

We present a detailed study of the chemical diversity of the metal-poor Milky Way (MW) using data from the GALAH DR3 survey. Considering 17 chemical abundances relative to iron ([X/Fe]) for 9,923 stars, we employ Principal Component Analysis (PCA) and Extreme Deconvolution (XD) to identify 10 distinct stellar groups. This approach, free from chemical or dynamical cuts, reveals known populations, including the accreted halo, thick disc, thin disc, and in-situ halo. The thick disc is characterised by multiple substructures, suggesting it comprises stars formed in diverse environments. Our findings highlight the limited discriminatory power of magnesium in separating accreted and disc stars. Elements such as Ba, Al, Cu, and Sc are critical in distinguishing disc from accreted stars, while Ba, Y, Eu and Zn differentiate disc and accreted stars from the in-situ halo. This study demonstrates the potential power of combining a latent space representation of the data (PCA) with a clustering algorithm (XD) in Galactic archaeology, in providing new insights into the galaxy's assembly and evolutionary history.

astro-ph.GA↗

arXiv2024

The accreted Galaxy: An overview of TESS metal-poor accreted stars candidates

The Milky Way is a mosaic of stars from different origins. In particular, metal-poor accreted star candidates offer a unique opportunity to better understand the accretion history of the Milky Way. In this work, we aim to explore the assembly history of the Milky Way by investigating accreted stars in terms of their ages, dynamical properties, and chemical abundances. We also aim to better characterize the impact of incorporating asteroseismic information on age and chemical abundance calculations of metal-poor accreted stars for which TESS data is available. In this study, we conducted an in-depth examination of 30 metal-poor accreted star candidates, using TESS and Gaia data, as well as MIKE spectra. We find satisfactory agreement between seismic and predicted/spectroscopic surface gravity (log g) values, demonstrating the reliability of spectroscopic data from our methodology. We found that while age determination is highly dependent on the log g and asteroseismic information used, the overall chemical abundance distributions are similar for different log g. However, we found that calcium (Ca) abundances are more sensitive to the adopted log g. Our study reveals that the majority of our stars have properties compatible to those reported for the Gaia-Sausage-Enceladus, with a minority of stars that might be associated to Splash. We found an age distribution with a median of 11.3 Gyr with lower and upper uncertainties of 4.1 and 1.3 Gyr respectively when including asteroseismic information. As regarding some key chemical signatures we note that these stars are metal-poor ([Fe/H]) < -0.8), alpha-rich ([alpha]/Fe] > 0.2), copper-poor ([Cu/Fe] < 0 ) and with chemical abundances typical of accreted stars. These findings illustrate the importance of multi-dimensional analyses in unraveling the complex accretion history of the Milky Way.

astro-ph.GA↗

arXiv2024

Generation Constraint Scaling Can Mitigate Hallucination

Addressing the issue of hallucinations in large language models (LLMs) is a critical challenge. As the cognitive mechanisms of hallucination have been related to memory, here we explore hallucination for LLM that is enabled with explicit memory mechanisms. We empirically demonstrate that by simply scaling the readout vector that constrains generation in a memory-augmented LLM decoder, hallucination mitigation can be achieved in a training-free manner. Our method is geometry-inspired and outperforms a state-of-the-art LLM editing method on the task of generation of Wikipedia-like biography entries both in terms of generation quality and runtime complexity.

cs.CL↗

arXiv2024

Needle in the Haystack for Memory Based Large Language Models

Current large language models (LLMs) often perform poorly on simple fact retrieval tasks. Here we investigate if coupling a dynamically adaptable external memory to a LLM can alleviate this problem. For this purpose, we test Larimar, a recently proposed language model architecture which uses an external associative memory, on long-context recall tasks including passkey and needle-in-the-haystack tests. We demonstrate that the external memory of Larimar, which allows fast write and read of an episode of text samples, can be used at test time to handle contexts much longer than those seen during training. We further show that the latent readouts from the memory (to which long contexts are written) control the decoder towards generating correct outputs, with the memory stored off of the GPU. Compared to existing transformer-based LLM architectures for long-context recall tasks that use larger parameter counts or modified attention mechanisms, a relatively smaller size Larimar is able to maintain strong performance without any task-specific training or training on longer contexts.

cs.CL↗

arXiv2024

A Deep Dive into the Trade-Offs of Parameter-Efficient Preference Alignment Techniques

Large language models are first pre-trained on trillions of tokens and then instruction-tuned or aligned to specific preferences. While pre-training remains out of reach for most researchers due to the compute required, fine-tuning has become affordable thanks to parameter-efficient methods such as LoRA and QLoRA. Alignment is known to be sensitive to the many factors involved, including the quantity and quality of data, the alignment method, and the adapter rank. However, there has not yet been an extensive study of their effect on downstream performance. To address this gap, we conduct an in-depth investigation of the impact of popular choices for three crucial axes: (i) the alignment dataset (HH-RLHF and BeaverTails), (ii) the alignment technique (SFT and DPO), and (iii) the model (LLaMA-1, Vicuna-v1.3, Mistral-7b, and Mistral-7b-Instruct). Our extensive setup spanning over 300 experiments reveals consistent trends and unexpected findings. We observe how more informative data helps with preference alignment, cases where supervised fine-tuning outperforms preference optimization, and how aligning to a distinct preference boosts performance on downstream tasks. Through our in-depth analyses, we put forward key guidelines to help researchers perform more effective parameter-efficient LLM alignment.

cs.CL↗

arXiv2024

NeuroPrune: A Neuro-inspired Topological Sparse Training Algorithm for Large Language Models

Transformer-based Language Models have become ubiquitous in Natural Language Processing (NLP) due to their impressive performance on various tasks. However, expensive training as well as inference remains a significant impediment to their widespread applicability. While enforcing sparsity at various levels of the model architecture has found promise in addressing scaling and efficiency issues, there remains a disconnect between how sparsity affects network topology. Inspired by brain neuronal networks, we explore sparsity approaches through the lens of network topology. Specifically, we exploit mechanisms seen in biological networks, such as preferential attachment and redundant synapse pruning, and show that principled, model-agnostic sparsity approaches are performant and efficient across diverse NLP tasks, spanning both classification (such as natural language inference) and generation (summarization, machine translation), despite our sole objective not being optimizing performance. NeuroPrune is competitive with (or sometimes superior to) baselines on performance and can be up to $10$x faster in terms of training time for a given level of sparsity, simultaneously exhibiting measurable improvements in inference time in many cases.

cs.LG↗

arXiv2024

Boundary Exploration for Bayesian Optimization With Unknown Physical Constraints

Bayesian optimization has been successfully applied to optimize black-box functions where the number of evaluations is severely limited. However, in many real-world applications, it is hard or impossible to know in advance which designs are feasible due to some physical or system limitations. These issues lead to an even more challenging problem of optimizing an unknown function with unknown constraints. In this paper, we observe that in such scenarios optimal solution typically lies on the boundary between feasible and infeasible regions of the design space, making it considerably more difficult than that with interior optima. Inspired by this observation, we propose BE-CBO, a new Bayesian optimization method that efficiently explores the boundary between feasible and infeasible designs. To identify the boundary, we learn the constraints with an ensemble of neural networks that outperform the standard Gaussian Processes for capturing complex boundaries. Our method demonstrates superior performance against state-of-the-art methods through comprehensive experiments on synthetic and real-world benchmarks. Code available at: https://github.com/yunshengtian/BE-CBO

cs.LG↗

arXiv2024

EDGE -- Dark matter or astrophysics? Breaking dark matter heating degeneracies with HI rotation in faint dwarf galaxies

Low-mass dwarf galaxies are expected to reside within dark matter haloes that have a pristine, `cuspy' density profile within their stellar half-light radii. This is because they form too few stars to significantly drive dark matter heating through supernova-driven outflows. Here, we study such simulated faint systems ($10^4 \leq M_{\star} \leq 2\times 10^6 \, M_\mathrm{\odot}$) drawn from high-resolution (3 pc) cosmological simulations from the `Engineering Dwarf Galaxies at the Edge of galaxy formation' (EDGE) project. We confirm that these objects have steep and rising inner dark matter density profiles at $z=0$, little affected by galaxy formation effects. But five dwarf galaxies from the suite also showcase a detectable HI reservoir ($M_{\mathrm{HI}}\approx 10^{5}-10^{6} \, M_\mathrm{\odot}$), analogous to the observed population of faint, HI-bearing dwarf galaxies. These reservoirs exhibit episodes of ordered rotation, opening windows for rotation curve analysis. Within actively star-forming dwarfs, stellar feedback easily disrupts the tenuous HI discs ($v_ϕ \approx 10\, \mathrm{km} \, \mathrm{s}^{-1}$), making rotation short-lived ($\ll 150 \, \mathrm{Myr}$) and more challenging to interpret for dark matter inferences. In contrast, we highlight a long-lived ($\geq 500 \, \mathrm{Myr}$) and easy-to-interpret HI rotation curve extending to $\approx 2\, r_{1/2, \text{3D}}$ in a quiescent dwarf, that has not formed new stars since $z=4$. This stable gas disc is supported by an oblate dark matter halo shape that drives high-angular momentum gas flows. Our results strongly motivate further searches for HI in rotation curves in the observed population of HI-bearing low-mass dwarfs, that provide a key regime to disentangle the respective roles of dark matter microphysics and galaxy formation effects in driving dark matter heating.

astro-ph.GA↗