Searcharxiv⌕ Search

arXiv subjects

Achintya Kundu

Publications and source records attributed to Achintya Kundu.

13 recordsLinked to original sources
arXiv2024

Enhancing Training Efficiency Using Packing with Flash Attention

Padding is often used in tuning LLM models by adding special tokens to shorter training examples to match the length of the longest sequence in each batch. While this ensures uniformity for batch processing, it introduces inefficiencies by including irrelevant padding tokens in the computation and wastes GPU resources. Hugging Face SFT trainer has always offered the option to use packing to combine multiple training examples, allowing for maximal utilization of GPU resources. However, up till now, it did not offer proper masking of each packed training example. This capability has been added to Hugging Face Transformers 4.44. We analyse this new feature and show the benefits across different variations of packing.

cs.LG↗
arXiv2024

Efficiently Distilling LLMs for Edge Applications

Supernet training of LLMs is of great interest in industrial applications as it confers the ability to produce a palette of smaller models at constant cost, regardless of the number of models (of different size / latency) produced. We propose a new method called Multistage Low-rank Fine-tuning of Super-transformers (MLFS) for parameter-efficient supernet training. We show that it is possible to obtain high-quality encoder models that are suitable for commercial edge applications, and that while decoder-only models are resistant to a comparable degree of compression, decoders can be effectively sliced for a significant reduction in training time.

cs.LG↗
arXiv2023

Transfer-Once-For-All: AI Model Optimization for Edge

Weight-sharing neural architecture search aims to optimize a configurable neural network model (supernet) for a variety of deployment scenarios across many devices with different resource constraints. Existing approaches use evolutionary search to extract models of different sizes from a supernet trained on a very large data set, and then fine-tune the extracted models on the typically small, real-world data set of interest. The computational cost of training thus grows linearly with the number of different model deployment scenarios. Hence, we propose Transfer-Once-For-All (TOFA) for supernet-style training on small data sets with constant computational training cost over any number of edge deployment scenarios. Given a task, TOFA obtains custom neural networks, both the topology and the weights, optimized for any number of edge deployment scenarios. To overcome the challenges arising from small data, TOFA utilizes a unified semi-supervised training loss to simultaneously train all subnets within the supernet, coupled with on-the-fly architecture selection at deployment time.

cs.LG↗
arXiv2022

Robustness and Personalization in Federated Learning: A Unified Approach via Regularization

We present a class of methods for robust, personalized federated learning, called Fed+, that unifies many federated learning algorithms. The principal advantage of this class of methods is to better accommodate the real-world characteristics found in federated training, such as the lack of IID data across parties, the need for robustness to outliers or stragglers, and the requirement to perform well on party-specific datasets. We achieve this through a problem formulation that allows the central server to employ robust ways of aggregating the local models while keeping the structure of local computation intact. Without making any statistical assumption on the degree of heterogeneity of local data across parties, we provide convergence guarantees for Fed+ for convex and non-convex loss functions under different (robust) aggregation methods. The Fed+ theory is also equipped to handle heterogeneous computing environments including stragglers without additional assumptions; specifically, the convergence results cover the general setting where the number of local update steps across parties can vary. We demonstrate the benefits of Fed+ through extensive experiments across standard benchmark datasets.

cs.LG↗
arXiv2021

Magnesium Contact Ions Stabilize the Tertiary Structure of Transfer RNA: Electrostatics Mapped by Two-Dimensional Infrared Spectra and Theoretical Simulations

Ions interacting with hydrated RNA play a central role in defining its secondary and tertiary structure. While spatial arrangements of ions, water molecules, and phosphate groups have been inferred from X-ray studies, the role of electrostatic and other noncovalent interactions in stabilizing compact folded RNA structures is not fully understood at the molecular level. Here, we demonstrate that contact ion pairs of magnesium (Mg2+) and phosphate groups embedded in local water shells stabilize the tertiary equilibrium structure of transfer RNA (tRNA). Employing dialyzed tRNAPhe from yeast and tRNA from Escherichia coli, we follow the population of Mg2+ sites close to phosphate groups of the ribose-phosphodiester backbone step by step, combining linear and nonlinear infrared spectroscopy of phosphate vibrations with molecular dynamics simulations and ab initio vibrational frequency calculations. The formation of up to six Mg2+/phosphate contact pairs per tRNA and local field-induced reorientations of water molecules balance the phosphate-phosphate repulsion in nonhelical parts of tRNA, thus stabilizing the folded structure electrostatically. Such geometries display limited sub-picosecond fluctuations in the arrangement of water molecules and ion residence times longer than 1 μs. At higher Mg2+ excess, the number of contact ion pairs per tRNA saturates around 6 and weakly interacting ions prevail. Our results suggest a predominance of contact ion pairs over long-range coupling of the ion atmosphere and the biomolecule in defining and stabilizing the tertiary structure of tRNA.

physics.chem-ph↗
arXiv2021

Phosphate Vibrations Probe Electric Fields in Hydrated Biomolecules: Spectroscopy, Dynamics, and Interactions

Electric interactions have a strong impact on the structure and dynamics of biomolecules in their native water environment. Given the variety of water arrangements in hydration shells and the femto- to subnanosecond time range of structural fluctuations, there is a strong quest for sensitive noninvasive probes of local electric fields. The stretching vibrations of phosphate groups, in particular the asymmetric (PO2)- stretching vibration νAS(PO2)-, allow for a quantitative mapping of dynamic electric fields in aqueous environments via a field-induced redshift of their transition frequencies and concomitant changes of vibrational line shapes. We present a systematic study of νAS(PO2)- excitations in molecular systems of increasing complexity, including dimethyl phosphate (DMP), short DNA and RNA duplex structures, and transfer RNA (tRNA) in water. A combination of linear infrared absorption, two-dimensional infrared (2D-IR) spectroscopy, and molecular dynamics (MD) simulations gives quantitative insight in electric-field tuning rates of vibrational frequencies, electric field and fluctuation amplitudes, and molecular interaction geometries. Beyond neat water environments, the formation of contact ion pairs of phosphate groups with Mg2+ ions is demonstrated via frequency upshifts of the νAS(PO2)- vibration, resulting in a distinct vibrational band. The frequency positions of contact geometries are determined by an interplay of attractive electric and repulsive exchange interactions.

physics.chem-ph↗
arXiv2020

Aqueous Contact Ion Pairs of Phosphate Groups with Na$^+$, Ca$^{2+}$ and Mg$^{2+}$ -- Structural Discrimination by Femtosecond Infrared Spectroscopy and Molecular Dynamics Simulations

The extent of contact and solvent shared ion pairs of phosphate groups with Na$^+$, Ca$^{2+}$ and Mg$^{2+}$ ions in aqueous environment and their relevance for the stability of polyanionic DNA and RNA structures is highly debated. Employing the asymmetric phosphate stretching vibration of dimethyl phosphate (DMP), a model system of the sugar-phosphate backbone of DNA and RNA, we present linear infrared, femtosecond infrared pump-probe and absorptive 2D-IR spectra that report on contact ion pair formation via the presence of blue shifted spectral signatures. Compared to the linear infrared spectra, the nonlinear spectra reveal contact ion pairs with increased sensitivity because the spectra accentuate differences in peak frequency, transition dipole moment strength, and excited state lifetime. The experimental results are corroborated by long time scale MD simulations, benchmarked by density functional simulations on phosphate-ion-water clusters. The microscopic interpretation reveals subtle structural differences of ion pairs formed by the phosphate group and the ions Na$^+$, Ca$^{2+}$ and Mg$^{2+}$. Intricate properties of the solvation shell around the phosphate group and the ion are essential to explain the experimental observations. The present work addresses a challenging to probe topic with the help of a model system and establishes new experimental data of contact ion pair formation, thereby underlining the potential of nonlinear 2D-IR spectroscopy as an analytical probe of phosphate-ion interactions in complex biological systems.

physics.chem-ph↗
arXiv2020

Change of Hydration Patterns upon RNA Melting Probed by Excitations of Phosphate Backbone Vibrations

The water hydration shell has decisive impact on the structural and functional properties of RNA. Changes of RNA structure upon melting and in biochemical processes are accompanied by a change of hydration patterns, a process which is barely characterized. To discern hydration geometries around the backbone phosphate groups of an RNA double helix at the molecular level, we combine two-dimensional infrared spectroscopy of phosphate vibrations with theoretical simulations. There are three distinct coexisting hydration motifs of the RNA A-helix, an ordered chain-like arrangement of water molecules with links between neighboring phosphate groups, separate local hydration shells of up to six water molecules, and hydrated phosphate/counterion contact pairs. RNA disordering upon melting is connected with a transition from predominant ordered water structures to local hydration shells around phosphate units. Structural fluctuations are dominated by librational water motions occurring on a 300 fs time scale, without exchange between hydration motifs.

physics.chem-ph↗
arXiv2019

Contact Ion Pairs of Phosphate Groups in Water -- Two-Dimensional Infrared Spectroscopy of Dimethyl-Phosphate and Ab-Initio Simulations

The interaction of phosphate groups with ions in an aqueous environment has a strong impact on the structure and folding processes of DNA and RNA. The dynamic variety of ionic arrangements, including both contact pairs and water separated ions, and the molecular coupling mechanisms are far from being understood. In a combined experimental and theoretical approach, we address the properties of contact ion pairs of the prototypical system dimethyl-phosphate with Na$^+$, Ca$^{2+}$, and Mg$^{2+}$ ions in water. Linear and femtosecond two-dimensional infrared (2D-IR) spectroscopy of the asymmetric (PO$_2$)$^-$ stretching vibration separates and characterizes the different species via their blue-shifted vibrational signatures and 2D-IR lineshapes. Phosphate-magnesium contact pairs stand out as the most compact geometry while the contact pairs with Ca$^{2+}$ and Na$^+$ display a wider structural variation. Microscopic density functional theory simulations rationalize the observed frequency shifts and reveal distinct differences between the contact geometries.

physics.chem-ph↗
arXiv2019

Hydrated Excess Protons in Acetonitrile/Water Mixtures - Solvation Species and Ultrafast Proton Motions

The solvation structure of protons in aqueous media is highly relevant to electric properties and to proton transport in liquids and membranes. At ambient temperature, polar liquids display structural fluctuations on femto- to picosecond time scales with a direct impact on proton solvation. We apply two-dimensional infrared (2D-IR) spectroscopy for following proton dynamics in acetonitrile/water mixtures with the Zundel cation H$_5$O$_2^+$ prepared in neat acetonitrile as a benchmark. The 2D-IR spectra of the proton transfer mode of H$_5$O$_2^+$ demonstrate stochastic large-amplitude motions in the double-minimum proton potential, driven by fluctuating electric fields. In all cases the excess proton is embedded in a water dimer, forming an H$_5$O$_2^+$ complex as major solvation species. This observation is rationalized by quantum mechanics/molecular mechanics molecular dynamics simulations including up to 4 water molecules embedded in acetonitrile. The Zundel motif interacts with its closest water neighbor in an H$_7$O$_3^+$ unit without persistent proton localization.

physics.chem-ph↗
arXiv2017

Convex optimization over intersection of simple sets: improved convergence rate guarantees via an exact penalty approach

We consider the problem of minimizing a convex function over the intersection of finitely many simple sets which are easy to project onto. This is an important problem arising in various domains such as machine learning. The main difficulty lies in finding the projection of a point in the intersection of many sets. Existing approaches yield an infeasible point with an iteration-complexity of $O(1/\varepsilon^2)$ for nonsmooth problems with no guarantees on the in-feasibility. By reformulating the problem through exact penalty functions, we derive first-order algorithms which not only guarantees that the distance to the intersection is small but also improve the complexity to $O(1/\varepsilon)$ and $O(1/\sqrt{\varepsilon})$ for smooth functions. For composite and smooth problems, this is achieved through a saddle-point reformulation where the proximal operators required by the primal-dual algorithms can be computed in closed form. We illustrate the benefits of our approach on a graph transduction problem and on graph matching.

math.OC↗