arXiv subjects

Gargi Ghosh

Publications and source records attributed to Gargi Ghosh.

At least 37 records · Page 2Linked to original sources

arXiv2024

Contractive Hilbert modules on quotient domains

Let the complex reflection group $G(m,p,n)$ act on the unit polydisc $\mathbb D^n$ in $\mathbb C^n.$ A $\boldsymbolΘ_n$-contraction is a commuting tuple of operators on a Hilbert space having $$\overline{\boldsymbolΘ}_n:=\{\boldsymbolθ(z)=(θ_1(z),\ldots,θ_n(z)):z\in\overline{\mathbb D}^n\}$$ as a spectral set, where $\{θ_i\}_{i=1}^n$ is a homogeneous system of parameters associated to $G(m,p,n).$ A plethora of examples of $\boldsymbolΘ_n$-contractions is exhibited. Under a mild hypothesis, it is shown that these $\boldsymbolΘ_n$-contractions are mutually unitarily inequivalent. These inequivalence results are obtained concretely for the weighted Bergman modules under the action of the permutation groups and the dihedral groups. The division problem is shown to have negative answers for the Hardy module and the Bergman module on the bidisc. A Beurling-Lax-Halmos type representation for the invariant subspaces of $\boldsymbolΘ_n$-isometries is obtained.

math.FA↗

arXiv2024

MoMa: Efficient Early-Fusion Pre-training with Mixture of Modality-Aware Experts

We introduce MoMa, a novel modality-aware mixture-of-experts (MoE) architecture designed for pre-training mixed-modal, early-fusion language models. MoMa processes images and text in arbitrary sequences by dividing expert modules into modality-specific groups. These groups exclusively process designated tokens while employing learned routing within each group to maintain semantically informed adaptivity. Our empirical results reveal substantial pre-training efficiency gains through this modality-specific parameter allocation. Under a 1-trillion-token training budget, the MoMa 1.4B model, featuring 4 text experts and 4 image experts, achieves impressive FLOPs savings: 3.7x overall, with 2.6x for text and 5.2x for image processing compared to a compute-equivalent dense baseline, measured by pre-training loss. This outperforms the standard expert-choice MoE with 8 mixed-modal experts, which achieves 3x overall FLOPs savings (3x for text, 2.8x for image). Combining MoMa with mixture-of-depths (MoD) further improves pre-training FLOPs savings to 4.2x overall (text: 3.4x, image: 5.3x), although this combination hurts performance in causal inference due to increased sensitivity to router accuracy. These results demonstrate MoMa's potential to significantly advance the efficiency of mixed-modal, early-fusion language model pre-training, paving the way for more resource-efficient and capable multimodal AI systems.

cs.AI↗

arXiv2024

Holomorphic retracts in the Lie ball and the tetrablock

In this article, we study various properties of holomorphic retracts in Lempert domains. We associate the existence and the related form of holomorphic retracts with the linear ones, provide non-trivial examples and discuss their properties in a quite general setting. Later we specialize on two Lempert domains which are the Lie ball of dimension three and its $2$-proper holomorphic image, that is, the tetrablock and give a complete description of holomorphic retracts in these domains.

math.CV↗

arXiv2024

Text Quality-Based Pruning for Efficient Training of Language Models

In recent times training Language Models (LMs) have relied on computationally heavy training over massive datasets which makes this training process extremely laborious. In this paper we propose a novel method for numerically evaluating text quality in large unlabelled NLP datasets in a model agnostic manner to assign the text instances a "quality score". By proposing the text quality metric, the paper establishes a framework to identify and eliminate low-quality text instances, leading to improved training efficiency for LM models. Experimental results over multiple models and datasets demonstrate the efficacy of this approach, showcasing substantial gains in training effectiveness and highlighting the potential for resource-efficient LM training. For example, we observe an absolute accuracy improvement of 0.9% averaged over 14 downstream evaluation tasks for multiple LM models while using 40% lesser data and training 42% faster when training on the OpenWebText dataset and 0.8% average absolute accuracy improvement while using 20% lesser data and training 21% faster on the Wikipedia dataset.

cs.CL↗

arXiv2023

Toeplitz operators on the weighted Bergman spaces of quotient domains

Let $G$ be a finite pseudoreflection group and $Ω\subseteq \mathbb C^d$ be a bounded domain which is a $G$-space. We establish identities involving Toeplitz operators on the weighted Bergman spaces of $Ω$ and $Ω/G$ using invariant theory and representation theory of $G.$ This, in turn, provides techniques to study algebraic properties of Toeplitz operators on the weighted Bergman space on $Ω/G.$ We specialize on the generalized zero-product problem and characterization of commuting pairs of Toeplitz operators. As a consequence, more intricate results on Toeplitz operators on the weighted Bergman spaces on some specific quotient domains (namely symmetrized polydisc, monomial polyhedron, Rudin's domain) have been obtained.

math.CV↗

arXiv2023

Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuning

We present CM3Leon (pronounced "Chameleon"), a retrieval-augmented, token-based, decoder-only multi-modal language model capable of generating and infilling both text and images. CM3Leon uses the CM3 multi-modal architecture but additionally shows the extreme benefits of scaling up and tuning on more diverse instruction-style data. It is the first multi-modal model trained with a recipe adapted from text-only language models, including a large-scale retrieval-augmented pre-training stage and a second multi-task supervised fine-tuning (SFT) stage. It is also a general-purpose model that can do both text-to-image and image-to-text generation, allowing us to introduce self-contained contrastive decoding methods that produce high-quality outputs. Extensive experiments demonstrate that this recipe is highly effective for multi-modal models. CM3Leon achieves state-of-the-art performance in text-to-image generation with 5x less training compute than comparable methods (zero-shot MS-COCO FID of 4.88). After SFT, CM3Leon can also demonstrate unprecedented levels of controllability in tasks ranging from language-guided image editing to image-controlled generation and segmentation.

cs.LG↗

arXiv2023

MAViL: Masked Audio-Video Learners

We present Masked Audio-Video Learners (MAViL) to train audio-visual representations. Our approach learns with three complementary forms of self-supervision: (1) reconstruction of masked audio and video input data, (2) intra- and inter-modal contrastive learning with masking, and (3) self-training by reconstructing joint audio-video contextualized features learned from the first two objectives. Pre-training with MAViL not only enables the model to perform well in audio-visual classification and retrieval tasks but also improves representations of each modality in isolation, without using information from the other modality for fine-tuning or inference. Empirically, MAViL sets a new state-of-the-art on AudioSet (53.1 mAP) and VGGSound (67.1% accuracy). For the first time, a self-supervised audio-visual model outperforms ones that use external supervision on these benchmarks.

cs.CV↗

arXiv2023

ALERT: Adapting Language Models to Reasoning Tasks

Current large language models can perform reasonably well on complex tasks that require step-by-step reasoning with few-shot learning. Are these models applying reasoning skills they have learnt during pre-training and reason outside of their training context, or are they simply memorizing their training corpus at finer granularity and have learnt to better understand their context? To tease apart these possibilities, we introduce ALERT, a benchmark and suite of analyses for assessing language models' reasoning ability comparing pre-trained and finetuned models on complex tasks that require reasoning skills to solve. ALERT provides a test bed to asses any language model on fine-grained reasoning skills, which spans over 20 datasets and covers 10 different reasoning skills. We leverage ALERT to further investigate the role of finetuning. With extensive empirical analysis we find that language models learn more reasoning skills such as textual entailment, abductive reasoning, and analogical reasoning during finetuning stage compared to pretraining state. We also find that when language models are finetuned they tend to overfit to the prompt template, which hurts the robustness of models causing generalization problems.

cs.CL↗

arXiv2023

LIMA: Less Is More for Alignment

Large language models are trained in two stages: (1) unsupervised pretraining from raw text, to learn general-purpose representations, and (2) large scale instruction tuning and reinforcement learning, to better align to end tasks and user preferences. We measure the relative importance of these two stages by training LIMA, a 65B parameter LLaMa language model fine-tuned with the standard supervised loss on only 1,000 carefully curated prompts and responses, without any reinforcement learning or human preference modeling. LIMA demonstrates remarkably strong performance, learning to follow specific response formats from only a handful of examples in the training data, including complex queries that range from planning trip itineraries to speculating about alternate history. Moreover, the model tends to generalize well to unseen tasks that did not appear in the training data. In a controlled human study, responses from LIMA are either equivalent or strictly preferred to GPT-4 in 43% of cases; this statistic is as high as 58% when compared to Bard and 65% versus DaVinci003, which was trained with human feedback. Taken together, these results strongly suggest that almost all knowledge in large language models is learned during pretraining, and only limited instruction tuning data is necessary to teach models to produce high quality output.

cs.CL↗

arXiv2023

On the spectra of Commuting Graphs

We describe the complete spectra of Laplacian, signless Laplacian, and adjacency matrices associated with the commuting graphs of a finite group using group theoretic information. We provide a method to find the center of a group by only using the Laplacian of the commuting graph of the group. The graph invariants (such as diameter, clique number, and mean distance) of the commuting graph associated with a finite group are determined. We produce a number of examples to illustrate the applications of our results.

math.GM↗

arXiv2023

CiT: Curation in Training for Effective Vision-Language Data

Large vision-language models are generally applicable to many downstream tasks, but come at an exorbitant training cost that only large institutions can afford. This paper trades generality for efficiency and presents Curation in Training (CiT), a simple and efficient vision-text learning algorithm that couples a data objective into training. CiT automatically yields quality data to speed-up contrastive image-text training and alleviates the need for an offline data filtering pipeline, allowing broad data sources (including raw image-text pairs from the web). CiT contains two loops: an outer loop curating the training data and an inner loop consuming the curated training data. The text encoder connects the two loops. Given metadata for tasks of interest, e.g., class names, and a large pool of image-text pairs, CiT alternatively selects relevant training data from the pool by measuring the similarity of their text embeddings and embeddings of the metadata. In our experiments, we observe that CiT can speed up training by over an order of magnitude, especially if the raw data size is large.

cs.CV↗

arXiv2022

Toeplitz operators on the Hardy spaces of quotient domains

Let $Ω$ be either the unit polydisc $\mathbb D^d$ or the unit ball $\mathbb B_d$ in $\mathbb C^d$ and $G$ be a finite pseudoreflection group which acts on $Ω.$ Associated to each one-dimensional representation $\varrho$ of $G,$ we provide a notion of the (weighted) Hardy space $H^2_\varrho(Ω/G)$ on $Ω/G.$ Subsequently, we show that each $H^2_\varrho(Ω/G)$ is isometrically isomorphic to the relative invariant subspace of $H^2(Ω)$ associated to the representation $\varrho.$ For $Ω=\mathbb D^d,$ $G=\mathfrak{S}_d,$ the permutation group on $d$ symbols and $\varrho = $ the sign representation of $\mathfrak{S}_d,$ the Hardy space $H^2_\varrho(Ω/G)$ coincides to well-known notion of the Hardy space on the symmetrized polydisc. We largely use invariant theory of the group $G$ to establish identities involving Toeplitz operators on $H^2(Ω)$ and $H^2_\varrho(Ω/G)$ which enable us to study algebraic properties (such as generalized zero product problem, characterization of commuting Toeplitz operators, compactness etc.) of Toeplitz operators on $H^2_\varrho(Ω/G).$

math.CV↗

arXiv2022

On analytic structure of weighted shifts on generalized directed semi-trees

Inspired by natural classes of examples, we define generalized directed semi-tree and construct weighted shifts on the generalized directed semi-trees. Given an $n$-tuple of directed directed semi-trees with certain properties, we associate an $n$-tuple of multiplication operators on a Hilbert space $\mathscr{H}^2(β)$ of formal power series. Under certain conditions, $\mathscr{H}^2(β)$ turns out to be a reproducing kernel Hilbert space consisting of holomorphic functions on some domain in $\mathbb C^n$ and the $n$-tuple of multiplication operators on $\mathscr{H}^2(β)$ is unitarily equivalent to an $n$-tuple of weighted shifts on the generalized directed semi-trees. Finally, we exhibit two classes of examples of $n$-tuple of operators which can be intrinsically identified as weighted shifts on generalized directed semi-trees.

math.FA↗

arXiv2022

CM3: A Causal Masked Multimodal Model of the Internet

We introduce CM3, a family of causally masked generative models trained over a large corpus of structured multi-modal documents that can contain both text and image tokens. Our new causally masked approach generates tokens left to right while also masking out a small number of long token spans that are generated at the end of the string, instead of their original positions. The casual masking object provides a type of hybrid of the more common causal and masked language models, by enabling full generative modeling while also providing bidirectional context when generating the masked spans. We train causally masked language-image models on large-scale web and Wikipedia articles, where each document contains all of the text, hypertext markup, hyperlinks, and image tokens (from a VQVAE-GAN), provided in the order they appear in the original HTML source (before masking). The resulting CM3 models can generate rich structured, multi-modal outputs while conditioning on arbitrary masked document contexts, and thereby implicitly learn a wide range of text, image, and cross modal tasks. They can be prompted to recover, in a zero-shot fashion, the functionality of models such as DALL-E, GENRE, and HTLM. We set the new state-of-the-art in zero-shot summarization, entity linking, and entity disambiguation while maintaining competitive performance in the fine-tuning setting. We can generate images unconditionally, conditioned on text (like DALL-E) and do captioning all in a zero-shot setting with a single model.

cs.CL↗

arXiv2021

VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding

We present VideoCLIP, a contrastive approach to pre-train a unified model for zero-shot video and text understanding, without using any labels on downstream tasks. VideoCLIP trains a transformer for video and text by contrasting temporally overlapping positive video-text pairs with hard negatives from nearest neighbor retrieval. Our experiments on a diverse series of downstream tasks, including sequence-level text-video retrieval, VideoQA, token-level action localization, and action segmentation reveal state-of-the-art performance, surpassing prior work, and in some cases even outperforming supervised approaches. Code is made available at https://github.com/pytorch/fairseq/tree/main/examples/MMPT.

cs.CV↗

arXiv2021

VLM: Task-agnostic Video-Language Model Pre-training for Video Understanding

We present a simplified, task-agnostic multi-modal pre-training approach that can accept either video or text input, or both for a variety of end tasks. Existing pre-training are task-specific by adopting either a single cross-modal encoder that requires both modalities, limiting their use for retrieval-style end tasks or more complex multitask learning with two unimodal encoders, limiting early cross-modal fusion. We instead introduce new pretraining masking schemes that better mix across modalities (e.g. by forcing masks for text to predict the closest video embeddings) while also maintaining separability (e.g. unimodal predictions are sometimes required, without using all the input). Experimental results show strong performance across a wider range of tasks than any previous methods, often outperforming task-specific pre-training. Code is made available at https://github.com/pytorch/fairseq/tree/main/examples/MMPT.

cs.CV↗

arXiv2021

HTLM: Hyper-Text Pre-Training and Prompting of Language Models

We introduce HTLM, a hyper-text language model trained on a large-scale web crawl. Modeling hyper-text has a number of advantages: (1) it is easily gathered at scale, (2) it provides rich document-level and end-task-adjacent supervision (e.g. class and id attributes often encode document category information), and (3) it allows for new structured prompting that follows the established semantics of HTML (e.g. to do zero-shot summarization by infilling title tags for a webpage that contains the input text). We show that pretraining with a BART-style denoising loss directly on simplified HTML provides highly effective transfer for a wide range of end tasks and supervision levels. HTLM matches or exceeds the performance of comparably sized text-only LMs for zero-shot prompting and fine-tuning for classification benchmarks, while also setting new state-of-the-art performance levels for zero-shot summarization. We also find that hyper-text prompts provide more value to HTLM, in terms of data efficiency, than plain text prompts do for existing LMs, and that HTLM is highly effective at auto-prompting itself, by simply generating the most likely hyper-text formatting for any available training data. We will release all code and models to support future HTLM research.

cs.CL↗

arXiv2021

Multi-task Retrieval for Knowledge-Intensive Tasks

Retrieving relevant contexts from a large corpus is a crucial step for tasks such as open-domain question answering and fact checking. Although neural retrieval outperforms traditional methods like tf-idf and BM25, its performance degrades considerably when applied to out-of-domain data. Driven by the question of whether a neural retrieval model can be universal and perform robustly on a wide variety of problems, we propose a multi-task trained model. Our approach not only outperforms previous methods in the few-shot setting, but also rivals specialised neural retrievers, even when in-domain training data is abundant. With the help of our retriever, we improve existing models for downstream tasks and closely match or improve the state of the art on multiple benchmarks.

cs.CL↗