arXiv subjects

Darshan Singh

Publications and source records attributed to Darshan Singh.

7 recordsLinked to original sources

arXiv2026

MINERVA-Cultural: A Benchmark for Cultural and Multilingual Long Video Reasoning

Recent advancements in video models have shown tremendous progress, particularly in long video understanding. However, current benchmarks predominantly feature western-centric data and English as the dominant language, introducing significant biases in evaluation. To address this, we introduce MINERVA-Cultural, a challenging benchmark for multicultural and multilingual video reasoning. MINERVA-Cultural comprises high-quality, entirely human-generated annotations from diverse, region-specific cultural videos across 18 global locales. Unlike prior work that relies on automatic translations, MINERVA-Cultural provides complex questions, answers, and multi-step reasoning steps, all crafted in native languages. Making progress on MINERVA-Cultural requires a deeply situated understanding of visual cultural context. Furthermore, we leverage MINERVA-Cultural's reasoning traces to construct evidence-based graphs and propose a novel iterative strategy using these graphs to identify fine-grained errors in reasoning. Our evaluations reveal that SoTA Video-LLMs struggle significantly, performing substantially below human-level accuracy, with errors primarily stemming from the visual perception of cultural elements. MINERVA-Cultural will be publicly available under https://github.com/google-deepmind/neptune?tab=readme-ov-file\#minerva-cultural

cs.CV↗

arXiv2025

Rethinking Cross-lingual Gaps from a Statistical Viewpoint

Any piece of knowledge is usually expressed in one or a handful of natural languages on the web or in any large corpus. Large Language Models (LLMs) act as a bridge by acquiring knowledge from a source language and making it accessible when queried using target languages. A cross-lingual gap is a drop in accuracy incurred when querying knowledge in a target language rather than the source language. Existing research focused on modeling or training failures leading to cross-lingual gaps. In this work, we take an alternative view to characterize the nature of cross-lingual error, and hypothesize that the variance of responses in the target language is a key cause of this gap. For the first time, we formalize the cross-lingual gap in terms of biased and unbiased errors. We empirically validate our hypothesis through multiple inference-time interventions that control variance and reduce the cross-lingual gap. We demonstrate a few test-time ensemble methods that reduce response variance, and thereby improve source-target transfer scores by up to 12 absolute points yielding relative gains of 8% to over 50% across various LLMs.

cs.CL↗

arXiv2024

No Detail Left Behind: Revisiting Self-Retrieval for Fine-Grained Image Captioning

Image captioning systems are unable to generate fine-grained captions as they are trained on data that is either noisy (alt-text) or generic (human annotations). This is further exacerbated by maximum likelihood training that encourages generation of frequently occurring phrases. Previous works have tried to address this limitation by fine-tuning captioners with a self-retrieval (SR) reward. However, we find that SR fine-tuning has a tendency to reduce caption faithfulness and even hallucinate. In this work, we circumvent this bottleneck by improving the MLE initialization of the captioning system and designing a curriculum for the SR fine-tuning process. To this extent, we present (1) Visual Caption Boosting, a novel framework to instill fine-grainedness in generic image captioning datasets while remaining anchored in human annotations; and (2) BagCurri, a carefully designed training curriculum that more optimally leverages the contrastive nature of the self-retrieval reward. Jointly, they enable the captioner to describe fine-grained aspects in the image while preserving faithfulness to ground-truth captions. Our approach outperforms previous work by +8.9% on SR against 99 random distractors (RD100) (Dessi et al., 2023); and +7.6% on ImageCoDe. Additionally, existing metrics to evaluate captioning systems fail to reward diversity or evaluate a model's fine-grained understanding ability. Our third contribution addresses this by proposing self-retrieval from the lens of evaluation. We introduce TrueMatch, a benchmark comprising bags of highly similar images that uses SR to assess the captioner's ability to capture subtle visual distinctions. We evaluate and compare several state-of-the-art open-source MLLMs on TrueMatch, and find that our SR approach outperforms them all by a significant margin (e.g. +4.8% - 7.1% over Cambrian) while having 1-2 orders of magnitude fewer parameters.

cs.CV↗

arXiv2024

Does the Amati Correlation Exhibit Redshift-Driven Heterogeneity in Long GRBs?

Long gamma-ray bursts (GRBs) offer significant insights into cosmology due to their high energy emissions and the potential to probe the early universe. The Amati relation, which links the intrinsic peak energy to the isotropic energy, is crucial for understanding their cosmological applications. This study investigates the redshift-driven heterogeneity of the Amati correlation in long GRBs. Analyzing 221 long GRBs with redshifts from 0.034 to 8.2 we divided the dataset based on redshift thresholds of 1.5 and 2. Using Bayesian marginalization and Reichart's likelihood approach, we found significant differences in the Amati parameters between low and high redshift subgroups. These variations, differing by approximately $2\sigma$ at $z = 1.5$ and more than $1\sigma$ at $z = 2$, suggest an evolution in the GRB population with redshift, possibly reflecting changes in host galaxy properties. However, selection effects and instrumental biases may also contribute. Our results challenge the assumption of the Amati relation's universality and underscore the need for larger datasets and more precise measurements from upcoming missions like Transient High-Energy Sky and Early Universe Surveyor (THESEUS) and enhanced X-ray Timing and Polarimetry mission (eXTP) to refine our understanding of GRB physics.

astro-ph.CO↗

arXiv2024

VELOCITI: Benchmarking Video-Language Compositional Reasoning with Strict Entailment

A fundamental aspect of compositional reasoning in a video is associating people and their actions across time. Recent years have seen great progress in general-purpose vision or video models and a move towards long-video understanding. While exciting, we take a step back and ask: are current models good at compositional reasoning on short videos? To this end, we introduce VELOCITI, a benchmark to study Video-LLMs by disentangling and assessing the comprehension of agents, actions, and their associations across multiple events. We adopt the Video-Language Entailment setup and propose StrictVLE that requires correct classification (rather than ranking) of the positive and negative caption. We evaluate several models and observe that even the best, LLaVA-OneVision (44.5%) and Gemini-1.5-Pro (49.3%), are far from human accuracy at 93.0%. Results show that action understanding lags behind agents, and negative captions created using entities appearing in the video perform worse than those obtained from pure text manipulation. We also present challenges with ClassicVLE and multiple-choice (MC) evaluation, strengthening our preference for StrictVLE. Finally, we validate that our benchmark requires visual inputs of multiple frames making it ideal to study video-language compositional reasoning.

cs.CV↗

arXiv2024

SRL-CLIP: Efficient CLIP Video Adaptation via Structured Semantic Role Labels

Adapting CLIP for videos has gained popularity due to its semantic and rich representation. While CLIP is a good starting point, it typically undergoes post-pretraining (contrastive finetuning) on large video narration or caption datasets (e.g. HowTo100M, WebVid2.5M). However, such narrations or captions often lack comprehensive information needed to represent a video holistically. As the learning signal from text is sparse, the visual learning is inefficient and adaptation requires millions of samples to post-pretrain. In this work, we ask: is it possible to efficiently adapt CLIP for general and holistic video understanding? We use videos labeled with structured and dense Semantic Role Labels (SRLs) that capture actions, people or objects, their attributes, adverbs (manner), and location in a structured format representing the entire video in a holistic way. We generate rule-based captions from SRLs and demonstrate that simple contrastive finetuning on a mere 23k video-caption pairs is adequate to learn powerful, transferable representations applicable across a diverse range of video understanding tasks that require varying levels of perceptual granularity. Our adapted CLIP model, SRL-CLIP, exhibits comparable or superior performance on zero-shot text-to-video retrieval compared to state-of-the-art models that possess 4-8x more parameters and are post-pretrained on up to 6000x more data. SRL-CLIP surpasses CLIP on multiple video benchmarks, underscoring the efficient learning and improved representations.

cs.CV↗

arXiv2022

Investigating the Evolution of Amati Parameters with Redshift

Gamma Ray Bursts (GRB) are among the brightest objects in the Universe and hence can be observed up to a very high redshift. Properly calibrated empirical correlations between intensity and spectral correlations of GRBs can be used to estimate the cosmological parameters. However, the possibility of the evolution of GRBs with the redshift is a long-standing puzzle. In this work, we used 162 long-duration GRBs to determine whether GRBs below and above a certain redshift have different properties. The GRBs are split into two groups, and we fit the Amati relation for each group separately. Our findings demonstrate that estimations of the Amati parameters for the two groups are substantially dissimilar. We perform simulations to investigate whether the selection effects could cause the difference. Our analysis shows that the differences may be intrinsic, and the selection effects are not their true origin.

astro-ph.CO↗