arXiv subjects

Saptarshi Sinha

Publications and source records attributed to Saptarshi Sinha.

13 recordsLinked to original sources

arXiv2026

Whareformer: Learning to Track What is Where in Long Egocentric Videos

The recently established 'Out of Sight, Not out of Mind' (OSNOM) task for egocentric videos focuses on tracking objects that are moved by the camera wearer, online, maintaining knowledge of instance locations throughout the video even when they leave the field of view or become heavily occluded. In this paper, we propose the first learning-based solution to the OSNOM task: Whareformer, a transformer-based model with two components: an updatable memory of established tracks and a track assignment module that associates observations with existing tracks in a feed-forward manner. Whareformer jointly reasons over evolving object appearance (what) and updated 3D location (where), and employs a dedicated New Track token to reason about novel objects. Thanks to its design choices of using relative distances and evolving track representations, Whareformer is trained on a small set of 56 videos but achieves SOTA performance on 260 long test videos from three datasets: EPIC-KITCHENS-100 (unseen videos), IT3DEgo, and HD-EPIC, with significant absolute improvements over prior work.

cs.CV↗

arXiv2026

Prime and Reach: Synthesising Body Motion for Gaze-Primed Object Reach

Human motion generation is a challenging task that aims to create realistic motion imitating natural human behaviour. We focus on the well-studied behaviour of priming an object/location for pick up or put down - that is, the spotting of an object/location from a distance, known as gaze priming, followed by the motion of approaching and reaching the target location. To that end, we curate, for the first time, 23.7K gaze-primed human motion sequences for reaching target object locations from five publicly available datasets, i.e., HD-EPIC, MoGaze, HOT3D, ADT, and GIMO. We pre-train a text-conditioned diffusion-based motion generation model, then fine-tune it conditioned on goal pose or location, on our curated sequences. Importantly, we evaluate the ability of the generated motion to imitate natural human movement through several metrics, including the 'Reach Success' and a newly introduced 'Prime Success' metric. Tested on 5 datasets, our model generates diverse full-body motion, exhibiting both priming and reaching behaviour, and outperforming baselines and recent methods.

cs.CV↗

arXiv2025

HD-EPIC: A Highly-Detailed Egocentric Video Dataset

We present a validation dataset of newly-collected kitchen-based egocentric videos, manually annotated with highly detailed and interconnected ground-truth labels covering: recipe steps, fine-grained actions, ingredients with nutritional values, moving objects, and audio annotations. Importantly, all annotations are grounded in 3D through digital twinning of the scene, fixtures, object locations, and primed with gaze. Footage is collected from unscripted recordings in diverse home environments, making HDEPIC the first dataset collected in-the-wild but with detailed annotations matching those in controlled lab environments. We show the potential of our highly-detailed annotations through a challenging VQA benchmark of 26K questions assessing the capability to recognise recipes, ingredients, nutrition, fine-grained actions, 3D perception, object motion, and gaze direction. The powerful long-context Gemini Pro only achieves 38.5% on this benchmark, showcasing its difficulty and highlighting shortcomings in current VLMs. We additionally assess action recognition, sound recognition, and long-term video-object segmentation on HD-EPIC. HD-EPIC is 41 hours of video in 9 kitchens with digital twins of 413 kitchen fixtures, capturing 69 recipes, 59K fine-grained actions, 51K audio events, 20K object movements and 37K object masks lifted to 3D. On average, we have 263 annotations per minute of our unscripted videos.

cs.CV↗

arXiv2024

Every Shot Counts: Using Exemplars for Repetition Counting in Videos

Video repetition counting infers the number of repetitions of recurring actions or motion within a video. We propose an exemplar-based approach that discovers visual correspondence of video exemplars across repetitions within target videos. Our proposed Every Shot Counts (ESCounts) model is an attention-based encoder-decoder that encodes videos of varying lengths alongside exemplars from the same and different videos. In training, ESCounts regresses locations of high correspondence to the exemplars within the video. In tandem, our method learns a latent that encodes representations of general repetitive motions, which we use for exemplar-free, zero-shot inference. Extensive experiments over commonly used datasets (RepCount, Countix, and UCFRep) showcase ESCounts obtaining state-of-the-art performance across all three datasets. Detailed ablations further demonstrate the effectiveness of our method.

cs.CV↗

arXiv2023

MILA: Memory-Based Instance-Level Adaptation for Cross-Domain Object Detection

Cross-domain object detection is challenging, and it involves aligning labeled source and unlabeled target domains. Previous approaches have used adversarial training to align features at both image-level and instance-level. At the instance level, finding a suitable source sample that aligns with a target sample is crucial. A source sample is considered suitable if it differs from the target sample only in domain, without differences in unimportant characteristics such as orientation and color, which can hinder the model's focus on aligning the domain difference. However, existing instance-level feature alignment methods struggle to find suitable source instances because their search scope is limited to mini-batches. Mini-batches are often so small in size that they do not always contain suitable source instances. The insufficient diversity of mini-batches becomes problematic particularly when the target instances have high intra-class variance. To address this issue, we propose a memory-based instance-level domain adaptation framework. Our method aligns a target instance with the most similar source instance of the same category retrieved from a memory storage. Specifically, we introduce a memory module that dynamically stores the pooled features of all labeled source instances, categorized by their labels. Additionally, we introduce a simple yet effective memory retrieval module that retrieves a set of matching memory slots for target instances. Our experiments on various domain shift scenarios demonstrate that our approach outperforms existing non-memory-based methods significantly.

cs.CV↗

arXiv2023

Use Your Head: Improving Long-Tail Video Recognition

This paper presents an investigation into long-tail video recognition. We demonstrate that, unlike naturally-collected video datasets and existing long-tail image benchmarks, current video benchmarks fall short on multiple long-tailed properties. Most critically, they lack few-shot classes in their tails. In response, we propose new video benchmarks that better assess long-tail recognition, by sampling subsets from two datasets: SSv2 and VideoLT. We then propose a method, Long-Tail Mixed Reconstruction, which reduces overfitting to instances from few-shot classes by reconstructing them as weighted combinations of samples from head classes. LMR then employs label mixing to learn robust decision boundaries. It achieves state-of-the-art average class accuracy on EPIC-KITCHENS and the proposed SSv2-LT and VideoLT-LT. Benchmarks and code at: tobyperrett.github.io/lmr

cs.CV↗

arXiv2022

Difficulty-Net: Learning to Predict Difficulty for Long-Tailed Recognition

Long-tailed datasets, where head classes comprise much more training samples than tail classes, cause recognition models to get biased towards the head classes. Weighted loss is one of the most popular ways of mitigating this issue, and a recent work has suggested that class-difficulty might be a better clue than conventionally used class-frequency to decide the distribution of weights. A heuristic formulation was used in the previous work for quantifying the difficulty, but we empirically find that the optimal formulation varies depending on the characteristics of datasets. Therefore, we propose Difficulty-Net, which learns to predict the difficulty of classes using the model's performance in a meta-learning framework. To make it learn reasonable difficulty of a class within the context of other classes, we newly introduce two key concepts, namely the relative difficulty and the driver loss. The former helps Difficulty-Net take other classes into account when calculating difficulty of a class, while the latter is indispensable for guiding the learning to a meaningful direction. Extensive experiments on popular long-tailed datasets demonstrated the effectiveness of the proposed method, and it achieved state-of-the-art performance on multiple long-tailed datasets.

cs.CV↗

arXiv2022

Class-Difficulty Based Methods for Long-Tailed Visual Recognition

Long-tailed datasets are very frequently encountered in real-world use cases where few classes or categories (known as majority or head classes) have higher number of data samples compared to the other classes (known as minority or tail classes). Training deep neural networks on such datasets gives results biased towards the head classes. So far, researchers have come up with multiple weighted loss and data re-sampling techniques in efforts to reduce the bias. However, most of such techniques assume that the tail classes are always the most difficult classes to learn and therefore need more weightage or attention. Here, we argue that the assumption might not always hold true. Therefore, we propose a novel approach to dynamically measure the instantaneous difficulty of each class during the training phase of the model. Further, we use the difficulty measures of each class to design a novel weighted loss technique called `class-wise difficulty based weighted (CDB-W) loss' and a novel data sampling technique called `class-wise difficulty based sampling (CDB-S)'. To verify the wide-scale usability of our CDB methods, we conducted extensive experiments on multiple tasks such as image classification, object detection, instance segmentation and video-action classification. Results verified that CDB-W loss and CDB-S could achieve state-of-the-art results on many class-imbalanced datasets such as ImageNet-LT, LVIS and EGTEA, that resemble real-world use cases.

cs.CV↗

arXiv2021

Scale-free networks may not necessarily witness cooperation

Networks with a scale-free degree distribution are widely thought to promote cooperation in various games. Herein, by studying the well-known prisoner's dilemma game, we demonstrate that this need not necessarily be true. For the very same degree sequence and degree distribution, we present a variety of possible behaviour. We reassess the perceived importance of hubs in a network towards the maintenance of cooperation. We also reevaluate the dependence of cooperation on network clustering and assortativity.

physics.soc-ph↗

arXiv2021

Mass testing and proactiveness affect epidemic spreading

The detection and management of diseases become quite complicated when pathogens contain asymptomatic phenotypes amongst their ranks, as evident during the recent COVID-19 pandemic. Spreading of diseases has been studied extensively under the paradigm of Susceptible - Infected - Recovered - Deceased (SIRD) dynamics. Various game-theoretic approaches have also addressed disease spread, many of which consider S, I, R, and D as strategies rather than as states. Remarkably, most studies from the above approaches do not account for the distinction between the symptomatic or asymptomatic aspect of the disease. It is well-known that precautionary measures like washing hands, wearing masks and social distancing significantly mitigate the spread of many contagious diseases. Herein, we consider the adoption of such precautions as strategies and treat S, I, R, and D as states. We also attempt to capture the differences in epidemic spreading arising from symptomatic and asymptomatic diseases on various network topologies. Through extensive computer simulations, we examine that the cost of maintaining precautionary measures as well as the extent of mass testing in a population affects the final fraction of socially responsible individuals. We observe that the lack of mass testing could potentially lead to a pandemic in case of asymptomatic diseases. Network topology also seems to play an important role. We further observe that the final fraction of proactive individuals depends on the initial fraction of both infected as well as proactive individuals. Additionally, edge density can significantly influence the overall outcome. Our findings are in broad agreement with the lessons learnt from the ongoing COVID-19 pandemic.

physics.soc-ph↗

arXiv2021

Topology dependent payoffs can lead to escape from prisoner's dilemma

The maintenance of cooperation in the presence of spatial restrictions has been studied extensively. It is well-established that the underlying graph topology can significantly influence the outcome of games on graphs. Maintenance of cooperation could be difficult, especially in the absence of spatial restrictions. The evolution of cooperation would naturally depend on payoffs. However, payoffs are generally considered to be invariant in a given game. A natural yet unexplored question is whether the topology of the underlying structures on which the games are played, possesses no role whatsoever in the determination of payoffs. Herein, we introduce the notion of cooperator graphs and defector graphs as well as a new form of game payoff, which is weakly dependent on the underlying network topology. These concepts are inspired by the well-known microbial phenomenon of quorum sensing. We demonstrate that even with such a weak dependence, the fundamental game dynamics and indeed the very nature of the game may be altered. Such changes in the nature of a game have been well-reported in theoretical and experimental studies.

physics.soc-ph↗

arXiv2020

Class-Wise Difficulty-Balanced Loss for Solving Class-Imbalance

Class-imbalance is one of the major challenges in real world datasets, where a few classes (called majority classes) constitute much more data samples than the rest (called minority classes). Learning deep neural networks using such datasets leads to performances that are typically biased towards the majority classes. Most of the prior works try to solve class-imbalance by assigning more weights to the minority classes in various manners (e.g., data re-sampling, cost-sensitive learning). However, we argue that the number of available training data may not be always a good clue to determine the weighting strategy because some of the minority classes might be sufficiently represented even by a small number of training data. Overweighting samples of such classes can lead to drop in the model's overall performance. We claim that the 'difficulty' of a class as perceived by the model is more important to determine the weighting. In this light, we propose a novel loss function named Class-wise Difficulty-Balanced loss, or CDB loss, which dynamically distributes weights to each sample according to the difficulty of the class that the sample belongs to. Note that the assigned weights dynamically change as the 'difficulty' for the model may change with the learning progress. Extensive experiments are conducted on both image (artificially induced class-imbalanced MNIST, long-tailed CIFAR and ImageNet-LT) and video (EGTEA) datasets. The results show that CDB loss consistently outperforms the recently proposed loss functions on class-imbalanced datasets irrespective of the data type (i.e., video or image).

cs.CV↗

arXiv2015

Slow poisoning and destruction of networks: Edge proximity and its implications for biological and infrastructure networks

We propose a network metric, edge proximity, ${\cal P}_e$, which demonstrates the importance of specific edges in a network, hitherto not captured by existing network metrics. The effects of removing edges with high ${\cal P}_e$ might initially seem inconspicuous but are eventually shown to be very harmful for networks. Compared to existing strategies, the removal of edges by ${\cal P}_e$ leads to a remarkable increase in the diameter and average shortest path length in undirected real and random networks till the first disconnection and well beyond. ${\cal P}_e$ can be consistently used to rupture the network into two nearly equal parts, thus presenting a very potent strategy to greatly harm a network. Targeting by ${\cal P}_e$ causes notable efficiency loss in U.S. and European power grid networks. ${\cal P}_e$ identifies proteins with essential cellular functions in protein-protein interaction networks. It pinpoints regulatory neural connections and important portions of the neural and brain networks, respectively. Energy flow interactions identified by ${\cal P}_e$ form the backbone of long food web chains. Finally, we scrutinize the potential of ${\cal P}_e$ in edge controllability dynamics of directed networks.

physics.soc-ph↗