arXiv subjects

Yash Agarwal

Publications and source records attributed to Yash Agarwal.

6 recordsLinked to original sources

arXiv2026

Lost in Reconstruction: Aligning Action Representations with Language in Vision-Language-Action Models

Action verbs describe not only the physical outcomes of actions, but also how those actions are performed. Yet action representations in vision-language-action models (VLAs) are typically optimized for reconstruction under L1/L2 losses in raw action space, where numerical proximity need not reflect linguistically meaningful distinctions. On BridgeV2, we show that action trajectories contain verb-grounding information beyond visual state changes, and that reconstruction-only discrete tokenization systematically erodes this information. To address this problem, we introduce SALT, a Semantically ALigned action Tokenizer that augments a VQ-VAE-style tokenizer with an auxiliary objective requiring a frozen vision-language model to recover the episode instruction from quantized action latents. Policies trained with SALT achieve 71.9% average success in SimplerEnv, compared with 42.7% for a reconstruction-only VQ-VAE tokenizer and 31.2% for FAST. SALT also develops verb-specialized codes while maintaining reconstruction fidelity. These results show that robot action trajectories provide a source of language grounding and that preserving this structure in action representations can substantially improve language-conditioned control.

cs.RO↗

arXiv2026

Carnot: Interpretable, Interactive, and Optimized Execution of Deep Research Queries

Enterprises increasingly seek to query data lakes using natural language via AI-driven tools like semantic operators or deep research agents. However, the latter operates as an opaque black box, hiding its intermediate reasoning and data retrieval steps, and failing to expose controls for managing API costs and execution latency. Meanwhile, the former can be prohibitively expensive for enterprise-scale data lakes. Consequently, analysts using these systems lack the agency to intercept hallucinated premises, verify intermediate results, or correct the system's trajectory. We present Carnot, an interactive execution engine for AI-driven analytics. Carnot compiles natural language requests into physical execution graphs and surfaces them through an interactive notebook interface. Rather than waiting blindly for a final output, users can critique the plan, incrementally execute operators, inspect intermediate data, or directly edit the underlying code or semantic operator instructions. Carnot's query optimizer will optimize the query with respect to cost or latency constraints provided by the user. Our demo will showcase how Carnot helps users achieve efficient and verifiable insights on workloads motivated by real enterprise use cases.

cs.DB↗

arXiv2025

Interactive Virtual Games: Winners for Deep Cognitive Assessment

Studies of human cognition often rely on brief, highly controlled tasks that emphasize group-level effects but poorly capture the rich variability within and between individuals. A suite of minigames built on the novel pixelDOPA platform was designed to overcome these limitations by embedding classic cognitive task paradigms in a 3D virtual interactive environment with continuous behavior logging. Four of the minigames explore constructs that overlap established NIH Toolbox tasks, including processing speed, rule shifting, inhibitory control and working memory. Across a clinical sample of 66 participants collected outside a controlled laboratory setting, large correlations (r = 0.47-0.92) between the pixelDOPA tasks and NIH Toolbox counterparts were found. Process-informed metrics improved both task convergence and data quality. Test-retest analyses revealed high reliability (ICC = 0.71-0.85) for all minigames. Beyond endpoint metrics, movement and gaze trajectories revealed stable, idiosyncratic profiles of gameplay strategy, with unsupervised clustering differentiating participants by their navigational and viewing behaviors. These trajectory-based features showed lower within-person variability than between-person variability, facilitating participant identification across repeated sessions. Game-based tasks can therefore retain the psychometric rigor of standard cognitive assessments while providing new insights into dynamic individual-specific behaviors. By leveraging a highly engaging, fully customizable game engine, comprehensive behavioral tracking can boost the power to detect individual differences without sacrificing group-level inference. This possibility reveals a path toward cognitive measures that are both psychometrically robust and deployable in less-than-ideal settings, while capturing richer behavioral data than traditional paradigms.

cs.HC↗

arXiv2024

YesBut: A High-Quality Annotated Multimodal Dataset for evaluating Satire Comprehension capability of Vision-Language Models

Understanding satire and humor is a challenging task for even current Vision-Language models. In this paper, we propose the challenging tasks of Satirical Image Detection (detecting whether an image is satirical), Understanding (generating the reason behind the image being satirical), and Completion (given one half of the image, selecting the other half from 2 given options, such that the complete image is satirical) and release a high-quality dataset YesBut, consisting of 2547 images, 1084 satirical and 1463 non-satirical, containing different artistic styles, to evaluate those tasks. Each satirical image in the dataset depicts a normal scenario, along with a conflicting scenario which is funny or ironic. Despite the success of current Vision-Language Models on multimodal tasks such as Visual QA and Image Captioning, our benchmarking experiments show that such models perform poorly on the proposed tasks on the YesBut Dataset in Zero-Shot Settings w.r.t both automated as well as human evaluation. Additionally, we release a dataset of 119 real, satirical photographs for further research. The dataset and code are available at https://github.com/abhi1nandy2/yesbut_dataset.

cs.CV↗

arXiv2022

Enabling hand gesture customization on wrist-worn devices

We present a framework for gesture customization requiring minimal examples from users, all without degrading the performance of existing gesture sets. To achieve this, we first deployed a large-scale study (N=500+) to collect data and train an accelerometer-gyroscope recognition model with a cross-user accuracy of 95.7% and a false-positive rate of 0.6 per hour when tested on everyday non-gesture data. Next, we design a few-shot learning framework which derives a lightweight model from our pre-trained model, enabling knowledge transfer without performance degradation. We validate our approach through a user study (N=20) examining on-device customization from 12 new gestures, resulting in an average accuracy of 55.3%, 83.1%, and 87.2% on using one, three, or five shots when adding a new gesture, while maintaining the same recognition accuracy and false-positive rate from the pre-existing gesture set. We further evaluate the usability of our real-time implementation with a user experience study (N=20). Our results highlight the effectiveness, learnability, and usability of our customization framework. Our approach paves the way for a future where users are no longer bound to pre-existing gestures, freeing them to creatively introduce new gestures tailored to their preferences and abilities.

cs.HC↗

arXiv2021

Convolutional encoder decoder network for the removal of coherent seismic noise

Seismologists often need to gather information about the subsurface structure of a location to determine if it is fit to be drilled for oil. However, there may be electrical noise in seismic data which is often removed by disregarding certain portions of the data with the use of a notch filter. Instead, we use a convolutional encoder decoder network to remove such noise by training the network to take the noisy shot record as input and remove the noise from the shot record as output. In this way, we retain important information about the data collected while still removing coherent noise in seismic data.

physics.geo-ph↗