Searcharxiv⌕ Search
Home/Archive/arXiv

arXiv · 2611.08895

Localizing and Editing the Goal Representation of a Vision-Language-Action Model

Sudhanshu Srivastava·Faisal Aqlan
View original source ↗Read source document ↗Export metadata ↓

Abstract

Operators can state safety constraints for robots as prohibitions such as "do not put the bowl on the stove". We study whether OpenVLA follows such prohibitions and where it represents the goal it executes. The 10 tasks of LIBEROGoal share one scene, so we can label the goal that each rollout executes. We combine closed-loop rollouts with activation patching, linear probes, activation steering and runtime monitors. On LIBERO-Goal, negation had no detectable effect on which task OpenVLA executed. The model executed the forbidden task in 79% of negated rollouts and completed 78% of plain tasks. A goal probe gives similar read-outs under "not X" and "X". Activation patching shows that the first action depends mostly on the instruction tokens before layer 12 and on the last prompt token from layer 16 onward. A fixed steering vector at the instruction tokens at layer 8 redirected 12 of 18 rollouts. Fixed vectors and prefill patches at layers 12 to 20 redirected none of 158. From layer 16, only patches at every decoded action token redirected the robot (15 of 18 rollouts at layer 24). When the instruction changes during a rollout, the layer-32 goal read-out detects non-compliance better than failure detectors, and an external probe on image and instruction features performs comparably. These results support enforcing prohibitions outside the instruction and monitoring goal read-outs against the specification. Goal edits in OpenVLA should act before layer 12 or patch every action token.

Explore related subjects

cs.RO

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Sudhanshu Srivastava, Faisal Aqlan. 2026-11-06. Localizing and Editing the Goal Representation of a Vision-Language-Action Model. https://arxiv.org/abs/2611.08895

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Sketch-MoMa: Teleoperation for Mobile Manipulator via Interpretation of Hand-Drawn Sketches

To use assistive robots in everyday life, a remote control system with common devices, such as 2D devices, is helpful to control the robots anytime and anywhere as intended. Hand-drawn sketches are one of the intuitive ways to control robots with 2D devices. However, since similar sketches have different intentions from scene to scene, existing work needs additional modalities to set the sketches' semantics. This requires complex operations for users and leads to decreasing usability. In this paper, we propose Sketch-MoMa, a teleoperation system using the user-given hand-drawn sketches as instructions to control a robot. We use Vision-Language Models (VLMs) to understand the user-given sketches superimposed on an observation image and infer drawn shapes and low-level tasks of the robot. We utilize the sketches and the generated shapes for recognition and motion planning of the generated low-level tasks for precise and intuitive operations. We validate our approach using state-of-the-art VLMs with 7 tasks and 5 sketch shapes. We also demonstrate that our approach effectively specifies the detailed motions, such as how to grasp and how much to rotate. Moreover, we show the competitive usability of our approach compared with the existing 2D interface through a user experiment with 14 participants.

cs.RO↗
arXiv2026

Evolving Drone Morphologies for Agile Flight

The design of multicopter drones has remained almost the same since their inception. While conventional designs work well in many cases, they may not be optimal in specific tasks. This paper revisits rotary drone design by exploring which body morphologies are best suited for different objectives and constraints. An evolutionary algorithm is used to produce drone morphologies that performs better than standard configurations at thrust-to-weight ratio, maneuverability and size. To generate a range of drones with performance trade-offs between them, the non-dominated sorting genetic algorithm II is used. The evolved drone bodies outperform a standard multicopters in at least one of the three objectives. The Pareto front extrema show improvement of 489.2%, 23% and 4.8% in maneuverability, thrust-to-weight ratio and size, respectively, when compared to a standard quadcopter. The improvement in maneuverability is attributed to the tilt angles of the motors, while the increase in thrust-to-weight ratio is due to the higher number of propellers. Experiments show that the performance improvement transfers to a real world drone racing task, where the evolved drone outperforms the standard drone in terms of number of gates passed,flight speed and lap times.

cs.RO↗
arXiv2026

VLD: Visual Learned Distance as an Interface for Reinforcement Learning Navigation

Training end-to-end policies from image data to directly predict navigation actions for robotic systems has proven inherently difficult. Existing approaches often suffer from either the sim-to-real gap during policy transfer or limited action-labeled training data. To address this problem, we introduce Visual Learned Distance (VLD), a scalable framework for image-goal navigation that decouples perception learning from policy learning. We first train a self-supervised distance-to-goal predictor on about 3,000 hours of internet-scale and simulated video, providing a distance signal that a reinforcement learning (RL) policy can minimize. The RL policy is trained separately in simulation using privileged geometric distances perturbed by GeoNoise, a learned noise model that reproduces the viewpoint- and overlap-dependent errors of the trained distance predictor. At deployment, the policy consumes VLD predictions zero-shot, without retraining, inheriting ``where to go'' from large-scale visual training while retaining the robust navigation behaviors learned in simulation. This error modeling is key to the transfer: with VLD inputs, GeoNoise-trained policies reach $58.7\%$ success on HM3D and $76.2\%$ on Gibson, whereas policies trained with generic temporally correlated noise collapse to $4.9\%$ and $11.5\%$. On a physical robot, the VLD-driven policy succeeds in $88\%$ of trials, compared with $60\%$ for a strong end-to-end image-goal baseline that outperforms it in simulation. Our decoupled design thus provides an alternative and scalable path toward reliable navigation policies that transfer from simulation to the real world.

cs.RO↗