arXiv subjects

Gunhee Lee

Publications and source records attributed to Gunhee Lee.

7 recordsLinked to original sources

arXiv2026

Pri4R: Learning World Dynamics for Vision-Language-Action Models with Privileged 4D Representation

Humans learn not only how their bodies move, but also how the surrounding world responds to their actions. In contrast, while recent Vision-Language-Action (VLA) models exhibit impressive semantic understanding, they often fail to capture the spatiotemporal dynamics governing physical interaction. In this paper, we introduce Pri4R, a simple yet effective approach that endows VLA models with an implicit understanding of world dynamics by leveraging privileged 4D information during training. Specifically, Pri4R augments VLAs with a lightweight point track head that predicts 3D point tracks. By injecting VLA features into this head to jointly predict future 3D trajectories, the model learns to incorporate evolving scene geometry within its shared representation space, enabling more physically aware context for precise control. Due to its architectural simplicity, Pri4R is compatible with dominant VLA design patterns with minimal changes. During inference, we run the model using the original VLA architecture unchanged; Pri4R adds no extra inputs, outputs, or computational overhead. Across simulation and real-world evaluations, Pri4R significantly improves performance on challenging manipulation tasks, including a +10% gain on LIBERO-Long and a +40% gain on RoboCasa. We further show that 3D point track prediction is an effective supervision target for learning action-world dynamics, and validate our design choices through extensive ablations. Project page: https://jiiiisoo.github.io/Pri4R/

cs.CV↗

arXiv2025

MASH-VLM: Mitigating Action-Scene Hallucination in Video-LLMs through Disentangled Spatial-Temporal Representations

In this work, we tackle action-scene hallucination in Video Large Language Models (Video-LLMs), where models incorrectly predict actions based on the scene context or scenes based on observed actions. We observe that existing Video-LLMs often suffer from action-scene hallucination due to two main factors. First, existing Video-LLMs intermingle spatial and temporal features by applying an attention operation across all tokens. Second, they use the standard Rotary Position Embedding (RoPE), which causes the text tokens to overemphasize certain types of tokens depending on their sequential orders. To address these issues, we introduce MASH-VLM, Mitigating Action-Scene Hallucination in Video-LLMs through disentangled spatial-temporal representations. Our approach includes two key innovations: (1) DST-attention, a novel attention mechanism that disentangles the spatial and temporal tokens within the LLM by using masked attention to restrict direct interactions between the spatial and temporal tokens; (2) Harmonic-RoPE, which extends the dimensionality of the positional IDs, allowing the spatial and temporal tokens to maintain balanced positions relative to the text tokens. To evaluate the action-scene hallucination in Video-LLMs, we introduce the UNSCENE benchmark with 1,320 videos and 4,078 QA pairs. Extensive experiments demonstrate that MASH-VLM achieves state-of-the-art results on the UNSCENE benchmark, as well as on existing video understanding benchmarks.

cs.CV↗

arXiv2024

Probing Visual Language Priors in VLMs

Despite recent advances in Vision-Language Models (VLMs), they may over-rely on visual language priors existing in their training data rather than true visual reasoning. To investigate this, we introduce ViLP, a benchmark featuring deliberately out-of-distribution images synthesized via image generation models and out-of-distribution Q&A pairs. Each question in ViLP is coupled with three potential answers and three corresponding images: one that can be resolved by text priors alone and two that demand visual reasoning. Although, humans achieve near-perfect accuracy, modern VLMs falter; for instance, GPT-4 achieves only 66.17% on ViLP. To alleviate this, we propose a self-improving framework in which models generate new VQA data, then apply pixel-level and semantic corruptions to form "good-bad" image pairs for self-training. Our training objectives compel VLMs to focus more on the actual visual inputs, and we demonstrate their effectiveness in boosting the performance of open-source VLMs, including LLaVA-v1.5 and Cambrian.

cs.CV↗

arXiv2024

OmniSplat: Taming Feed-Forward 3D Gaussian Splatting for Omnidirectional Images with Editable Capabilities

Feed-forward 3D Gaussian splatting (3DGS) models have gained significant popularity due to their ability to generate scenes immediately without needing per-scene optimization. Although omnidirectional images are becoming more popular since they reduce the computation required for image stitching to composite a holistic scene, existing feed-forward models are only designed for perspective images. The unique optical properties of omnidirectional images make it difficult for feature encoders to correctly understand the context of the image and make the Gaussian non-uniform in space, which hinders the image quality synthesized from novel views. We propose OmniSplat, a training-free fast feed-forward 3DGS generation framework for omnidirectional images. We adopt a Yin-Yang grid and decompose images based on it to reduce the domain gap between omnidirectional and perspective images. The Yin-Yang grid can use the existing CNN structure as it is, but its quasi-uniform characteristic allows the decomposed image to be similar to a perspective image, so it can exploit the strong prior knowledge of the learned feed-forward network. OmniSplat demonstrates higher reconstruction accuracy than existing feed-forward networks trained on perspective images. Our project page is available on: https://robot0321.github.io/omnisplat/index.html.

cs.CV↗

arXiv2022

StyLandGAN: A StyleGAN based Landscape Image Synthesis using Depth-map

Despite recent success in conditional image synthesis, prevalent input conditions such as semantics and edges are not clear enough to express `Linear (Ridges)' and `Planar (Scale)' representations. To address this problem, we propose a novel framework StyLandGAN, which synthesizes desired landscape images using a depth map which has higher expressive power. Our StyleLandGAN is extended from the unconditional generation model to accept input conditions. We also propose a '2-phase inference' pipeline which generates diverse depth maps and shifts local parts so that it can easily reflect user's intend. As a comparison, we modified the existing semantic image synthesis models to accept a depth map as well. Experimental results show that our method is superior to existing methods in quality, diversity, and depth-accuracy.

cs.CV↗

arXiv2020

Threat from being Social: Vulnerability Analysis of Social Network Coupled Smart Grid

Social Networks (SNs) have been gradually applied by utility companies as an addition to smart grid and are proved to be helpful in smoothing load curves and reducing energy usage. However, SNs also bring in new threats to smart grid: misinformation in SNs may cause smart grid users to alter their demand, resulting in transmission line overloading and in turn leading to catastrophic impact to the grid. In this paper, we discuss the interdependency in the social network coupled smart grid and focus on its vulnerability. That is, how much can the smart grid be damaged when misinformation related to it diffuses in SNs? To analytically study the problem, we propose the Misinformation Attack Problem in Social-Smart Grid (MAPSS) that identifies the top critical nodes in the SN, such that the smart grid can be greatly damaged when misinformation propagates from those nodes. This problem is challenging as we have to incorporate the complexity of the two networks concurrently. Nevertheless, we propose a technique that can explicitly take into account information diffusion in SN, power flow balance and cascading failure in smart grid integratedly when evaluating node criticality, based on which we propose various strategies in selecting the most critical nodes. Also, we introduce controlled load shedding as a protection strategy to reduce the impact of cascading failure. The effectiveness of our algorithms are demonstrated by experiments on IEEE bus test cases as well as the Pegase data set.

eess.SY↗

arXiv2019

Charge transport layer dependent electronic band bending in perovskite solar cells and its correlation to device degradation

Perovskite solar cells (PSCs) have shown remarkably improved power-conversion efficiency of around 25%. However, their working principle remains arguable and the stability issue has not been solved yet. In this report, we revealed that the working mechanism of PSCs is explained by a dominant pn junction occurring at the different interface depending on electron transport layer, and charges are accumulated at the corresponding dominant junction initiating device degradation. Locations of a dominant pn junction, the electric field, and carrier-density distribution with respect to electron-transport layers in the PCS devices were investigated by using the electron-beam-induced current measurement and Kelvin probe force microscopy. The amount of accumulated charges in the devices was analyzed using the charge-extraction method and the degradation process of devices was confirmed by SEM measurements. From these observations, we identified that the dominant pn junction appears at the interface where the degree of band bending is higher compared to the other interface, and charges are accumulated at the corresponding junction where the device degradation is initiated, which suggests that there exists a strong correlation between PSC working principle and device degradation. We highlight that an ideal pin PSC that can minimize the degree of band bending should be designed for ensuring long-term stability, via using proper selective contacts

physics.app-ph↗