arXiv subjects

Jaewoo Jeong

Publications and source records attributed to Jaewoo Jeong.

14 recordsLinked to original sources

arXiv2026

Ego-Human Motion Prediction with 3D-Aware LLM

Anticipating human motion from an egocentric perspective is fundamental for proactive assistance in AR/VR, human-robot collaboration, and embodied AI. While recent works incorporate language as a semantic prior to reduce the ill-posed nature of egocentric forecasting, they largely neglect the 3D spatial and semantic context that governs how motion unfolds, and treat pose and language prediction as separate inference streams. We introduce Ego3DLM, built on two core principles: accurate motion forecasting requires explicit spatial and semantic understanding of the 3D environment, and pose and language must be predicted holistically in a single pass, since motion is inherently tied to the semantic interpretation of actions being performed. Given three-point tracking, 3D scene features, and egocentric video, Ego3DLM simultaneously decodes past pose, future pose, past narration, and future narration in a single autoregressive pass, grounding predicted poses and descriptions in one another to enforce cross-modal and temporal consistency. We adopt a three-stage training scheme: (1) spatial-semantic scene awareness pretraining; (2) holistic instruction tuning over all four outputs in a single pass; and (3) GRPO-based reinforcement finetuning with intra- and inter-modal rewards that directly optimize pose-language fidelity. Experiments on the Nymeria benchmark demonstrate that Ego3DLM achieves state-of-the-art performance across future motion prediction, past motion tracking, and motion description, showing that 3D scene grounding and holistic cross-modal prediction yield physically plausible and semantically coherent motion forecasts. The project page is available at https://jaewoo97.github.io/Ego3DLM/.

cs.CV↗

arXiv2026

Towards Functional Correctness of Large Code Models with Selective Generation

The hallucination of code generation models hinders their applicability to systems requiring higher safety standards. One critical bottleneck in addressing code hallucination is the difficulty of identifying the functional correctness of generated code, due to its unnatural form. We address this core bottleneck by automatically generating unit tests using dynamic code analysis tools, leveraging the \emph{executable nature} of code. Accordingly, we propose a \emph{selective code generator} that abstains from uncertain generations -- based on the functional correctness evaluated by generated unit tests -- to theoretically control the correctness among non-abstained answers, \ie the false discovery rate. Finally, we propose to use generated unit tests in evaluation as well as in learning for precise code evaluation, calling this paradigm \emph{FuzzEval}. We demonstrate the efficacy of our method along with the controllability of code hallucination and reasonable selection efficiency.

cs.SE↗

arXiv2025

Interaction-Merged Motion Planning: Effectively Leveraging Diverse Motion Datasets for Robust Planning

Motion planning is a crucial component of autonomous robot driving. While various trajectory datasets exist, effectively utilizing them for a target domain remains challenging due to differences in agent interactions and environmental characteristics. Conventional approaches, such as domain adaptation or ensemble learning, leverage multiple source datasets but suffer from domain imbalance, catastrophic forgetting, and high computational costs. To address these challenges, we propose Interaction-Merged Motion Planning (IMMP), a novel approach that leverages parameter checkpoints trained on different domains during adaptation to the target domain. IMMP follows a two-step process: pre-merging to capture agent behaviors and interactions, sufficiently extracting diverse information from the source domain, followed by merging to construct an adaptable model that efficiently transfers diverse interactions to the target domain. Our method is evaluated on various planning benchmarks and models, demonstrating superior performance compared to conventional approaches.

cs.RO↗

arXiv2025

Non-differentiable Reward Optimization for Diffusion-based Autonomous Motion Planning

Safe and effective motion planning is crucial for autonomous robots. Diffusion models excel at capturing complex agent interactions, a fundamental aspect of decision-making in dynamic environments. Recent studies have successfully applied diffusion models to motion planning, demonstrating their competence in handling complex scenarios and accurately predicting multi-modal future trajectories. Despite their effectiveness, diffusion models have limitations in training objectives, as they approximate data distributions rather than explicitly capturing the underlying decision-making dynamics. However, the crux of motion planning lies in non-differentiable downstream objectives, such as safety (collision avoidance) and effectiveness (goal-reaching), which conventional learning algorithms cannot directly optimize. In this paper, we propose a reinforcement learning-based training scheme for diffusion motion planning models, enabling them to effectively learn non-differentiable objectives that explicitly measure safety and effectiveness. Specifically, we introduce a reward-weighted dynamic thresholding algorithm to shape a dense reward signal, facilitating more effective training and outperforming models trained with differentiable objectives. State-of-the-art performance on pedestrian datasets (CrowdNav, ETH-UCY) compared to various baselines demonstrates the versatility of our approach for safe and effective motion planning.

cs.RO↗

arXiv2025

Multi-modal Knowledge Distillation-based Human Trajectory Forecasting

Pedestrian trajectory forecasting is crucial in various applications such as autonomous driving and mobile robot navigation. In such applications, camera-based perception enables the extraction of additional modalities (human pose, text) to enhance prediction accuracy. Indeed, we find that textual descriptions play a crucial role in integrating additional modalities into a unified understanding. However, online extraction of text requires the use of VLM, which may not be feasible for resource-constrained systems. To address this challenge, we propose a multi-modal knowledge distillation framework: a student model with limited modality is distilled from a teacher model trained with full range of modalities. The comprehensive knowledge of a teacher model trained with trajectory, human pose, and text is distilled into a student model using only trajectory or human pose as a sole supplement. In doing so, we separately distill the core locomotion insights from intra-agent multi-modality and inter-agent interaction. Our generalizable framework is validated with two state-of-the-art models across three datasets on both ego-view (JRDB, SIT) and BEV-view (ETH/UCY) setups, utilizing both annotated and VLM-generated text captions. Distilled student models show consistent improvement in all prediction metrics for both full and instantaneous observations, improving up to ~13%. The code is available at https://github.com/Jaewoo97/KDTF.

cs.CV↗

arXiv2024

Multi-agent Long-term 3D Human Pose Forecasting via Interaction-aware Trajectory Conditioning

Human pose forecasting garners attention for its diverse applications. However, challenges in modeling the multi-modal nature of human motion and intricate interactions among agents persist, particularly with longer timescales and more agents. In this paper, we propose an interaction-aware trajectory-conditioned long-term multi-agent human pose forecasting model, utilizing a coarse-to-fine prediction approach: multi-modal global trajectories are initially forecasted, followed by respective local pose forecasts conditioned on each mode. In doing so, our Trajectory2Pose model introduces a graph-based agent-wise interaction module for a reciprocal forecast of local motion-conditioned global trajectory and trajectory-conditioned local pose. Our model effectively handles the multi-modality of human motion and the complexity of long-term multi-agent interactions, improving performance in complex environments. Furthermore, we address the lack of long-term (6s+) multi-agent (5+) datasets by constructing a new dataset from real-world images and 2D annotations, enabling a comprehensive evaluation of our proposed model. State-of-the-art prediction performance on both complex and simpler datasets confirms the generalized effectiveness of our method. The code is available at https://github.com/Jaewoo97/T2P.

cs.CV↗

arXiv2024

T4P: Test-Time Training of Trajectory Prediction via Masked Autoencoder and Actor-specific Token Memory

Trajectory prediction is a challenging problem that requires considering interactions among multiple actors and the surrounding environment. While data-driven approaches have been used to address this complex problem, they suffer from unreliable predictions under distribution shifts during test time. Accordingly, several online learning methods have been proposed using regression loss from the ground truth of observed data leveraging the auto-labeling nature of trajectory prediction task. We mainly tackle the following two issues. First, previous works underfit and overfit as they only optimize the last layer of the motion decoder. To this end, we employ the masked autoencoder (MAE) for representation learning to encourage complex interaction modeling in shifted test distribution for updating deeper layers. Second, utilizing the sequential nature of driving data, we propose an actor-specific token memory that enables the test-time learning of actor-wise motion characteristics. Our proposed method has been validated across various challenging cross-dataset distribution shift scenarios including nuScenes, Lyft, Waymo, and Interaction. Our method surpasses the performance of existing state-of-the-art online learning methods in terms of both prediction accuracy and computational efficiency. The code is available at https://github.com/daeheepark/T4P.

cs.CV↗

arXiv2024

Ferrimagnetic Heusler tunnel junctions with fast spin-transfer torque switching enabled by low magnetization

Magnetic random access memory that uses magnetic tunnel junction memory cells is a high performance, non-volatile memory technology that goes beyond traditional charge-based memories. Today its speed is limited by the high magnetization of the memory storage layer. Here we show that fast and highly reliable switching is possible using a very low magnetization ferrimagnetic Heusler alloy, Mn3Ge. Moreover, the tunneling magnetoresistance is the highest yet achieved for a ferrimagnetic material at ambient temperature. Furthermore, the devices were prepared on technologically relevant amorphous substrates using a novel combination of a nitride seed layer and a chemical templating layer. These results show a clear path to the lowering of switching currents using ferrimagnetic Heusler materials and, therefore, to the scaling of high performance magnetic random access memories beyond those nodes possible with ferromagnetic devices.

cond-mat.mtrl-sci↗

arXiv2023

Improving Transferability for Cross-domain Trajectory Prediction via Neural Stochastic Differential Equation

Multi-agent trajectory prediction is crucial for various practical applications, spurring the construction of many large-scale trajectory datasets, including vehicles and pedestrians. However, discrepancies exist among datasets due to external factors and data acquisition strategies. External factors include geographical differences and driving styles, while data acquisition strategies include data acquisition rate, history/prediction length, and detector/tracker error. Consequently, the proficient performance of models trained on large-scale datasets has limited transferability on other small-size datasets, bounding the utilization of existing large-scale datasets. To address this limitation, we propose a method based on continuous and stochastic representations of Neural Stochastic Differential Equations (NSDE) for alleviating discrepancies due to data acquisition strategy. We utilize the benefits of continuous representation for handling arbitrary time steps and the use of stochastic representation for handling detector/tracker errors. Additionally, we propose a dataset-specific diffusion network and its training framework to handle dataset-specific detection/tracking errors. The effectiveness of our method is validated against state-of-the-art trajectory prediction models on the popular benchmark datasets: nuScenes, Argoverse, Lyft, INTERACTION, and Waymo Open Motion Dataset (WOMD). Improvement in performance gain on various source and target dataset configurations shows the generalized competence of our approach in addressing cross-dataset discrepancies.

cs.CV↗

arXiv2020

Size scaling, dynamics, and electro-thermal bifurcation of VO2 Mott oscillators

Traditional electronic devices are well-known to improve in speed and energy-efficiency as their dimensions are reduced to the nanoscale. However, this scaling behavior remains unclear for nonlinear dynamical circuit elements, such as Mott neuron-like spiking oscillators, which are of interest for bio-inspired computing. Here we show that shrinking micrometer-sized VO2 oscillators to sub-100 nm effective sizes, achieved using a nanogap cut in a metallic carbon nanotube (CNT) electrode, does not guarantee faster spiking. However, an additional heat source such as Joule heating from the CNT, in combination with small size and heat capacity (defined by the narrow volume of VO2 whose insulator-metal transition is triggered by the CNT), can increase the spiking frequency by ~1000x due to an electro-thermal bifurcation in the nonlinear dynamics. These results demonstrate that nonlinear dynamical switches operate in a complex phase space which can be controlled by careful electro-thermal design, offering new tuning parameters for designing future biomimetic electronics.

cond-mat.mtrl-sci↗

arXiv2019

Localized Triggering of the Insulator-Metal Transition in VO2 using a Single Carbon Nanotube

Vanadium dioxide (VO2) has been widely studied for its rich physics and potential applications, undergoing a prominent insulator-metal transition (IMT) near room temperature. The transition mechanism remains highly debated, and little is known about the IMT at nanoscale dimensions. To shed light on this problem, here we use ~1 nm wide carbon nanotube (CNT) heaters to trigger the IMT in VO2. Single metallic CNTs switch the adjacent VO2 at less than half the voltage and power required by control devices without a CNT, with switching power as low as ~85 $μW$ at 300 nm device lengths. We also obtain potential and temperature maps of devices during operation using Kelvin Probe Microscopy (KPM) and Scanning Thermal Microscopy (SThM). Comparing these with three-dimensional electrothermal simulations, we find that the local heating of the VO2 by the CNT play a key role in the IMT. These results demonstrate the ability to trigger IMT in VO2 using nanoscale heaters, and highlight the significance of thermal engineering to improve device behaviour.

cond-mat.mtrl-sci↗

arXiv2019

Fast Spiking of a Mott VO2-Carbon Nanotube Composite Device

The recent surge of interest in brain-inspired computing and power-efficient electronics has dramatically bolstered development of computation and communication using neuron-like spiking signals. Devices that can produce rapid and energy-efficient spiking could significantly advance these applications. Here we demonstrate DC-current or voltage-driven periodic spiking with sub-20 ns pulse widths from a single device composed of a thin VO2 film with a metallic carbon nanotube as a nanoscale heater. Compared with VO2-only devices, adding the nanotube heater dramatically decreases the transient duration and pulse energy, and increases the spiking frequency, by up to three orders of magnitude. This is caused by heating and cooling of the VO2 across its insulator-metal transition being localized to a nanoscale conduction channel in an otherwise bulk medium. This result provides an important component of energy-efficient neuromorphic computing systems, and a lithography-free technique for power-scaling of electronic devices that operate via bulk mechanisms.

physics.app-ph↗

arXiv2016

Mesoscopic structural phase progression in photo-excited VO2 revealed by time-resolved x-ray diffraction microscopy

Dynamical phase separation during a solid-solid phase transition poses a challenge for understanding the fundamental processes in correlated materials. Critical information underlying a phase transition, such as localized phase competition, is difficult to reveal by measurements that are spatially averaged over many phase separated regions. The ability to simultaneously track the spatial and temporal evolution of such systems is essential to understanding mesoscopic processes during a phase transition. Using state-of-the-art time-resolved hard x-ray diffraction microscopy, we directly visualize the structural phase progression in a VO2 film upon photoexcitation. Following a homogenous in-plane optical excitation, the phase transformation is initiated at discrete sites and completed by the growth of one lattice structure into the other, instead of a simultaneous isotropic lattice symmetry change. The time-dependent x-ray diffraction spatial maps show that the in-plane phase progression in laser-superheated VO2 is via a displacive lattice transformation as a result of relaxation from an excited monoclinic phase into a rutile phase. The speed of the phase front progression is quantitatively measured, and is faster than the process driven by in-plane thermal diffusion but slower than the sound speed in VO2. The direct visualization of localized structural changes in the time domain opens a new avenue to study mesoscopic processes in driven systems.

cond-mat.mes-hall↗

arXiv2014

Extreme Ultraviolet Transient Grating Spectroscopy of Vanadium Dioxide

Nonlinear spectroscopy in the extreme ultraviolet (EUV) and soft x-ray spectral range offers the opportunity for element selective probing of ultrafast dynamics using core-valence transitions (Mukamel et al., Acc. Chem. Res. 42, 553 (2009)). We demonstrate a step on this path showing core-valence sensitivity in transient grating spectroscopy with EUV probing. We study the optically induced insulator-to-metal transition (IMT) of a VO2 film with EUV diffraction from the optically excited sample. The VO2 exhibits a change in the 3p-3d resonance of V accompanied by an acoustic response. Due to the broadband probing we are able to separate the two features.

physics.optics↗