arXiv subjects

Jinyeon Kim

Publications and source records attributed to Jinyeon Kim.

4 recordsLinked to original sources

arXiv2026

Ludi${}_{\scriptscriptstyle 0.1}$: An Agentic System for Socially Intelligent Robots

Robot foundation models have substantially advanced perception and control, but natural human-robot collaboration requires more than executing isolated commands. A robot must recognize ambiguity, maintain context across turns, communicate its intentions, and revise ongoing behavior as the user's intent changes. We present $\scriptstyle\mathsf{Ludi}_{\scriptscriptstyle 0.1}$, an agentic system for socially intelligent robots that integrates interactive speech, multimodal reasoning, memory, navigation, and learned manipulation. Its decision-making core is a fine-tuned vision-language model trained on multi-turn interaction traces spanning ambiguous requests, clarifications, corrections, interruptions, mixed social and task dialogue, and multi-step tasks. A purpose-built harness manages the model-tool interaction loop, while specialized navigation and manipulation policies execute physical skills. Ludi${}_{\scriptscriptstyle 0.1}$ demonstrates a practical path toward fluid human-robot collaboration today while producing the multimodal interaction traces needed to develop a more deeply integrated foundation model for robots and people.

cs.RO↗

arXiv2024

ReALFRED: An Embodied Instruction Following Benchmark in Photo-Realistic Environments

Simulated virtual environments have been widely used to learn robotic agents that perform daily household tasks. These environments encourage research progress by far, but often provide limited object interactability, visual appearance different from real-world environments, or relatively smaller environment sizes. This prevents the learned models in the virtual scenes from being readily deployable. To bridge the gap between these learning environments and deploying (i.e., real) environments, we propose the ReALFRED benchmark that employs real-world scenes, objects, and room layouts to learn agents to complete household tasks by understanding free-form language instructions and interacting with objects in large, multi-room and 3D-captured scenes. Specifically, we extend the ALFRED benchmark with updates for larger environmental spaces with smaller visual domain gaps. With ReALFRED, we analyze previously crafted methods for the ALFRED benchmark and observe that they consistently yield lower performance in all metrics, encouraging the community to develop methods in more realistic environments. Our code and data are publicly available.

cs.RO↗

arXiv2023

Context-Aware Planning and Environment-Aware Memory for Instruction Following Embodied Agents

Accomplishing household tasks requires to plan step-by-step actions considering the consequences of previous actions. However, the state-of-the-art embodied agents often make mistakes in navigating the environment and interacting with proper objects due to imperfect learning by imitating experts or algorithmic planners without such knowledge. To improve both visual navigation and object interaction, we propose to consider the consequence of taken actions by CAPEAM (Context-Aware Planning and Environment-Aware Memory) that incorporates semantic context (e.g., appropriate objects to interact with) in a sequence of actions, and the changed spatial arrangement and states of interacted objects (e.g., location that the object has been moved to) in inferring the subsequent actions. We empirically show that the agent with the proposed CAPEAM achieves state-of-the-art performance in various metrics using a challenging interactive instruction following benchmark in both seen and unseen environments by large margins (up to +10.70% in unseen env.).

cs.RO↗

arXiv2021

Lack of Fusion in Additive Manufacturing: Defect or Asset?

Rapid cooling rates and stochastic interactions between the heat source and feedstock in additive manufacturing (AM) result in strong anisotropy and process-induced defects deteriorating the tensile ductility and fatigue resistance of printed parts. We show that by deliberately introducing a high density of lack of fusion (LoF) defects, a processing regime that has been avoided so far, followed by pressure assisted heat treatment, we can print Ti-6Al-4V with reduced texture and exceptional properties surpassing that of wrought, cast, forged, annealed, and solution-treated and aged counterparts. Such improvement is achieved through the formation of low aspect ratio {\alpha}-grains around LoF defects upon healing, surrounded by {\alpha}-laths. This occurrence is attributed to surface energy reduction and recrystallization events taking place during healing of LoF defects. Our approach to design duplex microstructures is applicable to a wide range of AM processes and alloys and can be used in the design of damage tolerant microstructures.

cond-mat.mtrl-sci↗