arXiv subjects

Payam Jome Yazdian

Publications and source records attributed to Payam Jome Yazdian.

3 recordsLinked to original sources

arXiv2026

SalsaAgent: A multimodal embodied language model for interactive dance generation

Embodied interaction with humanoids depends on bidirectional nonverbal reactivity, coordination, and synchrony to convey cues and move with a partner. For socially interactive embodied agents, reactive motion generation requires expressive full-body motion that remains contextually appropriate while maintaining spatial and temporal synchrony. We present SalsaAgent, a language model that generates expressive, full-body salsa follower motions in reaction to a human leader and music. We formulate partner interaction as nonverbal token passing, extending the vocabulary of a large language model (LLM) to process discrete motion tokens, pairwise relation tokens, and audio tokens. Our method introduces full-body and pairwise-relation tokenizers, aligns language and motion tokens with automatically derived text descriptions of skeleton dynamics, and applies a two-stage token-to-diffusion pipeline. Subjective and objective evaluations show improved motion quality, two-person spatial coordination, and music and partner coordination relative to prior baselines.

cs.CV↗

arXiv2026

CoMPAS3D: A Dataset and Benchmark for Interactive Motion

Socially interactive humanoid robots must engage with humans through their bodies, adapting in real time to a partner's movement, intent, and abilities. This requires models that understand not just how bodies move, but what movement means in a shared social context. Yet evaluation frameworks for interactive motion generation do not measure whether generated follower motion is legible within a shared movement vocabulary, nor whether it is appropriate to the partner's proficiency level. This gap has two causes: existing frameworks rely on kinematic metrics such as FID and beat alignment that cannot measure either property, and existing datasets lack the move annotations and proficiency variation needed. Salsa is well-suited as an evaluation domain: improvised, dyadic, and governed by a move vocabulary and judging criteria covering timing, musicality, technique, difficulty, partnering, and originality. We present CoMPAS3D, a motion capture dataset of improvised partner salsa paired with an evaluation framework covering kinematic quality, two objective metrics (move legibility and proficiency appropriateness), and six competition-based subjective dimensions. The dataset includes 3 hours of improvisation by 18 dancers spanning beginner, intermediate, and professional levels, with over 2,800 expert-annotated segments covering move types, errors, and stylistic elements. We define three benchmarks: move classification (analogous to transcription), proficiency estimation (fluency assessment), and follower generation (dialogue response). Fine-tuned vision-language models perform strongly on objective metrics applied to ground-truth motion sequences. Applied to Duolando and InterGen, the metrics reveal failures that kinematic metrics miss. Human evaluations confirm the gap between generated and ground-truth motion. CoMPAS3D, annotations, benchmark code, and baseline results are publicly available.

cs.LG↗

arXiv2025

MotionScript: Natural Language Descriptions for Expressive 3D Human Motions

We introduce MotionScript, a novel framework for generating highly detailed, natural language descriptions of 3D human motions. Unlike existing motion datasets that rely on broad action labels or generic captions, MotionScript provides fine-grained, structured descriptions that capture the full complexity of human movement including expressive actions (e.g., emotions, stylistic walking) and interactions beyond standard motion capture datasets. MotionScript serves as both a descriptive tool and a training resource for text-to-motion models, enabling the synthesis of highly realistic and diverse human motions from text. By augmenting motion datasets with MotionScript captions, we demonstrate significant improvements in out-of-distribution motion generation, allowing large language models (LLMs) to generate motions that extend beyond existing data. Additionally, MotionScript opens new applications in animation, virtual human simulation, and robotics, providing an interpretable bridge between intuitive descriptions and motion synthesis. To the best of our knowledge, this is the first attempt to systematically translate 3D motion into structured natural language without requiring training data.

cs.CV↗