arXiv · 2610.19295
Learning Keypoint Constraints from Visual Demonstrations for Safe Motion Planning
Abstract
We present Keypoint Constraint Learning (KCL), a framework for learning task-relevant motion constraints from visual demonstrations and for planning safely with the extracted constraints. To make constraint learning tractable from RGB-D observations, we represent demonstrations using sparse 3D object-centric keypoints and semantic mask centroids extracted through segmentation, visual feature clustering, and point tracking. In this reduced geometric space, we learn keypoint dynamics and infer distance- and orientation-based constraints that make the expert trajectories feasible and approximately locally optimal by enforcing KKT optimality conditions in the reduced state space. We then plan with the learned constraints under uncertainty, using robustness margins derived from accumulated prediction error, and provide a theoretical analysis of how keypoint perception error and learned dynamics error affect the recovered constraints. We evaluate KCL on visual demonstrations from navigation to rigid/deformable manipulation, showing that it recovers multi-stage geometric constraints and outperforms baselines in downstream safe planning.
Explore related subjects
Keep this discovery
Explore connections, maps & timelines
Zheng Qiu, Zhouyu Zhang, Ninghan Zhong, Glen Chou. 2026-10-13. Learning Keypoint Constraints from Visual Demonstrations for Safe Motion Planning. https://searcharxiv.com/abs/2610.19295
Citations retain the original authors and DOI and include this site’s record URL. If this discovery service contributed to your research, you can also cite Searcharxiv by Indiciae: BibTeX · RIS.