Video-Grounded Verification for Long-Horizon Visual Imitation Learning

Visual imitation learning aims to translate a human demonstration video into an action plan or robot-executable code that enables a robot to faithfully reproduce the demonstrated behavior. Recent approaches based on vision-language models (VLMs) rely on these models’ ability to translate visual observations into action plans or codes. However, in long-horizon visual imitation learning, errors in temporal ordering, spatial grounding, or plan–code consistency can accumulate over time, i.e., compounding errors, resulting in significant deviations from the intended behavior. To address this problem, we propose LongVIL, a framework that introduces video-grounded verification into both plan and code generation. LongVIL first generates a structured action plan and verifies each step against the demonstration video through segment-level temporal consistency and end-frame spatial grounding. It then translates the verified plan into robot-executable code and verifies whether each code step faithfully implements the verified action plan. We further introduce LongVILBench, a benchmark for long-horizon demonstration-to-code imitation with 150 tabletop manipulation tasks and 300 human demonstration videos recorded under two visual conditions, featuring sequences of up to 18 steps, six spatial relation types, and three difficulty levels. Experiments on LongVILBench and SeeDo show that LongVIL consistently improves long-horizon visual imitation by reducing temporal, spatial, and plan–code errors.