GIF: Agentic Generation of Interactive and Functional Object Compositions for Robot Learning

Long Xu^*,1,3,Zhiqi Zhang^*,1,2,Mi Yan^*,1,2,Shengliang Deng^1,4,Chong Xia^1,5,Mingyu Dong^1,5,Jiayi Chen^1,2,Jiangran Lyu^1,2,Fei Gao^1,3,Zhizheng Zhang^†,1,He Wang^†,1,2

¹ Galbot ² Peking University ³Zhejiang University ⁴ The University of Hong Kong ⁵Tsinghua University

^* Equal contribution · ^† Corresponding authors

CoRL 2026

P Paper⌘ Code coming soon▶ Video

Overview of GIF. Left: diverse, interactive and functional task compositions, shown individually and arranged in a kitchen scene. Cyan labels match each composition to its placement in the scene; gray labels identify the independently interactive components within each composition (e.g., a pot and its lid). Right: the generated assets support robot training data synthesis in simulation (a) and deployment of the trained policy on a real robot (b).

Abstract

Generate contact-precise scenes for interaction, at scale.

Robot manipulation foundation models require scalable evaluation and data generation across diverse scenarios, with simulation providing an environment for both. Automated scene generation offers a promising path, yet prior work has largely emphasized coarse-grained scene layouts rather than fine-grained functional object compositions.

Motivated by this gap, we present GIF, an agentic Generation framework for Interactive and Functional object compositions. In this framework, we recast this problem as disentangled reconstruction followed by relative pose recovery. CoGen produces instance-disentangled meshes with coarse initial poses leveraging complementary strengths of 2D and 3D generative models. GPRM refines the relative pose under joint geometric and physical guidance, and a VLM verifier selects the candidate that best matches the structured specification. We further construct a benchmark spanning eight representative contact-geometry classes and compare with state-of-the-art generators; GIF improves both asset quality and relation matching, while reducing collision rate to below 1%. Finally, we synthesize data for policy learning, revealing diversity scaling in both simulation and real-world deployment.

Video walkthrough

From asset generation to real-robot deployment.

Method

The GIF pipeline from multimodal input canonicalization through CoGen, GPRM, VLM verification, and policy training

Canonicalize

An agent turns a language or image prompt into a foreground reference and a structured scene specification.

CoGen

Complementary direct reconstruction and erase-and-regenerate branches produce clean, disentangled 3D assets.

GPRM

Geometry- and physics-guided modules recover contact-precise, collision-free relative poses.

Verify

A vision-language verifier inspects simulated rollouts and selects the scene that best matches the specification.

Learn

Verified scenes seed diverse trajectories for VLA training and direct real-robot deployment.

Instance-disentangled meshes
Occlusion-aware hidden geometry
Independently interactive objects
Precise functional contact

Benchmark

8 representative object-interaction relations. 120 scenes.

Grounded in real human manipulation from Ego4D, the benchmark covers 8 representative pairwise contact-geometry relations: containment, capping, resting, leaning, hooking, slotting, spanning, and pegging. Each relation includes 3 manipulation tasks, each with 5 semantic variants: 24 tasks and 120 scenes in total.

C.1

Containment

Flower–vase · Pen–organizer · Pot–stand

15 variants ↗

C.2

Capping

Pot–lid · Teacup–lid · Bottle–cork

15 variants ↗

C.3

Resting

Hat–head · Coffee machine · Egg–carton

15 variants ↗

C.4

Leaning

Phone–stand · Shoe–stand · Steamer–lid

15 variants ↗

C.5

Hooking

Cup–rack · Hat–rack · Spoon–hook

15 variants ↗

C.6

Slotting

Plate–rack · Bottle–rack · Book–stand

15 variants ↗

C.7

Spanning

Fork–plate · Brush–stand · Ladle–stand

15 variants ↗

C.8

Pegging

Toss ring · Jewelry holder · Mug–rack

15 variants ↗

Trajectory synthesis

Verified compositions become executable robot data.

The same pipeline scales from a single contact-rich rollout to parallel generation across object identities, appearances, and functional relations.

One contact-precise rolloutCover a pot with a lid

Tasks synthesized in parallelServing and object placement

Different relations, same pipelineInsertion, slotting, stacking, and more

Real-robot results

The generated diversity reaches the real robot.

GIF turns functional compositions into varied simulation rollouts, then uses the resulting data to train policies that execute the same instruction on physical hardware.

Serve sausage in a paper box

Generated containers and food assets produce varied training rollouts before the learned policy is deployed on hardware.

SimulationGenerated task assets

Real robotPolicy deployment

Cover a pot with a lid

Asset diversity changes the geometry seen in simulation while preserving the same functional relation required by the real task.

SimulationGenerated task assets

Real robotPolicy deployment

Real-robot and simulation success increase as the number of GIF-generated task assets grows from 2 to 40

The payoff

Generated diversity transfers to the real robot.

As the asset budget grows from 2 to 40, both real-world tasks improve monotonically.

Serve sausage0.20→0.90

Cover pot with lid0.00→0.75

Real-robot success rate · 2 → 40 generated task assets

Extension

From object pairs to multi-object scenes.

Iterative expansion from an object pair to a multi-object scene, with image generation and simulation reconstruction at each step — Preliminary extension to a multi-object scene.

Citation

Build on GIF.

Accepted at CoRL 2026. Read the manuscript, explore the code, or cite the conference paper.

Read the paperCode coming soon

@inproceedings{xu2026gif,
  title   = {GIF: Agentic Generation of Interactive and
             Functional Object Compositions for Robot Learning},
  author  = {Xu, Long and Zhang, Zhiqi and Yan, Mi and
             Deng, Shengliang and Xia, Chong and Dong, Mingyu and
             Chen, Jiayi and Lyu, Jiangran and Gao, Fei and
             Zhang, Zhizheng and Wang, He},
  booktitle = {Conference on Robot Learning (CoRL)},
  year    = {2026}
}