Pose6DAug: Physically Plausible Multi-view Object Swapping for Robot Data Augmentation

Given a successful episode, Pose6DAug swaps only the manipulated object in 3D, which preserves the original action trajectory, grasp contact, and cross-view geometry. This enables synthesizing new, physically grounded demonstrations.

Before

Original successful episode (source object)

After

Augmented episode (target object swapped in 3D)

Abstract

Vision-language-action (VLA) policies have shown strong potential for general-purpose manipulation, yet they often fail on novel, out-of-distribution objects whose appearance or geometry deviates from the training distribution. The standard remedy is to collect multi-view teleoperation data for every failure case, but this scales poorly in both cost and time. We introduce Pose6DAug, a failure-driven data augmentation framework that turns a policy's own successful episodes into targeted demonstrations for its failure modes, without any new data collection. Our key insight is that each successful episode already encodes a physically valid action trajectory together with calibrated multi-view observations. By swapping only the manipulated object while preserving this trajectory, we obtain new and physically grounded demonstrations. However, naive 2D video editing breaks multi-view consistency and physical plausibility, particularly under heavy occlusion and egocentric viewpoints. Our method instead operates directly in 3D, anchoring the target object with an explicit mesh driven by a temporally coherent 6D pose trajectory, ensuring geometrically consistent renderings across all camera views. Fine-tuning a VLA on data augmented by our method improves success rates by 16.5% relative to the state-of-the-art baseline on novel objects, while preserving in-distribution performance.

Method

Pose6DAug overview — **Failure-driven object-swap augmentation.** From a policy's failures on novel objects, we retrieve successful in-distribution episodes and, preserving their action trajectory, replace the manipulated object with the target object via a 3D mesh driven by a 6D pose sequence under rigid coupling — rendering physically plausible, multi-view-consistent videos for fine-tuning.

Our pipeline decomposes object-centric augmentation into three stages: (1) target mesh reconstruction and 6D pose extraction, (2) object augmentation (rotation, translation along the approach axis, and rescaling), and (3) 3D mesh-pose guided video composition. Because every camera renders the same 3D mesh at a shared world-frame pose, multi-view consistency holds by construction — in contrast to per-view 2D editing, which introduces cross-camera discrepancies.

**Augmentation details.** Combinations of rotation, flipping, translation, and vertical rescaling keep the swapped mesh within a graspable range of the gripper, ensuring physically plausible contact after substitution.

Geometry-guided rendering

Inpainted background + mesh rendered along the 6D pose trajectory

Consistent rendering across exocentric & egocentric views

Quantitative Results

Base model: GR00T-1.5 on the RoboCasa365 Counter-to-Cabinet task.

**Table 1. Performance on failure episodes** (instances the base policy never solved). Success Rate (%) and Turnover Ratio (%), split into in-distribution (ID) and out-of-distribution (OOD); averaged over 3 seeds. Higher is better.
Method	Properties		Success Rate (%)			Turnover Ratio (%)
Method	Multi-view	Sim-free	ID	OOD	Avg.	ID	OOD	Avg.
Base Policy	–	–	0.0	0.0	0.0	0.0	0.0	0.0
MimicGen	✓	✗	14.7	17.3	15.8	16.4	18.1	17.2
VACE	✗	✓	12.8	21.2	16.4	14.9	22.4	18.2
Pose6DAug (Ours)	✓	✓	21.2	24.7	22.8	24.6	24.2	24.5

**Table 2. Augmentation-only training on the 8 hardest unseen objects** (20 episodes each). Trained purely on synthesized demonstrations; class recovery = at least one success per instance.
Method	Success Rate	Objects Recovered
Base Policy	15/160 (9.4%)	–
MimicGen	9/160 (5.7%)	2 / 8
VACE	24/160 (15.0%)	5 / 8
Pose6DAug (Ours)	34/160 (21.2%)	7 / 8

Qualitative Results

Comparison with baselines

VACE

Ours

Object-swap augmentations (three camera views)

Teapot → Teapot

Cupcake → Cupcake

Qualitative comparison across views — **Qualitative comparison** across left/right (exocentric) and wrist (egocentric) views. Red circles mark multi-view inconsistencies; green rectangles mark physically implausible trajectories. Pose6DAug stays consistent across all views.

BibTeX

@article{lee2026pose6daug,
  title   = {Pose6DAug: Physically Plausible Multi-view Object Swapping
             for Robot Data Augmentation},
  author  = {Lee, Jonghoon and Park, Seong Hyeon and Jeon, Byungwoo
             and Lee, Minha and Shin, Jinwoo},
  journal = {arXiv preprint arXiv:2606.20118},
  year    = {2026}
}