RealSimLoop: Online Real-to-Sim Adaptation via Differentiable Reduced-Order Simulation with Vision Feedback

Abstract

Observing a deformable object reveals its surface motion, but many applications also need its internal stress, material properties, and interaction forces. RealSimLoop combines differentiable physical simulation with vision feedback to keep a simulated object aligned with its real-world behavior.

A neural reduced-order model makes both simulation and gradient computation more efficient. Coupled with differentiable 3D Gaussian Splatting, it allows camera observations to guide material-parameter updates within a sliding time window. This continuous adaptation tracks changing stiffness and compensates for simulation mismatch. We demonstrate the framework through robotic manipulation, multi-material structures, temperature-dependent stiffness tracking, force prediction, and stress visualization.

Method Overview

RealSimLoop pipeline connecting reduced-order simulation, geometry mapping, differentiable rendering, and a sliding window of vision feedback.

The forward pass simulates deformation in a neural subspace, maps the deformed mesh to 3D Gaussians, and renders the scene. Image differences accumulated over recent frames provide gradients for updating material properties and Gaussian orientations, closing the loop between observation and simulation.

Efficient physical simulation

A learned neural subspace reduces the number of simulation variables and accelerates forward and backward computation.

Dense visual feedback

Differentiable rendering connects camera images to physical parameters through the simulated geometry.

Continuous adaptation

A sliding window of recent observations supports online updates as material properties and interactions change.

Applications and Results

Deformable Object Manipulation

Two robot arms twist and stretch a soft bar while visual feedback adapts its simulated material properties. After adaptation, shape-control commands optimized in simulation are applied to the physical robot setup. The resulting geometry more closely matches the scanned target, and the simulation also recovers the internal von Mises stress field.

Three dual-arm robot target shapes, their simulated results, scan-distance distributions, and reconstructed stress fields. — Physical target shapes and adapted simulations are compared using scanned point clouds. The error distributions and stress maps show the geometric and physical information available after adaptation.

Multi-Material Identification and Stress Reconstruction

A cable-driven Eiffel Tower structure contains two silicone materials with different stiffnesses. RealSimLoop estimates their Young's moduli separately, aligns the simulated deformation with visual observations, and renders the reconstructed stress field from new viewpoints.

Multi-material Eiffel Tower: observed and simulated shapes, novel-view stress maps, stiffness convergence, and computation time. — Material estimates converge toward the two reference stiffnesses. The reduced-order implementation lowers computation time from 1.36 s to 0.18 s per frame in this experiment, a 7.56× speedup.

External force prediction

The reconstructed physical state also supports cable-force prediction. Below, predicted tensions are compared with force-sensor measurements across five target configurations.

Cable-driven target configurations and predicted versus measured forces for three cables. — Predicted and measured cable tensions across changing target configurations.

Tracking Temperature-Dependent Stiffness

A soft specimen combines a polycaprolactone truss with a silicone body. As it cools from 45°C to 27°C, its stiffness increases. Online adaptation follows the changing mechanical response, reflected in both the measured push force and the estimated Young's modulus.

Physical, thermal, and simulated views at three temperatures, with push-force and estimated-stiffness curves over time. — Representative frames during cooling, together with force measurements and the online stiffness estimate.

Online Adaptation to Changing Materials

In a controlled dynamic experiment, Young's modulus changes in steps from 10 to 40 kPa. The online estimate follows these changes, while an offline fit retains a single constant value. Continuous updates reduce deformation error as the material changes.

Deformations at four stiffness levels, stress fields over time, and online versus offline stiffness and error curves. — Deformation at different stiffness levels, stress reconstruction during impact, and the resulting parameter-tracking and simulation-error curves.

Evaluation

Computational Efficiency

The neural subspace accelerates online identification across the reported reduced-order experiments. Times below include forward simulation, gradient computation, and 3D Gaussian Splatting on an NVIDIA RTX 4090.

Average computation time per frame (paper, Table 1)
Experiment	Full space	Reduced order	Speedup
Material Bar	3.30 s	0.53 s	6.23×
Eiffel Tower	1.36 s	0.18 s	7.56×
Dinosaur	7.77 s	1.01 s	7.69×
Gingerbread Joy	1.46 s	0.58 s	2.52×

These are experiment-specific timings. The high-speed ball case uses full-space simulation because the reduced-order solver did not converge.

Vision feedback versus marker-based observations

The dinosaur experiment compares dense image feedback with sparse marker trajectories. Vision feedback produces a more stable material estimate and lower deformation error in this comparison.

Dinosaur experiment comparing visual and marker observations, spatial errors, stiffness estimates, and L2 error distributions. — Observation types, reconstructed deformations, material estimates, and error distributions for vision-based and marker-based adaptation.

Ablation: subspace dimension, window size, and camera count

Subspace dimension and window size trade computation time for reconstruction accuracy. Camera count also affects the cost and quality of visual feedback; the plots report these trends in the dinosaur experiment.

Ablation plots of L2 error and update time against subspace dimension, sliding-window size, and number of cameras. — Accuracy and update-time tradeoffs under different parameter settings.