RESEARCH
3D motion reconstruction from monocular video without 3D ground truth
Overview
This research aims to learn how things move in 3D from the 2D motion seen in video, without 3D ground truth such as motion capture. The learned model is then used to recover 3D motion from ordinary videos taken with a still camera.
Collecting 3D ground truth takes special equipment and a lot of effort. Instead, I start from rough 3D estimated by a classical method (NRSfM) on videos where the camera moves only a little, and learn from there. This rough 3D contains errors. The hard part is learning without memorizing those errors, by checking against the 2D observations and correcting as it goes.
Method

STEP 1
Video from a slightly rotating camera
Training uses videos taken while the camera rotates only a little. Points on the body surface are tracked in 2D.

STEP 2
Rough 3D from a classical method
NRSfM (non-rigid structure from motion) gives a rough 3D trajectory for each point.

STEP 3
Learn motion with a diffusion model
A Transformer diffusion model is trained with each point's trajectory as one token. It alternates between re-estimating the 3D that fits the 2D and retraining on it (EM).

STEP 4
Recover motion from a still camera
The model estimates 3D motion from monocular videos with a still camera that were not used in training.
Research question
How small can the camera rotation in the training videos get?
The smaller the camera rotation, the less a classical method can pin down depth. I am studying how far a diffusion model with EM can push past this limit.
How it differs from prior work
Akhter et al. 2008
Nonrigid Structure from Motion in Trajectory Space(NeurIPS 2008)
What it doesRepresents point trajectories with a fixed set of waves (DCT) to recover 3D from 2D
In this researchUses a learned generative model instead of fixed waves
Valmadre & Lucey 2012
General Trajectory Prior for Non-Rigid Reconstruction(CVPR 2012)
What it doesStudied how the amount of camera rotation relates to the accuracy of 3D reconstruction
In this researchUses the limit that small rotations cannot be reconstructed as the baseline to compare against
Rozet et al. 2024 (DiEM)
Learning Diffusion Priors from Observations by Expectation Maximization(NeurIPS 2024)
What it doesLearns a diffusion model with EM from degraded observations only
In this researchDeals with 3D motion and starts from NRSfM
Ji et al. 2024
Unsupervised 3D Pose Estimation with Non-Rigid Structure-from-Motion Modeling(WACV 2024)
What it doesBuilds a diffusion model into NRSfM
In this researchTrains the diffusion model without 3D ground truth
Datasets
Points on the human body surface
People in dance videos
Cloth simulation
A real octopus arm
Faces (planned next)
How I work
01
I write down the conditions to compare and the criteria for judging before running an experiment.
02
I check with independent code reviews and recalculation that 3D ground truth does not leak into training or inference.
03
I keep results that did not work, as results that show the limits.
Current status
In progress. Some conditions show improvement, and some metrics still do not beat a simple baseline (placing depth flat).
NEXT — 2027.04From April 2027, I plan to join the master's program in Information and Communications Engineering at the School of Engineering, Institute of Science Tokyo, in the Kenji Suzuki Lab.
Kenji Suzuki Lab