Language日本語English

Back to top

RESEARCH

3D motion reconstruction from monocular video without 3D ground truth

Computer Vision Lab, Kyoto Institute of Technology

Overview

This research aims to learn how things move in 3D from the 2D motion seen in video, without 3D ground truth such as motion capture. The learned model is then used to recover 3D motion from ordinary videos taken with a still camera.

Collecting 3D ground truth takes special equipment and a lot of effort. Instead, I start from rough 3D estimated by a classical method (NRSfM) on videos where the camera moves only a little, and learn from there. This rough 3D contains errors. The hard part is learning without memorizing those errors, by checking against the 2D observations and correcting as it goes.

Method

  1. Points on the body surface tracked in 2D, and the camera rotating slightly around the body

    STEP 1

    Video from a slightly rotating camera

    Training uses videos taken while the camera rotates only a little. Points on the body surface are tracked in 2D.

  2. The 2D points and the rough 3D points from classical NRSfM, seen from the side

    STEP 2

    Rough 3D from a classical method

    NRSfM (non-rigid structure from motion) gives a rough 3D trajectory for each point.

  3. Diagram of a diffusion model that turns noise into 3D point trajectories, with each trajectory as one token

    STEP 3

    Learn motion with a diffusion model

    A Transformer diffusion model is trained with each point's trajectory as one token. It alternates between re-estimating the 3D that fits the 2D and retraining on it (EM).

  4. Points in an image from a still camera, and their recovered 3D positions seen from the side

    STEP 4

    Recover motion from a still camera

    The model estimates 3D motion from monocular videos with a still camera that were not used in training.

Research question

How small can the camera rotation in the training videos get?

The smaller the camera rotation, the less a classical method can pin down depth. I am studying how far a diffusion model with EM can push past this limit.

How it differs from prior work

Datasets

How I work

  1. 01

    I write down the conditions to compare and the criteria for judging before running an experiment.

  2. 02

    I check with independent code reviews and recalculation that 3D ground truth does not leak into training or inference.

  3. 03

    I keep results that did not work, as results that show the limits.

Current status

In progress. Some conditions show improvement, and some metrics still do not beat a simple baseline (placing depth flat).

NEXT — 2027.04From April 2027, I plan to join the master's program in Information and Communications Engineering at the School of Engineering, Institute of Science Tokyo, in the Kenji Suzuki Lab.

Kenji Suzuki Lab