1
2
Recent works on dynamic neural field reconstruction assume the input from synchronized multi-view videos whose poses are known. The input constraints often are not satisfied in real-world setups, making the approach impractical. We demonstrate that unsynchronized videos from unknown poses can generate dynamic neural fields as long as the videos capture human motion. Humans are one of the most common dynamic subjects captured in videos whose poses can be estimated using state-of-the-art libraries. While noisy, the estimated human shape and pose parameters provide a decent initialization point to start the highly non-convex and under-constrained problem of training a consistent dynamic neural representation. Given the pose and shape parameters of humans in individual frames, we formulate methods to calculate the time offsets between videos, followed by camera pose estimations that analyze the 3D joint positions. Then, we train the dynamic neural fields employing multiresolution grids while we concurrently refine both time offsets and camera poses. The setup still involves optimizing many parameters, therefore, we introduce a robust progressive learning strategy to stabilize the process. Experiments show that our approach achieves accurate spatiotemporal calibration and high-quality scene reconstruction in challenging conditions.
From unsynchronized videos with unknown camera poses, we aim to calibrate accurate time offsets and camera poses while reconstructing dynamic 3D scenes. We first extract human motion from each video independently. Then we estimate time offsets and global camera poses by aligning human motions (Initialization stage). We further refine both camera poses and time offsets by jointly optimizing them with dynamic NeRF with progressive training (Refinement stage).
Baseball
Office1
Office2
Office3
Tennis
Our initialization stage estimates good initial camera poses by aligning human motions (first few frames of videos). Then our refinement stage of joint optimization of calibration parameters and dynamic NeRF produces near-perfect camera poses.
Baseball
Office1
Office2
Office3
Tennis
Baseball
Office1
Office2
Office3
Tennis
@article{Choi2024HCP,
author = {Choi, Changwoon and Kim, Jeongjun and Cha, Geonho and Kim, Minkwan and Wee, Dongyoon and Kim, Young Min},
title = {Humans as a Calibration Pattern: Dynamic 3D Scene Reconstruction from Unsynchronized and Uncalibrated Videos},
year = {2024},
journal = {arXiv preprint arXiv:2412.19089},
}