I have been working on a project at the university involving the reconstruction of the world as seen in a video. The goal was to see if I could implement a Structure from Motion approach and test it on a few challenging videos to see whether it works and if I had any good ideas on how to improve it. The outcome was this Report On Sparse Reconstruction From Video. The overall implementation succeeded, but there is still work to be done in improving the evaluation of the results and making the approach generally ‘better’.

 

Abstract: The goal of this experimentation is to do a sparse 3D reconstruction from videos without any prior knowledge on the camera calibration. Because of the large (and increasing) availability of online videos on websites such as youtube, it is interesting again to look at 3D reconstruction from video.The problem of estimating Structure from Motion (as this is called) has already been researched extensively in the past and but remains interesting to improve because of the large amount of data publicly available. With the use of this large set of online videos we would be able to automatically create 3D reconstructions of scenery all around the world. When combining this with meta-data such as title, description and GPS-coordinates, these reconstructions could be linked to actual places in the world, thus allowing for a searchable 3D earth.

To show off the most fancy results achieved with my little piece of software, here we go

Reconstruction of the road scene

Reconstruction of the road scene

This image is a snapshot taken from the application that has done the 3D reconstruction on a scene that was originally used to demonstrate the ACTS (Automatic Camera Tracking System) sofware, which was mainly created by Guofeng Zhang. Here is an image of the first frame of this video.

First frame of the video

First frame of the video

Some of the technology and terminology used in this project (to give an idea) are: Structure from Motion (SfM), Single- and Multiview Geometry, Epipolar Geometry, Triangulation, Homography, Fundamental Matrix, SIFT Feature Detection, Sparse Bundle Adjustment (SBA), Plane Fitting using RANSAC (Random Sampling)

If you are interested (for educational purposes) in the software implementation, feel free to send me a message.