Abstract

This thesis presents the research that has been done on a framework for people tracking in 3D, which is under the supervision of Nico van der Aa and Robby T. Tan. We developed this framework as a potential product for Noldus IT to evaluate human behaviour in the Restaurant of the Future. We focus on three different aspects in the framework: steering pan-tilt-zoom cameras, calibrating these cameras and detecting people and estimating their body poses.

We evaluated the functionality of pan-tilt-zoom cameras to see whether they can be used to their full potential in a tracking framework. We concluded that the image quality, even when panning or tilting at a speed which captures walking people, is sufficient for pose estimation. A major disadvantage of the PTZ camera is that its accuracy is not high enough for it to be used in our framework. The camera can be in a different position and produce different images even though it has been sent to the same pan-tilt-zoom state.

We researched the calibration of static cameras using Zhang’s method [1] and concluded that the convergence of internal parameters is questionable. After that, we looked at auto-calibration techniques such as [2, 3, 4, 5], to keep the cameras calibrated throughout their on-line pan, tilt and zoom movements. We did not find an auto calibration method immediately suitable for our framework but we did find interesting approaches that could potentially fulfill this task. In order for this to work we would have to create a 3D model of the scene beforehand.

In order to estimate the poses of the people in the scene, we made use of the Pictorial Structures framework presented in [6]. We evaluated the accuracy of the method on different datasets and identified its strengths and weaknesses. We showed that the amount of correctly estimated bodyparts on different datasets is around 55\%. Though, the method is too slow to be used in our real-time framework as it takes around 90 minutes on an average PC for a pose estimation on the images produced by our PTZ cameras. We showed that the method could be sped up around 60 times using a faster detection algorithm, at the cost of around 18\% accuracy. Still, the pose estimation will take more than a minute thus other speed ups will have to be found to be able to use it in our framework.

System Pipeline

Steering

This video demonstrates automatic pan-tilt-zoom steering over an IP network based on tracking results.

Calibration

Keeping cameras calibrated throughout an online panning, tilting and zooming stage, in order to track people and keep them in the view, requires more accurate steering than is possible with state-of-the-art PTZ cameras. A difference as small as between these images:

results in a 3D reconstruction as far off as seen here:

Pose Estimation

Using a Windows version of the available code from [2], we obtained the following results:

References

[1] Z. Zhang. – A Flexible new technique for camera calibration. IEEE PAMI, 22:1330-1334, 1998.
[2] B. Micusik and T. Pajdla. – Simultaneous surveillance camera calibration and foot-head homology estimation from human detections. CVPR, 2010.
[3] A. Sinha and M. Pollefeys. – Pan-tilt-zoom camera calibration and high-resolution mosaic generation. Computer Vision and Image Understanding, 103:170-183, 2006.
[4] S. Sinha, M. Pollefeys, and L. McMillan. – Camera network calibration from dynamic silhouettes. CVPR, 2004.
[5] L. Wang, S. You, and U. Neumann. – PTZ camera calibration for augmented virtual environments. In IEEE International Conference on Multimedia and Expo, 2009
[6] M. Andriluka, S. Roth, and B. Schiele. – Pictorial structures revisited: People detection and articulated pose estimation. CVPR, 2009

Official master thesis project page

As a small part of my graduation I worked with PTZ cameras. To demonstrate steering over HTTP/IP using the Axis VAPIX API I have created a small application that can track an orange object (a Wuppie) and steers the camera accordingly.

PTZ Camera is tracking orange object

The whole video can be seen here:  http://vimeo.com/25667397. It might be interesting for anyone trying to steer PTZ cameras through c++ code. This one works on Microsoft Windows using COM-interfaces provided by Axis. It uses CURL to send commands over HTTP and OpenCV to visualize.

Source code is available here

How to compile the program

At first, download the code and experiments from http://www.d2.mpi-inf.mpg.de/node/381. I am currently running Microsoft Windows, so I created a Mint Linux virtual machine using Oracle VM VirtualBox. A fresh install is always good to see what dependencies are needed. Following the README.txt accompanying the code I first installed QT, Boost, Matlab and Google Protocol Buffers. The PNG libraries, libboost-program-options-dev and libstdc++5 were also needed but not stated in the README.

sudo apt-get install libqt4-dev
sudo apt-get install libpng++-dev
sudo apt-get install libboost-program-options-dev
sudo apt-get install libstdc++5
sudo apt-get install libstdc++5-dev

Matlab version is 7 R14. Google Protocol Buffers version 2.3.0.

Following the README I continued to create symlinks to the library and include directories like this:

  ln -s /home/frank/andriluka/matlab/extern/include/ include_mat
  ln -s /home/frank/andriluka/matlab/bin/glnx86/ lib_mat
  ln -s /usr/local/include/google/protobuf/ include_pb
  ln -s /usr/local/lib/ lib_pb

The installed directory for the application is /home/frank/andriluka/partapp-r2 and these commands were run from there. Next step is to go into the src/libs and src/apps dir and run qmake; make. This was not working right away as there were some problems with the code. This is what I ran into and how I solved it:

Problem I:
../libKMA2/gauss_iir/../ImageContent/imageContent.h:25:17: fatal error: png.h:
No such file or directory

Solution:
sudo apt-get install libpng++-dev

Problem II:
../libAdaBoost/AdaBoost.h:128:20: error: ‘uint’ was not declared in this scope

Solution:
#include <sys/types.h>
In Adaboost.h:22 

Problem III:
../libMatlabIO/matlab_cell_io.hpp:65:17: error: ‘mwIndex’ was not declared in this scope
matlab_io.hpp:87:5: error: ‘mwSize’ was not declared in this scope

Solution (hack?):
In matlab file matrix.h
#define mwSize int
#define mwIndex int

Problem IV:
../libAnnotation/annorect.h: In member function ‘int AnnoRect::w() const’:
../libAnnotation/annorect.h:72:38: error: ‘abs’ was not declared in this scope

Solution:
In annorect.h:
#include <stdlib.h>

Problem V:
ImageContent/ImageContent.cpp:383:21: error: ‘atoi’ was not declared in this scope

Solution:
In imageContent.h:
#include <stdlib.h>

Problem VI:
kmaimagecontent.h:74:52: error: ‘strcpy’ was not declared in this scope

Solution:
kmaimagecontenmat.h
#include <string.h>

Probleem VII:
annotation.cpp:251:69: error: ‘stable_sort’ was not declared in this scope

Solution:
#include <algorithm>
In annotation.h 

Probleem VIII:
xmlhelpers.cpp:48:33: error: ‘atoi’ was not declared in this scope

Solution:
xmlhelpers.h:
#include <string.h>

Problem IX:
/usr/bin/ld: cannot find -lboost_program_options

Solution:
; sudo apt-get install libboost-program-options-dev

Problem X:
../../../lib/Release/libPartDetect.so: undefined reference to `mxCreateCellArray'
../../../lib/Release/libPartDetect.so: undefined reference to `mxIsCell'
../../../lib/Release/libMatlabIO.so: undefined reference to `mxCreateNumericArray@v7.0'
../../../lib/Release/libMatlabIO.so: undefined reference to `mxIsSingle@v7.0'
../../../lib/Release/libPartDetect.so: undefined reference to `mxGetCell'
../../../lib/Release/libMatlabIO.so: undefined reference to `matPutVariable@v7.0'
../../../lib/Release/libMatlabIO.so: undefined reference to `mxGetPr@v7.0'
../../../lib/Release/libPartDetect.so: undefined reference to `matPutVariable'
../../../lib/Release/libMatlabIO.so: undefined reference to `mxDestroyArray@v7.0'
../../../lib/Release/libMatlabIO.so: undefined reference to `matClose@v7.0'
../../../lib/Release/libMatlabIO.so: undefined reference to `matOpen@v7.0'
../../../lib/Release/libMatlabIO.so: undefined reference to `mxGetDimensions@v7.0'
../../../lib/Release/libMatlabIO.so: undefined reference to `matGetNextVariable@v7.0'
../../../lib/Release/libPartDetect.so: undefined reference to `mxSetCell'
../../../lib/Release/libMatlabIO.so: undefined reference to `mxGetNumberOfDimensions@v7.0'
../../../lib/Release/libPartDetect.so: undefined reference to `mxCreateNumericArray'
../../../lib/Release/libPartDetect.so: undefined reference to `mxCalcSingleSubscript'
together with this:
/usr/bin/ld: warning: libmx.so, needed by ../../../lib/Release/libMatlabIO.so, not found (try using -rpath or -rpath-link)
/usr/bin/ld: warning: libmat.so, needed by ../../../lib/Release/libMatlabIO.so, not found (try using -rpath or -rpath-link)

Solution:
I think the linker has a problem because these things are not in the library path. When I add include dir to it
; LD_LIBRARY_PATH=/home/frank/andriluca/matlab/extern/include it gives same errors
When I add matlab lib dir, as I think it should be since this dir has got the libmat.so in it, I get:
export LD_LIBRARY_PATH=/home/frank/andriluca/matlab/bin/glnx86/
/usr/bin/ld: error while loading shared libraries: libstdc++.so.5: cannot open shared object file: No such file or directory

So this is a new problem. I get this when I run any program too, gvim gives same error... This path must be wrong. Nope, I think it's a libstdc++ version
dependency problem. libmat and libmx from the matlab are depending on libstdc++.so.5 and I think I'm using 6.

New plan:
; sudo apt-get install libstdc++5
; export LD_LIBRARY_PATH=/home/frank/andriluca/matlab/bin/glnx86/
It seems to compile with warning about 5/6 version problem now.

Problem XI:
Now a system wide problem is this:
/home/frank/andriluca/matlab/bin/glnx86/libz.so.1: no version information
available (required by /usr/lib/libbfd-2.21.0-system.20110327.so)
So I think that the libs in the matlab thing are interfering with my system now. 

Solution:
I will remove LD_LIBRARY_PATH for now again and only add it when compiling this program

The program is now working. Go into the code_test directory to test (following the README):

Test the compiled code
- Issue the following commands in the code_test subdictory:
   ../run_partapp.sh --expopt ./expopt/exp_code_test.txt --part_detect --find_obj
   ../run_partapp.sh --expopt ./expopt/exp_code_test.txt --eval_segments
   This will estimate part positions on several example images and visualize the results.
   Compare the images in the ./log_dir/exp_code_test/part_marginals/seg_eval_images with the images in ./images_result

After comparing the images it seems they are very similar meaning the code works!

How to run the program on your own data

I am not sure how to do this yet. The commandline interface is not very user friendly. It is said that the command

   ../run_partapp.sh --expopt ./expopt/exp_code_test.txt --part_detect --find_obj

should be run in the code_test directory in order to create results in te directory log_dir. Though, the data that is in log_dir is being used by the application. Let’s start from the beginning:

; ../run_partapp.sh
argc: 2
argv[0]: ../bin/Release/partapp
reading command line parameters
command line options::
  --help                 help message
  --expopt arg           experiment parameters
  --train_class          train part detector
  --train_bootstrap      train part detector, adding hard negative samples
  --pidx arg             0-based index of the part
  --bootstrap_prep       create bootstrapping dataset (crops of objects with
                         some background)
  --bootstrap_detect     run previously trained classifer on bootstrapping
                         images
  --bootstrap_showrects  show top negatives on bootstrapping images
  --first arg            index of first image
  --numimgs arg          number of images to process
  --part_detect          run part detector on the test set
  --find_obj             find_obj
  --pc_learn             estimate prior on part configurations with maximum
                         likelihood
  --save_res             save object recognition results in al/idl formats
  --eval_segments        evaluate part localization according to Ferrari's
                         criteria
  --distribute           split processing into multiple chunks, keep track of
                         the chunks which must be processed
  --ncpu arg             number of chunks
  --batch_num arg        current chunk
  --vis_parts            visualize ground-truth part positions

From here, let’s start to try and run the part detection trainer on a set of images. The images we will be training on are the training images for the Ramanan people set, they are stored in the folder ramanan_people_train_h200. I expect to start the program to train it’s appearance model on these 200 images.

/home/frank/andriluka/partapp-r2/code_test2
; ../run_partapp.sh --train_class
'expopt' parameter missing

Let’s see what is needed in this experiment options file. When looking at the other expopt files in partapp-experiments-r2/expopt , these are the options that I found:

; cat abcparams4_trainall_rounds500_shape5_sparse_jb_radius24.txt
boosting_rounds: 500
desc_size: 24
desc_step: 12
window_desc_step_ratio: 1

num_train_pos: 212
num_train_jitter: 1138
num_train_neg: 1000
num_train_bootstrap: 350

feature_type: "SHAPEv5"

; cat exp_buffy_hog_detections.txt
train_dataset: "./ramanan_train_upperbody_h180.al"
validation_dataset: "./ramanan_train_upperbody_h180.al"
test_dataset: "./buffy_hog_detections.al"

part_conf: "./part_conf6_buffy_root1.txt"
abc_param: "./abcparams4_trainall_rounds500_shape5_sparse_jb_radius24.txt"
log_dir: "./log_dir"

min_object_scale: 1.0
max_object_scale: 1.0
num_scale_steps: 1

min_part_rotation: -180
max_part_rotation: 180
num_rotation_steps: 24

flip_orientation: false

scorehist_nbins: 200
scorehist_def_pw: 10
scorehist_def_ph: 10

object_height_width_ratio: 1.2

compute_part_marginals: true
num_pose_samples: 0
strip_border_detections: 0.3

I am now trying to use the first file to see if that is the data that is needed for training. Well…. no:

libprotobuf ERROR google/protobuf/text_format.cc:169] Error parsing text-format 
ExpParam: 1:16: Message type "ExpParam" has no field named "boosting_rounds".

So let’s try the other file? This is not the right approach… Let’s look in the code and use expopt/exp_code_test.txt as configuration file. Oh man I would kill for a better README..

<post not complete yet>

Abstract: The goal of this experimentation is to do a sparse 3D reconstruction from videos without any prior knowledge on the camera calibration. Because of the large (and increasing) availability of online videos on websites such as youtube, it is interesting again to look at 3D reconstruction from video.The problem of estimating Structure from Motion (as this is called) has already been researched extensively in the past and but remains interesting to improve because of the large amount of data publicly available. With the use of this large set of online videos we would be able to automatically create 3D reconstructions of scenery all around the world. When combining this with meta-data such as title, description and GPS-coordinates, these reconstructions could be linked to actual places in the world, thus allowing for a searchable 3D earth.

To show off the most fancy results achieved with my little piece of software, here we go

Reconstruction of the road scene

This image is a snapshot taken from the application that has done the 3D reconstruction on a scene that was originally used to demonstrate the ACTS (Automatic Camera Tracking System) sofware, which was mainly created by Guofeng Zhang. Here is an image of the first frame of this video.

First frame of the video

Some of the technology and terminology used in this project (to give an idea) are: Structure from Motion (SfM), Single- and Multiview Geometry, Epipolar Geometry, Triangulation, Homography, Fundamental Matrix, SIFT Feature Detection, Sparse Bundle Adjustment (SBA), Plane Fitting using RANSAC (Random Sampling)

If you are interested (for educational purposes) in the software implementation, feel free to send me a message.

If you want to see how this was done, check the bspline-drawing.zip here (don’t expect programming beauty).

Some good reading on this can be found here in Thomas Sederbergs’ notes and a good applet with demonstration here http://www.ibiblio.org/e-notes/Splines/Bezier.htm.

So, onto the images!

Compositor source image (photo by GrandiJoos)

Compositor destination image

Composited image

It can be seen that the input image got stretched a lot on the bottom part. There is a preciseness parameter for which 1/preciseness determines how many times the output width (height) can be larger then the input width (height). Check out the source code for details on this. Also, try adding your own images!

Download the source code here, it has been tested on Linux only.