sharing computer sciencey experimentations

As a small part of my graduation I worked with PTZ cameras. To demonstrate steering over HTTP/IP using the Axis VAPIX API I have created a small application that can track an orange object (a Wuppie) and steers the camera accordingly.

PTZ Camera is tracking orange object

The whole video can be seen here:  http://vimeo.com/25667397. It might be interesting for anyone trying to steer PTZ cameras through c++ code. This one works on Microsoft Windows using COM-interfaces provided by Axis. It uses CURL to send commands over HTTP and OpenCV to visualize.

At the moment I am working with a program created by Mykhaylo Andriluka that is able to detect people in images in order to estimate their poses using a method called Pictorial Structures. The program is downloadable from his website but is not easy to compile and run. Here is what I did in order to get it to work so that it might help someone else too.

How to compile the program

At first, download the code and experiments from http://www.d2.mpi-inf.mpg.de/node/381. I am currently running Microsoft Windows, so I created a Mint Linux virtual machine using Oracle VM VirtualBox. A fresh install is always good to see what dependencies are needed. Following the README.txt accompanying the code I first installed QT, Boost, Matlab and Google Protocol Buffers. The PNG libraries, libboost-program-options-dev and libstdc++5 were also needed but not stated in the README.

sudo apt-get install libqt4-dev
sudo apt-get install libpng++-dev
sudo apt-get install libboost-program-options-dev
sudo apt-get install libstdc++5
sudo apt-get install libstdc++5-dev

Matlab version is 7 R14. Google Protocol Buffers version 2.3.0.

Following the README I continued to create symlinks to the library and include directories like this:

  ln -s /home/frank/andriluka/matlab/extern/include/ include_mat
  ln -s /home/frank/andriluka/matlab/bin/glnx86/ lib_mat
  ln -s /usr/local/include/google/protobuf/ include_pb
  ln -s /usr/local/lib/ lib_pb

The installed directory for the application is /home/frank/andriluka/partapp-r2 and these commands were run from there. Next step is to go into the src/libs and src/apps dir and run qmake; make. This was not working right away as there were some problems with the code. This is what I ran into and how I solved it:

Problem I:
../libKMA2/gauss_iir/../ImageContent/imageContent.h:25:17: fatal error: png.h:
No such file or directory

Solution:
sudo apt-get install libpng++-dev

Problem II:
../libAdaBoost/AdaBoost.h:128:20: error: ‘uint’ was not declared in this scope

Solution:
#include <sys/types.h>
In Adaboost.h:22 

Problem III:
../libMatlabIO/matlab_cell_io.hpp:65:17: error: ‘mwIndex’ was not declared in this scope
matlab_io.hpp:87:5: error: ‘mwSize’ was not declared in this scope

Solution (hack?):
In matlab file matrix.h
#define mwSize int
#define mwIndex int

Problem IV:
../libAnnotation/annorect.h: In member function ‘int AnnoRect::w() const’:
../libAnnotation/annorect.h:72:38: error: ‘abs’ was not declared in this scope

Solution:
In annorect.h:
#include <stdlib.h>

Problem V:
ImageContent/ImageContent.cpp:383:21: error: ‘atoi’ was not declared in this scope

Solution:
In imageContent.h:
#include <stdlib.h>

Problem VI:
kmaimagecontent.h:74:52: error: ‘strcpy’ was not declared in this scope

Solution:
kmaimagecontenmat.h
#include <string.h>

Probleem VII:
annotation.cpp:251:69: error: ‘stable_sort’ was not declared in this scope

Solution:
#include <algorithm>
In annotation.h 

Probleem VIII:
xmlhelpers.cpp:48:33: error: ‘atoi’ was not declared in this scope

Solution:
xmlhelpers.h:
#include <string.h>

Problem IX:
/usr/bin/ld: cannot find -lboost_program_options

Solution:
; sudo apt-get install libboost-program-options-dev

Problem X:
../../../lib/Release/libPartDetect.so: undefined reference to `mxCreateCellArray'
../../../lib/Release/libPartDetect.so: undefined reference to `mxIsCell'
../../../lib/Release/libMatlabIO.so: undefined reference to `mxCreateNumericArray@v7.0'
../../../lib/Release/libMatlabIO.so: undefined reference to `mxIsSingle@v7.0'
../../../lib/Release/libPartDetect.so: undefined reference to `mxGetCell'
../../../lib/Release/libMatlabIO.so: undefined reference to `matPutVariable@v7.0'
../../../lib/Release/libMatlabIO.so: undefined reference to `mxGetPr@v7.0'
../../../lib/Release/libPartDetect.so: undefined reference to `matPutVariable'
../../../lib/Release/libMatlabIO.so: undefined reference to `mxDestroyArray@v7.0'
../../../lib/Release/libMatlabIO.so: undefined reference to `matClose@v7.0'
../../../lib/Release/libMatlabIO.so: undefined reference to `matOpen@v7.0'
../../../lib/Release/libMatlabIO.so: undefined reference to `mxGetDimensions@v7.0'
../../../lib/Release/libMatlabIO.so: undefined reference to `matGetNextVariable@v7.0'
../../../lib/Release/libPartDetect.so: undefined reference to `mxSetCell'
../../../lib/Release/libMatlabIO.so: undefined reference to `mxGetNumberOfDimensions@v7.0'
../../../lib/Release/libPartDetect.so: undefined reference to `mxCreateNumericArray'
../../../lib/Release/libPartDetect.so: undefined reference to `mxCalcSingleSubscript'
together with this:
/usr/bin/ld: warning: libmx.so, needed by ../../../lib/Release/libMatlabIO.so, not found (try using -rpath or -rpath-link)
/usr/bin/ld: warning: libmat.so, needed by ../../../lib/Release/libMatlabIO.so, not found (try using -rpath or -rpath-link)

Solution:
I think the linker has a problem because these things are not in the library path. When I add include dir to it
; LD_LIBRARY_PATH=/home/frank/andriluca/matlab/extern/include it gives same errors
When I add matlab lib dir, as I think it should be since this dir has got the libmat.so in it, I get:
export LD_LIBRARY_PATH=/home/frank/andriluca/matlab/bin/glnx86/
/usr/bin/ld: error while loading shared libraries: libstdc++.so.5: cannot open shared object file: No such file or directory

So this is a new problem. I get this when I run any program too, gvim gives same error... This path must be wrong. Nope, I think it's a libstdc++ version
dependency problem. libmat and libmx from the matlab are depending on libstdc++.so.5 and I think I'm using 6.

New plan:
; sudo apt-get install libstdc++5
; export LD_LIBRARY_PATH=/home/frank/andriluca/matlab/bin/glnx86/
It seems to compile with warning about 5/6 version problem now.

Problem XI:
Now a system wide problem is this:
/home/frank/andriluca/matlab/bin/glnx86/libz.so.1: no version information
available (required by /usr/lib/libbfd-2.21.0-system.20110327.so)
So I think that the libs in the matlab thing are interfering with my system now. 

Solution:
I will remove LD_LIBRARY_PATH for now again and only add it when compiling this program

The program is now working. Go into the code_test directory to test (following the README):

Test the compiled code
- Issue the following commands in the code_test subdictory:
   ../run_partapp.sh --expopt ./expopt/exp_code_test.txt --part_detect --find_obj
   ../run_partapp.sh --expopt ./expopt/exp_code_test.txt --eval_segments
   This will estimate part positions on several example images and visualize the results.
   Compare the images in the ./log_dir/exp_code_test/part_marginals/seg_eval_images with the images in ./images_result

After comparing the images it seems they are very similar meaning the code works!

How to run the program on your own data

I am not sure how to do this yet. The commandline interface is not very user friendly. It is said that the command

   ../run_partapp.sh --expopt ./expopt/exp_code_test.txt --part_detect --find_obj

should be run in the code_test directory in order to create results in te directory log_dir. Though, the data that is in log_dir is being used by the application. Let’s start from the beginning:

; ../run_partapp.sh
argc: 2
argv[0]: ../bin/Release/partapp
reading command line parameters
command line options::
  --help                 help message
  --expopt arg           experiment parameters
  --train_class          train part detector
  --train_bootstrap      train part detector, adding hard negative samples
  --pidx arg             0-based index of the part
  --bootstrap_prep       create bootstrapping dataset (crops of objects with
                         some background)
  --bootstrap_detect     run previously trained classifer on bootstrapping
                         images
  --bootstrap_showrects  show top negatives on bootstrapping images
  --first arg            index of first image
  --numimgs arg          number of images to process
  --part_detect          run part detector on the test set
  --find_obj             find_obj
  --pc_learn             estimate prior on part configurations with maximum
                         likelihood
  --save_res             save object recognition results in al/idl formats
  --eval_segments        evaluate part localization according to Ferrari's
                         criteria
  --distribute           split processing into multiple chunks, keep track of
                         the chunks which must be processed
  --ncpu arg             number of chunks
  --batch_num arg        current chunk
  --vis_parts            visualize ground-truth part positions

From here, let’s start to try and run the part detection trainer on a set of images. The images we will be training on are the training images for the Ramanan people set, they are stored in the folder ramanan_people_train_h200. I expect to start the program to train it’s appearance model on these 200 images.

/home/frank/andriluka/partapp-r2/code_test2
; ../run_partapp.sh --train_class
'expopt' parameter missing

Let’s see what is needed in this experiment options file. When looking at the other expopt files in partapp-experiments-r2/expopt , these are the options that I found:

; cat abcparams4_trainall_rounds500_shape5_sparse_jb_radius24.txt
boosting_rounds: 500
desc_size: 24
desc_step: 12
window_desc_step_ratio: 1

num_train_pos: 212
num_train_jitter: 1138
num_train_neg: 1000
num_train_bootstrap: 350

feature_type: "SHAPEv5"

; cat exp_buffy_hog_detections.txt
train_dataset: "./ramanan_train_upperbody_h180.al"
validation_dataset: "./ramanan_train_upperbody_h180.al"
test_dataset: "./buffy_hog_detections.al"

part_conf: "./part_conf6_buffy_root1.txt"
abc_param: "./abcparams4_trainall_rounds500_shape5_sparse_jb_radius24.txt"
log_dir: "./log_dir"

min_object_scale: 1.0
max_object_scale: 1.0
num_scale_steps: 1

min_part_rotation: -180
max_part_rotation: 180
num_rotation_steps: 24

flip_orientation: false

scorehist_nbins: 200
scorehist_def_pw: 10
scorehist_def_ph: 10

object_height_width_ratio: 1.2

compute_part_marginals: true
num_pose_samples: 0
strip_border_detections: 0.3

I am now trying to use the first file to see if that is the data that is needed for training. Well…. no:

libprotobuf ERROR google/protobuf/text_format.cc:169] Error parsing text-format 
ExpParam: 1:16: Message type "ExpParam" has no field named "boosting_rounds".

So let’s try the other file? This is not the right approach… Let’s look in the code and use expopt/exp_code_test.txt as configuration file. Oh man I would kill for a better README..

<post not complete yet>

I have been working on a project at the university involving the reconstruction of the world as seen in a video. The goal was to see if I could implement a Structure from Motion approach and test it on a few challenging videos to see whether it works and if I had any good ideas on how to improve it. The outcome was this Report On Sparse Reconstruction From Video. The overall implementation succeeded, but there is still work to be done in improving the evaluation of the results and making the approach generally ‘better’.

Abstract: The goal of this experimentation is to do a sparse 3D reconstruction from videos without any prior knowledge on the camera calibration. Because of the large (and increasing) availability of online videos on websites such as youtube, it is interesting again to look at 3D reconstruction from video.The problem of estimating Structure from Motion (as this is called) has already been researched extensively in the past and but remains interesting to improve because of the large amount of data publicly available. With the use of this large set of online videos we would be able to automatically create 3D reconstructions of scenery all around the world. When combining this with meta-data such as title, description and GPS-coordinates, these reconstructions could be linked to actual places in the world, thus allowing for a searchable 3D earth.

To show off the most fancy results achieved with my little piece of software, here we go

Reconstruction of the road scene

This image is a snapshot taken from the application that has done the 3D reconstruction on a scene that was originally used to demonstrate the ACTS (Automatic Camera Tracking System) sofware, which was mainly created by Guofeng Zhang. Here is an image of the first frame of this video.

First frame of the video

Some of the technology and terminology used in this project (to give an idea) are: Structure from Motion (SfM), Single- and Multiview Geometry, Epipolar Geometry, Triangulation, Homography, Fundamental Matrix, SIFT Feature Detection, Sparse Bundle Adjustment (SBA), Plane Fitting using RANSAC (Random Sampling)

If you are interested (for educational purposes) in the software implementation, feel free to send me a message.

Made this little application (c++) that can draw cubic/quadratic uniform/non-uniform b-splines based on clicking and dragging of points. This is one of these concepts I never really got into, since it was not really necessary, but now it’s getting clear! Now, on to the non-uniform rational b-splines… hurray!

If you want to see how this was done, check the bspline-drawing.zip here (don’t expect programming beauty).

Some good reading on this can be found here in Thomas Sederbergs’ notes and a good applet with demonstration here http://www.ibiblio.org/e-notes/Splines/Bezier.htm.

Today I came across a small application which I made during my computer vision course. It can place and stretch (or rectify) a selected area (a convex quadrilateral) from one image into an area in the other image.  The most important concept that is used is the Homography Matrix. This is a matrix representing the projective transformation from one image into the other and can be found by OpenCV’s function cvFindHomography(). I believe given a set of point correspondences, a homography can be found using either a SVD approach or a RANSAC approach. The OpenCV documentation is not clear about this and just uses “a regular method using all the points”, for now that’s fine by me :)

So, onto the images!

Compositor source image (photo by GrandiJoos)

Compositor destination image

Composited image

It can be seen that the input image got stretched a lot on the bottom part. There is a preciseness parameter for which 1/preciseness determines how many times the output width (height) can be larger then the input width (height). Check out the source code for details on this. Also, try adding your own images!

Download the source code here, it has been tested on Linux only.