CS291A – Mixed and Augmented Reality

Assignment 3: OpenCV Camera Calibration and OpenGL Augmentation

Change Log:
Thu., January 14: original post
Mon. January 25: extended deadline to Thu., Jan. 28


For this assignment, you will familiarize yourself with the OpenCV and OpenGL libraries by calibrating a camera using a checkerboard. OpenCV and OpenGL are still two of the most common libraries/tools used by researchers for vision, image processing, and augmented reality applications. Through this assignment you will:

The assignment has been broken into the following sections. Please read all of the instructions before beginning the assignment.

1. Install OpenCV library

Instructions for installing OpenCV can be found at several places on the internet and there are quite a few different approaches to getting it done on your own Linux, Windows, or MacOS computers. We strongly recommend to use the last known stable version, which currently is 4.3.0.

No matter how you get your installation done, we strongly recommend the use of cmake for your assignments. If you don't have CMake on your system, install it from cmake.org. It is sufficient to use the binary cmake distribution for your platform. Also, take 15 minutes to familiarize yourself with CMake, e.g. by going over the FAQ. In addition to using CMake and command-line development, you are also welcome to use an IDE of your choice.

CSIL

CMake and OpenCV are already installed for you on CSIL, so you can go directly to Testing OpenCV

Your own Windows system

You can use Windows 10's Windows Subsystem for Linux (WSL), but there is currently no camera device (web cam) support.

You can also use MinGW or Visual Studio for your development. MinGW has the advantage that you would be using the same GNU compilers as on CSIL. Visual Studio is obviously a more elaborate IDE.

Your own MacOS system

The following approaches worked well for us:

  1. Installation via Homebrew
  2. Installation via Macports
  3. Compiling from Source with CMake:

2. Test OpenCV library with first programs

Here is a zipped file with all the 4.x.x compatible OpenCV samples: samples.zip.

Just unzip, and you can find the samples for c++ and python in the 'cpp' and 'python' subdirectories.

C++: after installing cmake and opencv, go to the resulting directory in a shell, and type 'cmake .' and 'make'. Some programs will expect access to resources (files), which you can provide as command line parameters, or you can execute the compiled programs from the samples/data/ directory.

Python: Go to the samples/python subdirectory and run the examples with "python3 <sample>.py <params...>"

A few useful links:

OpenCV 4.5.1
OpenCV 4.5.1 Tutorials
 OpenCV 4.5.1 Documentation

( OpenCV 2.4 Tutorials, sometimes more conceptually complete)

3. OpenCV/GL Sample Code Skeleton

We provide you with this basic skeleton file for an integrated OpenCV/OpenGL application to help you get started. A corresponding CMakeLists.txt is here. And here is a version for Python.

Currently, the skeleton program uses OpenCV to provide camera streams (either from an integrated/attached web cam or from playback of a video file provided as a command-line parameter) and uses OpenGL to render these camera frames to an OpenGL viewport. It also places a 3D teapot (flat shaded, unlit) and coordinate axes in the center of the viewport. 

4. Implement Camera Calibration

As an introducing exercise of programming with OpenCV, you are tasked to calibrate the camera that took this set of images with OpenCV calibration routines.  The video on which you can test the calibration is available as checkerboard.mov (Quicktime H264) or checkerboard.avi (DivX codec). This uses the 8x6 (inner corners count) checkerboard pattern that we recommend you use.

(If you run into problems with the codecs, try this uncompressed video (600MB), which has this set of images associated. Note, though, that the checkerboard pattern in this video is 10x7 (inner corners count), and OpenCV needs to be told about that. The 8x6 detection will not work on this one and vice versa.)
  1. For each image, use the findChessboardCorners method to obtain 2D locations of chessboard corners (locations where a black square meets a black square).  Here we denote the chessboard corners as xij for corner i and camera image j.
     
  2. Specify 3D locations for the corners (e.g. Xi = [0 0 0], [1 0 0], [2 0 0], ..., [0 1 0]... ).
    The result of camera calibration is to find parameters K, Ri, ti such that
                                        xij = proj( K [ Rj | tj ] Xi )
                                        where proj( [U V W] ) = [ U/W, V/W ] for all corners I = 1...M and camera images j = 1...N. 
    K is the matrix of intrinsic parameters shared by all camera images and [ Rj | tj ] is the matrix of extrinsic parameters for camera image j.
     
  3. Pass the 3D corner points and 2D corner locations to the calibrateCamera method to calculate the camera matrices and distortion coefficients.  Use the four-element  distortion coefficient model.
     
  4. Calculate re-projection error using estimated coefficients:
    In order to verify the results of our calibration, we can calculate the re-projection error. By using the estimated camera matrices, we can project the 3D points for chessboard corners back to 2D image points. Comparing the projected points with the extracted chessboard corners gives an error metric for the calibration procedure.
    Use the projectPoints method to project the 3D checkerboard corners to 2D points for each image.  Then compute the total re-projection error (sum squared difference or SSD):
                                                    err = sumij( ||xij – proj( K [ Rj | tj ] Xi )||2 )

    Also compute the average error in pixels:
                                                    avg pixel error = sqrt( meanij(||xij – proj( K [ Rj | tj ] Xi )||2 ) )

More details for the abovementioned functions and more can be found at http://docs.opencv.org/2.4/doc/tutorials/calib3d/camera_calibration/camera_calibration.html or
https://docs.opencv.org/4.3.0/d9/d0c/group__calib3d.html#details,

or, for Python, at

https://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_calib3d/py_calibration/py_calibration.html


Once you have calibrated the test images, you should calibrate your own camera. Print out this checkerboard pattern and take several pictures of it from different angles (make sure that your printout stays flat while acquiring images).

Run the same calibration technique for the images taken with your camera.

Write out the following intrinsic camera parameters in a small text file, with one parameter per line (some of these you will need in the OpenGL part of the assignment): horizontal field of view, vertical field of view, focal length (using 0.0f for both apertureWidth and apertureHeight), principal point, aspect ratio parameters fx and fy, and distortion parameters k1, k2, p1, and p2.

Report all results and answer the following questions in a README.TXT: 

How do the intrinsic calibration parameters of your camera compare to the camera for the image sequence we provided (e.g. argue which camera lens has a wider angle, which one more distortion, and distortion of what kind)?
How does the average error compare? Can you explain the difference in error?

 

4. Augment images with OpenGL renderings

For this part, you will use OpenGL to overlay different kinds of 3D graphics on top of the calibration images (and on top of live feeds from your own cameras). The two augmentations we want you to add to the scene are: a) Small spheres at the checkerboard corners and b) the GLUT teapot.

OpenGL

A few useful OpenGL links:

Depending on your OS, OpenGL may or may not need to be installed.
 

Rendering Spheres with OpenGL

Using your knowledge of the real camera parameters gained from your OpenCV camera calibration, you will use OpenGL to render spheres at the inner corners on the chessboard.

The main challenge to this part of the assignment will be to align the OpenCV and OpenGL coordinate systems, which is most easily accomplished by setting up an OpenGL camera that uses the measured intrinsic and extrinsic parameters from the real camera. For the intrinsic parameters, read in the small text file that you wrote out during camera calibration. Undistort the camera image according to the four undistortion parameters before you display the background image. Use OpenCV calls to determine the extrinsic parameters on a frame-by-frame basis. 

During OpenCV calibration of your camera, you assigned 3D world coordinates to the chessboard corners (Xi = [0 0 0], [1 0 0], [2 0 0], ..., [0 1 0]...). These denote your world coordinates for OpenGL.

Create two different Augmented Reality overlays for your input frames, which can be alternated between by pressing the space bar:
  1. Render a sphere at each chessboard corner with the glutSolidSphere(...) command.
  2. Render a teapot hovering over the center of the calibration pattern (glutSolidTeapot(...))
Hints:

 

 

Submission

The Deadline for code and results is 11:59pm on Tue., Jan 26 Thu, Jan. 28th. You are free to develop on your own platform of choice, but all code would idealy be able to be tested by me, so write a README.TXT file describing exactly how to build and run your example. Remember to cite any sources used in your README file as well!