Camera Calibration for Blender

Download

You can get the latest version of the extension module and script bundle here (includes everything, even this documentation): http://www.warpax.com/pytsai/pytsai-0.1.tar.gz

Once you've read this file, check out the tutorial: tutorial.html

Introduction

Welcome to the wonderful world of camera calibration, direct from the comfort of your favourite 3D studio program: Blender!

So what is camera calibration? Well, imagine that you have a real camera in the real world, and you have taken a picture of a real scene. Now imagine that you have modeled at least some of that real scene in Blender, using dimensions and angles from the real world. Let's suppose that you want to put a camera at the same location in the Blender scene, relative to the modeled objects, as the real camera occupied in the real scene, relative to the real objects. That's what camera calibration does for you! You can think of it as setting up a camera in a virtual scene at the same spot as the camera in a real scene. I'm sure you can think of many fun uses for that kind of technology, ne? :-) In the world of visual effects (I'm speaking from the experience of watching many DVD extras disks here :-), I've heard this method referred to by various names including camera matching and camera tracking. The current technique only works for still images so far, so it doesn't truly do camera "tracking" just yet (unless you're into manual labour in a big way!).

Calibrating the Camera

In the current method, the parts of the scene that are modeled in Blender must be Empties. This is because they represent point locations. In the real world, they can be the corners of shapes, specially designed markers (eg: ping-pong balls), the corners of a checkerboard pattern on a plane, etc. These points, whose 3D locations are known, are called the calibration geometry or calibration target. Not all calibration geometry in the literature are point sets: sometimes other things such as sets of line segments are used. However, we are restricted to points in the current method. The task of matching up the points in the virtual scene with the points in the image of the real scene is called the correspondence problem, and is solved manually here. The user of the script must select the Blender Empties and assign them a corresponding point in the image. Naturally, a funky GUI is supplied for just this purpose.

Once the calibration target has been defined and corresponding image space points have been assigned, the camera can then be calibrated. The method determines two main sets of data about the camera. Firstly, it determines the position and orientation of the camera, which is a 6 degree-of-freedom (DOF) quantity. The position and orientation are sometimes referred to as the exterior orientation or exterior parameters of the camera. The method also determines the Blender lens value of the camera, which is the interior orientation or interior parameter. Finally, the method does a first-order approximation for radial distortion of the image, supplying a centre of distortion and a single distortion constant. These last three parameters (Cx, Cy, kappa1) can be retrieved quite simply if required, but are currently ignored because Blender has no representation for them in its own camera model.

What Makes Camera Calibration Difficult and How Do We Do It?

If we take two sets of 3D points, and are asked to find a rigid body transformation which makes them align, it is a relatively easy task. There is an exact solutions available for three points, and a number of strategies available for minimizing the error from more than three points. A good, general, least-squares technique is given in this paper:

Challis, J.H. (1995) A Procedure for Determining Rigid Body Transformation Parameters, J. Biomech. 28(6):733-737.

However, the problem in camera calibration is quite different. In camera calibration, we have lost information about the 3D scene: specifically, how far the points are away from the camera, or their depth. We are now no longer trying to find a mapping from 3D to 3D (which as pointed out above is quite simple), but instead we are trying to find a mapping from 3D to 2D. This loss of information complicates the problem tremendously!

The technique used here was described by Roger Tsai:

Tsai, R.Y. (1986) An Efficient and Accurate Camera Calibration Technique for 3D Machine Vision. Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Miami Beach, FL, pp. 364-374.

Tsai, R.Y. (1987) A Versatile Camera Calibration Technique for High-Accuracy 3D Machine Vision Metrology Using Off-the-Shelf TV Cameras and Lenses. IEEE Journal of Robotics and Automation, Vol. RA-3, No. 4, pp. 323-344.

Incidentally, if anyone has complete copies of those papers, can you please (please, please, please...) send me copies?! I have partial, very poorly-photocopied versions. The code used as the basis for the current implementation was written by Reg Willson et al., and is available online here:

http://www-2.cs.cmu.edu/afs/cs.cmu.edu/user/rgw/www/TsaiCode.html

I simply wrote a Python wrapper around the C code available on that website, and modified it to continue (returning an error flag) when an error occurs, rather than bailing out instantly using the exit() function.

Using the Camera Calibration Script

The camera calibration script depends upon a Python extension module which is a wrapper around the code listed above. This wrapper defines a Python module called Tsai, and a lower-level C module called pytsai. Before you can use the Blender script, you must therefore install the extension module. This is done using the normal distutils. Under Linux (sorry Windoze users, I must admit ignorance of your requirements):

$ python setup.py build

$ su

# python setup.py install

Then you can fire up Blender:

Set up a scene with some Empties placed at the location of calibration points. The Empties do not need to be specially named or anything, and can be arbitrarily connected to other geometry in the scene (parented, etc.). The Empties can optionally be added later while the script is running, but I tend to set them up in advance and save the file in case the script segfaults on me or something (grin).
Load and run the script (use the menus of the Text window if you can't remember the shortcuts - I can't).
Click on the Load Image button to load an image to use for the calibration. (IMPORTANT: The image must be in RAW TGA format for now.) The image can be moved in the script window by dragging with the middle mouse button. You can change the zoom using either the mouse wheel or (perhaps if you don't have a mouse wheel) the Zoom numeric button. Pressing ZKEY will reset the zoom to 1.0.
Select Empties in the 3D scene (in the 3D view) and add calibration points to the image. If everything is working correctly, when you select an Empty in the 3D view, the script window will change automatically to give you the option of adding a corresponding point to the image. Points can be added by clicking the Add button or by pressing the SPACEBAR. This will bring up a pair of red cross-hairs that show the prospective location of the image point. Note that the offset of the cross-hairs from the mouse pointer is deliberate; it is designed to assist in seeing the actual intersection of the two lines (yes, I know it can be off-putting, but you get used to it). When working with image points after they have been added, GKEY, AKEY and XKEY work as expected. The only minor point is that GKEY drag mode cannot be cancelled by pressing ESCKEY - you must actually move the point(s) somewhere and finalize the drag once you have started it. If you require fine-tuning of position that is cumbersome with the mouse, you can use the arrow keys to move around one screen pixel at a time. Points can be selected using the right mouse button, and multiple selections can be achieved by holding down either SHIFT key. Points can be deleted using the Delete button (which does not prompt for confirmation) or by pressing the XKEY (which does prompt for comfirmation).
Once correspondence has been achieved between scene Empties and the image points, calibrate the camera. When you select or create a new camera, the script should change to offer calibration options. Set the options as you require (see the Notes below for some hints) and click the Calibrate button.

Notes

The following are some general notes about using the script. This documentation is far from complete, but should hopefully give you enough to be going on with:

Two types of calibration targets are possible. You must use the correct calibration script option for each target type, otherwise the calibration will definitely fail (or be wildly inaccurate):

Coplanar - all calibration points lie in the same plane (they must have z = 0 in their 3D space coordinates, and must not approximate a line).
Noncoplanar - calibration points occupy 3D space (they must not approximate a line or plane).

Two calibration routines exist:

Partial optimization - non-linear optimization is performed only for the Tz, f, and kappa1 parameters (which are the z-camera coordinate, focal length and radial distortion constant respectively).
Full optimization - non-linear optimization is performed for all parameters. This method currently seems to have some kind of bug, which I'm looking into (I may have introduced a bug myself during my wild hacking of Willson's code).

Calibration will fail if the world space origin is near to the vertical line that passes through the centre of the image. For now, just avoid this condition (don't have the scene origin anywhere near the vertical centre of the image). You can manually offset the scene if you need to. In the future, an option will be provided to mathematically shift the origin of the entire scene during calibration and then move it back again afterwards if necessary.
For coplanar calibration, avoid looking directly along the z-axis (ie: straight down at the target). If you do so, it is not possible to distinguish depth effects from focal length effects. Your calibration may fail or be wildly inaccurate. Willson recommends a minimum angle of 30 degrees from the z-axis, although that is arbitrary.
To increase the accuracy of calibration, for both target types, consider the following:

"Perspective effects" increase the accuracy. These can be achieved by having calibration points located over a wide range of depths in the scene. This will particularly help to pin down the camera z-coordinate and its focal length.
More points will increase the calibration accuracy. Be creative about including more points in your calibration set (provided they are accurate). If, for example, you have a pole with a calibration point at the top, place a few calibration points down along its length. For coplanar checkerboard targets, use a denser grid of checkers. If you are indoors and have a regular grid pattern of ceiling tiles, maybe include their corners in your set. Always beware, though, that architecture is sometimes notoriously non-square! If in doubt, measure things using reliable techniques first.

The Python extension module contains many externally-visible symbols (many non-static functions, and three main global variables), instead of just the module init function. The chances of a naming conflict occurring between these symbols and symbols in either the Python interpreter or Blender itself seems to be quite remote (I changed the names of the three global variables to something more "specific" just in case), but it is worth me noting this problem anyway. The Python documentation is very strict about this issue: these symbols should not be visible. The only problem is that enforcing that condition would require some major re-working of the library. Maybe there is a shortcut that I've missed? C++ namespaces maybe? Please let me know if you are aware of an easy solution.

Jonathan Merritt (j.merritt@pgrad.unimelb.edu.au),
PhD Student (Equine Biomechanics),
The University of Melbourne Equine Centre,
240 Princes Highway,
Werribee, Vic. 3030.

Last updated: 04-Nov-2004 (20041104).