The following outlines a project to convert security video camera footage captured as fisheye images into both panoramic and perspective projections. A fisheye image is captured as it samples the entire visual field however panoramic and perspective images can be more natural to inspect. The goal is for real time operation which has been achieved with extensive optimisation. The warping is done entirely on the CPU without use of a GPU which is the more obvious approach for high performance warping.

The general options include the dimensions of the output image as well as the aperture of the output panoramic or perspective frustum. Some other requirements arise from imperfect fisheye capture such as the fisheye not being centered on the input image, the fisheye not be aligned with the intended axis, and the fisheye not being a full hemisphere.

The usual approach for such warping is to perform the inverse mapping. That is, one needs to consider each pixel in the output image and map backwards to find the closest pixel in the input image (fisheye). In this way every pixel in the output image is found (compared to a forward mapping), it also means that the performance is governed but the resolution of the output image irrespective of the size of the input image.

Software: downfish2persp

The utilities described here were written in C/C++ for a UNIX based system (eg: Linux and Mac OS-X) but the algorithms/code can readily be modified for other operating systems and programming languages. For each utility the usage message is provided, the example images are provided along with the command line that generated them.

Usage: downfish2persp [options] fisheyeimage
Options
   -w n     perspective image width, default = 400
   -h n     perspective image height, default = 300
  -ap n     aperture of perspective (degrees), default = 70
  -cp x y   center of the perspective view, default = 0,45
  -af n     aperture of fisheye (degrees), default = 180
  -cf x y   center of the fisheye image, default is image center
   -r n     radius of the fisheye image, default is half the image width
   -a n     antialiasing level, default = 1 (no antialiasing)

Input image example, courtesy Peter Murphy.

A critical consideration is antialiasing, required when sampling any discrete signal. The approach here is a simple supersampling antialiasing, that is, each pixel in the output image is subdivided into a 2x2, 3x3....grid and the inverse mapping applied to the subsamples. The final value for the output pixel is the weighted average of the inverse mapped subsamples. There is a sense in which the image plane is considered to be a continuous function. Since the number of samples that are inverse mapped is the principle determinant of performance, high levels of antialiasing can be very expensive, typically 2x2 or 3x3 are sufficient especially for images captured from video in which neighbouring pixels are not independent in the first place.

Antialias: 3x3 downfish2persp -cf -cf 380 353 -r 360

Antialias: None

In general the jagged edges are more noticeable in features with a sharp colour/intensity boundary. The other form of aliasing that occurs because of insufficient samples of high frequency structure is less common in video material, more common in computer graphics.

Antialias: 3x3 downfish2persp -cf 300 262 -r 310 -cp -90 60 -ap 90

Antialias: None

The vertical aperture is automatically adjusted to match the width and height.

downfish2persp -a 3 -cf 380 353 -r 360 -cp 110 50 -ap 100 -w 700 -h 300

downfish2persp -a 3 -cf 380 353 -r 360 -cp 110 50 -ap 60 -w 300 -h 500

The center of the fisheye on the input image can be found by projecting lines along vertical structure in the scene. Where these lines intersect is a close approximation to the center of the fisheye, assuming the camera is mounted vertically.

The naming conventions and axis alignment are shown below. The camera is modelled as having a position (same as fisheye camera), a view direction and an up vector which is always assumed to be in the negative z direction. The right vector is derived from these and then the up vector is recomputed to ensure an orthographic camera coordinate system. The case where the camera points straight down (along the z axis) is treated as a special case. A straightforward addition would be to allow the camera up vector to be a user option permitting cameras the roll.

Downward pointing fisheye to panoramic warping

Software: downfish2pano

Usage: downfish2pano [options] fisheyeimage
Options
   -w n     panoramic image width, default = 600
   -h n     panoramic image height, default = 150
   -t n n   start and stop longitude for panoramic, default = 0..360
  -cf x y   center of the fisheye image, default is image center
   -r n     radius of the fisheye image, default is half the image width
   -a n     antialiasing level, default = 1 (no antialising)

The input image used for the following examples is the same one used above for the perspective warping. The grey regions below correspond to where the fisheye is clipped by the image rectangle, ie: the fisheye is not complete.

downfish2pano -a 3 -cf 380 353 -r 360

A subset of the panoramic by choosing a longitude range.

downfish2pano -a 3 -cf 380 353 -r 360 -t 90 270 -w 600 -h 300

Front pointing fisheye to panoramic warping

Software: frontfish2pano

In this case the fisheye is pointing horizontally, as such a traditional panoramic image is limited to half the normal horizontal extent.

Usage: frontfish2pano [options] fisheyeimage
Options
   -w n     panoramic image width, default = 600
   -h n     panoramic image height, default = 300
  -ap n     vertical aperture of panoramic, default = 100
  -af n     aperture of fisheye (degrees), default = 180
  -cf x y   center of the fisheye image, default is image center
   -r n     radius of the fisheye image, default is half the image width
  -fa n     angle for non level fisheye, default = 0
   -a n     antialiasing level, default = 1 (no antialising)

Input fisheye

Default settings.

frontfish2pano -a 3 -cf 430 240 -r 330

Correct for the fact that the camera is not quite horizontal, this is the reason the vertical structure doesn't appear vertical in the panoramic images.

frontfish2pano -a 3 -cf 430 240 -r 330 -fa -20

Set the vertical aperture of the panoramic.

frontfish2pano -a 3 -cf 430 240 -r 330 -fa -20 -ap 110

Set the angle of the fisheye, this example is slightly less than 180 degrees.

frontfish2pano -a 3 -cf 430 240 -r 330 -fa 20 -ap 110 -af 170

The coordinate system for fisheye and panoramic system is shown below.