Over years of equipment testing we have sought to provide readers with objective information on the performance of the cameras we evaluate. Until recently, the only quantifiable data we could capture was related to capture lag, burst mode and camera response times. But that has changed. From June 2005 we have implemented a new, more objective testing system called Imatest that allows readers to compare the performance of similarly-featured cameras quantifiably.

 

Over years of equipment testing we have sought to provide readers with objective information on the performance of the cameras we evaluate. Until recently, the only quantifiable data we could capture was related to capture lag, burst mode and camera response times. But that has changed. From June 2005 we have implemented a new, more objective testing system called Imatest that allows readers to compare the performance of similarly-featured cameras quantifiably.

The Imatest system, which was developed by Norman Koren in Boulder, Colorado (USA), was first released in September 2004 after months of rapidly-evolving beta tests. It was immediately picked up by the leading online camera reviewers but we believe Photo Review is the first Australian magazine to use it.

We are using Imatest to measure the following camera characteristics for each camera and lens we review:

  • Image sharpness and the camera’s ability to record fine detail (modulation transfer function or MTF) and, hence, the optimum output size for the camera;
  • Chromatic aberration;
  • Colour recording quality, tonal response and colour noise;
  • Camera tonal response and dynamic range.

Other characteristics will be added as we fine-tune our testing procedures, giving Photo Review subscribers an accurate ‘picture’ of how each camera performs. We don’t have enough space to publish the Imatest graphs in our magazine reviews. Instead they will be posted in our Tech Focus section with each camera review.

How to Read the Graphs

See Canon EOS 350D review and Nikon D70s review to see how the results are presented in the context of a review. The following information can be gleaned from the graphs:

Graph 1. Spatial frequency response of the camera/lens system: This graph represents the camera’s ability to produce sharp pictures and is shown as a modulation transform function (specifically MTF50, where contrast drops to half its low frequency level). The figures in red are corrected for standardised sharpening.

The MTF50 you need depends on the size of the prints you produce. Hence, the graph also provides a figure for line widths per pixel height. Dividing the MTF50 LW/PH figure by 150 will give you the optimum print size for printing images with the highest quality inkjet printer. (Dividing by 80 gives you the maximum print size recommended for this camera at normal viewing distances).

Other figures on this graph show the degree of under- or over-sharpening applied by the camera’s internal processing system and the ‘ideal’ megapixel rating for the camera. Note: It is rare for the ‘ideal megapixel’ count to equal the total megapixel count of the camera. In the real world, performance is excellent if it is half the actual megapixels. The MTF at Nyquist frequency is an indicator of the effectiveness of the anti-aliasing filter. Figures under 0.1 indicate negligible aliasing (jagged edges), while over 0.3 shows serious aliasing with the likelihood of Moire patterns becoming visible. Note: anti-aliasing can affect image sharpness, so a very low Nyquist figure could indicate image softening.

Graph 2. Chromatic Aberration: The chromatic aberration (CA) plot charts the edge profiles for the red, green and blue channels, while the black line shows the edge profile of the luminance channel. The closer these profiles correspond, the less the CA. Imatest measures CA in pixels. Values between 0 and 0.5 indicate negligible CA. Between 0.5 and 1.0, CA is low and only visible in enlarged images if you really look for it. From 1.0 to 1.5 it is moderate and noticeable at high print magnifications. Over 1.5, CA is severe enough to be visible at low-to-moderate magnifications.

Graph 3.L*a*b* colour error: This chart shows how well the camera reproduces image colours in comparison to the ideal MacBeth ColorChecker colours photographed. The different ColorChecker values are plotted out on the CIELAB colour space, with the small squares representing the ideal and the circles showing the camera’s rendition. Zero colour saturation is at the centre of the figure, with saturation increasing as you move out radially. Hue varies in a circle around the centre.

In a camera with theoretically perfect colour reproduction, the circles would lie on top of the squares. However, most digital cameras boost the saturation in some areas of the spectrum. This can be seen where the circles fall outside the corresponding squares radially from the centre of the graph. Colour casts can be seen when the circles are displaced to the right or left of the squares.

The mean camera saturation (%) gives an indication of the average saturation of the camera colours, but should not be seen as a strong indicator of image quality because it can be easily adjusted in image editors. Images straight from the camera often look better with extra saturation (120% is not uncommon in compact digital cameras), but a camera that boosts saturation runs the risk of losing detail in highly saturated objects.

Colour error (delta-E, corrected for saturation) is the mean (average) distance between the camera and ideal colours, after normalisation to ideal average saturation. It is a fairly significant indicator of colour quality. The root mean square (RMS) error, denoted by sigma (øÆ’) gives greater weight to large errors and may be a better overall indicator of colour accuracy

Graph 4. Colour Analysis: This chart shows an image of the ColorChecker with the correct colours superposed in the centre of each patch. The colours in the central squares are corrected for the difference in luminance between the exposed and the ideal values. The colours in the small rectangles to the right of the central squares are uncorrected. The grey patches below show the exaggerated White Balance error. This is much worse (more obvious) than reality and the colour shifts and noise shown in these patches may not be visible in photographs, especially in the vicinity of strongly saturated colours.

Granularity in the Exaggerated White Balance error squares can also indicate elevated noise levels. It should be rare in the graphs we publish for each camera because they come from shots taken at the camera’s lowest ISO setting.

For those with a technical bent, full documentation for Imatest – including details of how to interpret the graphs – can be downloaded free of charge from www.imatest.com. The testing software can also be purchased from the site for $US59.

The Imatest system is still evolving and we are only using part of the information available. As our expertise with this system develops we plan to fine-tune our testing procedures so we can exploit its full capabilitiesr and offer a full suite of objective results for each camera we test. Subscribers can view each camera’s test results on this site in advance of the magazine’s publication date.