Manufacturing variability

Respected camera and lens manufacturers carry out exhaustive quality control testing (shown here in a Panasonic lens factory) before products leave their factories so any units that leave the factory with defects can be seen as very rare aberrations. (Source: Panasonic.)  

Way back when digital photography was but a dream, manufacturing variability was a regular topic of discussion in the photography courses I attended at Sydney TAFE. The issue arose when students were comparing equipment from different manufacturers.

Our lecturer pointed out that cameras and lenses coming off the manufacturers’ production lines are not necessarily identical. Some variability is inevitable in the manufacturing process, even with today’s relatively tight quality controls.

This, he added, can explain why one photographer is totally satisfied with the camera and/or lens he/she is using while another with exactly the same brand of camera and/or lens has nothing good to say about it. Back in those days, the difference between the best sample and the worst from a production line could be substantial. Modern manufacturing techniques have reduced it, but the problem still remains.

Equipment manufacturers normally run quality control tests by collecting samples at random from each production line. This will show whether the production process is operating to a set standard, but it can’t prevent manufacturing variability.

There has been a recent situation where products that met the manufacturer’s standards when they left the production line have developed problems over time due to manufacturing faults. This has resulted in the manufacturer being obliged to repair or exchange the faulty equipment. While it’s been worrying and inconvenient to consumers who purchased affected products, this situation is, fortunately, relatively rare.

Sigma’s internally-developed ‘A1’ testing system claims to test every lens off the production line so, in theory at least, sample variability should never occur. So far we have never needed to return any Sigma products, although they represent a small percentage of the units we test. US-based company, Lens rentals, tests hundreds of lenses and cameras and reports periodically on their findings when products fail to meet the grade. You can browse their website at www.bit.ly/no-perfect-lens.

How Photo Review is affected

From time to time we are reminded of manufacturing variability when processing the results of our Imatest tests. Having used Imatest for more than a decade, we can usually estimate roughly how any camera and lens combination should perform. Occasionally we encounter a camera /lens combination that doesn’t match these expectations.

When this occurs, we repeat our tests, taking extra care to avoid any errors that may have caused the unexpected drop in performance. If the second set of tests fails to produce the results we expect, it is probably because either the camera or lens is ‘out of spec’.

The next question to ask is whether the average photographer would be able to actually see any difference in performance between a camera or lens that is performing to expectations and one that is slightly ‘out of spec’. In many cases you would need to resort to some serious pixel peeping to pick up any differences. Chances are, however, that most people wouldn’t perceive any difference, even between the best and worst copies of the equipment in question.

However, Photo Review specialises in technical reviews based on objective test results. Reporting the results that were below expectations would mean the equipment received a lower rating than it would otherwise be entitled to ““ and that simply isn’t fair. You can’t downgrade any piece of equipment based on a sample of one.

So any equipment that we suspect may be ‘out of spec’ is returned to the local distributor and we wait for a replacement and repeat our tests. The chances of having two samples with the same variability are very low and, so far, we have always found the replacement performed much as we expected.

Will you be affected?

Some readers will wonder whether they could experience sample variability when they buy new equipment. Going by our experience over the 15 years Photo Review has been reviewing products, we think it’s extremely unlikely.

Out of the hundreds of products we’ve tested in that time, only four items failed to pass muster in our Imatest tests and were returned for replacement. The chance of this happening to somebody who buys one or two pieces of new equipment in an average year is incredibly small.

But that doesn’t mean it can’t happen. Which means in turn, you must be VERY sure the fault lies with the equipment and not the methods you are using to assess it.

Our assessments are made with a standardised test set-up. We take care to align test cameras with their backs parallel to the test target and make sure the target is evenly lit. The self-timer is used to trip the shutter to minimise potential camera shake.

Test shots are taken at different distances from the target, depending upon the selected focal length setting. A representative sample of focal lengths is tested (for zoom lenses we test each marked setting), along with each aperture at 1/3EV steps up to about the middle of the range then 1EV steps to the minimum aperture.

Where raw files are captured they are processed with Adobe Camera Raw (our first choice) ““ or the supplied software if the camera isn’t supported by ACR at the time of the tests. Test results are processed with Imatest software, which indicates whether they are within, above or below expectations for the sensor’s resolution. These are objective measurements.

Before we had Imatest, most assessments had to be subjective, although we also used a technique we learned at TAFE. This involved photographing a document page covered with printed type (it used to be the classified ads page from a broadsheet paper).

Provided your camera is set up parallel to the target, you can clearly see differences in the sharpness of the type between the centre and edges of the sheet and determine variations in sharpness with different focal lengths. The only item missing is quantitative measurements, which Imatest now supplies.

What can you do?

If you suspect your new equipment isn’t quite up to scratch, the first thing you need to do is quantify the problem. Nobody will take you seriously unless you have examples that clearly show what you’re complaining about.

You don’t need elaborate test equipment; carrying out a test like the one described above should clarify the issue. If the set-up has been arranged with care, the shooting parameters are correct, and you have been meticulously objective in assessing and documenting the results, this test should show you whether your lens is performing as it should.

If you have doubts about a camera’s performance, test it with at least one lens you know performs well. (running tests with two different lenses will provide a more reliable base on which to make your judgments). Run each set of tests at least twice to ensure it has been done correctly and each time check carefully to ensure your set-up and shooting parameters are correct ““ and standardised.

If you have any doubts after that, check the online reviews. Not just one but at least four or five. Go to sites that use standardised technical testing; not those that base their evaluations on subjective criteria. Ignore forums; they’re usually subjective, often uninformed and vulnerable to bias.

Then if ““ and only if ““ you have clear evidence that your new camera or lens isn’t up to scratch, you should return it to the shop you purchased it from. Under the Australian Consumer Law, when you buy products and services they come with automatic guarantees that they will work and do what you asked for. Details can be found on the ACCC website (http://www.accc.gov.au/).

Under the Australian Consumer Law (ACL), these automatic consumer guarantees apply to many products and services you buy, regardless of any other warranties suppliers sell or give to you.

Reputable manufacturers also provide warranties against defects, which guarantee that defective products will be replaced or repaired. These warranties are usually limited by time. A warranty against defects is provided in addition to the consumer guarantees and does not limit or replace them.

Both consumer protection laws and warranties are normally restricted to goods purchased in Australia, where they are a very effective combination. If you shop offshore the same rights probably won’t apply and you may find it difficult to get a repair, replacement or refund when the business is not based in Australia. For more information visit www.bit.ly/camera-buy-local  

Article by Margaret Brown  

Excerpt from Photo Review Issue 62

Subscribe to Photo Review magazine