Upscaling advancements in AI-based software

Back in the early 21^st Century, the megapixel count of a camera’s sensor was the most important parameter for evaluating a camera. Each marginal increase in resolution was promoted in marketing materials and we were led to believe that Camera A was superior to Camera B purely because it had more effective megapixels.

Today we’ve reached a point where 20 to 26 megapixels is considered the norm for most mid-level cameras. Moving up from 20 to 40 megapixels (the typical resolution of ‘flagship’ professional and semi-professional cameras) represents a doubling in resolution potential, which is quantifiable. However, unless you plan to make very large (larger than A2) prints of your shots, 20 megapixels should be good enough – particularly as an increasing number of cameras with in-built image stabilisation (IBIS) provide pixel-shift high resolution modes.

Olympus (now OM System) cameras were the first with pixel-shift image upsizing, with most other manufacturers now including it.

Olympus (now OM System) cameras were the first to include a High Res Shot Mode, but they were soon followed by Panasonic, Pentax, Sony, Fujifilm and, more recently, Canon and Nikon. Each manufacturer only provides this facility in some cameras, although OM Systems has it in most models.

We’ve now reached a stage where the sensors in most consumer cameras deliver excellent performance, with low electronic noise, abundant dynamic range and very little noise at high ISO sensitivity. However, you may find times when the resolution of an image file is too low for your purpose – and that’s when the ability to upscale comes into play.

Reasons to upscale

The main situations where you may need to upscale an image because the original resolution was too low:
1. You want to print images from small-sensor cameras at larger sizes like A3+ and A2.
2. You want to print a frame from an HD or VGA video.
3. The main subject was further from the camera than the lens could reach so you’ve had to crop the image.
4. You want to make better-looking prints from older or lower-quality images.

AI upscaling can also be used to improve drone photos by ‘intelligently’ adding details, particularly to zoomed-in shots in order to make them more printable. In a similar fashion, it can help to bring out surface details in photographs of the Moon and enhance colours and structure in images of star clusters and nebulae.

This image from a 12-megapixel Olympus TG-5 waterproof camera with a 6.17 x 4.55 mm sensor has been used as an example for what the latest upscaling software can do.

How upscaling works

Until the arrival of AI processing, image upscaling was carried out by simple interpolation, which added new pixels based on the hues and tones of neighbouring pixels. As long as the number of additional pixels created was relatively low, compared with the original image, most methods delivered acceptable results.

More sophisticated systems that evolved subsequently used deep convolutional neural networks based on machine learning, which did a better job with photographic images. Deep learning is a subset of machine learning that has also been used to upscale video frames in real-time, particularly by video game developers.

The latest AI-based upscaling systems are based on extensively trained models that have ‘learned’ how detailed parts of a picture should appear when they are enlarged. As a result, the interpolation can usually make the upscaled image appear more detailed and sharper.

Image upscaling software is designed to increase the size and resolution of an original image file (shown in the top left corner) without compromising image quality.

But even AI upscaling has its limits. Success depends on the quality of the original file, how much you attempt to enlarge it and the software you use. It is also affected by how well the software can analyse the photo content to create the right patterns. These will be different for landscapes, portraits, animals (pets or wildlife), plants, architecture, food, fashion – or any particular genre.

While AI upscaling may add some clarity to lower-quality images, it can’t ‘fix’ out-of-focus photos; nor can it put in detail that wasn’t captured in the first place. At best it can create an impression that the photo is sharp when viewed from an appropriate distance. This is particularly true when you work on older images and those from small-sensor cameras.

Noise-reduction isn’t part of current processing, although it can be applied separately, either before or after upscaling. But, again, the more an image is enlarged, the greater the chances noise will still be visible after processing.

Software
If you simply want to try your hand at upscaling an image to see how it works, you can find plenty of ‘free’ upscalers with a Google search. Some are better than others but most will limit the number of images you can upload before you have to pay. Some won’t even let you download an upscaled image until you’ve committed to subscribing to or purchasing the application.

These two screen grabs show the user interfaces plus examples from freeware apps, Pixelcut (top) and Spyne (below).

Some applications that offer ‘image resizing’ can’t attain a useful level of upscaling. For example, Adobe Express limits the output size to a maximum of 6000 pixels on the longest size of images that can be upscaled – and you’ll be required to sign up for an account to access this feature. (There’s a 30-day free trial period in which you can cancel the subscription.) However, it provides a quick way to resize images for virtually any social media you can think of – as well as for other common business and personal usages.

The best software we’ve found so far is the Super Resolution function in the ‘Enhance’ tool set in Adobe Photoshop, which is accessed by right-clicking on the image. It increases resolution by a factor of approximately four and we were able to produce an 8000 x 5344 pixel image from an original 12-megapixel (4014 x 3016 pixel) image.

The dialog box (shown in the screen grabs on this page) shows roughly how long the process takes and the end result is saved as a separate DNG raw file in the same folder as the original image. The results look very impressive – as long as they aren’t magnified too much.

The Super Resolution upsizing function in Adobe Camera Raw, the bundled raw file converter in Adobe Photoshop. Below it are screen grabs showing the results obtained at 100% magnification when the image is enlarged to 2x the original (middle) and 4x the original (bottom screen grab).

We made some A2 prints from shots taken with an Olympus TG-5 waterproof camera on a recent kayaking trip. When framed, they look quite good from the correct viewing distance. But when examined closely, shadow details have been sacrificed and darker areas in the image appear relatively flat. On-screen, the image is also noisy, although the printing process tends to make this less noticeable.

Luminar Neo provides upscaling as one of its AI-based ‘extensions’. It’s a simple drag-and-drop process with very limited adjustability.

Recent versions of Luminar Neo come with AI-based ‘extensions’, one of which, Upscale AI, lets you enlarge an image by up to 6x.

The Gigapixel upscaler from Topaz can recover details from most low-resolution images – within some limits. This illustration shows (from top) 2x, 4x and 6x magnifications from our original source image. Note the reduction in quality as magnification is increased.

Topaz Labs offers the dedicated upscale, Gigapixel 8, with advanced options to recover facial details of older photos that lack sharpness. Upscale low-resolution, compressed images up to 6x and experience a dramatic enhancement of details and textures.

Article by Margaret Brown (see Margaret’s photography pocket guides)

Excerpt from Photo Review Issue 101

See Photo Review Membership options