Getting creative with Generative Fill

If you’ve updated your editing software lately – or purchased a new image editing application – it probably includes generative AI tools. That’s to be expected, and it should be welcomed if it makes our editing easier and more rewarding.

Photographers have always manipulated their images – and mid-20^th Century photographers like Man Ray and Jerry Uelsmann are mainly remembered for their mastery of composite printing and photomontage. Their images were obviously faked – but the skills required to produce them were at the highest levels for image editing at the time.

These before and after images illustrate some of the things you can do with the latest generative AI tools.

You can learn a lot about editing by exploring the AI-based tools in the latest software, and also find out what they can and can’t accomplish. Regular software updates let you track changes and see if they provide meaningful improvements.

We’ve used the Beta version of Adobe Photoshop for this demonstration because it receives frequent updates and it’s the most sophisticated, editing-friendly editor we’ve found. New, adjustable layers are created for every generative AI addition so you can see what’s been done and adjust individual layers when required.

Our source image has the remains of a vehicle in the foreground and a bland and uninteresting sky. As a first step we’ll eliminate the derelict vehicle and make the foreground uniformly sandy. This may not always be necessary as the software is able to replace a selected area with whatever you specify in the prompt box; but replacements are usually more successful when the selected area is relatively uniform.

Use the Lasso tool to outline the area you want to replace and click on Select > Modify > Feather to soften the edges of the selection so the new content blends in better with the rest of the image. We set the Feather to 20 pixels for a 5184 x 3888 pixel image.

Now click on the Edit tag and select Generative Fill from the dropdown menu. Leave the prompt box that pops up blank and choose the best option from the three provided by the software. If none of the three is suitable, click on the Generate tab and try again.

Having removed the vehicle, we can insert new content into the foreground. Use the Lasso tool to select where you want it to go and feather the edges of the selection, as before.

Select Generative Fill again, but enter a prompt describing what you’d like to put in the selected area. We prompted the software to insert a ‘picnic rug and campfire’.

TIP: Avoid prompting the software to insert human forms as the software has problems reproducing credible people. It’s best to stick with easy-to-replicate things.

All three attempts at creating a picnic rug and campfire were unsatisfactory because the software put the campfire in the centre of the rug – an illogical place. Note the three layers in the panel to the right of the workspace.

Given the repeated failures with the initial prompt, our next attempt was simpler and more specific. This time we prompted the software to insert ‘a campfire’ into the same area as we selected before.

The first option provided is the best match – although it’s not perfect. But, because the layers are independently editable, further changes are easy to make.

 Lock in that selection by clicking on Layer > Flatten Image.

Next, it’s time to do something about that bland, uninteresting sky. Because campfires are normally lit in the evening or at night, we’ll turn the sky into a night sky – and make it stormy as well. First, however, we need to select the sky.

Previous tests showed us that if all the sky is selected, the software interprets the ‘Sky lounge’ seats as a building and attempts to make the structure larger and more impressive. We don’t want that so we’ll keep the border of the selection well above the horizon as shown in the screen grab, avoiding the post near the centre of the frame so the software won’t augment that, too.

After feathering the selection, we click on it and select Generative Fill from the dropdown menu. This time we prompt the software to generate a ‘stormy night sky’.

This time one of the second round of fill generations was ‘good enough’ – although not perfect. Here’s where having the ability to edit individual layers is really useful.

There’s still a lot wrong with this image: the foreground is unnaturally bright, the flames are too intense and the clouds don’t come down low enough and are the wrong colour. Selective editing can fix all of these problems.

First we’ll make the foreground look more like it’s night by clicking on the Background layer and selecting Image > Adjustments > Brightness/Contrast.

Don’t worry about the sky as we’ll adjust that layer separately. Clicking on Image > Adjustments > Vibrance lets you pull back on the Vibrance and Saturation until the foreground looks more natural. The Burn tool was also used to subdue the highlights in the fire.

Now it’s time to fix up the unnatural-looking sky. Click on the upper layer in the panel and select Image > Adjustments > Brightness/Contrast, adjusting both sliders until a satisfactory tonal rendition is reached. We’ll also adjust the colour of the sky to make it fit in better with the overall tones in the image.

The next thing to do is to make the sky fit in better with the foreground, which involves several steps. With the top layer selected we select Edit > Transform > Scale. A message pops up, stating: Smart filters applied to layers in this group will be temporarily turned off while the transform is being previewed. They will be applied after committing the transform. Click on OK and ignore it. Grab the corners or edges of the layer and stretch them out until the light area below the clouds overlaps the background layer.

Rescaling the sky.

Now select the Eraser tool from the left toolbar, set a reasonably large brush size with maximum softness and opacity and carefully erase as much as you can of the light area below the clouds until the background layer shows through. Pull back the opacity to 50% and re-adjust the brush size to fine-tune your adjustments. We’ve made additional adjustments to the brightness and contrast of this layer using the Image > Adjustment tools, just as you would with a normal edit.

You may need to change the brush size and opacity several times during this process to obtain the results you want.

Next we’ll add a bolt of lightning for a bit more drama. First we need to flatten the layers by selecting Layer > Flatten Image. Then, use the Lasso tool to select the area where the lightning bolt will go. Go to Edit > Generative Fill and prompt the software to generate a lightning bolt in the area you’ve selected.

The result will inevitably be too bright so adjust the opacity of the layer (outlined in red) until it blends in better with the rest of the scene. Use the eraser brush to help blend the lightning bolt into the clouds. We also blurred lightning slightly and transformed the selection to align it better with the clouds.

In addition we selected an area around the top of the lightning bolt, feathered the edges by 50 pixels and created a new Layer via Copy. This layer was darkened and its colour balance was adjusted to add a touch more yellow. The edges were lightly erased to make the darkened patch of sky blend in better with the rest in the final image. (These are all normal editing adjustments.)

Summing up

Using AI tools effectively requires skill and experience and now is a good time to begin developing both.

It’s also important to acknowledge the use of AI tools in the images we create, so our skills are recognised and viewers can clearly separate the ‘real’ from the ‘generated’ elements in each image.

Article by Photo Review tech editor Margaret Brown.

UPDATE: 26 April, 2024.
On 24 April, Adobe released a significant update to Photoshop (Beta) with improvements to its generative AI functions so we thought we’d try out the new software on the source image used for this article. The results were, indeed, a significant improvement: Two out of the three options presented when we asked the software to generate a picnic rug and campfire in the selected foreground area separated the campfire from the picnic rug in the first round of attempts. The examples are shown below.

First attempt.

Second attempt.