Photoshop Astrophotography, DSO Processing using Layer Masks Tutorial

This is a short introduction to processing astrophotography images, specifically DSO’s using layer masks in Photoshop. The process is quite complex at first, but as with all processing, the more you do it and practice, the easier it becomes, and the better the results.

This tutorial is essentially creating star masks. Stars can be destructive to a good DSO image, it comes from stretching the image to gain exposure in the DSO, but in doing so making we also make stars brighter. By using masks we can isolate what we stretch and what we control. The same applies to sharpening, adding sharpening without masks can make stars even brighter and detailed, which ends up drowning out the more subtle DSO around them.

We can also use masks to control noise reduction, which is great for reducing noise in the background, but can destroy detail in the DSO.

Below is a walkthrough of three key mask processes which achieves a well-balanced image.

1. Star Mask tutorial: A break down of creating a star mask in Photoshop.
2. Sharpening Mask: Sharpening detail in DSO’s without affecting stars.
3. Masked Noise Reduction: Reducing background noise without affection detail in DSO’s.

A few things to note before we get started:

• I’m demonstrating this on a luminance layer, I’ll look to do a video on integrating the workflow on LRGB soon. The fundamentals are the same, with slight differences in respect to preserving star colour
• The image is calibrated using darks and flats, but I haven’t applied a gradient removal in Photoshop yet. There’s a tiny one in the lower left corner, so excuse that for the purpose of this tutorial.
• I’ll be using Photoshop CC 2017, If you’re using Photoshop Elements you may notice some features missing. Sorry I can’t support these!

Star Mask Tutorial

1) Layer creation

Create 3 duplicate layers of your base layer (Right click > Duplicate layer). Name one as your ‘base stars‘, this will be the base star layer which controls their brightness. Rename the other ‘Working L‘, this will be our main luminance layer we use for processing. Name the last layer ‘Stretch Worker‘, this is a temporary layer we’ll use to create masks.

2) Rough Stretch

Start off with your Stretch Worker layer and do a rough stretch and brighten the image to give a decent dynamic range, don’t worry about being precise with the stretching as we’re getting rid of this layer later on. Our aim with this layer is to brighten as many of the stars that we can, from the bright ones to most of the dim ones, in order to create a layer mask. We’re not enhancing the DSO at this stage, so ignore trying to bring out detail in it.

2) Black cut

Bring up levels (Image > Adjustments > Levels) and start bringing the black point to the left – this will cut the background information, and start to cut information in the DSO, too. This is perfectly fine and what we’re after – In this stage we’re looking to cut all information in the image except the stars – You’ll need to play carefully at this stage as to get rid of as much background information as possible without starting to remove fainter stars.

Now use the ‘polygon lasso tool’ to select the DSO, then use curves (Image > Adjustments > Curves) to drop the blacks (the left of the image) and the mid-tones, but saving the highlights – See my image below to get a rough idea of the curve shape in relation to the histogram. Dropping the blacks and mid tones eradicate most of the DSO and background information but keep the stars which are over the DSO. If we don’t do this, our mask won’t include the stars over the DSO and they’ll become brighter than the rest of the stars in the image when we process our DSO, making it look unnatural.

3) Mask Layer

Select your ‘Stretch Worker‘ layer (Command/Ctrl + A) and copy it (Command/Ctrl + C). Then Select your ‘Working L‘ layer, and create a layer mask. With the layer mask selected (Alt + Click on mask), paste your ‘Working Stretch‘ layer in it (Command/Ctrl + V).

Invert the mask layer (Command/Ctrl + I). Click on your ‘Working L‘ layer. You’ll now have effectively a star layer, where you can make adjustments to your Working L layer while preserving the brightness of the stars. You can control your star brightness, colour and sharpness from this point forward with the bottom ‘Star base‘ layer.

You can now hide or delete the ‘Stretch Worker’ layer – we don’t need it anymore.

4) Touch up

In my example (M31 Andromeda), you may have an instance where brighter parts of your DSO (cores of galaxies, brighter parts of nebulea, etc.) are being affected by the star layer. We don’t want this, so:

1. Activate the mask layer (alt + Click on mask)
2. Select the ‘Brush’ tool.
3. Open the primary colour palette box.
4. Use the colour selector and click on a white area of your mask.
5. Using a soft edge brush, with around 50% ‘flow’, paint over the bright parts of the DSO so they blend with the background colour.

5) Refining the Mask

To tidy things up, right click on your mask and click ‘Refine Mask‘, in the window that pop’s up, use the feather slider to add feather to the mask – how much will depend on the size/resolution of your image, I added 0.6-0.9px, but you’ll need to adjust it to your own size. You’re looking to soften the mask edge on your stars so any adjustments you make aren’t so harsh.

6) Key Stretch

With your ‘Working L‘ layer selected do a ‘key stretch’ using ‘Levels’ and/or ‘Curves’, this will be your main image stretch, so make it good. No need to be exact at this stage as we can touch up the stretch later, but you’ll want to carefully bring out all the detail in the DSO while keeping an eye on noise.

7) Feather Touch up

After the stretch, you might see some anomalies around the stars, this is where the darker base star layer is showing through around the edges of the stars.

To fix it, we can either adjust the feather on the main mask or use the brush tool to paint the mask in these dark halos to soften them up.

8) Finish!

From this point, you have a layer to work on with your DSO, and a separate layer for your stars. You can use your usual workflow to carry on, and ‘Merge Layers’ if you wish to adjust on just one layer, although you may want to preseve the masking separation to adjust the balance between stars and DSO later down the line.

From this point, I’ve continued on with my workflow in case you’re after more food for thought:

Extended: Sharpening

When you’re happy with your mask and main stretch, we’re going to sharpen the DSO. We’ll use a mask again to just sharpen the DSO, as sharpening the background will only enhance noise, make any gradients worse and potentially start brightening dimmer stars.

1) Duplicate layer

Duplicate your Working L layer and rename it ‘Sharpening‘, delete the mask which duplicates with it as we’ll be doing a fresh mask.

2) DSO Selection and Mask

Use the magic wand tool and click on the DSO, you’ll need to adjust the tolerance on the magic wand tool until you select just the DSO and not the background around it. Don’t worry about being precise, if you manage to select some of the background we can adjust the mask with the brush.

Now we need to refine this selection to include elements of the DSO our main selection may have missed. Stay on the Magic Wand tool, and select the ‘add to selection’ button. Now start clicking on the parts of the DSO the mask may have missed. You may have to adjust your tolerence lower as you go, and undo stages here and there to make sure you’re still selecting the DSO without stars – This bits fiddly, but important!

3) Create Mask

Once you have a rough selection, create a layer mask from it (Click the mask button). This will create a mask where we adjust just your DSO.

If you’re having trouble making the selection, maybe your DSO is quite dim, you can add a temporary adjustment levels layer, stretch it out some more then use the selector with it active. Just make sure to delete the adjustment layer after.

4) DSO Mask Touch up

If you have parts of the DSO missing in the mask, click alt+click on mask to view the mask, and using the same technique as step (4) earlier in this tutorial, paint over those missed areas so your mask includes the whole of the DSO.

You might be wondering why we don’t just use the brush tool and do a crude layer mask, just painting the sharpening mask over the DSO. In short, doing this will include any stars that are over the DSO – we don’t want to sharpen these as it’ll make those stars brighter, and unbalance them in the image. Using the magic wand eliminates these stars being selected.

5) Sharpen Layer

Now with this masked layer, you can add sharpening which should just affect the DSO. Everyone has their own ways of sharpening, I use ‘Smart Sharpen‘ as I think this has some great adjustments.

Extended: Noise Reduction

Now to reduce noise. You may want to apply this to the whole image, in which case you can ignore these steps, but I prefer to do a masked noise reduction, as I don’t want to reduce detail in my DSO, and I don’t want to add to the noise reduction that my sharpening has applied.

1) Duplicate layer

Duplicate your Sharpening layer, and call it ‘Noise reduce’.

2) Copy + Invert Mask

The easiest way to mask noise reduction is to just take our sharpening mask, and invert it, assuming we’re only reducing noise where we haven’t sharpened (the dark background).

To do so:

1. alt + click on the sparpening mask to activate it.
2. Select All (ctrl + A), and copy (ctrl + C).
3. On your duplicated layer for noise reduction, create a new blank mask and select it (alt + click on the new blank mask) to activate it.
4. Paste (ctrl + V) the sharpening mask in to the new mask.
5. Invert the mask (ctrl + I)

3) Denoise

Select your noise layer and apply your denoise technique. I use Noiseless CK – A Photoshop plugin with an array of noise reduction settings, but Photoshop has its own good denoise software.

General Touchup

We now have 3 main layers to our image! A ‘Working L’ layer which serves as a backup, and source for any pixels masks may have missed, A ‘Sharp’ layer which is just sharpening our DSO, and a ‘Noise red’ layer, which is controlling the ugly noise in the dark parts of our image. You can use the ‘Opacity’ slider on these layers to adjust how much sharpening and noise reduction you apply, even after making the edits to the masked layers.

Now we can now do some final adjustments to the overall image. You may want to stretch the image some more, denoise/sharpen the whole image and do local touch-ups. If you’re working with an RGB image, you might want to do some final colour balancing.

Here’s an example of what the image is like with and without these layer masks, on the left is just a stretched, sharpened and denoised image. On the right, our masked version.

Do feel free to leave a comment below with any further thoughts, or if you think something can be improved!