Removing a known background in presence of noise

[Starlite]

Hi all,

 

I'm trying to remove known backgrounds from some images, which is easy enough in itself -- except some of the images and backgrounds are overlaid with random "noise." I'm wondering what's a good way to get a "good enough" result.

 

The basic setup is that I have two images (screenshots from a game). The first image is the background. The second image is the background plus an overlay image. I'm after the overlay image. So I insert the second image as a layer over the top of the first image and set the new layer's blend mode to "Difference". Flatten and voila! Works a treat.

 

Trouble is, there's additional, semi-random information overlaid on both the first and second images in some cases (eg: a rain animation).

 

So, I take multiple screenshots of both images. Each will have a different noise pattern. I'd like to combine all the versions of image 1 into a base background, and all the versions of image 2 into a base overlay, but I can't figure out a good way to do it.

 

Ideally, I figured to use a "Majority Report" approach; that is, for each pixel, take the value that appears in the majority of screenshots. Or, assuming there's no identical match, take the average of the closest two values. I definitely can't see any way to do this within Paint.NET.

 

Alternatively, I figured to average the three screenshots, but I can't see that feature, either.

 

So now I'm wondering, is there a plugin that will do either of these things?

 

Or if not, is there another way to get much the same result. Assuming the background image has three versions (1a, 1b, 1c) and the overlay image has three versions (2a, 2b, 2c), maybe I could do some clever maths.

 

i) a Difference a Difference b (from top to bottom) inverts the brightness of the noise differences -- not really a solution.

ii) 2a+2b+2c Difference 1a+1b+1c might result in some effective averaging, but the additions could easily clip.

iii) Something tricky with setting the opacity...

 

OK, so I'm clearly not a pro at image manipulation, and certainly not blending algebra. But I'd love to learn!

 

What do people suggest?

 

(Big thanks in advance.)

 

[MJW]

Your strategy of essentially using a median of a number of images seems like a good one, but I don't see how it can be done within PDN. Other than by using the blend modes, which don't really do what you want to do, the only way I know to have multiple images "active" at the same time is to have one in the active layer, and one in the clipboard. For the median idea to work, you need more than just two images. I can't see any way to break a median-type operation into a series of pairwise operations, though perhaps there is.

 

You could average together a power-of-two number of layers reasonably simply. Just average them pairwise. For example, for four layers, average layers 1 and 2; then average layers 3 and 4; then average the two averaged layers. To average two layers, set the opacity of the top layer to 50% and merge the layers.

 

Actually, you should be able to average three layers by setting the opacity of the lower layer to 100%, the next layer to 50%, and the top layer to 33%. The same idea should work for more layers. Just set the layer opacities to 100%/n, where n is the layer number, numbering upward.

 

For removing noise, using the mean average isn't, I would think, nearly as good as using the median.

[Starlite]

Thanks, MJW.

 

I actually had a feeling about the opacity. I haven't worked much with image manipulation (my expertise is more in compression).

 

Thanks for pointing out that setting 50% opacity is the same as averaging. I think that will give a perfectly adequate result, even if there are better theoretical options.

 

I might experiment with the 33% suggestion, but as you point out, it can effectively be replicated with three 50% blends, near enough.

 

My only other option was to write some Matlab code, and I'm not even sure if my Matlab licence is still valid, so this will save a lot of time!

 

Thanks again.

[MJW]

Three 50% blends won't produce the correct average. The last layer will get more weight. Specifically, the weights will be 25%, 25%, 50%.

[Starlite]

Thanks. I've never properly understood alpha blending.

[Starlite]

With the help of MJW's advice, I came up with an amazingly good result, which I thought I'd share.

 

What I realised was that the noise (animated snow) is almost entirely different on each image. That is, snow particles rarely overlap between images. So taking the difference between two otherwise identical images results in just the snow particles on a black background. I reasoned that this could be subtracted back from the combination of the two original images, factoring in the fact that each contributes half of the intensity.

 

So, if the two background images are BG1 and BG2:

1. Create two copies of each, as alternating layers.
2. Set the top layer to Difference mode and merge down to give BG1-BG2.
3. Set the top layer of the lower pair to 127 Opacity (50%) and merge down to give an average of the two, BG1+BG2.
4. Now set the topmost of the two combined layers to Difference AND 127 Opacity and merge down, which I'll call BG1/BG2.

The result is a superbly clean and sharp rendition of the background. Frankly, I was astonished at the quality of the result.

Repeating the same process with the foreground images, I can Difference the FG and BG results to get just the desired overlay image.

 

Analysing, step 2 creates an image of the noise (snow particles) in both layers. Step 3 gives an average of the two layers, which also halves the intensity of any snow particles that don't appear in both images. Step 4 subtracts half of the snow particles from the averaged image, effectively removing them. Of course, if there are any snow flakes that appear in the same location on both images, they won't be removed, but this can be handled with additional source images.

 

Unfortunately, I have other images where the noise is more fog-like, and I suspect this process won't work. In which case it will be back to simple averaging.

 

Nonetheless, this is exactly the kind of layer algebra that I was looking for. Yay!