Bold Brushstrokes v2.0beta (porting my effect to the new Paint.NET 5 GPU stuff)

[Robot Graffiti]

BoldBrush.zip

 

I'm porting my Bold Brushstrokes plugin to be a PropertyBasedGpuImageEffect. The new version draws the brushstrokes into a Direct2D CommandList and then uses a graph of IDeviceEffect shaders to copy the alpha channel from the original image and do the impasto effect.

 

In this version the strokes are SVGs of various crappy blob shapes I drew in Inkscape; in the previous version they were just straight lines with different endcaps.

 

This is all basically working but I'm not finished. The new impasto effect I added to it isn't quite right, and I'm thinking of maybe splitting that off into a separate effect.

 

The hardest part so far was trying to trim ComputeSharp to make my plugin file smaller. In the end I ended up trimming ComputeSharp.Core and leaving ComputeSharp.D2D1 untrimmed. Trimming ComputeSharp.D2D1 with ILlink.dll kept stripping off the IDisposable interface from a type and then the plugin would crash when ComputeSharp used that type in a using block. Eh, not important, the plugin is small enough now I guess.

 

Do let me know if you have suggestions.

[Ego Eram Reputo]

Niccccceeeeeee! This is a great tool. I'm going to have to spend some time playing with the different settings.

 

Looks like its time to repaint the old Breezehome.....

 

[user.by]

We're waiting stabil relase in short time.

[lynxster4]

Very good @Robot Graffiti!  I find the Impasto effect intriguing.  Lots of variables to play with.  Thank you.

Looking forward to the official release!   ☺️

[Rick Brewster]

This is looking really good  Quality and performance are fantastic.

 

Okay here are some of my notes from looking through the disassembled code in ILSpy:

• I'm sure this effect is mostly intended to work with opaque images, but it might be worth exploring if it can be made to work on images with transparent (alpha=0) or translucent (A between 1 and 254) images.
  • Your shaders appear to operate in straight alpha space (e.g. OverlayImpastoShader subtracting a constant value from RGB, calculating Hlsl.Distance() between colors in RestoreEdgesShader), while all of the D2D effects you're utilizing are meant to operate in premultiplied alpha space. The mismatch can cause weird coloring artifacts when not everything is opaque.
  • You can investigate this by using UnPremultiplyEffect for the inputs on your shaders (e.g. input -> UnPremultiplyEffect -> shaderEffect), and then applying the PremultiplyEffect to its output (e.g. shader effect -> PremultiplyEffect -> nextEffect). @BoltBait can relate to the pain of alpha management
• It's possible to make it so that changing the properties that affect the shaders -- basically everything in CreateShaderGraph() -- perform much faster.
  • Your rendering code starts with a large analysis pass, then it creates a command list with a lot of drawing instructions. Then it creates an effect graph that operates on those command lists.
  • What if you only had to update the effect graph when properties like PreserveFineDetails, ImpastoDirection, and ImpastoPercent are changed? (that is, what if you could just re-use instead of rebuilding the command lists?)
  • You can do this in two parts.
    • First, implement OnInspectTokenChanges(). Compare the old and new token: if only any those 3 properties changes, return InspectTokenAction.UpdateOutput. Otherwise return RecreateOutput.
      • InspectTokenAction.RecreateOutput causes OnCreateOutput() to be called every time any property changes. Then, OnUpdateOutput() is called (which, if you don't implement it, is of course just a no-op: everything was done in OnCreateOutput()).
      • InspectTokenAction.UpdateOutput will result in only OnUpdateOutput() being called (although OnCreateOutput() is always called at least once!).
    • Second, move the call to CreateShaderGraph() into OnUpdateOutput(). This will mean you'll have to store your command lists into fields, and dispose of them in OnInvalidateDeviceResources().
    • The update speed will now be much faster -- possibly instant -- when modifying only those 3 properties.
• In DrawBlurredBackground, you're creating a compatible device context (essentially a bitmap) and drawing a scaled-down version of the source image using the HighQualityBicubic sampling mode.
  • This will probably cause your plugin to crash when used with very large images. Down-sizing a very large image to a very small size is not something that Direct2D does a good job with (the quality is fine, it just won't work). You may get an IntermediateTooLargeException because each output region needs "too much" from the source image.
  • Instead, consider not using the compatible device context or the down-sizing and instead using GaussianBlurEffect with the Optimization property set to Speed.
  • If you still prefer the down-sizing approach (it might look or perform better), you will need to manually mipmap the image. In Paint.NET, in the Move Selected Pixels tool, I get around this by chaining ScaleEffects. So instead of resizing the image directly to, say, 1/64th of its size, I use two ScaleEffects that each reduce the size by 1/8th and I set the Cached property to true (necessary to avoid aforementioned exception/crash). You can look at how I do this by opening up ILSpy and looking for the Affined2DTransformEffect2 (which is internal). I may need to make this available for plugins to use, but there is some robustness that I haven't implemented yet.
• I recommend not using ColorBgra in favor of ColorBgra32. The first one is the older/legacy BGRA32 struct, and all the new stuff is based around ColorBgra32 and friends in the PaintDotNet.Imaging namespace. This isn't a high priority, as there are implicit casting operators to make these easier to interop between.

[Robot Graffiti]

Thanks Rick. Good notes.

 

I didn't know about the IntermediateTooLargeException - I haven't tested it on any images larger than about 4000x4000 pixels.

 

The results of DrawBlurredBackground are 99% covered up by the brushstrokes on the default settings. The background doesn't have to look good at all, it just has to be less detailed than the brushstrokes and each part has to be a colour that doesn't look very out of place in that vague area. So, I was trying to go for speed over quality there.

[Rick Brewster]

GaussianBlurEffect with Optimization=Speed should be very fast. To speed it up further you could pre-downscale the input by a fixed amount like 25% using ScaleEffect. That should avoid the hazards with down-sizing.

 

Edit: and of course you'd need to un-downscale it. So the effect graph would be input -> scale(down) -> blur -> scale(up)

[Robot Graffiti]

I'm working through your notes.

 

Use ColorBgra32 instead of ColorBgra: easy, done.

Updating the shader graph instead of rebuilding it when possible: done. The performance gain is tiny on my machine (I have an OK CPU and a very very old GPU so for me 95% of the time is spend rendering 15000 little brushstrokes) but it should help for people with weak CPUs and current GPUs.