Luminance Blender

[_koh_]

  

On 9/3/2024 at 3:40 PM, _koh_ said:

Feels like
when R:G:B is consistent, sRGB gradation looks linear
when R+G+B is consistent, linear gradation looks linear
this... kinda making sense.

 

I finally understanded what I want blending to do in this case, so I made an effect which blends layers in a gamma encoded luminance color space instead of a gamma encoded RGB one. In this way, it will create visually linear gradation for any combination of colors.

 

sRGB / linear / gamma encoded luminance

These samples are like best / worst case scenarios for sRGB / linear, and this effect could handle them fairly well. Actually it 100% pixel matches sRGB for gray scale blending.

 

source code

protected override void OnInitializeRenderInfo(IGpuImageEffectRenderInfo renderInfo) {
    renderInfo.ColorContext = GpuEffectColorContext.ScRgb;
    base.OnInitializeRenderInfo(renderInfo);
}

protected override IDeviceImage OnCreateOutput(PaintDotNet.Direct2D1.IDeviceContext DC) {
    var gray = (float R, float G, float B) => new Matrix5x4Float(R,R,R,0, G,G,G,0, B,B,B,0, 0,0,0,1, 0,0,0,0);
    var image = (IDeviceImage)new EmptyEffect(DC);
    foreach (var o in Environment.Document.Layers) if (o.Visible) {
        using var idst = image;
        using var iimg = DC.CreateImageFromBitmap(o.GetBitmapBgra32(), null, BitmapImageOptions.DoNotCache);
        using var ilum = new ColorMatrixEffect(DC, iimg, gray(0.2126f, 0.7152f, 0.0722f));
        using var imul = new LinearToSrgbEffect(DC, ilum);
        using var ivec = Shader<Render>([iimg, ilum, imul]);
        using var imix = o.BlendMode != LayerBlendMode.Normal ? new MixEffect(DC, idst, ivec, o.BlendMode.ToMixMode()) : ivec;
        using var isrc = o.Opacity != 1 ? new OpacityEffect(DC, imix, o.Opacity) : imix;
        image = new MixEffect(DC, idst, isrc, LayerBlendMode.Normal.ToMixMode());
    }
    using var ovec = image;
    using var olum = new ColorMatrixEffect(DC, ovec, gray(0.2126f, 0.7152f, 0.0722f));
    using var omul = new SrgbToLinearEffect(DC, olum);
    using var oimg = Shader<Render>([ovec, olum, omul]);
    return oimg.CreateRef();
}

[D2DInputCount(3), D2DInputSimple(0), D2DInputSimple(1), D2DInputSimple(2)]
[D2DShaderProfile(D2D1ShaderProfile.PixelShader50), D2DGeneratedPixelShaderDescriptor]
internal readonly partial struct Render() : ID2D1PixelShader {
    public float4 Execute() => D2D.GetInput(0) * Hlsl.Max(D2D.GetInput(2), 1e-6f) / Hlsl.Max(D2D.GetInput(1), 1e-6f);
}

 

dll + full source code LuminanceBlender.zip
You need PDN 5.1 to build & run this one.

[_koh_]

It requires a layer for each color, so this effect itself has no real use.

You can convert an image to this color space, calculate brush color for it, draw something, then bring it back to standard color space in your effect maybe.

[_koh_]

Maybe these samples are better.
sRGB / linear / gamma encoded luminance

 

Basically it adjusts linear RGB vector size to match human perception, but still keeps vector direction. One downside I can think of is while luminance stays 0-1 range trough out the process, RGB values won't. RGB(0, 0, 1) being something like (0, 0, 4.2). Normal blending has no problem with this, but it breaks some other blend modes.

 

Added the UI and linear color blend mode. You still need a layer for each color because built-in brush works in gamma encoded RGB, but this one may have some non drawing use.
source code + dll LuminanceBlender.zip

[_koh_]

Use gamma encoded luminance / linear just for normal blending.
Going by the formula, blend modes other than normal work better in gamma encoded RGB anyway.
source code + dll LuminanceBlender.zip

 

Yeah... too many color space conversions.

protected override IDeviceImage OnCreateOutput(PaintDotNet.Direct2D1.IDeviceContext DC) {
    var space = (int)Token.GetProperty(PropertyNames.Space).Value;
    var compo = (IDeviceEffect)new EmptyEffect(DC);
    using var enc = Environment.Document.ColorContext.CreateRef();
    using var dec = DC.CreateLinearizedColorContextOrScRgb(enc);
    using var cnv = DC.CreateColorContext(DeviceColorSpace.ScRgb);
    foreach (var o in Environment.Document.Layers) if (o.Visible) {
        using var input = compo;
        using var image = (IDeviceEffect)new BitmapSourceEffect2(DC); image.SetValueRef(BitmapSourceProperty.BitmapSource, o.GetBitmapBgra32());
        using var blend = o.BlendMode != LayerBlendMode.Normal ? new MixEffect(DC, compo, image, o.BlendMode.ToMixMode(), MixAlphaMode.Straight) : image;
        using var layer = o.Opacity != 1 ? new HlslBinaryOperatorEffect(DC, blend, HlslBinaryOperator.Multiply, new Vector4(1, 1, 1, o.Opacity)) : blend;
        using var lcode = new ColorManagementEffect(DC, layer, enc, dec, ColorManagementAlphaMode.Straight);
        using var ccode = new ColorManagementEffect(DC, compo, enc, dec, ColorManagementAlphaMode.Straight);
        using var lconv = space == 1 ? Convert(lcode, dec, cnv, ConvertGammaMode.LinearToSrgb) : lcode;
        using var cconv = space == 1 ? Convert(ccode, dec, cnv, ConvertGammaMode.LinearToSrgb) : ccode;
        using var merge = new MixEffect(DC, cconv, lconv, LayerBlendMode.Normal.ToMixMode(), MixAlphaMode.Straight);
        using var mconv = space == 1 ? Convert(merge, dec, cnv, ConvertGammaMode.SrgbToLinear) : merge;
        using var mcode = new ColorManagementEffect(DC, mconv, dec, enc, ColorManagementAlphaMode.Straight);
        compo = mcode.CreateRef();
    }
    return compo;
}

private IDeviceImage Convert(IDeviceImage input, IDeviceColorContext dec, IDeviceColorContext cnv, ConvertGammaMode mode) {
    var gray = (float R, float G, float B) => new Matrix5x4Float(R,R,R,0, G,G,G,0, B,B,B,0, 0,0,0,1, 0,0,0,0);
    using var iconv = new ColorManagementEffect(DC, input, dec, cnv, ColorManagementAlphaMode.Straight);
    using var lumin = new ColorMatrixEffect(DC, iconv, gray(0.2126f, 0.7152f, 0.0722f), ColorMatrixAlphaMode.Straight);
    using var gamma = new ConvertGammaEffect(DC, lumin, mode, 1, ConvertGammaAlphaMode.Straight);
    using var image = Shader<Render>([input, lumin, gamma]);
    return image.CreateRef();
}

[_koh_]

Use gamma curve of the image's color space to encode / decode the luminance.
With this, grayscale blending will look the same as the original in any color space.
source code + dll LuminanceBlender.zip

It's a bit difficult to recreate this image with the built-in brush, so this effect may have some use after all.
add new layer -> draw something -> run this effect -> merge it down

[_koh_]

Use scRGB for composition instead of the linearized color.
Need to use scRGB to calculate luminance anyway, so this is better.
Color space conversions per layer 12 -> 9. LuminanceBlender.zip

Feels like I can do the same for other cases but scRGB could have non 0-1 range values so not always going to work.
Weighted average works fine. So in this case, normal blending does work but some blend modes don't.
 

protected override IDeviceImage OnCreateOutput(PaintDotNet.Direct2D1.IDeviceContext DC) {
    var gamma = (bool)Token.GetProperty(PropertyNames.Gamma).Value;
    var compo = (IDeviceEffect)new EmptyEffect(DC);
    using var enc = Environment.Document.ColorContext.CreateRef();
    using var dec = DC.CreateColorContext(DeviceColorSpace.ScRgb);
    foreach (var o in Environment.Document.Layers) if (o.Visible) {
        using var input = compo;
        using var image = (IDeviceEffect)new BitmapSourceEffect2(DC); image.SetValueRef(BitmapSourceProperty.BitmapSource, o.GetBitmapBgra32());
        using var blend = o.BlendMode != LayerBlendMode.Normal ? new MixEffect(DC, compo, image, o.BlendMode.ToMixMode(), MixAlphaMode.Straight) : image;
        using var layer = o.Opacity != 1 ? new HlslBinaryOperatorEffect(DC, blend, HlslBinaryOperator.Multiply, new Vector4(1, 1, 1, o.Opacity)) : blend;
        using var lcode = new ColorManagementEffect(DC, layer, enc, dec, ColorManagementAlphaMode.Straight);
        using var ccode = new ColorManagementEffect(DC, compo, enc, dec, ColorManagementAlphaMode.Straight);
        using var lconv = gamma ? Convert(lcode, dec, enc) : lcode;
        using var cconv = gamma ? Convert(ccode, dec, enc) : ccode;
        using var merge = new MixEffect(DC, cconv, lconv, LayerBlendMode.Normal.ToMixMode(), MixAlphaMode.Straight);
        using var mconv = gamma ? Convert(merge, enc, dec) : merge;
        using var mcode = new ColorManagementEffect(DC, mconv, dec, enc, ColorManagementAlphaMode.Straight);
        compo = mcode.CreateRef();
    }
    return compo;
}

private IDeviceImage Convert(IDeviceImage input, IDeviceColorContext src, IDeviceColorContext dst) {
    var gray = (float R, float G, float B) => new Matrix5x4Float(R,R,R,0, G,G,G,0, B,B,B,0, 0,0,0,1, 0,0,0,0);
    using var lumin = new ColorMatrixEffect(DC, input, gray(0.2126f, 0.7152f, 0.0722f), ColorMatrixAlphaMode.Straight);
    using var gamma = new ColorManagementEffect(DC, lumin, src, dst, ColorManagementAlphaMode.Straight);
    using var image = Shader<Render>([input, lumin, gamma]);
    return image.CreateRef();
}

 

[Rick Brewster]

23 hours ago, _koh_ said:

using var image = (IDeviceEffect)new BitmapSourceEffect2(DC);

 

You should use DC.CreateImageFromBitmap() with the DoNotCache option, potentially also with UseStraightAlpha.

 

BitmapSourceEffect2 is a "low level" effect and will not handle certain cases, such as large bitmaps. CreateImageFromBitmap() works with any input bitmap size, and a wider variety of pixel formats. It uses BitmapSourceEffect2 under the hood.

 

I also specify this sort of recommendation in the documentation.

[Rick Brewster]

You may also find it interesting to experiment with @saucecontrol's scRGB-v2.icc color profile (applied via Image -> Color Profile in PDN v5.1). You'll only get 8 bits of precision, and thus you'll get banding in darker colors, but it's still useful.

[_koh_]

2 hours ago, Rick Brewster said:

BitmapSourceEffect2 is a "low level" effect and will not handle certain cases, such as large bitmaps.

 

Ah, thanks.

When I switched to WorkingSpace + straight alpha, this one felt more explicit but didn't know about other limitations.

 

Feels like what I actually want in this case is LayerInfo.UncachedImage which respects RenderInfo.

[Rick Brewster]

I might just mark or hide BitmapSourceEffect and BitmapSourceEffect2. They only seem to pull folks in the wrong direction.

[_koh_]

Just BitmapSourceEffect2 -> CreateImageFromBitmap()
source code + dll LuminanceBlender.zip

This B -> G gradation this effect created doesn't look linear to me, but when converted to the luminance, it's linear in sRGB values.
So while it's working as intended, my assumption which is visually linear in luminance makes gradation visually linear is a bit questionable.
At least it keeps the brush size consistent in this way.

 

edit:

If I use RGB average instead of the luminance, it matches sRGB blending for black -> white gradation, and matches linear blending for R -> G -> B gradation, but it loses consistent brush size.

 

gray(0.2126f, 0.7152f, 0.0722f) -> gray(1/3f, 1/3f, 1/3f)