Getting an error when trying to use HistogramEffect (CodeLab sample)

[_koh_]

Code below is functional because I commented out the given part.
I'm within a DrawingScope so I can call DrawImage() and can visualize IEnumerable<float>. Fine.

protected override IDeviceImage OnCreateOutput(IDeviceContext deviceContext) {
    var graph = deviceContext.CreateCommandList();
    using (graph.UseBeginDraw(deviceContext)) {
        var brush = deviceContext.CreateSolidColorBrush(new(1, 0, 0, 1));
        var scale = Math.Min(Environment.Document.Size.Width, Environment.Document.Size.Height) / 256f;
        deviceContext.Transform = new(scale, 0, 0, -scale, 0, Environment.Document.Size.Height);
        deviceContext.DrawCurve(Hist().Select((v, i) => new Point2Float(i, v * 256)).ToArray(), brush, 1.6f);
    }
    return new MixEffect(deviceContext, Environment.SourceImage, graph, MixMode.CompositeSourceOver);

    IEnumerable<float> Hist() {
        var hist = new HistogramEffect(deviceContext);
        hist.Properties.Input.Set(Environment.SourceImage);
        hist.Properties.ChannelSelect.SetValue(ChannelSelector.R);
        //deviceContext.DrawImage(hist);
        //return hist.Properties.HistogramOutput.GetValue();
        deviceContext.DrawImage(new EmptyEffect(deviceContext));
        return Enumerable.Range(0, 256).Select((i) => i / 256f);
    }
}

 

But if I uncomment DrawImage(hist), I get this error.

System.ArgumentException: Value does not fall within the expected range. (InteropError.InvalidArgument 0x80070057)

 

What am I doing it wrong?
 

[Rick Brewster]

HistogramEffect is a weird one. First, be sure to look at the Direct2D documentation -- in CodeLab, put the cursor on "HistogramEffect" and press F1, which will take you to the wrapper's documentation, which has a "For more information, see Histogram effect" link. (this is a very important technique to know when working with Direct2D effects in Paint.NET!)

 

You can't just call DrawImage() inside of a command list's drawing scope like that and have it work. You have to actually draw it to a real target, you can't just use it in a command list and use its result like that because you aren't actually drawing when you create a command list. The drawing is deferred until after OnCreateOutput(), where Paint.NET takes the image you've given it (an "image" can be a command list btw) and draws it once for each output tile.

 

What you'll want to do is to create a temporary "compatible" device context to draw into "for real". ICompatibleDeviceContext maps to the Direct2D interface ID2D1BitmapRenderTarget. See here and here for some more info. 

 

HistogramEffect can only operate on images up to 4096x4096 pixels, so you'll have to break up the image into those chunks, run histogram on each one, and aggregate the results according to what you want to do with the data.

 

Here's an example CodeLab script I wrote that shows how to use it:

 

protected override IDeviceImage OnCreateOutput(IDeviceContext deviceContext)
{
    StringBuilder text = new StringBuilder();
    using (ICompatibleDeviceContext cdc = deviceContext.CreateCompatibleDeviceContext(desiredPixelSize: new SizeInt32(1, 1), desiredPixelFormat: DevicePixelFormats.Prgba128Float))
    {
        // The HistogramEffect only works on images up to 4096x4096 in size. So we must loop through tiles of that size and draw each one separately.
        SizeInt32 docSize = this.Environment.Document.Size;
        const int tileEdgeLength = 4096;
        for (int tileTop = 0; tileTop < docSize.Height; tileTop += tileEdgeLength)
        {
            int tileBottom = Math.Min(tileTop + tileEdgeLength, docSize.Height);
            for (int tileLeft = 0; tileLeft < docSize.Width; tileLeft += tileEdgeLength)
            {
                int tileRight = Math.Min(tileLeft + tileEdgeLength, docSize.Width);
                RectInt32 tileRect = RectInt32.FromEdges(tileLeft, tileTop, tileRight, tileBottom);
                using CropEffect sourceImageTile = new CropEffect(cdc, this.Environment.SourceImage, tileRect);

                using HistogramEffect histogramEffect = new HistogramEffect(deviceContext);
                histogramEffect.Properties.Input.Set(sourceImageTile);
                
                cdc.BeginDraw();
                cdc.Clear();
                cdc.DrawImage(histogramEffect);
                cdc.EndDraw();

                text.AppendLine($"for tile {tileRect}:");

                IReadOnlyList<float> data = histogramEffect.Properties.HistogramOutput.GetValue();
                for (int i = 0; i < data.Count; ++i)
                {
                    if (i != 0) 
                    {
                        text.Append(", ");
                    }

                    text.Append(data[i]);
                }
                text.AppendLine();
                text.AppendLine();
            }
        }
    }

    IDirectWriteFactory dwFactory = this.Environment.DirectWriteFactory;
    ICommandList commandList = deviceContext.CreateCommandList();
    using (commandList.UseBeginDraw(deviceContext))
    {
        deviceContext.Clear(((ColorRgba128Float)LinearColors.White) with { A = 0.25f });
        using ITextFormat textFormat = dwFactory.CreateTextFormat("Segoe UI", null, FontWeight.Normal, FontStyle.Normal, FontStretch.Normal, 64);
        using ISolidColorBrush fillBrush = deviceContext.CreateSolidColorBrush(LinearColors.Black);
        deviceContext.DrawText(text.ToString(), textFormat, new RectFloat(Point2Float.Zero, this.Environment.Document.Size), fillBrush);
    }

    GaussianBlurEffect2 sourceBlurred = new GaussianBlurEffect2(deviceContext);
    sourceBlurred.Properties.Input.Set(this.Environment.SourceImage);
    sourceBlurred.Properties.StandardDeviation.SetValue(StandardDeviation.FromRadius(30));
    sourceBlurred.Properties.BorderMode.SetValue(BorderMode.Hard);
    OpacityEffect dimSourceBlurred = new OpacityEffect(deviceContext, sourceBlurred, 0.5f);

    CompositeEffect output = new CompositeEffect(deviceContext, sourceBlurred, commandList);
    return output;
}

 

Here's what it looks like after running on a 5472 x 3648 image:

 

[Rick Brewster]

This is the first use I know of for HistogramEffect, so be sure to show us your result when you get it working

[_koh_]

Wow thanks a lot. That's a bit more code than I expected especially that tiling part.

 

I currently have functional histogram in my BitmapEffect version which is done in a totally different manner, and evaluating if I can replicate this in GpuEffect.

 

1 hour ago, Rick Brewster said:

You can't just call DrawImage() inside of a command list's drawing scope like that and have it work. You have to actually draw it to a real target, you can't just use it in a command list and use its result like that because you aren't actually drawing when you create a command list.

 

Yeah, now I totally get that. I somehow overlooked that when I coded this one.

Now I'm wondering, is it possible to have input + shader output histograms on GPU?

[Rick Brewster]

2 minutes ago, _koh_ said:

Now I'm wondering, is it possible to have input + shader output histograms on GPU?

 

This may be possible by writing a compute shader with an analysis transform. While IAnalysisTransform (aka ID2D1AnalysisTransform) is available in PDN's D2D wrappers, it can't be used yet because compute shaders are not yet supported (the docs for ID2D1AnalysisTransform erroneously state it can be used with an ID2D1DrawTransform, which is for pixel shaders, but it doesn't work).

[Rick Brewster]

For now you can run the histogram like above to gather the data, aggregate it how you need, and then pipe _that_ into a pixel shader's input (constants buffer). You'd use ComputeSharp.D2D1 for that. But you can't have a histogram + compute shader w/ analysis transform yet. And I'm not even sure HistogramEffect can be used in that way anyway.

[Rick Brewster]

I suppose you could write your own histogram pixel shader, but you would need to write it as a full custom pixel shader with a "transform mapper." So instead of putting the output into an effect property, you would emit the data into an image (texture) that is like 256x1 (if you want 256 "bins" for 1 channel) or probably 256x3 (256 bins for each color channel).

[Rick Brewster]

Compute Shader support may be coming in 2024, @sergiopedri is planning to add it to ComputeSharp.D2D1, and then I can add support in PDN's D2D wrappers

[_koh_]

41 minutes ago, Rick Brewster said:

I suppose you could write your own histogram pixel shader, but you would need to write it as a full custom pixel shader with a "transform mapper." So instead of putting the output into an effect property, you would emit the data into an image (texture) that is like 256x1 (if you want 256 "bins" for 1 channel) or probably 256x3 (256 bins for each color channel).

 

So I can control where to write and can create "array" texture and use it as an input for following shader? That's cool.

Too much things I don't know about and currently I'm fairly limited when it comes to data structure.

[_koh_]

Couldn't resist to see how it looks so I created DrawImage() shader output to CompatibleDeviceContext version.
I believe this means my shader running 4 times (output + RGB histogram) so I'm not going to use it, but at least it worked.

IDeviceImage Graph(Func<Vector4, Vector4> func, IDeviceImage iimg, IDeviceImage oimg, float alpha) {
    var hist = (IDeviceImage img) => {
        using var cdc = deviceContext.CreateCompatibleDeviceContext(desiredPixelFormat: DevicePixelFormats.Prgba128Float);
        using var tile = new CropEffect(cdc, img, RectFloat.Intersect(new(0, 0, 4096, 4096), new(0, 0, Environment.Document.Size)));
        using var hist = new HistogramEffect(cdc); hist.SetInput(0, tile);
        var data = (ChannelSelector ch) => {
            hist.Properties.ChannelSelect.SetValue(ch);
            cdc.BeginDraw();
            cdc.DrawImage(hist);
            cdc.EndDraw();
            return hist.Properties.HistogramOutput.GetValue();
        };
        var o = (R: data(ChannelSelector.R), G: data(ChannelSelector.G), B: data(ChannelSelector.B));
        return Enumerable.Range(0, 256).Select((i) => new Vector4(o.R[i], o.G[i], o.B[i], 0));
    };

    var tpnt = Enumerable.Range(0, 256).Select((i) => new Vector4(i + 0.5f)).Select((c) => (c, func(c / 256) * 256));
    var ipnt = hist(iimg).Select((v, i) => (new Vector4(i), v * 8192));
    var opnt = hist(oimg).Select((v, i) => (v * 8192, new Vector4(i)));
    var list = new (IEnumerable<(Vector4 X, Vector4 Y)>, Func<Vector4, float>, ColorRgb96Float)[] {
        (ipnt, (c) => c.X, new(4/8f, 0, 0)),
        (ipnt, (c) => c.Y, new(0, 2/8f, 0)),
        (ipnt, (c) => c.Z, new(0, 0, 4/8f)),
        (opnt, (c) => c.X, new(6/8f, 0, 0)),
        (opnt, (c) => c.Y, new(0, 3/8f, 0)),
        (opnt, (c) => c.Z, new(0, 0, 6/8f)),
        (tpnt, (c) => c.X, new(8/8f, 0, 0)),
        (tpnt, (c) => c.Y, new(0, 4/8f, 0)),
        (tpnt, (c) => c.Z, new(0, 0, 8/8f)),
    };

    var graph = deviceContext.CreateCommandList();
    using var _ = graph.UseBeginDraw(deviceContext);
    using var brush = deviceContext.CreateSolidColorBrush(new());
    var scale = Math.Min(Environment.Document.Size.Width, Environment.Document.Size.Height) / 256f;
    deviceContext.Transform = new(scale, 0, 0, -scale, 0, Environment.Document.Size.Height);
    foreach (var (p, v, c) in list) {
        brush.Color = new(c, alpha);
        deviceContext.DrawCurve(p.Select((o) => new Point2Float(v(o.X), v(o.Y))).ToArray(), brush, 1.6f);
    }
    return graph;
}

 

Some findings.

 

While built-in effects output image size information, my shader doesn't. So not to skew histogram data, I needed to crop shader output to document size to add size information.

 

I was curious if I give 3 HistogramEffect to CompositeEffect then draw it to CDC, my shader need to run only once. But I got invalid graph error so it didn't work.

 

This is multi pass rendering thing after all, and while GpuEffect makes multi pass rendering simpler (e.g. I can blur my rendering result), if I want to rely on D2D drawing I still need to know what to draw beforehand. So having compute shader may not change this particular case.
 

[_koh_]

Full source code by the way.LightBalanceGPU.zip

[_koh_]

OK I've found a solution. I think now my shader running only once.
Problem is, my shader is relatively simple and HistogramEffect is basically slow, so I have no idea I'm making it faster.
 

var cdc = deviceContext.CreateCompatibleDeviceContext(Environment.Document.Size, null, DevicePixelFormats.Prgba128Float);
using (cdc.UseBeginDraw()) cdc.DrawImage(shader);
var (input, output) = (shader.GetInput(0), cdc.Bitmap);

return new MixEffect(deviceContext, output, Graph(func, input, output, alpha), LayerBlendMode.Normal.ToMixMode());

 

Another finding.
HistogramEffect gives me an error if I access Properties.Bins even by reading it, so I believe something going wrong somewhere.

 

Full source code. LightBalanceGPU.zip

[Rick Brewster]

1 hour ago, _koh_ said:

HistogramEffect gives me an error if I access Properties.Bins even by reading it, so I believe something going wrong somewhere

What error? Is it like ArgumentException? I think I see a typo in the wrapper code, but should be an easy fix

 

What happens if you try to read/write it this way:

// instead of this
//hist.Properties.Bins.SetValue(bins); 

// try this for setting
hist.SetValue((int)HistogramProperty.Bins, DeviceEffectPropertyType.UInt32, (uint)bins);

// or this for getting
int bins = hist.GetValue<int>((int)HistogramProperty.Bins);

 

[Rick Brewster]

Yes, if your shader needs the output from HistogramEffect, this is the correct way to do it. You are certainly using a ton of memory though! You could reduce the compatible DC to DevicePixelFormats.Pbgra32 which probably won't affect results much but will reduce the memory use of that compatible DC by 75%. (You can also experiment with Prgba64Float but I'm renaming that in the next update so I'd advise against releasing a plugin with it, but you can use new DevicePixelFormat(DxgiFormat.R16G16B16A16_Float, AlphaMode.Premultiplied) directly instead.). Shaders will still execute at Float32 precision, it's only command list output precision that should be affected.

 

For much more memory savings, instead of creating your own composition image with Compo(), consider using Environment.Document.Image instead. That's a pre-made composition of the whole image (aka Document), taking into account per-layer blending and opacity. It does all of the composition on the CPU in the exact same way it's done in the canvas -- which means non-linear blending (your code is doing everything in linear, which is great, but produces a different result). This will save an ENORMOUS amount of GPU memory if you are working with multiple layers, since this only needs to be stored on the GPU as a single BGRA32 image (converted on-the-fly to RGBA128F).

 

It looks like Dec() is an sRGB->Linear conversion, and Enc() is Linear->sRGB. You can use SrgbToLinearEffect and LinearToSrgbEffect instead. Simiarly, Mul() is PremultiplyEffect and Div() is UnPremultiplyEffect. Instead of handling these yourself, you could build an effect chain and split your shader into multiple shaders.

 

So your graph would be:

 

input -> UnPremultiplyEffect -> shader1 (Pad) -> SrgbToLinear -> shader2 (HDR, LDR) -> LinearToSrgb -> shader3 (Fit) -> PremultiplyEffect

 

Only shader1 would need both inputs.

 

This would result in two small simple shaders instead of 1 bigger one, and is probably a more natural way to express this in D2D's effect graph system. Plus then you don't have to implement Srgb<->Linear or (Un)Premultiply yourself, and can rely on D2D's and my optimizations in this area (for both perf and quality).

 

[Rick Brewster]

Also, this:

private float3 If(bool3 c, float3 a, float3 b) => new(c.R ? a.R : b.R, c.G ? a.G : b.G, c.B ? a.B : b.B);

should be replaceable with just:

c ? a : b

... At least, that's how it can be done in HLSL directly (IIUC). But I don't think we have that support in ComputeSharp.D2D1's C#->HLSL transpiler yet. I'll send the request over to @sergiopedri to see if we can figure out a way to provide this.

[_koh_]

39 minutes ago, Rick Brewster said:

What error? Is it like ArgumentException? I think I see a typo in the wrapper code, but should be an easy fix

 

What happens if you try to read/write it this way:

// instead of this
//hist.Properties.Bins.SetValue(bins); 

// try this for setting
hist.SetValue((int)HistogramProperty.Bins, DeviceEffectPropertyType.UInt32, (uint)bins);

// or this for getting
int bins = hist.GetValue<int>((int)HistogramProperty.Bins);

 

 

OK this worked. Thanks!

Haven't narrowed down the condition but if I uncomment that line I get this error. Do you need full stack trace?

 

System.ArgumentException: Value does not fall within the expected range. (InteropError.InvalidArgument 0x80070057)

 

[Rick Brewster]

50 minutes ago, _koh_ said:

Do you need full stack trace?

No, I know exactly the typo -- the wrapper's property is giving you an Int32 property accessor, but it needs to be a Int32-As-UInt32 property accessor. The underlying native effect property is UInt32, so I need to tell Direct2D that when retrieving the property before reinterpret-casting it to Int32. It's a tiny change, it'll be in the next update.

[_koh_]

50 minutes ago, Rick Brewster said:

You could reduce the compatible DC to DevicePixelFormats.Pbgra32 which probably won't affect results much but will reduce the memory use of that compatible DC by 75%.

 

For sure I should do this.

 

53 minutes ago, Rick Brewster said:

For much more memory savings, instead of creating your own composition image with Compo(), consider using Environment.Document.Image instead.

 

That's what I get when Ctrl+Shift+C in PDN right? Each layer blended in integer.

A bit difficult to explain, but when a layer having opacity = 16, it's like the layer using 12bit color instead of 8, and when I use excessive parameters I can see the differences between FP blended images and INT blended images. But I agree this is "Why you even need this?" kind of thing.

 

1 hour ago, Rick Brewster said:

input -> UnPremultiplyEffect -> shader1 (Pad) -> SrgbToLinear -> shader2 (HDR, LDR) -> LinearToSrgb -> shader3 (Fit) -> PremultiplyEffect

 

While I know this, I thought having them in one liners makes overall code simpler, but that's just my taste. 

 

1 hour ago, Rick Brewster said:

ALso, Pad and Fit may not be necessary?

 

Standard way is

c' = c / 255 -> (do things) -> c = trunc(c' * 255 + 0.5)

But I'm team

c' = (c + 0.5) / 256 -> (do things) -> c = trunc(c' * 256)

In this way, I don't need to worry about div 0 or 255 converted into infinity in the HDR color space. And if I want dithering, I can simply swap 0.5 to random noise so it's more convenient.

 

1 hour ago, Rick Brewster said:

should be replaceable with just:

c ? a : b

... At least, that's how it can be done in HLSL directly (IIUC). But I don't think we have that support in ComputeSharp.D2D1's C#->HLSL transpiler yet. I'll send the request over to @sergiopedri to see if we can figure out a way to provide this.

 

Yeah, documents saying HLSL do this, but ComputeSharp don't have bool operators for bool vectors and have bit operators instead for whatever the reason.

So if we can have the conditional operator, that's very welcome!

[Rick Brewster]

6 hours ago, _koh_ said:

That's what I get when Ctrl+Shift+C in PDN right? Each layer blended in integer.

A bit difficult to explain, but when a layer having opacity = 16, it's like the layer using 12bit color instead of 8, and when I use excessive parameters I can see the differences between FP blended images and INT blended images. But I agree this is "Why you even need this?" kind of thing.

Yes it's the same as if you had used Ctrl+Shift+C

 

The two benefits here are 1) your input pixels will match what PDN shows the user, and 2) you won't copy every layer to the GPU. If there's a lot of layers, #2 matters a LOT. Most people do not have 24GB GeForce 4090s or 48GB RTX 6000s 😂 Eventually I'll be increasing the precision of this, adding linear blending, etc. but that's not today. If you use Document.Image then you'll get any of those upgrades for free, so that's benefit #3.

 

You can also do this by using "streaming" images, which won't copy it to the GPU, it'll stream on demand and not use any extra GPU memory. Performance will be lower however. So if you really want to do the layer compositing yourself, and in linear space, and want to save GPU memory, I can show you how to do that.

 

[_koh_]

Umm. Maybe I should drop this from the published version but I'm using this in some cases and will keep it for myself anyway so I want to hear. Thanks in advance!

Hope this is as easy as mark buffers "mapped" or something like that. haha

[_koh_]

This is the deference by the way. Only visible when using very low opacity layer + making the image very dark / bright.

[Rick Brewster]

btw in PDN v5.1, I'm hoping to include a HistogramEffect2 that I've just got working. Instead of outputting a normalized array of floats for the chosen channel, it will output an array of Vector256<long> with direct value counts for all channels. It will also work on images up to 1M x 1M px (so, 1 terapixel 😂).

 

I'm using this for porting over Auto-Level, Levels, and Curves over to GPU rendering.

[_koh_]

This MS histogram is a bit cumbersome to use so that's good to hear.

In the latest version, I'm mapping input histogram to output histogram like my CPU version and I need to use at least 4096 bins to do that, but MS histogram only supports up to 1024 bins so I'm scanning the image 4 times then back calculating it. So if new histogram supports higher bin count, that's even better.
 

private Vector4[] Histogram(IDeviceImage image, int prec) {
    using var idc = DC.CreateCompatibleDeviceContext(null, new(1024, 1024), DevicePixelFormats.Prgba128Float);
    using var odc = DC.CreateCompatibleDeviceContext(null, new(), DevicePixelFormats.Prgba128Float);
    var (bins, span) = (Math.Min(prec, 4) * 256, Math.Max(prec, 4) / 4);
    var data = new Vector4[span * bins];

    var d = new RectInt32(0, 0, Environment.Document.Size);
    var t = new RectInt32(0, 0, idc.PixelSize);
    for (t.Offset(0, -t.Y); t.Y < d.Height; t.Offset(0, t.Height))
    for (t.Offset(-t.X, 0); t.X < d.Width;  t.Offset(t.Width,  0)) {
        var r = RectInt32.Intersect(d, t);
        using (idc.UseBeginDraw()) idc.DrawImage(image, null, r, compositeMode: CompositeMode.SourceCopy);
        foreach (var n in new[] {0, 1, 2}) for (var j = 0; j < span; j++) {
            var (l, o) = (255f / 256, 0.5f / 256 - (j - span + 1f) / (span * bins));
            using var size = new CropEffect(DC, idc.Bitmap, new(0, 0, r.Size));
            using var tran = new ColorMatrixEffect(DC, size, new(l,0,0,0, 0,l,0,0, 0,0,l,0, 0,0,0,1, o,o,o,0));
            using var hist = new HistogramEffect(DC);
            hist.Properties.Input.Set(tran);
            hist.Properties.Bins.SetValue(bins);
            hist.Properties.ChannelSelect.SetValue((ChannelSelector)n);
            using (odc.UseBeginDraw()) odc.DrawImage(hist);
            var (v, s) = (hist.Properties.HistogramOutput.GetValue(), (float)r.Area / d.Area);
            for (var i = 0; i < bins; i++) data[span * i + j][n] += v[i] * s;
        }
    }

    var sum = (Span<Vector4> a) => {var x = Vector4.Zero; foreach (var v in a) x += v; return x;};
    for (var i = 0; i < data.Length; i++) data[i] -= sum(data.AsSpan(Math.Max(i - span + 1, 0)..i));
    return data;
}

 

full source code LightBalanceGPU.zip

[Rick Brewster]

Supporting bin counts that high is problematic. My current implementation has poor performance at 1024. I’m trying to improve it but I have not been successful. Running multiple passes at lower bin counts will likely be better.

 

e.g. multiply image values by 8 (like with HlslBinaryOperatorEffect) and build histogram with 512 bins. Repeat with (image*8)-0.5, then (image*8)-1, etc. My histogram implementation also skips over non-finite pixels, so you could use a pixel shader to flush pixels outside of the [0,1] range: return (input == Hlsl.Saturate(input)) ? input : (float4)float.NegativeInfinity

[Rick Brewster]

Actually I think I got 4096 bins to work well ... so that seems plausible