Antialiased selection for plugins?

[MJW]

As I understand it, the selection is stored internally in a way that provides an altialiased boundary. Is there any method for a plugin to obtain a representation of the selection boundary that's more precise then that obtained by testing whether pixels are inside or outside the selection?

[Rick Brewster]

The selection is stored as a polygon, and then rasterized to an 8-bit alpha mask for the purposes of rendering. The 8-bit mask is essentially a cache, however, and there's no per-pixel manipulation.

 

What are you trying to do?

[MJW]

I have a plugin called Paste Warp+ that distorts an image in the clipboard to fit the shape of the selection. I'm hoping to eliminate the aliasing that results from the  roughness of the selection boundary. If IsVisible() with floating-point arguments works to subpixel precision, that would probably provide the information I need. If the finer-precision information for the selection isn't available, I will probably try linearly or quadratically interpolating between the selection-boarder pixels, since I'm much more concerned with eliminating the stair-stepping along the border than I am in exactly matching the selection.

 

 

[MJW]

I should check my own code more carefully. I already linearly interpolate between the selection-border pixels when in AA mode. That should smooth the border, though when I used the plugin recently, I believe I saw some very noticeable stair steps, even in AA mode. I'll need to investigate further.

 

I'd still be interested in knowing if the selection boundary can be obtained by plugins in a more precise version than that obtained by testing whether pixels are selected or not selected.

[Rick Brewster]

Effects only have access to the selection via the PdnRegion (which is just a wrapped GDI+ Region), which unfortunately doesn't give you access to the polygon. Internally the selection is a GeometryList which is basically just a PointF[][] (a "poly-polygon"). Having access to that would make it (relatively...) straightforward to determine the outline (I'm not sure how you treat "holes" for your algorithm), and then render a mesh, instead of doing per-pixel IsVisible testing.

 

If that sounds useful I can look into providing access in the next update.

[MJW]

Thank you Rick. The Paste Warp+ algorithm is very simplistic, and doesn't work with holes, or anything much fancier than convex regions.

 

I think before I make any requests for additional features, I'll try to figure out if my linear interpolation of the border pixels solves the problem I thought I saw, and if it doesn't, whether it's a weakness in the method or a bug in my implementation.

 

Not that it matters, but I don't actually do a per-pixel IsVisible() test; I do the following:

 

Rectangle[] selectionRectangles = EnvironmentParameters.GetSelection(Src.Bounds).GetRegionScansInt();
foreach (Rectangle r in selectionRectangles)
{
    for (int y = r.Top; y < r.Bottom; y++)
    {
        int selY = y - selTop;
        if (xBounds[selY].Start > r.Left)
            xBounds[selY].Start = r.Left;
        if (xBounds[selY].End < r.Right)
            xBounds[selY].End = r.Right;
    }

    for (int x = r.Left; x < r.Right; x++)
    {
        int selX = x - selLeft;
        if (yBounds[selX].Start > r.Top)
            yBounds[selX].Start = r.Top;
        if (yBounds[selX].End < r.Bottom)
            yBounds[selX].End = r.Bottom;
    }
}

 

I can't remember from where I took that idea of determining the selected pixels by  iterating over the selection rectangles, but it seems like it would be more efficient. (That is, assuming GetRegionsScansInt() doesn't call IsVisible a bunch of time to determine the selection rectangles!)

[MJW]

Oh, now I know what caused the aliasing I was seeing. I created the selection with the Magic Wand. I should have remembered, but didn't, a recent comment about the Magic Wand and aliased selections:

 

 

It also make me realize that my linear interpolation along the boundary will not solve all the problems. It's somewhat of a relief to know there's no easy solution, since it means I can move on to writing new plugins, rather than improving one from way back when.