What tricks come to mind include:
- only drawing the visible portion of the canvas (after transformations)
- keeping a merged copy in memory of the visible portion of the canvas for all layers above or below the current, I don't know if PDN does this but it would be faster than merging each pixel per layer on every frame to redraw it. So this means 3 images in total: one for all layers above (merged), one for the active layer, one for all layers below (merged). For large images, you'd probably want to keep the temporary copies saved to disk or only generated for the visible portion of the canvas to avoid overloading RAM. I believe paint.net layers automatically have the ability to save & load from disk already, so it's probably doing that much at least
- load and unload the image in chunks to handle extremely large images. Loading it all in memory would quickly fail for a 10x10k image
- use GPU not CPU. Older version of paint.net were not GPU optimized
I'm sure Rick can pour over it in more detail and with better accuracy since I'm not actually a paint.net dev. I just have a plugin.
You could just copy a thin rectangle of pixels near the edges and put them over the rough edges so they're nice and straight.
Or take Pixey's edge blur approach and copy all the modified edges to some layer that you run AA's assistant on since you wanted those edges to 'blend' better.
It looks like the entire canvas is visible in this picture. You have it fully occupied by opaque pixels from the two layers above the one you're posting, so you won't see anything. (Also, the size of the image you're posting is much larger than the canvas, so most of it will be lost when you're done pasting.)
Click the checkmarks next to the layers (bottom right) that are above the current layer, and they will be invisible. Then you can see what you're posting. Or move the current layer above them, either or.
NinthDesertDude's post in Technical question about drawing of bitmaps and primitives was marked as the answer