Before I get too far down this path, what are the limitations and thoughts on creating plugins that use generative AI tools?

[RLH]

First, I have read much of the guidelines about what's acceptable around here, as well as what Effects can do. I still have a couple of questions I have regarding what the plugin features are capable of and, further, what is allowed to be published on here.  I'd rather not write a series of plugins that do not meet some form of community standard.

 

Some personal background.  I've used Paint.Net for decades. Every time I get a new PC, whether personal or for work, one of the first downloads I download and install has been this application, along with Visual Studio, Notepad++ and Sql Server.  I love it, and I consider it essential.  I'm a developer and not an artist, so I love the simplicity of this tool, paired with functionality that's well beyond Windows Paint, but also isn't something as complex as PhotoShop.

 

However, this is my first time ever considering writing a plug-in.  In my case, I'd like to use currently available tools to provide plugins that offer new features that we are seeing in other image tools that leverage AI such as prompt-based image generation, inpainting or upscaling. It is not my intent to roll my own versions of these features, but rather, take implementations already in .Net that can work with Stable Diffusion, and place rather simple, straight forward dialogs that allow barebones implementations of these features.

 

My question is, at least within the terms of acceptable plugin behavior, is this allowed, and can it be done?  I'm pretty sure I could write an image generator, as well as an inpainting plug-in, but what about upscaling?  Specifically, does the plugin library allow for the image canvas resizing, or am I stuck using the canvas, layers or selections at the fixed sizes that they are within memory.  Obviously, if you have a 1024x1024 image and want to scale it 2-4x, you need to have the ability to resize the canvas, but you can't do that in a plugin effect, then you can implement an upscaler.

 

I'm not trying to stir anything up, and apologies if others are asking these questions.  And to be clear, this is not me stating "Hey! it'd be great if one of you capable, and competent devs would do this!"  No, I'd like to do this as a both learning experience, and to offer a feature to the community that's quickly becoming ubiquitous on all other platforms. I've working with generative AI imaging for several months now, and I'd just like to make a few simple implementations of these features for my favorite image editor.

 

I'd appreciate any advisement on this topic of discussion.   Any tips for getting started, limitations I'm not thinking of or, of course, any guidance that tells me why this isn't possible with PDN.  Thank you.

[_koh_]

Effect plugins can't resize the image, but they can stretch part of the image to full image.

Scaler effect I recently made works in this way. Stretch the selection to the image or shrink the image to the selection.

But yeah, I still need to change the image size manually.

[BoltBait]

You cannot change the canvas size. That said, you can resize an image and place it on the clipboard.

 

Also, you are limited to changing the pixels that are on the active layer of the current image.

[_koh_]

Ah, you are not going to do per pixel rendering in this case, so just receive the result in the clipboard and paste into new image could be more convenient.

[RLH]

On 11/14/2024 at 12:20 PM, _koh_ said:

Ah, you are not going to do per pixel rendering in this case, so just receive the result in the clipboard and paste into new image could be more convenient.

If that's an option, then that could work.  Inpainting should work on an image.  If the guideline.  I may have to get creative and paint over only whatever is selected.  Typically, inpainting over an image is derived by either selecting a color, transparent pixels or a mask image.

 

Creating a new image from a prompt would be easy--you have to specify a canvas size, so either default to the size of the current layer and fill it with the rendered contents.

 

[_koh_]

Copy the original image to the clipboard -> user manually resize the canvas -> plugin fits the clipboard image to the canvas

Plugin can do this and maybe this is the most straightforward way to do scaling. Seems like I need to modify my scaler plugin.