New packaging requirements for plugins starting with PDN 4.3 / .NET 5

[Rick Brewster]

I've added a new rule for plugins regarding packaging and installation when a plugin needs to use additional DLLs or "shared libraries": https://forums.getpaint.net/topic/13129-rules-for-plugins-that-are-published-on-this-forum/

 

 

Quote

Plugins that use shared libraries must be packaged and installed correctly
If your plugin is just 1 DLL, then it can go directly into the Effects or FileTypes directories. However, if you have multiple DLLs, and/or are using a shared library (like, say, NewtonsoftJson.dll), then your plugin must be packaged and installed correctly. It must: 1) target PDN 4.3+, 2) it must be installed to its own folder within Effects/FileTypes, and 3) it must have a .deps.json file, which is automatically created by Visual Studio at build time. This ensures that plugins can load their own private copies of any additional DLLs, and that plugin dependencies won't interfere with each other or with the app itself.

 

Starting with PDN 4.3, plugins are loaded into isolated load contexts (AssemblyLoadContext, more info here). This makes it possible for a plugin to load its own dependencies without having them conflict with other plugins that may have dependencies with the same assembly name, but a different version. This also avoids conflict with any DLLs that the app itself is using. For instance, if a plugin wants to use NewtonsoftJson, that's fine -- but it's important to make sure that the plugin doesn't use the version of this DLL that the app is using because it could cause the plugin the break in the future when the app uses vNext but the plugin needs vPrevious.

 

Please ask any questions, or post here if you need help getting this to work for your plugin.

[Rick Brewster]

If your plugin is just 1 DLL then it doesn't need to worry about this. Most (all?) CodeLab plugins won't need this, in other words.

 

You can also merge your dependencies into your DLL using merging or linking, which then avoids the need for a separate folder or the .deps.json file. I haven't done this with a DLL so I'm not sure what's required, but @BoltBait and @toe_head2001 use this with CodeLab so it's definitely possible and they have experience here.

[midora]

Maybe I missed something, but if a shared library must be installed in each plugin folder which uses the shared library then it's no longer a shared library or? It's just a library.

 

[Rick Brewster]

It's a "shared" library from the perspective of your plugin. Probably not the best word to be using, it's more of a legacy term now.

[midora]

Then all libraries are shared. It's just not the case that there is a single place to replace and update a library used from different plugins if there is a bug or an enhancement like a new supported language.

[Rick Brewster]

If a plugin has DLL dependencies, then the plugin needs to be put into its own folder, with the dependencies, and with a .deps.json file. It doesn't matter what terms are used, that's what has to happen.

[otuncelli]

.deps.json is meaningless for unmanaged dll dependencies. But tbh, subfolder is great feature for both plugin managability and code isolation. I'm encouraging my plugin' users to install in its own subfolder even it has no shared dlls.

Edited October 7, 2021 by otuncelli

[Rick Brewster]

.deps.json still has use for native dependencies. Look at the .deps.json for an application, you'll see references to them in there.

[otuncelli]

48 minutes ago, Rick Brewster said:

.deps.json still has use for native dependencies. Look at the .deps.json for an application, you'll see references to them in there.

 

I checked paintdotnet.deps.json file. C++/CLI libraries have references (e.g. PaintDotNet.SystemLayer.Native.x86/x64/arm64.dll) but C libraries don't. I doubt that Visual Studio is that clever to generate .deps.json for p/invoked C libraries.

 

I'm going to reference one of the bundled plugins here. WebP plugin has native libraries but I can't find any reference for WebP_x64/x86/ARM64.dll file in paintdotnet.deps.json file and it doesn't have WebPFileType.deps.json as well.

 

Should they be manually added?

Edited October 7, 2021 by otuncelli
typo

[MJW]

I know very little about this stuff, so forgive me if this comment doesn't make much sense. That said . . .

 

As I understand it, the original idea behind DLLs as shared libraries is to avoid having multiple copies of the same code. The new plugin requirement seems to be that each plugin must include its own copy of the "shared" library. If that's so, is there any advantage for a plugin to use separate DLLs rather than link in the libraries into a single DLL at built time, the way BoltBait does with CodeLab? If there isn't, I think that's the method that should be encouraged, provided it isn't too difficult to do.

 

Specifically, does using separate DLLs use less memory space if multiple plugins share the same libraries, even though each plugin must have a copy of the DLL on the hard drive? Or does the AssemblLoadContext stuff mean each has its own copy in memory of the library routines it uses. (Again, forgive me if this is a nonsensical question.)

 

EDIT: When I looked into something related to this a long time ago, I was surprised and disappointed that Visual Studio doesn't provide any method for statically linking in libraries when the program is built. I downloaded the program BoltBait uses for building CodeLab, but I've never tried using it.

[Reptillian]

Looks like I will have to make some changes to my g'mic-pdn project though I find it to be difficult to make changes to suit the new rule.

Edited October 7, 2021 by Reptillian

[NinthDesertDude]

2 hours ago, Rick Brewster said:

If a plugin has DLL dependencies, then the plugin needs to be put into its own folder, with the dependencies, and with a .deps.json file. It doesn't matter what terms are used, that's what has to happen.

Due to supporting tablet pressure sensitivity, I now have 2 DLLs. The winforms solution doesn't involve deps.json anywhere and doesn't output it. I can't find a source describing this process. I could manually make the file, which I probably will have to do, but it's a little iffy whether it'll have correct data or not when I do that.

[toe_head2001]

1 hour ago, NinthDesertDude said:

The winforms solution doesn't involve deps.json anywhere and doesn't output it.

 

If you're compiling with .NET Framework instead of .NET 5, you will not see a .deps.json file.

[Rick Brewster]

3 hours ago, MJW said:

As I understand it, the original idea behind DLLs as shared libraries is to avoid having multiple copies of the same code. The new plugin requirement seems to be that each plugin must include its own copy of the "shared" library. If that's so, is there any advantage for a plugin to use separate DLLs rather than link in the libraries into a single DLL at built time, the way BoltBait does with CodeLab? If there isn't, I think that's the method should be encouraged, provided it isn't too difficult to do.

 

Yes, each plugin will load its own copy of any DLL dependencies they have. The benefits of having these "shared" is massively outweighed by the benefits of keeping plugins as isolated from each other as possible. Frankly, it's been hell having all the DLLs in one folder with the capability to stomp on each other's dependencies. I'm not concerned about memory usage, it hasn't proven to be an issue. DLLs don't really take up that much memory compared to whatever else gets loaded (the images you actually work with, i.o.w.). If you install a ton of plugins, memory usage goes up. If it's too much then your system probably doesn't have much memory to begin with and this really isn't the problem to optimize (go buy more RAM first).

 

Merging (aka linking) is a great solution, it does simplify plugin installation albeit at the expense of some added complexity for the developer. I'd encourage it more if I had an easy solution to offer up for folks to use. Maybe @BoltBait / @toe_head2001 have something to share here, a tutorial perhaps.

[BoltBait]

3 minutes ago, Rick Brewster said:

Merging (aka linking) is a great solution

 

Yeah, when it works. CodeLab is a perfect example.

 

4 minutes ago, Rick Brewster said:

Maybe @BoltBait / @toe_head2001 have something to share here, a tutorial perhaps.

 

I tried to do that with the update to my BBChart.dll plugin... it just didn't work.

 

So, I'm 50/50 wrt merging.

[Ego Eram Reputo]

Yep - I failed to bundle WFPMath.dll with my MathLaTeX plugin. I'll have to have another look at it.

[Reptillian]

11 hours ago, Rick Brewster said:

2) it must be installed to its own folder within Effects/FileTypes

Requesting a quick clarification as I'm not sure what's the requirement is in my case.

 

I have GMICSharpNative folder inside the effects folder or document/app files folder. So, every dlls that utilize GMICSharp library must be inside the folder? That's what you mean?

 

As for the deps thing, going into .NET 5 with this setup is beyond me, so I can't do anything about that unfortunately though I will have assistance from @null54as he worked on this.

[Rick Brewster]

You can still put other DLLs into a 3rd level folder (e.g. Effects/YourPlugin/Folder), that's fine, assuming you load them using the correct path. Paint.NET won't find or load any plugin DLLs beyond the 2nd level folder. "Plugin DLL" meaning a DLL with an Effect or IFileTypeFactory[2] in it. Any other DLLs can be organized as you see fit into further subfolders.

[midora]

10 hours ago, Rick Brewster said:

It doesn't matter what terms are used, that's what has to happen.

It always matters which term is used to get an effective communication. I'm not complaining about your rules, just saying that using of shared libraries is no longer possible. And it makes no sense to invest time in multi-threading protection of shared objects.

[Rick Brewster]

I used the term "shared library" as a bridge term. Libraries not being "shared" is kind of an implementation detail at this point.

 

For the race condition in current releases of OBL, it's not too difficult to fix that (if it needed fixing, I mean). ConcurrentDictionary would have been fine. 

 

If you can provide an explanation of why a library would actually need to be shared, like if there's an important scenario it enables, I'm open to listening and maybe finding a way to enable it. It's important that the default policy for loading plugins is full isolation, but there's always room for relaxing that when it's important.

[midora]

That's fair enough.

 

I'm not sure that ConcurrentDirectory is needed. As far as I know Dictionary has no problem with concurrent read. Maybe there is an issue at startup if multiple plugins try to to initialize the Dictionary. A lock should be fine in this case.

 

[otuncelli]

5 hours ago, midora said:

That's fair enough.

 

I'm not sure that ConcurrentDirectory is needed. As far as I know Dictionary has no problem with concurrent read. Maybe there is an issue at startup if multiple plugins try to to initialize the Dictionary. A lock should be fine in this case.

 

 

You can also try wrapping the Dictionary to Lazy<T> if it needs to be initialized at startup and only once. This'll both solve the problem and improve the startup performance.

[VcSaJen]

I use Magick.NET library, which uses .deps.json for it's native DLLs. Paint.NET seemingly ignore .deps.json file located in my plugin's folder, resulting in error:

System.TypeInitializationException: The type initializer for 'NativeMagickSettings' threw an exception.
 ---> System.DllNotFoundException: Unable to load DLL 'Magick.Native-Q8-x64.dll' or one of its dependencies: The specified module could not be found. (0x8007007E)
   at ImageMagick.Environment.NativeMethods.X64.Environment_Initialize()
   at ImageMagick.Environment.NativeEnvironment.Initialize() in /_/src/Magick.NET/Native/Helpers/Environment.cs:line 50
   at ImageMagick.Environment.Initialize() in /_/src/Magick.NET/Helpers/Environment.cs:line 21
   at ImageMagick.MagickSettings.NativeMagickSettings..cctor() in /_/src/Magick.NET/Native/Settings/MagickSettings.cs:line 230
   --- End of inner exception stack trace ---
   at ImageMagick.MagickSettings.NativeMagickSettings..ctor() in /_/src/Magick.NET/Native/Settings/MagickSettings.cs:line 246
   at ImageMagick.MagickSettings..ctor() in /_/src/Magick.NET/Settings/MagickSettings.cs:line 33
   at ImageMagick.MagickImage..ctor() in /_/src/Magick.NET/MagickImage.cs:line 39
   at MyFileType.MyFileTypePlugin.OnLoad(Stream input) in C:\full_path\MyFileType\MyFileType\FileType.cs:line 210
   at PaintDotNet.FileType.Load(Stream input) in D:\src\pdn\src\Data\FileType.cs:line 498
   at PaintDotNet.Functional.Func.Eval[T1,TRet](Func`2 f, T1 arg1) in D:\src\pdn\src\Base\Functional\Func.cs:line 158

Magick.Native-Q8-x64.dll is located in runtimes/win-x64/native/Magick.Native-Q8-x64.dll directory inside my plugin's directory and present in .deps.json (targets -> .NETCoreApp,Version=v5.0 -> Magick.NET-Q8-AnyCPU/8.3.3 -> runtimeTargets -> runtimes/win-x86/native/Magick.Native-Q8-x86.dll).

[Rick Brewster]

@VcSaJen, can you provide a link to a ZIP of your packaged-up-plugin so I can try it out?

[VcSaJen]

@Rick Brewster It's been sent.

I dunno if "System.*.dll" files are necessary, so I didn't include them.

Edited October 23, 2021 by VcSaJen