Plugin indexing for faster startup

[MJW]

Recently there was a bit of discussion about how slowly PDN starts when there are lots of plugins. I thought of a modification that might help, though I'm not sure it's even possible.

 

Instead of searching through the plugins each time PDN is started to get the plugin info for the menus, PDN could have a special Add Plugin mode that would search through the plugins and store the information in a file. At start-up PDN could load the file, which would, I think, reduce startup time for those with a lot of plugins. Adding plugins is a much rarer event than starting PDN, so I think requiring a bit of extra effort when they're added would be worthwhile in order to speed startup.

[Ego Eram Reputo]

Most users would be confused by dual boot modes IMHO.

 

Better that paint.net operates the way it does and the user accepts the overhead created by a large number of plugins. They can always reduce the number of plugins loaded. I've found many plugins loaded are very rarely  used.

[MJW]

There wouldn't be a dual boot mode. Starting PDN would always read the index file. A user who wanted to add a plugin would start PDN and would then select Add Plugin, perhaps under the File menu, or perhaps as one of the icons in the upper-right corner. That would either just cause the index file to be regenerated, or it could start an Add Plugin dialog. The advantage to the dialog version is that it could allow the user to browse for the plugin, and if it weren't already in the Effects directory, it could copy it there. Then it would regenerate the index file. Plugins aren't normally in the Effects directory, so that would replace one step with another one; but the new step would be better integrated into PDN. The dialog would also allow the index file to be rebuilt without adding a plugin, in case the user wanted to delete plugins. Or it could have a separate delete feature. If PDN tried to run a plugin that was no longer there, it could ask the user if the index file should be regenerated. Adding plugins is not currently an operation that happens without some user knowledge and effort, so requiring an Add Plugin operation wouldn't be a big deal.

 

EDIT: I don't know if it's possible to overwrite a plugin file if the plugin's already been loaded. There's that annoying business of the File-in-use error that occurs when trying to replace a plugin file while PDN is running. If that's not possible, the Add Plugin dialog might not be the best way to do it, and just regenerating the index file might be better. (Assuming the whole idea isn't impossible or impractical.)

Edited October 30, 2015 by MJW

[racerx]

I have 305 plug-ins installed and PDN (v4.06) loads in 3 seconds on my (i7) editing desktop. Load speed isn't a concern for me. I do remember it being slower on older systems however.

[Eli]

I have 628 files inside my effects folder  . And it takes about 5 seconds before I can do anything with paint.net. I am sure I must have effects that I do not use but you never know. 

[Ishi]

I support a decent part of this idea. A plugin manager is cool and such a tool was suggested before.

 

I imagine a two column dialog box. The ones listed on the left column are all the plugins whose corresponding DLLs are detected to be present in the Effects directory. The ones listed on the right column are the plugins currently allowed by PDN to be loaded and appear on the Effects menu.

 

Then you have an arrow ---> that says Add and another arrow <--- that says Remove. Its pretty much self-explanatory,

 

I got this idea from the ribbon customization setting of MS Office. PDN would be more sophisticated if it had this. Here's a screenshot.

 

[Drydareelin]

My old Intel Pentium laptop takes about 20 seconds to load PDN 4.0.6. I don't have that many plugins either.

... Getting a new laptop today though

[skullbonz]

If people can't wait a few seconds for the program to start then maybe they shouldn't be graphic artists.  I'm surprised no one has asked for a plugin that automatically makes a picture for them. Oh wait, they already did,lol. 

[Rick Brewster]

Neat idea, but ... there are unfortunately way too many problems with it once we get past the idea stage.

 

For one, it's just begging for inconsistencies between the real list of plugins and "the index." It's partly the same reason why I will never add a "font cache" or whatever. There's already a "font cache" on the system, it's called the file system. You'd end up with 2 ways to delete plugins, one of which is secretly wrong (why can't you just delete it from the folder?!), and it would cause no end of confusion and trouble ("I deleted the plugin, why is it still in the Effects menu?!"). As for adding plugins, you already added the plugin by copying it to the right folder on your hard drive, why should I add another step to that? Yes, an "add plugin dialog" would help automate that, but then you'd still have people copying it to the folder manually and then wondering "I added the plugin, why isn't it showing up?!" Oh! Wait! paint.net could rescan the folder and look for inconsistencies between the index and the file system but then that would be its own performance problem and bug farm that we only have because we created an index ... And what if the index gets corrupted or broken somehow? Does that cause a crash? If the plugin index were in your user folder (vs. Program Files) it would be a security vulnerability and attack target. Looks like something/someone malicious added C:\virus\hahaha.dll to the plugin index, uh oh ... By requiring plugins to be in C:\Program Files\paint.net\Effects, it puts an administrator privilege barrier in front of a security attack like that which would be completely circumvented by having a plugin index in the user's folder (remember, plugins are code, and code can do bad things). If the plugin index were in C:\Program Files\paint.net then it would require administrator privilege to reindex the plugins if an inconsistency were detected, which means a UAC dialog that the user might click "no" on which would make the situtation even worse. And that tends to be a bug farm as well; the updater is the only place that paint.net requires admin/elevation, and that was not exactly a peaceful and painless feature to birth. What if recreating the plugin index encounters a problem? Does paint.net then crash, restart, and try to recreate the plugin index again? This would create an infinite loop and the user would be locked out of paint.net.

 

There are ways to solve many or all of these, but we're still greatly increasing the complexity of a system that already works for the sake of a bunch of people who have way too many plugins and who haven't upgraded to an SSD yet. Really, all you need is fewer plugins and/or an SSD and those are pretty cheap nowadays. This problem is solving itself over time as people upgrade or replace their computers (or install fewer plugins ...), and it's happening without any need for a "plugin index" that'll just get broken and have bugs. And it would take a TON of my time to engineer and debug this and it won't be fun and I'll hate it and only a few people will even benefit from it.

 

As for the screenshot Ishi posted: sure, paint.net would be "more sophisticated" with something like that. I think it'd also be rather confusing and overly complicated and unnecessary.

 

In addition, adding tons and tons of plugins tends to make things unstable, especially on 32-bit systems. The menus also become clumsy and annoying because WinForms doesn't handle it very well. It's also, to be honest, not a common situation when you consider all paint.net users. This would be optimizing and encouraging an uncommon, degenerate configuration.

 

I don't mean to squash your enthusiasm or discourage your ideas or anything, but ... this just won't be happening.

[Pavele]

Hi Rick,

 

A bit old discussion, but here are my 5 cents.

First of all I greatly appreciate all the efforts you do to develop and improve Paint.net! Good job an well done! Keep the good work!

Here is my case. I have been trying different plug-ins looking for useful ones. I have installed different packages (bunches). I liked some tools from some packages, but ended up with around 900 dlls (I know, I know) in the Effects folder. I have a SSD but still the startup time is couple of minutes (!). It is difficult to delete the right 800 redundant dlls. The question here is which one to delete. Which may be a common paradigm with installing a plug-in package. Therefore a plug in manager is a good idea in general.

As a long time c++/.net developer I would dare to propose having a dedicated app for plug-in management, possibly with the UI similar to what other people suggested or to what is commonly recognizable. I imagine this app will not start if Paint.net is running.

The idea of plug-in representation cache is not bad IMHO. The cache can be easily checked and synchronized on startup against dll's in  Effects (and  other trusted) folder using CRC, hash, file write time+size or whatever "match" criteria is best. There is no problem for the cache file(s) to be in user's data folder as it will not refer to absolute paths (which can be checked).

 

If anyone has an idea how to leave only the right dlls among all without losing a day - please do not hesitate to share it.

 

Regards

 

[Rick Brewster]

You can't use a CRC/hash for the index. The whole point of an index is to avoid all of that disk I/O on startup. If you use a CRC/hash then by definition you have to read the entire contents of the DLL. And the code in the DLL has to be executed in order to populate the menu.

 

At best you're looking at having this work (that is, loading all of the plugins) being done in background thread(s) while you do other stuff other than click on the Effects or Adjustments menu. And it _will_ interfere with things like File->Open. However, it may still be an improved experience, and it may happen.

 

A plugin manager is a different notion. Making it straightforward to figure out which DLLs go with which plugin, in a concentrated way, is definitely a good idea. Right now you can determine which DLL goes to which effect with the little tooltip on the menu item. But that doesn't tell you which other effects are in that DLL, and it's pretty clumsy.

 

I will fight against the idea of an index until my dying breath  Not just because I think it's a bad idea, but because I think it's a lazy idea. There are many other apps out there that index your darned fonts at startup. Ever boot up GIMP or VLC and have it make you wait for an extra minute because it has to index your fonts? That's really stupid. Paint.NET doesn't index your fonts and it has the fastest font chooser ever. No index needed, I just had to be smart, careful, and a little clever about how I did things. I haven't really applied that to effect loading yet.

[Ego Eram Reputo]

Random idea: Effect load profiles.

 

I run (at least) two versions of PDN installed side-by-side. I do this to have different sets of plugins available. The two versions gives me a minimal plugin and maximum plugin options with matching load times.

 

I would dearly love to be able to have some way of specifying:

 

a. Load plugins x,y & z  with each instance of PDN.

b. If minimal profile is selected, additionally load plugins a, b & c.

c. If maximum selected, additionally load plugins a, b, c, k, l, m, h, s..................................................... & w

or

d. Load zero plugins.

 

[Pavele]

13 hours ago, Rick Brewster said:

You can't use a CRC/hash for the index. The whole point of an index is to avoid all of that disk I/O on startup. If you use a CRC/hash then by definition you have to read the entire contents of the DLL. And the code in the DLL has to be executed in order to populate the menu.

 

At best you're looking at having this work (that is, loading all of the plugins) being done in background thread(s) while you do other stuff other than click on the Effects or Adjustments menu. And it _will_ interfere with things like File->Open. However, it may still be an improved experience, and it may happen.

 

A plugin manager is a different notion. Making it straightforward to figure out which DLLs go with which plugin, in a concentrated way, is definitely a good idea. Right now you can determine which DLL goes to which effect with the little tooltip on the menu item. But that doesn't tell you which other effects are in that DLL, and it's pretty clumsy.

 

I will fight against the idea of an index until my dying breath  Not just because I think it's a bad idea, but because I think it's a lazy idea. There are many other apps out there that index your darned fonts at startup. Ever boot up GIMP or VLC and have it make you wait for an extra minute because it has to index your fonts? That's really stupid. Paint.NET doesn't index your fonts and it has the fastest font chooser ever. No index needed, I just had to be smart, careful, and a little clever about how I did things. I haven't really applied that to effect loading yet.

 

Hi Rick,

 

Thanks for your reply. I do not want you to die by any mean so let's forget about the index (I see your point in using hash on DLLs and I do agree. IMHO file size + file times would give you pretty reliable and fast hash value in this case. The index will contain everything that Paint.net extracts from the DLL in the fastest to load form. But as we said - no dying.)

Some facilitated way of plugin management seems to be a good idea though.

In my case I was thinking of the possibility to have right click on an effect menu item, display a popup context menu with an option "remove from Paint.net", maybe along with other useful and related tasks. This will remove temporarily the item and permanently (remove the dlls) on the next startup. Just a simple to implement idea IMHO.

 

Again, thanks for your great work!

Regards.