RAM and Storage question vis-a-vis Number of Layers

[HankP]

I have a question about how PDN utilizes memory and stores info.  In particular, I want to know whether I am storing information inefficiently.

 

I am making a series of maps to accompany something that I am writing.  To simplify, say that I have three "groups" of Layers for the fill color of the maps.  One Layer is just the national fill, in a single color.  The next group of Layers is the States (let's say 20 Layers, one for each state on the map), with one color per state.  Next, there is a group of Layers (one for each state), with the fill of each state being a dozen or so colors, for the counties.  So, I have (for example) about 41 layers.

 

My question is whether I would save storage, save RAM and enhance speed and performance, if I were (a) to merge all the state Layers and (b) to merge all the county Layers, such that I would end-up with three Layers, rather than forty-one of them.  The number of colored pixels remains exactly the same, but the number of layers is decreased dramatically.

 

I have similar issues with other layers than the color fill.  There is a similar hierarchy (for example) for boundary lines.  Several different layers for rivers of increasing size and importance.  Multiple layers for towns/cities based upon size/population, etcetera.  I've not actually counted them, but my best guess is that this file has 100+ Layers.

 

First, would merging some of these Layers make the file smaller on storage?  Second, would this use less RAM?  Third, would this increase speed and performance?

 

I would rather not make this change while I am still manipulating the maps, but I am willing to do so if I can increase performance because (frankly) things are getting rather laggy.

 

Thanks.

Edited February 4 by HankP

[Tactilis]

Every time you add a layer, you increase the memory required.

 

Each new layer needs approximately another image width x image height x 4 (for the RGBA pixel value) bytes.

You can easily see this for yourself:

1. Start paint.net and resize the blank canvas to 2000 x 1000 pixels (for example).
2. Run Task Manager and type paint.net in the search field. You should now see just the paint.net process. Note the memory usage when it stabilises.
3. In paint.net type Ctrl+Shift+N to create a new layer. Note the increased memory usage.

4. Repeat step 3 again and again until you have, for example, 20 layers.
5. Now type Ctrl+Shift+F to flatten all the layers. Note the reduced memory usage.

So, yes, merging the layers will reduce the memory requirement. Whether this is significant depends on your image size and the amount of RAM you have.

 

Do you have a Hard Disk (HDD) or an SSD?  Each time you make an edit, paint.net has to write history data for Undo/Redo out to disk. An SSD will be considerably faster for this than writing to a HDD, so will reduce lag.


When you save to a .pdn file, the image data for each layer is saved in a compressed format. The degree of compression will depend on the nature of the image in each layer.

 

You can easily see this for yourself:

1. Start paint.net and resize the blank canvas to 2000 x 1000 pixels (for example).
2. Save the image to a .pdn file. Note the size of the file.

3. Type Ctrl+Shift+N to create a new layer. Save the file. Note the increased file size.

4. Repeat step 3 again and again until you have, for example, 20 layers. Save the file. Note the increased file size.

If instead of a blank (uniform colour) canvas, you create a colour gradient, you'll find that the image won't compress so well. Try it and see the increased image size.

The .pdn file for your multi-layer maps should compress well because each State's layer will, I assume, mostly comprise pixels of a uniform (i.e. transparent) colour plus a uniform colour for the State fill.


The bottom line is that having plenty of RAM (it's not that expensive) and an SSD will give you the biggest performance boost, and allow you to work with more layers.

The size of the .pdn file is likely to be a minor consideration in your situation.

 

[HankP]

Thanks much.  I suspected that it was more a RAM issue than a storage issue, but I wanted to be thorough.

 

You are right.  RAM is cheap.  I used to build all my own desktop computers, and it was easy to add RAM as I needed it.  I've not built a desktop in a while and use an HP Pavilion laptop as my primary.  The thing is so thin that it never occurred to me that I COULD add RAM.  I will look into that possibility immediately.

 

I have literally hundreds of layers, and it IS getting cumbersome.  DAMN but I wish someone would complete the task of that upgrade to group/folder Layers (hint, hint).

 

Is there a point of diminishing returns on adding RAM for PDN?  I have only 12gb currently, and it is more than adequate for absolutely everything that I do ... except for PDN.  I can essentially triple the RAM to 32gb for about $70.  I can go to 64gb for $120 (which is fine), but will going that high provide me with any real incremental benefit in using PDN, vis-a-vis 32gb?  ( I can even get up to 128gb, but would have trouble justifying $600 for RAM alone, just to speed-up one software app, LoL).

 

As an aside, Task Manager says that PDN is using about 3gb of memory right now, which is a TINY fraction of what should be use per THIS FORMULA that I found on the forum:  (Width x Height x 4) x (Layers +2)

 

Thanks again for the advice.

Edited February 4 by HankP

[Red ochre]

Interesting question and informative answer.

If you cannot add internal R.A.M., adding a quick 'thumb/flash drive' memory stick via U.S.B. and setting it to 'ready boost' sometimes helps.

In this particular case (with simple gradients), possibly worth investigating an S.V.G. editor too (e.g. Inkscape - free), for smaller file sizes, might be an alternative?. (I know, heretical!).

Only rough thoughts... personally I find working with vector graphics challenging to say the least!

[Rick Brewster]

Adding layers also has a linear effect on performance. Roughly speaking, having 100 layers will take 10x as much CPU time as only having 10 layers.

[HankP]

So, a Layer uses CPU time, even if it isn't doing anything?  It seems counter-intuitive until you think of it like a programmer, at which point I suppose it makes sense.  Thus, use as many Layers as necessary to get to the point of having the job done on one task, then when that task is complete merge them to speed CPU processing time and move to the next task?  That is hard for me, because I am constantly tweaking, but I suppose I can train myself.  (self-deprecating chuckle)

 

Can you provide any more info on whether there is a point of diminishing returns, where adding more RAM does not really improve performance much?  From what you said, I suppose that each Layer is eating processing time, regardless of whether you have 4gb of RAM or 32gb.  The effect of Layers on processing time is essentially constant, as to the amount of RAM?  But set that aside for a moment.  If I DO have (for example) 100 Layers, do I get any real, incremental benefit in performance between 32gb of RAM and 64gb of RAM?

 

Just an "old guy ponders life" point, but I remember when we discussed RAM in terms of KILObytes rather than GIGAbytes and storage in MEGAbytes rather than TERRAbytes.  I remember how excited I was for my first 10mb hard drive on an IBM PC-XT.

Edited February 4 by HankP

[Tactilis]

18 hours ago, HankP said:

So, a Layer uses CPU time, even if it isn't doing anything?

Each layer contributes to the rendered (i.e. flattened) image that you see on the screen.

 

Each time you make an edit, the layers have to be merged in accordance with their Opacity and Blend Mode to achieve the final image.

 

I'm not sure in v5.x to what extent the GPU handles this but for sure the CPU is still involved, so more layers means more CPU usage.


 

18 hours ago, HankP said:

Can you provide any more info on whether there is a point of diminishing returns, where adding more RAM does not really improve performance much?

Not without more information.

You have stated that you have 100s of layers but you haven't told us what your image width and height is.

Also, in paint.net could you go to Settings -> Diagnostics -> Copy to clipboard and paste the info here. This will tell us your laptop's hardware capabilities.
 

 

 

[Rick Brewster]

10 hours ago, Tactilis said:

I'm not sure in v5.x to what extent the GPU handles this

This is unchanged in 5.0. Layer composition still happens on the CPU and that is unlikely to change because of how much GPU VRAM would be required to hold all your layers. I tried it in the early days of 4.0’s development and it was a disaster. And you’d still have linear performance with respect to layer count.

 

Nowadays we have quite decent float32 performance on the CPU and I’m not too worried about it for the next engine upgrade.

 

It would work fine on a pro/workstation class card with lots of memory (up to 48GB!). These cards, such as the NVIDIA RTX 5000 Ada, have twice the RAM but cost 4x as much compared to the equivalent consumer card. So you get 32GB for $4000 versus a GeForce 4080 Super with 16GB for $1000. Not worth it!

[HankP]

Update:  I just decided to order two 32gb sticks.  RAM is cheap, and having "too much" won't hurt anything.  With 64gb,  I should be able to Paint.net the heck out of my maps.

Edited February 6 by HankP

[Tactilis]

7 hours ago, HankP said:

I just decided to order two 32gb sticks.

Did you confirm that your HP Pavillion will be able to use the full 64GB?

Many laptops have limitations that will provide access to 32GB total, even though you could physically fit 2 x 32GB.

 

[HankP]

10 hours ago, Tactilis said:

Did you confirm that your HP Pavillion will be able to use the full 64GB?  Many laptops have limitations that will provide access to 32GB total, even though you could physically fit 2 x 32GB.

I did. I may not understand exactly how PDN utilizes memory or CPU time, but I have been building my own computers for more than 30 years.  I can handle the simple hardware issues.  (chuckle)

 

Thank you for the follow-up and the advice upthread.

[HankP]

On 2/3/2024 at 9:04 PM, Rick Brewster said:

Adding layers also has a linear effect on performance. Roughly speaking, having 100 layers will take 10x as much CPU time as only having 10 layers.

I meant to ask this question earlier, but forgot.

 

Does this apply to ALL Layers, or only to visible Layers.  In other words, is PDN still actively processing Layers that are not even in use at the time?  (Unimportant Corollary Question: If so, why?)

 

I ask this because I have a tendency to us real-world maps as a lower-level Layer for tracing my (simplified) maps on higher Layers.  After I think I am finished with a real-world map Layer, I have a tendency to toggle it "off" and to then drop it down below an all-White background Layer and leave it there ... in case I want to come back an use it later to add something to my active map Layers.  Since I have rotated, resized and otherwise manipulated the real-world map Layer, I don't want to lose all that work, which (sometimes) can amount to hours of work.

 

If these (probably now extraneous) invisible and inactive Layers are eating materially into processing time (and RAM?), I may want to start moving them into a parallel file and deleting them from the main map file ... to better utilize resources.  It is a small hassle to bring that Layer BACK into the main map, if I want to use something else from that map in the new file, but if it frees material amounts of hardware resources it may be worth that effort.  (BTW, Intel i7 processor)

 

Thanks.

[Tactilis]

1 hour ago, HankP said:

If these (probably now extraneous) invisible and inactive Layers are eating materially into processing time (and RAM?)

RAM
Whether a layer is visible or not will make no difference to its memory requirements. It still has to hold the pixel data of approx W x H x 4 bytes; changing the visibility doesn't delete the pixels. You can confirm this yourself by creating multiple layers, as described in my reply to your initial post, and then toggling their visibility to off. The paint.net memory usage does not decrease.

CPU
It would make sense, but Rick will need to confirm, that non-visible layers are excluded from layer composition. Therefore the CPU usage will be reduced if you toggle layer visibility off.

 

You used the phrase "invisible and inactive Layers".
Note that all layers, apart from the one that is currently highlighted, are inactive, regardless of their visibility.
The highlighted layer is the active layer, regardless of its visibility. Active in paint.net terms means the layer that will be affected by drawing operations. You can still draw on the active layer while it is invisible.

I suspect that "inactive" in your sense simply means layers that you are not interested in at the present time. I can confirm that your lack of interest in them will not influence RAM or CPU usage 😜

 

 

[Rick Brewster]

2 hours ago, HankP said:

Does this apply to ALL Layers, or only to visible Layers

 

Rendering performance is only affected by visible layers. (Technically speaking, non-visible layers still require some processing but it's negligible.)

[HankP]

3 hours ago, Rick Brewster said:

 

Rendering performance is only affected by visible layers. (Technically speaking, non-visible layers still require some processing but it's negligible.)

Great info!  Many thanks

[HankP]

Update.  two words.  Greased Lightning.

 

PDN with 64gb of RAM runs like Usain Bolt.

 

I had not realized how laggy the app had gotten as I allowed my canvas sizes and number of layers to increase incrementally.  Right now, I have two files open with a canvas of about 30,000 x 24,000 pixels and two dozen Layers.  PDN is not batting an eye.  Task Manager says I am barely using 2/3 of the  RAM, even with those two outrageous files.  (They are very temporary)

Edited February 12 by HankP