To extend upon what Rick has said.
To a large extent performance of highly algorithmic code such as Paint.Net's filters is roughly proportional to the size in instructions of the hottest of the hot loops in the algorithm(1) and to some extent the size in bytes. If you are trying to do 64-bit math on a 32-bit X86 chip you will frequently end up not only with significant register pressure (which will cause you to go to cache more often) but also significantly more instructions. One of the primary reasons this occurs is a side effect of the fact that the chip requires we use two registers to hold a single 64-bit value, but it is even worse than that, most of the instructions which take 64-bit operands require that the same two 32-bit registers be used, so there will end up being a ton of instructions wasted moving things around from register to stack, from stack to register, etc..., and the hot inner loops often end up bigger.
Depending on the memory access characteristics of the algorithm the size of the inner loop may not end up being the defining characteristic of the performance of the algorithm, but it is frequently a good place to start when understanding these algorithms.
Footnote:
1) There are particular cases where more instructions can be faster (sometimes significantly so), for instance when getting around the processors abysmally slow idiv instruction, however these aren't the norm and getting around them requires highly specialized code which works only if you can properly constrain your inputs.
