While trying to reproduce the issue I came to realize that the issue is with my ol' trusty mouse and not with paint.net. Apparently my mouse wheel will sometimes register a click when I scroll which will cause the selection to move a few pixels.
Seems like paint.net is working fine as usual 😊