Nevermind... it just randomly fixed itself. Was trying to press X for ~10 mins and then it suddenly started working for some reason. Sorry for the spam.
Hovering over the button used to swap colors says the keybind should be "X", but pressing that does nothing. What is the keybind, or what am I doing wrong?