Hi guys, I believe I have fixed this issue on my system.
I searched around on my HD for something related to this Font Cache, and found the following directory:
C:\Windows\ServiceProfiles\LocalService\AppData\Local\
Inside were these files, which I deleted (made backups first just in case)
FontCache-S-1-5-21-1948862929-1947518890-988191046-1000-8192.dat
FontCache-System.dat
FontCache3.0.0.0.dat
once these files were deleted, it seemed to build this font cache almost instantly. Maybe one of the files was corrupt or had bad permissions? Anyway, still not 100% sure this fix is permanent but for now, it's worked for me.