#1. Are you used GMLMatting0.3-setup.exe for the GMLMatting.8bf installation?
If yes
#2. Drag both files GMLMatting.8bf & GMLMatting.ini from C:\Program Files (or C:\Program Files (x86)) to C:\Program Files\paint.net\Effects
#3. Check existence of the cxcore097.dll file in the С:\Windows\Sys or С:\Windows\SysWOW64 folder.
This filter works fine for me.