Sherline Lathe Conversion ?

Also what breakout board are you using. Some breakout boards (especially cheap ones) have slow optical isolators on their inputs, not suitable for reading encoders at any reasonable speed. You might need to bypass these. (That is essentially what Hugh did by routing his encoder inputs through the un-isolated inputs of his 540 rather than through an isolated breakout board.)

Your base thread speed may still be an issue as well. All though it should be able to read the encoder faster than 400 rpm. I think It ought to be able to read up to about 2000 rpm.