Retrofit small 5-axis mill - Ideas for EtherCAT drives

I don't think there's a lower limit for incremental encoder resolution on most devices; it's generally just a configurable 16-bit setting on the drive. It usually defaults to 1000/4000, but 256/1024 would probably work, and I know that Leadshine sells 5000/20000 steppers now, although with a long lead time.

Hmm. Why would they produce closed-loop steppers with 100 encoder steps per stepper step? They'd be able to measure much finer moves than the stepper can usually make. Are they doing some sort of *wild* dynamic microstepping or something? At peak RPM, this'd be almost a 1 MHz encoder signal. How much CPU are they hiding in their newer stepper drives, anyway?

It might be worth mentioning that the encoder resolution on the rtelligent steppers is 800 pulse per rev (6.2 micons on a 5mm pitch ballscrew). I don't know why you think 1000 ppr (5 microns) is not enough. So what is the ballscrew pithch, it may be finer?

There are many other sources of error when you start making things. Temperature and tool wear to start with. Good luck!

How much CPU are they hiding in their newer stepper drives, anyway?

Most likely very little CPU power. It'll likely be a FPGA with a hardware encoder counter capable of multi-MHz counting. The actual processing will only be polling the hardware counter periodically to get the position, then run the required calculations on that position (and probably still done in the FPGA for maximum performance).

High count encoders have little practical benefit on steppers (or most servo systems for that matter), but too low can cause issues with low speed tuning/stability. Using too high won't have any significant effect on performance (as long as the drive can handle them), but they won't actually benefit the machine in anyway.
Personally, I wouldn't pay any extra for more than a 1000 line encoder, but I wouldn't worry if higher was the only option.