This is a math problem in EMC2 motion control for sure. It is also a error I have observed on my mill with boring round holes. and other oddities. I did start looking over the code for positioning logic and there is yet ANOTHER issue with the 32b float format. there's only 6.92 decimal places possible for a 32bit float,
But: LinuxCNC uses 64-bit doubles throughout.
And, more importantly, it doesn't actually use the float to store position. The accumulated encoder counts are stored in a 64 bit accumulator and then the (float) position is calculated from that every time.
I don't think that there is any scope for rounding errors in this way of doing things.