The missing-tooth index scheme would generally be used with a single sensor (it's a way to save wires).
You seem to be using one pin for phase A and one for index (Z). I am not sure how that works with a missing-index scheme, I suspect that it will simply ignore the Z phase.
I think that you can (probably) use phases A and B (and no Z) to detect speed _and_ direction and rely on the missing tooth for index, but I don't think I actually tested that...
You have set counter-mode = true, so it should be able to work with a single phase (A)
Make sure that the spindle encoder position _increases_ by 1.0 for each revolution of the lathe spindle in the positive direction (ie, clockwise looking down the spindle axis towards the workpiece)
If it decreases then I am a bit puzzled, as it shouldn't be able to with only one phase pin and index
Does it detect the index? You should be able to test this by (at the command line) "halcmd sets spindle-index-enable 1" then rotating the spindle past index and using "halcmd show sig spindle-index-enable" and it should have set back to false.
