If you are not using reset mode (Not sure if the RPI can do this)
The step pulse width is the base period.
Drifting in a (mod) 45 degree angle suggests crosstalk to me.
Maybe a loose ground or 5V common in the step wiring?
That's it! Somehow the stranded wire connecting d- and p - to GND broke in the ferrule. After I replaced the wire, the drifting is gone!
Thanks!