Rough movement in cnc

It is almost certainly software stepping. (If you used the Stepconfig wizzard to build your Linuxcnc config it is software stepping.)

I'd hardly consider 4000mm/min fast. (I'm working with wood carving machines where high feed rates are important.)

If you are having realtime errors, they can easily cause your motion to not be smooth.
Run the real time test for at least a few hours with a couple of GLXgears windows open, and performing some other tasks, like updates, and surfing the net.

Reduce micro stepping settings on the drive. High micro stepping does not increase the resolvable accuracy of a stepper motor much (if at all) beyond about 4x micro-stepping.
for 4000mm/min, and a 25000ns jitter, I would suggest not having a step scale higher than about 100steps/mm (more like 50 steps/mm might be better.)

Chinese machine so most probably it uses chinese BOB with slow opto couplers, so set the latency to 100.000, then yoi can play with step timings a bit to get max speed.
Do this AFTER you check what Todd has sugested above.
Almost all cheap BOB's will cause all kinds of wierdness with anything under 100.000 for latency, had over 15 versions of them.

Sure I will do these today. Not sure how to change micro-stepping settings, but will google and see.

When you say set the latency to 100,000 are you saying I should set the max jitter to 100,000?

Re - BOB: Are you referring to these:
users.skynet.be/ldt/CNC%20electronics/BOB%27s/BOB4_descr.html

This looks like the one I have inside. I need to re-check. In that case, knowing the schematics will be really helpful.
Do you know any better boards that I can use to replace above without much changes? I like to have to support for a PWM and coolant control too. Although this board has a relay which switches on/off with coolant on/off button in EMC, relay does not seem to be connected to any power line so it useless.

I don't think that is what he meant. With a slow opto breakout board, you need to set the step timings as large as possible and still be able to meet your velocity goals. (I think that might be what he was referring to.)

Yes, that is max jitter.
I can check later, i have a bob that is plenty fast.
New chinese ones absolutely must have latency set at 100.000 or they just do not work, and the output wave form is pittyful, After seting latency you can try to shorten the step time and/or step space to find a maximum velocity it will handle, also lovering microsteping on the drives till you get the desired speed.
Thank you.