Jittery stepper pulses with mesa 7i76E [SOLVED]

Hi, I'm new to Linuxcnc, but have made cnc machines with other software/boards.
I've recently done a cnc conversion on a small manual lathe using open loop stepper motors and a mesa 7i76E board, and the configuration has been really very slick with pnconf.
However, the stepper movements are rather rough/scratchy and slightly noisy. The movements are very smooth with pulses from a signal generator, but looking at the pulse output from the mesa board on a 'scope there appears to be quite a lot of jitter in the output causing the rough movement.

I'm using a dedicated PC, which is an HP Proliant microserver. 
I've got the base thread latency below 50usec, after disabling stuff in the bios, and I am using a 1ms servo thread.
The PC has a Broadcom BCM5723 network card.
uname -a reports:
Linux cnclathe 4.19.0-17-rt-amd64 #1 SMP PREEMPT RT Debian 4.19.194-2 (2021-06-21) x86_64 GNU/Linux

If I reduce the servo thread to 500usec, the movements are much smoother, but I get some realtime delay errors, I suppose not surprisingly. A servo thread of 2ms seemed to make it worse.

The network card seems to use IRQ coalescing of 20usec, according to ethtool, and although it isn't an Intel card I tried setting it to zero. In fact it wouldn't support less than 1, and 1 made no difference.

If I ping the mesa board I get times of:
min 0.16ms, avg 0.201ms, max 0.34ms, mdev 0.026ms. I'm not sure how good/bad those figures are.

As I understand it, dpll is enabled on the mesa board by default, but I tried toggling the setting in halcmd to be sure.

I'm not sure what to try now, so any help would be very appreciated, please 
Many thanks
Phil

Histogram attached
 

 

Can you post your hal/ini files? This sounds like a setup error of some kind

Also a halscope plot of the following error might help in diagnosis

paw wrote:

However, the stepper movements are rather rough/scratchy and slightly noisy. The movements are very smooth with pulses from a signal generator, but looking at the pulse output from the mesa board on a 'scope there appears to be quite a lot of jitter in the output causing the rough movement.

Can you post a screenshot of the 'scope trace? If I'm guessing a bit, with a Mesa HW stepgen you should have a pretty smooth pulse train except that its frequency should be updated once every servo period. If your settings are wrong so that the frequency changes a lot every servo period you might perceive that as noise and jitter.

@PCW attached are my hal/ini files. I'm working on how to use halscope - I'll post again when I've got it going
Many thanks

Here is a plot of 7I76E stepgen following error at a 1 KHz servo thread rate.
These are with fairly  aggressive  velocity and acceleration limits = 1500 IPM
and 200 IPS/S = ~ 1/2G. Errors will be lower with lower velocity and acceleration limits,
but even at these extremes the actual stepgen position error is ~15 uinch peak:

 

Top trace is velocity in IPS, 5 IPS/div, middle trace is following error, 10 uinch/div
and bottom trace is DPLL error (basically Host PC jitter relative to DPLL reference)

I would suggest plotting the same variables for comparison

The hal/ini files look OK except I would use a larger DPLL pre-sample time, perhaps 100 usec
plotting the same variables I plotted will help diagnose the issue:

joint.0.vel-cmd
hm2_7i76e.0.dpll.phase-error-us
joint.0.f-error
 

@PCW Brilliant! Big thanks :-)That's got me going with halscope, and the change to the DPLL pre-sample time has made the movements perfectly smooth
I've had a fail with a screen dump, so I've attached halscope data files with before and after traces, I hope that is ok. If not I'll try harder. Anyway, hopefully you will see the hideous following error initially. Significantly reduced after the change. There is a pretty large DPLL error, which I assume means that I need to try for a bit better latency.
I've repeated my histogram, with the parameters as suggested by @tommylight (many thanks) and these reveal more latency than gets shown with no parameters - I'll attach. Also for completeness, @tommylight, the moves were still rough with MDI/gcode moves, and my step pulse times were as you suggested. 
One last thing - is it ok to put the altered setp DPLL line into the custom hal file so it doesn't get overwritten by pnconf? Or is that the wrong way to go about it? 

Also @ arvidb, Thanks for your help. PCW has solved it for me as above, but in answer to your suggestion, it was difficult to demonstrate the jitter on a still image from the oscilloscope because the jitter appeared as a regular pulse train which kept jumping about. I probably needed to do some fancy triggering to show it better. Anyway, thanks again.

A bit of explanation:

If you look at the log, the DPLL error is often less than -50 usec.
(the are some ~-57 usec values)
If the sample time was set to -50 usec, a -57 usec error would mean
that the DPLL would not have sampled the stepgen position before it
was read by LinuxCNC, meaning the position read would be the position
almost a whole servo period earlier. This will cause a large, bogus velocity
correction, resulting in the jittery motion you observed.

Basically what this means is there is considerable statistical
variation in host/Ethernet communication delays, causing a large
"baseline wander" in the average delay. I suspect this may be
BroadCom (driver or hardware) related.

That said, there is no real penalty to increasing the pre-sample period.
It does increase the time between reading the position and writing a new velocity,
(which causes a small error during acceleration) but you can compensate for this
with the PIDs FF2 parameter (by adding the DPLL pre-sample time to the FF2
parameter (in seconds so a  -100 usec sample time means add 0.0001 to FF2)