Hi all,
Sorry topic appeared first double then deleted all together (site is reacting a bit slow) Now back
I've been chasing an intermittent slowdown on my router for a while -- motion that should run smooth and at full commanded feed instead crawls and stutters, seemingly at random. I've now managed to localise and measure it precisely enough that I think it's worth a proper writeup rather than a vague "my machine is slow sometimes" post. Full data below; happy to share raw captures if useful.
SUMMARY
Motion on dense toolpaths sometimes degrades to roughly a quarter to a tenth of programmed feed. The degradation shows up part way through a session, persists across different programs, and is cured completely by restarting LinuxCNC -- a full reboot is not required, just restarting the LinuxCNC application.
I've localised the cause to the interpreter no longer reading ahead. In the degraded state the trajectory-planner queue never holds more than one move, so the planner has nothing to blend into and executes every segment as a discrete stop-start move.
MY SETUP
- LinuxCNC: 2.9.4, uspace
- Kernel: 6.1.0-30-rt-amd64 (PREEMPT_RT)
- GUI: QtDragon / qtvcp (qtdragon_hd)
- Motion card: Mesa 7i96s over Ethernet (hm2_eth, 100BASE-TX)
- Servo period: 1,000,000 ns (1 kHz)
- CPU: 4 cores @ 3.4 GHz, isolcpus=3 nohz_full=3 rcu_nocbs=3 irqaffinity=0-2
- Machine: 3-axis gantry router, X/Y/Y2/Z (Y is a slaved gantry pair)
- Axis limits: X 15000 mm/min @ 1200 mm/s^2; Y 8000 @ 1000; Z 3000 @ 300
THE MEASUREMENT
I ran the same air-move program twice, about ten minutes apart, with a LinuxCNC restart in between and nothing else changed. Two independent monitors ran throughout: a HAL sampler on the servo thread capturing per-joint velocity/position at 1 kHz, and a Python poller reading linuxcnc.stat().queue (trajectory-planner queue depth), .current_line, and /proc/<pid>/stat CPU for milltask and qtvcp, five times a second.
Test file: 2.84 mm median segments, G64 P0.05 Q0.2, F7000
(put this block inside a
tag when posting so the spacing holds -- it only lines up in a monospace font)
[code]
healthy (after restart) degraded (before)
-------------------------------- ----------------------- -----------------
mean feed while moving 112.2 mm/s 34.3 mm/s
retention of F7000 (116.67 mm/s) 96% 29%
TP queue depth 1039 -- full 0-1, never more
read-ahead 3,674 lines in 0.6 s ~5 lines/s
milltask CPU 15% burst, then idle 0.4%
In the healthy run the interpreter fills the entire 1039-move queue within six tenths of a second of pressing start and holds it saturated the whole run. In the degraded run it never gets more than one move ahead.
Second file: 0.54 mm median segments, G64 P0.01 Q0.05, F3600 (denser and tighter-tolerance than the file above, to rule out density as the cause)
healthy degraded
---------------------------- ------------ ----------
mean feed while cutting 56.7 mm/s 6.3 mm/s
retention of F3600 (60 mm/s) 94% 10.5%
TP queue depth 1781 -- full n/a
program advance 85 lines/s ~5 lines/s
THE ARITHMETIC CHECKS OUT
With one move queued the planner can't blend, so each segment accelerates and decelerates in isolation. For a segment of length L under acceleration a, a stop-start traverse peaks at sqrt(a*L) and averages roughly two thirds of that. For the first file: L = 2.84 mm, a = 1200 mm/s^2 gives a peak of 58 mm/s and a mean near 29 mm/s. Measured: 34.3 mm/s. No parameter was fitted to get that number.
THE DETAIL THAT RULED OUT A LOT OF OTHER EXPLANATIONS: RAPIDS ARE UNAFFECTED
In the same degraded capture where every G1 crawled at 12-30 mm/s, the opening G0 rapid held 136.3 mm/s, dead flat, for 2.25 seconds. A rapid is a single long move and needs no queue at all. So the axes, stepgens, Mesa link, servo thread, and acceleration limits are all demonstrably capable of full-speed smooth motion in the degraded state. Only motion that depends on queued look-ahead is affected.
WHAT I'VE RULED OUT BY DIRECT MEASUREMENT (NOT JUST REASONING)
- Interpreter CPU load -- milltask sits at 0.4% CPU while degraded. Not compute-bound, not falling behind -- barely working at all.
- Servo thread -- period measured at 1.006 ms in both states, thread execution ~20% of budget, no overrun.
- G-code density/geometry -- the denser file with the tighter blend tolerance runs better (94% at 0.54 mm / P0.01) than the sparser one.
- G64 P Q blending mode -- different tolerances made no measurable difference.
- File size -- 4k/15k/40k/70k/90k/110k-line programs all ran 94-95% in a healthy session.
- Memory -- qtvcp RSS tracks the size of the currently loaded program, released on load, not a cumulative leak.
- CPU frequency/isolation, NIC tuning, governor, thermals -- all measured directly, none correlated with the state.
- QtDragon's own GUI/graphics CPU load -- checked this after seeing it suggested elsewhere as a cause of large-file sluggishness in QtDragon. qtvcp's own CPU usage from the same captures: 3.67% mean / 49.9% peak (healthy) vs. 3.41% mean / 40.0% peak (degraded) -- flat to slightly lower while degraded, not elevated. Doesn't fit a "GUI thread starving the interpreter" mechanism.
WHAT I DON'T KNOW YET
Why read-ahead stops. It's a property of the session, not of any specific program -- it survives loading a different file, and a LinuxCNC restart (not a reboot) restores it completely. I haven't caught it in the act of transitioning, so I don't know if it's triggered by a specific event or degrades gradually.
Things I'm planning to check next, in rough order of promise -- any input on these, or other ideas, would be very welcome:
1. Whether anything in my config is busting interpreter read-ahead -- queue-busting words, M66, probing, a remap. None of these are present in my test programs, but the read-ahead machinery seems like the obvious place to look next.
2. Whether qtvcp's polling (stat.poll() at GUI rate) can interact with the interpreter's ability to queue, even though its CPU load doesn't change (see above) -- planning to try a run with the GUI closed/killed entirely (headless linuxcnc + halcmd) to see if degradation can even occur without qtvcp attached.
3. Whether the emcmot/emctask NML message channel saturates or throttles under some condition I haven't identified.
4. Whether this is a known 2.9.x regression -- I haven't found a matching report yet, which is part of why I'm posting this.
Operational workaround I'm using in the meantime: if a session has degraded performance, restart LinuxCNC and restart the session. Works, but obviously I'd rather find the actual cause.
Thanks for reading this far -- any pointers, even "check X" without a full answer, would help a lot.
grtz Pieter
