Strange motion offsets in one direction on one axis

Not sure it's a LinuxCNC issue, but I figure the folks here are likely also 'up' on likely hardware/software causes that might be related.

I was machining a pocket this afternoon, and ran into some really weird behavior where everything came out great, EXCEPT the walls on the -Y side of the pocket are over-cut in various ways. The attached image is hopefully clearer than my words (green line shows expected cuts, red line shows actual), but:

• The wall seems to get progressively more over-cut the further down the wall it goes- stairstep effect shown, with "reverse taper". Hard to measure the depths of the steps but they're big- maybe 0.02"? The very top of the wall is fairly close to on-size.
• The "notch" out of the otherwise-rectangular pocket is grossly overcut, except at the very end where it joins the larger area, where it's more or less correct. Notably, that should be a straight line, but isn't. The wall is all scalloped from the adaptive roughing- the finish pass didn't even touch it.
• All the other walls/dimensions look good. The +y wall has about 2 tenths of taper down the ~1" wall, nice surface finish, and the wall thickness is correct. The +X wall similarly measures correctly.
• The pocket is ~1x5"

The machine is using a Mesa 7i76E card, with 5mm lead ballscrews direct-driven by Clearpath servos. The servos are controlled in quadrature mode. Feedback pins are connected, though I'd have to recheck the math for how far from commanded position they'd have to get before tripping an error. Either way this doesn't obviously scream following error. All axes use linear bearings.

Tool was a YG1 uncoated carbide 3/8" end mill with 1/2" LOC, extended reach with reduced shank and ~2.125 LBS. The cuts weren't super heavy- adaptive roughing with ~2mm stepover, ~8mm stepdown, 7000 RPM, 0.0014 IPT/~29 IPM, leaving 0.1mm for a finish pass. I used air blast + MQL, no signs of chip welding or anything like that.

LinuxCNC is 2.10 from Git, grabbed... 6 months or so ago- probably should have used the 2.9 release, but for whatever reason I didn't. Compiled from source on Arch Linux. Machine is offline, so no updates to have changed anything/messed things up.

I reviewed the gcode coordinates manually and also viewed it in multiple different viewers. It looks entirely correct.

Any thoughts? Has there been a known trajectory bug in LinuxCNC I haven't found reference to? Does this sound like some kind of Ethernet/Mesa comms/latency/disconnect issue? Hardware problem? Nothing I've come up with seems to fit the data. I'm perfectly happy to try running updates/move to LinuxCNC 2.9.x (and probably will), but I like to have assignable causes to problems.

Let me know if there's any more detail I can supply, and THANK YOU!

Its not clear it's just one axis with the error.

Random uncommanded moves suggest a hardware issue to me,
either in the drives (encoder feedback error?) or the step/dir
interface (cross talk? EMI? marginal levels? bad connection?)

I'll try to make a better model tomorrow, but maybe the attached shows the gist of it better in cross section. The desired end result is a rectangular pocket. The result is as shown with the undercuts. The tops of both walls are in the right spots and of the correct thickness. The +Y wall is fine all the way down, the -Y wall is not. The end mill was a YG1 E5G98910 (3/8" diameter, 2-1/8" LBS, and 1/2" LOC, with a 0.345" shank, so it is possible for it to make some degree of undercut without rubbing). I'll see if I can measure how extreme the undercuts are tomorrow.

Cutting an internal pocket, climb cuts. The original material was a solid piece.

Possible multiple axes have errors, but I'm not seeing evidence of it. The one dimension in X I can measure is correct and shows no signs of misalignment between Z passes.


I agree this smells like a hardware issue- if so, it'll probably be a bear to track down. The motors are ClearPath servos, driven in quadrature mode. I believe the signals are 5V from the Mesa board, but would have to double check that. If there's some kind of encoder error, it's internal to the motor.

The cables are as provided from Teknic: motor -> cable -> soldered joint to circular connector -> connector -> soldered joint on other end of connector -> ferrule to 7i76E screw terminal. X and Y cables do run adjacent to one another (and power cables) for a few feet, and they're not shielded- though Teknic also sells 55' unshielded cables, so one might infer they don't believe it's important.

What's really hanging me up is that it seems to return to the correct place in +Y, despite apparently cutting incorrectly a variable amount in -Y.


Either way, I'll double check nothing is mechanically loose tomorrow (that would be the nicest answer) and put a scope on the Y axis signals while jogging X around, see what kind of cross-talk I get. I already got LinuxCNC 2.9.3 installed.

ETA: Mechanics appear fine- I can lean/pull on the Y axis and it'll move maybe 0.001", then springs back when I release it. Leadscrew retention nut appears tight.

Do let me know if things are still unclear.

Some debugging. Per update above, mechanically things seem fine- I can shove the Y axis around and it's definitely not moving as much as the undercuts previously mentioned. I measured those, and the deepest is maybe ~0.01"/10 mil/0.25mm. Not as bad as I thought, still entirely unacceptable.

See attached pictures for some scope traces of one of the Y axis quadrature phases. Note that the measurement setup wasn't great for this, so there might be more noise represented here than is present in normal use- scope leads were dangling from jumper wires wedged into the cable ends, so... yeah.

With nothing moving, the signal idles at 5V +/- ~1.2V. Moving the X axis doesn't seem to influence this at all. Enabling the spindle adds some noise, dropping the signal as low as 2.3V. That's quite possibly enough to create false motion- minimum input to the servo is listed at 4V, but it also says it has digital filtering against noise, so... who knows. I should look at both A and B phases at once, but ran out of time. The fourth picture shows the Y axis moving- it's pseudo-differential, which in this case I don't think is actually buying anything, since the motor inputs are opto-isolated.

All of the above seems suboptimal, but I'm not seeing how it would somehow create extra steps in -Y, then somehow seemingly exactly recover them when going to +Y. Unless I want to level-shift, I'm not aware of a way to get the 7i76E to output > 5V on the motor control pins.

Anything else I should check? I'll plan on getting traces of both A and B phases simultaneously. Is a 3D model of  the issue still helpful, or is the cross-section sketch sufficient?

Without a differential probe it's pretty hard so see,
but position recovery really seems like a drive or mechanical stall issue.

Can you set the drive following error limits tighter?

If it's possibly a drive strength issue, you could try single ended wiring.
(Since the drives have optoccoupler inputs this should have little effect on noise immunity)

 

Those OP pics are not cool. Re-check all your terminations when you'll have a spare 143hrs. Crimps, joints, everything linked to motion/control, look for something loose. I know, it's a nightmare, takes quite a bit of time and efforts, but I'll have to throw in the highly suspect "caused by vibration".

Checked a few things- still need to check filddling with error limits on the servo (requires me to dredge up and set up a Windows PC by the mill- servo software doesn't play nice with WINE):

• "Differential" noise isn't quite as bad as it initially looks- yes, still awful, but mostly the noise between the A+/A- pins at the servo end is in phase. The high-frequency noise that is out of phase is well below the minimum step time (750 ns). So... awful, but I think "OK". At least seems less likely to be the problem. See picture with math in it, measured at the end of the cable right where it would connect to the motor.
• Tying the A- pin to GND (single-ended signals instead of pseudo-differential) mostly just gives different noise, similarly bad. See picture without math in it, measured at the end of the cable right where it would connect to the motor.
• The noise gets far worse when I disable the e-stop on the machine. The e-stop removes power from the spindle drive and servos, so could be either one- though disconnecting the servo power plugs individually had no impact on noise. Spindle is a DMM DYN4 with all the recommended filtering on the power inputs, but still could be a problem.
• Drive strength seems OK- the 0-5V transitions are really sharp.
• Connectors do tend to be the bane of happiness (or at least reliable signals), but they seem to check out OK. Crimps all seem good (can pull on them with no movement), soldered joints all look fine, connectors are clean, secure, and haven't been exposed to any contaminants. All signals are reading ~0.3 ohms from the end of the cable to the screw terminal on the 7i76E, which is maybe 0.1~0.2 ohms higher than I get just touching the DMM leads together. I retorqued the screw terminals, which made no difference.
• I disconnected the servo from the leadscrew and turned it by hand. It appears to be slightly bent- not enough that it's hard to turn with my fingertips poking on the coupling, but definitely goes through phases of being easier/harder to turn. Might replace that on principle since it's easy to do and "not right", but it's not obviously the issue.

Will test removing spindle power to see if that's the signal noise issue, and also see if powering the 7i76E from an isolated 5V supply impacts anything. Should've tested those before posting this, but they just occurred to me now. (also changing the motor error threshold)

Thanks all for the help, even though it's not AFAICT LinuxCNC related! Really helps to have outside eyes on problems when your head is in the weeds.

1. Entirely plausible

2. Yes. The point of those was to highlight the noise seen is very high frequency, and seemingly unlikely to cause steps

3. Spindle is a BT30 taper with a manual drawbar- no ATC/belleville washers/pull studs/etc. End mill is held in a Nikken SK10 collet holder torqued to 35~40 ft-lbs, which is the recommended value. Tools are changed with an impact driver on the drawbar.

4. I should definitely re-run the code and see if I can reproduce. Haven't wanted to spend the time/material doing so, but I've wasted enough other time on this to dwarf both costs. I'll rerun both with the same tools originally used and also with a different one. May or may not have time to do that today.

5. I put an indicator on the tool and pushed on it. I can get it to deflect ~0.0005 (5 tenths) before I don't really want to push on it any harder. Seems to apply going any direction.

6. So climb vs conventional? I've thought about this, but the pocket was roughed with an adaptive/HSM toolpath, which started in the center of the pocket- so the cutter is roughing -Y to +Y on the +X side of the pocket, and +Y to -Y on the -X side. Correct me if my logic is wrong here, but if cut direction were the problem, wouldn't you expect to see issues in (eg) the +X+Y and -X-Y quadrants instead of the -Y half?


Grounding: One ground loop snuck in (shielded cable with shield connected at both ends), though disconnecting that has no impact on noise I can see. Will fix it anyway.

Beyond that, I wouldn't rule ground issues out, but am not seeing an obvious issue. The entire cabinet is aluminum and grounded. It's not a single point due to eg the filters, PSUs, and spindle drive all grounding through their chassis. The axis motors have a completely isolated supply (transformer-based)- as far as they're concerned, the world is the isolated 65V and a couple opto-couplers. The 24V and 5V negatives are all tied to a single bus bar- which is NOT ground. A multimeter shows an open circuit between 24V negative and the enclosure. The machine body has no dedicated ground, but is grounded indirectly via the spindle motor. The axis motor bodies are electrically connected to the machine, but that's isolated from their power supply.

Spindle drive has all the recommended filtering. I deviated slightly from the recommendations by omitting fuses on the logic power, replacing the contactor with an SSR, and placing said SSR before all the filtering, though I don't see an issue with that. Pictures show datasheet drawing and the schematic of how my machine is wired.

Noise is much better when motors are disabled- got it on my to-do list to see what's the root causes there. Guess is the spindle drive, but need to verify. No ferrites on anything at the moment- and no mention of them in the DYN4 (spindle) nor ClearPath (axis) manuals. Not to say they wouldn't help, just that the manuals don't call them out as important. The PSUs are MeanWell units, so at least not no-name. SOP is to not buy dubious PSUs- they can cause too many problems and take out too much valuable hardware to be worth the risk. Not to mention noise, as you say.