ESP32/S2/S3 LinuxCNC Controller (6 axis hardware step gen), USB plug-and-play

So on the PC side how is USB running real time. As this has been to the major drawback in getting USB devices, no real time usb stack, working within the Linuxcnc model.

The LinuxCNC host just sees a network adapter. This is presented over USB via the ESP32-S2/S3.

Take a look at the Remora project on this forum - it uses Ethernet.
Even the mesa cards use Ethernet.
This is the same.

IIRC the latency issues with USB that's been described will have been with trying to create USB HID devices and tuning endpoints to get them run as fast as possible. This is an entirely different approach...

With USB networking - it's all handled by the OS since it has to be low latency by design in the kernel. As long as the receiving device (MCU) can receive and process packets fast enough (just like mesa and remora) then there will be little-to-no delay or jitter.

The networking on the ESP32-S2/S3 is handled in hardware via a USB PHY (block on silicon) and hardware interrupts to ensure it's as fast as possible. Much better than any SPI Ethernet interfaces which is being used in other motion controller projects

I know that the Linuxcnc host will see a Ethernet adapter, in the same way it sees a SD card, flash drive or USB to SATA drive as a plain disk.
Yeah but you can't use a Mesa card with a USB to Ethernet adapter. No matter the hardware connected to a USB port, is it not at ransom to all the issues related to using a USB hardware in a realtime environment ?


As i understand it the need for real time is not for outright speed but for "things" to happen at a regular beat. This has been the issue with USB, whether a HID or not.

www.oreilly.com/library/view/linux-devic...0596005903/ch13.html
BULK
Bulk endpoints transfer large amounts of data. These endpoints are usually much larger (they can hold more characters at once) than interrupt endpoints. They are common for devices that need to transfer any data that must get through with no data loss. These transfers are not guaranteed by the USB protocol to always make it through in a specific amount of time. If there is not enough room on the bus to send the whole BULK packet, it is split up across multiple transfers to or from the device. These endpoints are common on printers, storage, and network devices.

The text I've highlight in bold would be a limitation in using USB to Ethernet with Linuxcnc, would it not ?

With mesa it's streaming data as fast as possible to LinuxCNC but that was before offloading to hardware became a thing.
Which is fine.

However with offloading to external hardware you can produce the very nearly the same results with IOs being monitored in real time (down to 50 microseconds) and any changes get raised to host immediately + transfer delay.

Encoders are counted in hardware and updated values are sent to the host the moment they change or based on a frequency depending on the application. 99% of use-cases won't require a sub-millisecond constant streaming of data from whatever peripheral you wish to monitor. Specifically, for axis related motor encoder positions, they must be counted in hardware on the MCU with the data streamed back to the host, the good news is that it already does via the current built-in stepper position feedback and so adding motor encoders would be fairly straightforward. This allows for LinuxCNC to continue to perform closed-loop PID control of motor positions.

Spindle/lathe RPMs for rigid tapping would only require a certain interval - say a 5ms period - the current RPM of the motor feeds into a tuned PID on the host (as long as the encoder pulse counts are handled on the MCU in hardware).

MCU hardware handles all aspects that need to be done at a regular intervals. LinuxCNC simply processes the received data as it arrives from the MCU however LinuxCNC can continue to run its own loops for hal components as normal.

There will always be latency with any encapsulated medium, however given the use-case, all IO is monitored via interrupts, hardware counters and as a result the latency between host and MCU is less of an issue. 

The UDP latency via USB is around a 1.2ms, which when compared to a native ethernet interface at 0.47ms approx is actually very good.

With regards to motion -
All axis commanded position is continuously streamed to the MCU via UDP.
When there is a positional difference the command is added to a pulse queue and sent to the RMT peripheral for step pulse generation based on the positional difference.
The firmware runs in absolute position mode continuously but at high speed (try not to compare to velocity vs position mode in LinuxCNC since this is done in hardware regardless of motor velocity) The ESP32 RMT hardware peripheral maintains a track of the pulses which are converted to a motor position co-ordinate and can be read at any time.

As the host and MCU can easily send 1000 UDP packets per second - each of which include every axis' new commanded position (not pulses, actual axis co-ords) in each packet then positional errors are a non-issue.

Axis position feedback is sent to LinuxCNC immediately once a new commanded position packet is received or until the motor(s) reach their targeted position co-ords. Equally LinuxCNC hal ensures the motors are always at their desired position by comparing desired vs actual position and ensuring packets are generated to ensure they match in every loop. If they don't it will throw a joint follower error.

I will be providing axis follower error scope outputs in LinuxCNC to prove this soon - I'm currently tweaking the LWIP stack it so it's a nice and true flat line.

I could have probably just said "its all offloaded" but that would have been easy!

Hope I've explained it clearly enough

EDIT: formatting and reworded
EDIT2: cornholio, RE USB BULK transfers, The wording on that link is very generalised. It depends on many things, but realistically speaking that's less of an issue today. You really need to push the bus alot for timing to even to start to become even a minor issue. It would require lots of high IO devices on the same roof hub, e.g  disks, 4k webcams etc, then *maybe* you'd see the odd stalled EP. The firmware handles those conditions anyway. Please take note of what I've mentioned above. The proof will be once I have optimised everything and show the results in detail via datasets. Please do take the time to watch the more recent YouTube videos that I've published for an initial idea of performance. The iperf udp jitter results I have should help ease any concerns. Will share in due course...
EDIT3: Do remember its use case specific. The data sizes exchanged between host and MCU over USB is tiny, less than 100 bytes per packet. That helps to additionally ensure the transfers are executed fast. I've tested a 1500 bytes packets (standard mtu) at 10k pps and latency is stable. Its all pretty standard stuff these days. 4G/5G dongles etc

Trust is earned. Scepticism is good!
If you breakdown a kernels handling of ethernet driver and compare the same to USB. It's surprising how many quirks there have been in both of them over the years, by differing maintainers and vendors. These days things are considerably better and more refined.

Take a modern USB webcam as an example. It's low latency and jitter. Any packet errors or delay presents itself instantly.
That's high bandwidth too. Not designed for noisy environments, however.

This application is low bandwidth. Noise can be mitigated to most degrees if designed well. USB gets a bad rap due to the variation in vendor specific firmware on the devices, commonly either due to rushed to market, cost savings or edge cases on the host controller, which is much less of an issue today though!
Cables vary so much too which compounds the problem.

USB stacks on most hardware are, generally speaking, very robust. It's like anything though, if you dig deep enough you'd find flaws, even with ethernet. Testing and validation is what matters.

With that in mind. Data driven approach and prove it is what I think will help address concerns.

I'll soon share two firmwares for both the S2 & S3 which contains a tuned stack that has:
1) iperf UDP server and a built-in standalone stepper generator + basic motor configurator in the firmware via a webui
2) iperf client with data logging for analysis across various bandwidth use-cases and varying intervals
3) host stress testing tool with varying parameters
4) Docs on how to run the series of tests

If a machine has its motors decoupled from its physial axes to allow for an unrestricted test it should replicate an as close as possible, real world environment.

I've tested the same here on a physical machine with a very noisy VFD, with 4 different hosts of differing ages (different USB phys).
Admittedly I've not ran 24 hour soaks on all, yet - only 2 hour for most, one has been for 24 via a RPI5. It's been stable. No encoder losses or positional drift (closed loop servos connected to a dedicated STM32 tracking positional errors). A high quality shielded and screened USB cable matters.
Sure the IO is 3.3V on a dev board but it does drive external stepper drivers fine, at least for testing purposes.

Should help provide real world stress and soak testing solution to help build confidence with latency, jitter and reliability.

*Bear in mind to use an FCC certified S2/S3 module not an exposed IC board - one that has an EMI can on top. e.g a proper espressif DevKitC-1 for $10

How does this sound?

You had me pulling out my lcnc gizmos... a high-speed 1um scale patched via USB to lcnc (hw counter). Complainless... jusslikemy wife!

Top secret stuff... no thinkers. 

Thanks Mecanix. That's cool. Interested to know more but its classified

ESP32-S2 Motion USB & XHC Wireless MPG Pendant USB device on RPI4