Writing modules for PREEMPT_RT
21 Sep 2016 21:48 - 21 Sep 2016 21:50 #80748
by DaBit
Replied by DaBit on topic Writing modules for PREEMPT_RT
Had a few hours of time available today, so I did some fooling around.
VMWare 12 virtual machine running default LinuxCNC Wheezy ISO on a Win10 host, using the Linux HIDRAW driver to send 32-byte packets to a custom HID device, with LinuxCNC running and executing code:
Not bad, considering it is running from within a virtual machine on top of a Windows host. And certainly good enough for demonstration purposes.
VMWare 12 virtual machine running default LinuxCNC Wheezy ISO on a Win10 host, using the Linux HIDRAW driver to send 32-byte packets to a custom HID device, with LinuxCNC running and executing code:
dabit@linuxcnc-vm:~/usb_tests$ sudo ./udev_test
Device found at /dev/hidraw1
Packet transmission time since last: 1.838086 msec
Packet transmission time since last: 2.992347 msec
Packet transmission time since last: 2.844372 msec
Packet transmission time since last: 3.039186 msec
Packet transmission time since last: 1.987344 msec
Packet transmission time since last: 1.982596 msec
Packet transmission time since last: 1.055455 msec
Packet transmission time since last: 2.027151 msec
Packet transmission time since last: 1.955042 msec
Packet transmission time since last: 1.839815 msec
Packet transmission time since last: 0.989827 msec
Packet transmission time since last: 0.996477 msec
Packet transmission time since last: 2.967388 msec
Packet transmission time since last: 2.066263 msec
..
..
Packet transmission time since last: 3.011264 msec
Packet transmission time since last: 1.981308 msec
Packet transmission time since last: 2.008413 msec
dabit@linuxcnc-vm:~/usb_tests$
Not bad, considering it is running from within a virtual machine on top of a Windows host. And certainly good enough for demonstration purposes.
Last edit: 21 Sep 2016 21:50 by DaBit.
Please Log in or Create an account to join the conversation.
22 Sep 2016 19:46 #80807
by jtc
Replied by jtc on topic Writing modules for PREEMPT_RT
not bad at all!
Please Log in or Create an account to join the conversation.
04 Oct 2016 21:54 - 04 Oct 2016 22:01 #81234
by DaBit
Replied by DaBit on topic Writing modules for PREEMPT_RT
Back from digging a few trenches using knobby tires on the two-wheeler.
I figured out that the userspace approach has one drawback: there is no strict synchronisation between the RT part and userspace part. That gives funny velocity and acceleration values if you use the CLOCK_MONOTONIC_RAW as the time source and positions as they are coming in.
Sure, I can somehow generate a timestamp from RT using a counter that increments every servo cycle, or even do everything I need in RT and pass complete frames through shared memory to a userspace program that sends it over USB, but somehow I feel that 'uspace' should be easier.
So, I spent the evening installing native Linux next to the Windows 10 on my work laptop (Alienware M17). I ended up using lubuntu 16.04 since i could not persuade Debian to see the existing Windows installation and I did not want to mess that up.
To make this lubuntu realtime I grabbed the 4.4.21 kernel from kernel.org, applied the preempt-rt patches, compiled the kernel, and compiled the master branch of LinuxCNC on top of it using --with-realtime=uspace
Latency test running for a couple of hours promise good results in the latency department:
Now, how do I proceed writing a component that can be hooked into the servo thread and accesses the hidraw device? Do I use halcompile just like a normal realtime component? If so, then how do I link extra libraries such as libudev? The documentation says: 'option extra_link_args "…" - (default: "") This option is ignored if the option userspace (see above) is set to no. '.
This is still a bit unclear to me.
Is there a 'writing LinuxCNC RT components in C' tutorial somewhere?
I figured out that the userspace approach has one drawback: there is no strict synchronisation between the RT part and userspace part. That gives funny velocity and acceleration values if you use the CLOCK_MONOTONIC_RAW as the time source and positions as they are coming in.
Sure, I can somehow generate a timestamp from RT using a counter that increments every servo cycle, or even do everything I need in RT and pass complete frames through shared memory to a userspace program that sends it over USB, but somehow I feel that 'uspace' should be easier.
So, I spent the evening installing native Linux next to the Windows 10 on my work laptop (Alienware M17). I ended up using lubuntu 16.04 since i could not persuade Debian to see the existing Windows installation and I did not want to mess that up.
To make this lubuntu realtime I grabbed the 4.4.21 kernel from kernel.org, applied the preempt-rt patches, compiled the kernel, and compiled the master branch of LinuxCNC on top of it using --with-realtime=uspace
Latency test running for a couple of hours promise good results in the latency department:
Now, how do I proceed writing a component that can be hooked into the servo thread and accesses the hidraw device? Do I use halcompile just like a normal realtime component? If so, then how do I link extra libraries such as libudev? The documentation says: 'option extra_link_args "…" - (default: "") This option is ignored if the option userspace (see above) is set to no. '.
This is still a bit unclear to me.
Is there a 'writing LinuxCNC RT components in C' tutorial somewhere?
Last edit: 04 Oct 2016 22:01 by DaBit.
Please Log in or Create an account to join the conversation.
05 Oct 2016 22:40 #81280
by DaBit
Replied by DaBit on topic Writing modules for PREEMPT_RT
OK, I have an RT component running that communicates over USB. Using straight .comp file/ halcompile --install did not work; somehow opening /dev/hidrawX (fd = open("/dev/hidraw0", O_RDWR|O_NONBLOCK) from within EXTRA_SETUP() fails. Using hm2_eth.c as a template and doing it early in rtapi_app_main works. No idea why.
Next problem: a call to write() does not return immediately but always blocks until the data frame is transmitted and I am never getting an EAGAIN or EWOULDBLOCK. This is bad since that might take a while. Using poll() doesn't improve this either. No idea how to handle this except running the component in a slower thread. Ideas welcome.
For reference: the test code.
Next problem: a call to write() does not return immediately but always blocks until the data frame is transmitted and I am never getting an EAGAIN or EWOULDBLOCK. This is bad since that might take a while. Using poll() doesn't improve this either. No idea how to handle this except running the component in a slower thread. Ideas welcome.
For reference: the test code.
#include <linux/input.h>
#include <linux/hidraw.h>
#include <sys/fcntl.h>
#include <sys/ioctl.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <sys/wait.h>
#include <fcntl.h>
#include <unistd.h>
#include <errno.h>
#include <poll.h>
#include <rtapi_slab.h>
#include <rtapi_ctype.h>
#include <rtapi_math64.h>
#include "rtapi.h"
#include "rtapi_app.h"
#include "rtapi_string.h"
#include "hal.h"
MODULE_LICENSE("GPL");
MODULE_AUTHOR("Bart van Hest");
MODULE_DESCRIPTION("Test module");
#ifdef MODULE_SUPPORTED_DEVICE
MODULE_SUPPORTED_DEVICE("test");
#endif
/* USB device to search for */
#define USBDEV_VID 0x0483
#define USBDEV_PID 0x5751
/* USB transmit and receive buffers */
static unsigned char xmitbuf[65], rcvbuf[65];
/* Component id */
static int comp_id;
/* USB device file descriptor */
static int usbfd=-1;
struct __comp_state {
struct __comp_state *_next;
hal_float_t *position_cmd_x;
hal_float_t *position_fb_x;
hal_float_t maxaccel_x;
hal_float_t maxvel_x;
hal_float_t position_scale_x;
};
struct __comp_state *__comp_first_inst=0, *__comp_last_inst=0;
static void _(struct __comp_state *__comp_inst, long period);
static int __comp_get_data_size(void);
#undef TRUE
#define TRUE (1)
#undef FALSE
#define FALSE (0)
#undef true
#define true (1)
#undef false
#define false (0)
static int export(char *prefix, long extra_arg) {
char buf[HAL_NAME_LEN + 1];
int r = 0;
int sz = sizeof(struct __comp_state) + __comp_get_data_size();
struct __comp_state *inst = hal_malloc(sz);
memset(inst, 0, sz);
r = hal_pin_float_newf(HAL_IN, &(inst->position_cmd_x), comp_id,
"%s.position-cmd-x", prefix);
if(r != 0) return r;
r = hal_pin_float_newf(HAL_OUT, &(inst->position_fb_x), comp_id,
"%s.position-fb-x", prefix);
if(r != 0) return r;
r = hal_param_float_newf(HAL_RW, &(inst->maxaccel_x), comp_id,
"%s.maxaccel-x", prefix);
if(r != 0) return r;
r = hal_param_float_newf(HAL_RW, &(inst->maxvel_x), comp_id,
"%s.maxvel-x", prefix);
if(r != 0) return r;
r = hal_param_float_newf(HAL_RW, &(inst->position_scale_x), comp_id,
"%s.position-scale-x", prefix);
if(r != 0) return r;
rtapi_snprintf(buf, sizeof(buf), "%s", prefix);
r = hal_export_funct(buf, (void(*)(void *inst, long))_, inst, 1, 0, comp_id);
if(r != 0) return r;
if(__comp_last_inst) __comp_last_inst->_next = inst;
__comp_last_inst = inst;
if(!__comp_first_inst) __comp_first_inst = inst;
return 0;
}
static int __comp_get_data_size(void) { return 0; }
/* Brute-force open a couple of /dev/hidrawX devices and check if it is our motion controller */
static int FindAndOpenUSBdevice(void) {
char buf[16];
int i, fd, res;
struct hidraw_devinfo info;
for (i=0;i<16;i++) {
rtapi_snprintf(buf, sizeof(buf), "/dev/hidraw%d", i);
fd = open(buf, O_RDWR|O_NONBLOCK);
// fd = open(buf, O_RDWR); /* Note: do not rely on O_NONBLOCK; there is a kernel bug that prevents nonblock on disconnect */
if (fd) {
/* Opening the device succeeded. Extract VID/PID and compare */
memset(&info, 0x0, sizeof(info));
res = ioctl(fd, HIDIOCGRAWINFO, &info);
if (res >= 0) {
if (info.vendor == USBDEV_VID && info.product == USBDEV_PID) {
/* We found our device. Return the fd. */
rtapi_print("Found device at %s\n",buf);
return fd;
}
}
/* Not the correct device or we could not extract VID/PID. Close fd */
close (fd);
}
}
return -1;
}
int rtapi_app_main(void) {
int ret;
rtapi_print("loading test module\n");
ret = hal_init("test");
if (ret < 0)
return ret;
comp_id = ret;
usbfd = FindAndOpenUSBdevice();
if (usbfd < 0) {
rtapi_print("ERROR opening device, fd=%d\n", usbfd);
} else {
rtapi_print("SUCCESS opening device\n");
}
ret = export("test", 0);
if(ret) {
hal_exit(comp_id);
} else {
hal_ready(comp_id);
}
return ret;
}
void rtapi_app_exit(void) {
if (usbfd > 0) {
close (usbfd);
}
hal_exit(comp_id);
rtapi_print("test module unloaded\n");
}
static void _(struct __comp_state *__comp_inst, long period) {
int ret;
ret=0;
if (ret==0) {
xmitbuf[0] = 0x0;
xmitbuf[1] = 0xde;
xmitbuf[2] = 0xad;
ret=write(usbfd, xmitbuf, 3);
//ret=send(usbfd, xmitbuf, 3,0);
}
*__comp_inst->position_fb_x = *__comp_inst->position_cmd_x;
}
Please Log in or Create an account to join the conversation.
05 Oct 2016 23:02 #81281
by andypugh
Replied by andypugh on topic Writing modules for PREEMPT_RT
Perhaps EXTRA_SETUP wasn't returning comp_ready? (or something similar)
The blocking call is a problem. You could get away with it in a userspace component.
There might be a way to set the call off, exit the sub, and watch for it finishing, But that's no more realtime than the userspace option, so I doubt there is any point.
The blocking call is a problem. You could get away with it in a userspace component.
There might be a way to set the call off, exit the sub, and watch for it finishing, But that's no more realtime than the userspace option, so I doubt there is any point.
Please Log in or Create an account to join the conversation.
06 Oct 2016 08:47 #81293
by DaBit
Replied by DaBit on topic Writing modules for PREEMPT_RT
The component would load, it was the fd=open(...) that failed. If comp_ready is not returned, halrun displays 'waiting for the component to become ready...' or something like that forever. Or at least until halrun -U is executed.
Also, rtapi_print did not work. I did not dive into it, but went the 'strip hm2_eth.c' route instead. I assume it is a user error somewhere.
The blocking is a problem indeed. For now I'l leave it this way and just reduce the servo frequency or run this component from another slower thread, and see where it ends. It is not meant to be 'production code' anyway, and other than the call that blocks until Mr. USB has a timeslot available to transmit my data (which should be guaranteed to be once a millisecond) the timing seems to be very consistent. Which is definitely a good thing.
If that proves to be troublesome, I will construct the data frames from within the RT component, pass them over to a userspace application, and transmit them from there. Then I might miss a frame now and then, which should be easy to cope with if acceleration and velocity is also transmitted.
At least I am proving the point that USB is not very suitable for this
Also, rtapi_print did not work. I did not dive into it, but went the 'strip hm2_eth.c' route instead. I assume it is a user error somewhere.
The blocking is a problem indeed. For now I'l leave it this way and just reduce the servo frequency or run this component from another slower thread, and see where it ends. It is not meant to be 'production code' anyway, and other than the call that blocks until Mr. USB has a timeslot available to transmit my data (which should be guaranteed to be once a millisecond) the timing seems to be very consistent. Which is definitely a good thing.
If that proves to be troublesome, I will construct the data frames from within the RT component, pass them over to a userspace application, and transmit them from there. Then I might miss a frame now and then, which should be easy to cope with if acceleration and velocity is also transmitted.
At least I am proving the point that USB is not very suitable for this
Please Log in or Create an account to join the conversation.
14 Oct 2016 15:03 #81643
by DaBit
Replied by DaBit on topic Writing modules for PREEMPT_RT
Wrote a bit more code. Not finished, but promising.
Debugging an USB device is not easy; once the debugger hits a breakpoint the host starts complaining.
On the LinuxCNC side I am running the HAL component at a 250Hz servo rate that communicates joint position, velocity and acceleration to the device. If a device with latency=1 enumerates USB promises that I can transmit an interrupt packet every millisecond. At 250Hz I have 4 milliseconds.
On the device side the stepgens consist of NCO's driven by a PI controller. Actual position is fed back to LinuxCNC. Joint velocity and position are used, acceleration is not at this moment.
With a siggen-generated position at a low frequency:
Once I demand more than the stepgen NCO can deliver:
This is looking quite OK imho.
I have been running this for almost 24 hours while using the computer for 'leisure tasks', and so far I have not seen a single hickup in the communication. I'm seeing a steady 250 packets/sec being delivered to the device.
I can also plug in and remove USB devices without the communication being disturbed. On other root hubs, but also on the very same root rub the motion device is plugged in. This does surprise me...
Debugging an USB device is not easy; once the debugger hits a breakpoint the host starts complaining.
On the LinuxCNC side I am running the HAL component at a 250Hz servo rate that communicates joint position, velocity and acceleration to the device. If a device with latency=1 enumerates USB promises that I can transmit an interrupt packet every millisecond. At 250Hz I have 4 milliseconds.
On the device side the stepgens consist of NCO's driven by a PI controller. Actual position is fed back to LinuxCNC. Joint velocity and position are used, acceleration is not at this moment.
With a siggen-generated position at a low frequency:
Once I demand more than the stepgen NCO can deliver:
This is looking quite OK imho.
I have been running this for almost 24 hours while using the computer for 'leisure tasks', and so far I have not seen a single hickup in the communication. I'm seeing a steady 250 packets/sec being delivered to the device.
I can also plug in and remove USB devices without the communication being disturbed. On other root hubs, but also on the very same root rub the motion device is plugged in. This does surprise me...
Please Log in or Create an account to join the conversation.
- tommylight
- Away
- Moderator
Less
More
- Posts: 19190
- Thank you received: 6433
15 Oct 2016 01:00 #81661
by tommylight
Replied by tommylight on topic Writing modules for PREEMPT_RT
Am still following this although i have no idea where would i use it for now.
I will most probably think of something.........
Keep up the good work.
I will most probably think of something.........
Keep up the good work.
Please Log in or Create an account to join the conversation.
18 Oct 2016 22:33 - 18 Oct 2016 22:38 #81810
by DaBit
It turns out that when you happen to make a mistake with an extra zero in all those nanoseconds for the servothread and you try to run that thread at 25Hz you get a lot of weird errors. Gmoccapy crashes with an 'onboard' error, etc. Hard to link that to a wrong servofrequency setting.
Once I got that figured out everything worked smoothly.
Excluding the motor that's $7,50 worth of hardware used to control the motor, and it runs on a laptop. I still have to try the VMware virtual machine.
I also use position, velocity and acceleration to drive the stepgenerator now (which doesn't make a huge difference; communication is stable at 250 updates/second), and I included somewhat more I/O on the board than just steppers because the pins and hardware units in the microcontroller are there.
Step frequency is currently limited to 30kHz, but that is because the code is not that well optimized. I need 2 interrupts for a single step pulse, and STM's hardware abstraction layer is not very efficient.
If I would implement a more sophisticated stepgen (using DMA to the I/O pins, for example), 100+ kHz should be possible.
Maybe it is even useable for a real low-end machine. At least the extra I/O makes it useable on my mill to provide some extra hardware knobs (I still want rapid/feed/spindle override knobs right on the table).
Replied by DaBit on topic Writing modules for PREEMPT_RT
It turns out that when you happen to make a mistake with an extra zero in all those nanoseconds for the servothread and you try to run that thread at 25Hz you get a lot of weird errors. Gmoccapy crashes with an 'onboard' error, etc. Hard to link that to a wrong servofrequency setting.
Once I got that figured out everything worked smoothly.
Excluding the motor that's $7,50 worth of hardware used to control the motor, and it runs on a laptop. I still have to try the VMware virtual machine.
I also use position, velocity and acceleration to drive the stepgenerator now (which doesn't make a huge difference; communication is stable at 250 updates/second), and I included somewhat more I/O on the board than just steppers because the pins and hardware units in the microcontroller are there.
Step frequency is currently limited to 30kHz, but that is because the code is not that well optimized. I need 2 interrupts for a single step pulse, and STM's hardware abstraction layer is not very efficient.
If I would implement a more sophisticated stepgen (using DMA to the I/O pins, for example), 100+ kHz should be possible.
Maybe it is even useable for a real low-end machine. At least the extra I/O makes it useable on my mill to provide some extra hardware knobs (I still want rapid/feed/spindle override knobs right on the table).
Last edit: 18 Oct 2016 22:38 by DaBit.
The following user(s) said Thank You: jtc, tommylight
Please Log in or Create an account to join the conversation.
- tommylight
- Away
- Moderator
Less
More
- Posts: 19190
- Thank you received: 6433
19 Oct 2016 00:35 #81814
by tommylight
Replied by tommylight on topic Writing modules for PREEMPT_RT
Now, this got interesting.
Respect for setting your sights on something, and seeing it through.
Respect for setting your sights on something, and seeing it through.
Please Log in or Create an account to join the conversation.
Time to create page: 0.098 seconds