Accessing libusb from a HAL component

I am working on a series of USB-connected I/O subsystems. The first is a hardware stepper indexer.

My idea is to build a HAL module that will connect to the EMC with HAL pins and talk to the hardware via libusb async I/O.
I'm working on Ubuntu 10.0.4 with an rtai kernel,

I, probably naively, thought I could just call libusb from within a HAL component. So I started with stepgen.c since it has the HAL pins I want. I ripped out the software indexer and I added the USB side to it.

I'm ready to compile, but my include files won't work. The libusb include is at /usr/include/libusb-1.0/libusb.h so I put this line in my code:

#include <libusb-1.0/libusb.h>

This is how I normally use libusb but using the linuxcnc Makefile, gcc couldn't find the header. I tried adding /usr/include to the makefile, which I realize is not the answer, but it didn't work. In frustration, I changed the source to an absolute path:

#include "/usr/include/libusb-1.0/libusb.h"

All this got me is a new set of errors from the headers that libusb.h tries to pull in. Starting with <stdint.h>. Of course, I could hard-code these, too, but I realize I am going down the wrong trail.

I assume this is something to do with the rtapi. Can I even access a normal kernel library like libusb? Can anyone set me on the right path, so to speak?

Hi

You haven't specifically said so, but I have to assume you are building a real time component.

Because these are kernel modules, they do not access the normal user space libraries, the headers used are largely from the linuxcnc source tree and /usr/src/linux/include, ie the kernel headers or source tree.

If you look in them, there are versions of some includes, but not the higher level ones.
I don't have a copy of libusb.h installed, so have no idea how it addresses the USB port and at what level.
(/usr/src/linux-headersxxxxxxx/include/linux/usb has the kernel specific usb headers)

You can write a userspace component which opens, reads, writes to etc a USB port and links to HAL pins. That way you can use the standard headers and libraries.

All depends what you are trying to do with the access and how fast it needs to poll etc.

I have writen user space components that quite happily poll a serial USB port connection every 1/4 second and get sufficient processor time to update HAL pins and send replies etc.

regards

I see what you're saying. I am trying to access libusb from within a kernel module. I can see why that would be problematic.

Well, perhaps I need to do it in userspace but I'm a little concerned about speed. I was hoping to do this in the 1mS servo thread. However, I know I can exchange 1,000 messages per second with my USB peripheral from a userspace program because I've done it in another application.

So, I'll try a userspace component and if that doesn't work I'll break open libusb and see if I can handle USB comms from within a kernel module. I'll report back.

Thanks!

After some research, I decided to go for adding the USB communications to the realtime component. Essentially, this means making the module act as a USB driver as well as exporting HAL hooks.

I have it communicating with the HAL and receiving messages from my USB hardware. However, when I try to send messages from one of the realtime functions, the entire OS locks up after a dozen or so messages.

The process of sending is allocating an URB, filling it with the message, submitting it to the USB core, and then waiting for a completion callback. Several steps, such as allocating the URB and the DMA buffer, require me to specify the memory allocation "mode" for want of a better term, e.g. GFP_KERNEL, GFP_ATOMIC, etc.

I wonder if my problem is that I am using the wrong allocation "mode". Any idea what it should be, if I'm calling kernel RAM allocation from within a real-time function?

Hi

You are obviously trying to re-implement low level USB access, but we don't know exactly what you are trying to achieve, so assisting is difficult outside the snippets presented.

However, when I try to send messages from one of the realtime functions, the entire OS locks up after a dozen or so messages........
........................................... Any idea what it should be, if I'm calling kernel RAM allocation from within a real-time function?

You will have to be more specific as to how you are sending messages and how you are allocating memory.

Real time components do not normally allocate memory outside of the shared area available. ie. hal_malloc for pins etc and rtapi_shmem for other blocks. (gross generalisation)
HAL works on a shared data area, containing structures of pins, params, signals, etc. etc. that can be referenced by calling for the lock, copying the relevant structure and then relinquishing the lock.

I have never tried using kmalloc inside a linuxcnc component.
However the literature seems to suggest that _GFB_DMA might be suitable for your DMA buffer and that you probably would not want to use GFB_KERNEL but rather GFB_ATOMIC when calling it from an process outside the kernel (but that is just from a quick scan).

Have a look at this www.makelinux.net/ldd3/chp-8-sect-1

If you look at the source code for linuxcnc you will see several examples of kmalloc being used
(/src/hal/drivers/ppmc.c /src/hal/drivers/hostmot2-xxxxxx)

A quick scan of ppmc shows it does use GFB_KERNEL, so a study of these may prove helpful

regards

mark.sullivan wrote:

After some research, I decided to go for adding the USB communications to the realtime component. Essentially, this means making the module act as a USB driver as well as exporting HAL hooks.

I have it communicating with the HAL and receiving messages from my USB hardware. However, when I try to send messages from one of the realtime functions, the entire OS locks up after a dozen or so messages.

The process of sending is allocating an URB, filling it with the message, submitting it to the USB core, and then waiting for a completion callback. Several steps, such as allocating the URB and the DMA buffer, require me to specify the memory allocation "mode" for want of a better term, e.g. GFP_KERNEL, GFP_ATOMIC, etc.

I wonder if my problem is that I am using the wrong allocation "mode". Any idea what it should be, if I'm calling kernel RAM allocation from within a real-time function?

The USB hardware has its own 1 ms clock, and everything happening on the bus is
synched to that, which will not quite sync with the RTAI servo thread.

Secondly, RT kernel modules must never block awaiting something to complete. If
it has not completed in a few microseconds, you must give up and check again
next time the thread is invoked.

My guess is you are blocked waiting for some completion from the USB, and the
thread overruns the next interrupt, which is pretty catastrophic for RTAI.

Jon

Having tried a number of things, it's kind of messy at this point but let me try to explain in detail.

rtai_app_main exports the HAL hooks like any realtime module. It also registers the module as a USB device driver by calling

rc = usb_register(&my_driver);

my_driver includes pointers to the probe and disconnect callbacks. When the driver is registered and the device is connected, usb_core calls my probe function.

The probe function retrieves a pointer to the device which we need later

my_dev = usb_get_dev(interface_to_usbdev(ptr_to_interface));

The probe function then allocates an urb for send and one for receive. (An urb is a usb message descriptor/control block).

send_urb = usb_alloc_urb(0,GFP_KERNEL);

Various fields of the urb are completed to set up the transfer.

usb_fill_int_urb(send_urb,my_dev,usb_sndintpipe(my_dev,1),
0,64,send_callback,my_dev,1);

The urb has to include pointers to the send or receive buffer. It's a DMA buffer and the urb needs both the CPU-space and the DMA-space addresses of the buffer:

send_urb->transfer_buffer = usb_buffer_alloc(blu_dev,64,GFP_KERNEL,&(send_urb->transfer_dma));
send_urb->transfer_flags |= URB_NO_TRANSFER_DMA_MAP;

The flag set in the second line lets the kernel know that the DMA address of the buffer is already populated.

Now we're ready to send and receive. In a real-time function called by the servo thread, I need to send a message to my device. I do this by loading the appropriate data into the send buffer allocated above and then submitting the urb to the usb_core.

memcpy(ptr_to_urb->transfer_buffer,my_message,64);

usb_submit_urb(ptr_to_urb,GFP_KERNEL);

Once the send completes and presumably is acknowledged by the device, usb_core calls my callback function. All the callback does in my case is set a flag to let the real-time send function know that it is OK to re-use the urb and buffer. So, assuming the callback has been called, the above two steps, loading the message into the buffer and submitting the urb, occur on every invocation of the servo thread function.

All of this works a few times. The analogous process for receiving works and I have run it for tens of thousands of messages, receiving one message every pass through the servo thread task. If I only let the send process happen a few times, say 10, then it works. If I let it run continuously, or even for something less than 100 times, then the system locks up solid.

On the surface, this sounds like some sort of memory leak but I am only allocating the buffers one time at startup so that doesn't seem right. And I've tried adding counters to see if the number of usb_submit_transfer calls matches the number of callbacks and they do match.

Asking what memory allocation mode to use is probably grasping at straws. But I'm at the straw-grasping point!

I don't think I'm blocking but I agree that would fit the symptoms well. None of the usb calls I'm using is supposed block.

I thought of the asynchrony of the rtai thread and the USB thread and I think I've designed around it.

Yes, the send is initiated by the HAL/RTAI timing and the actual transmission is working off the USB frame interval. USB can only transfer 1 of my messages per 1 mS frame. In other testing, I've verified that my device can sustain this rate.

So, if the servo thread runs even slightly faster than 1 / 1mS, then it would eventually try to send a message before the previous message completes. But I've taken this into account. I set a flag when each send completes and I check it before submitting another send. If the servo thread over-runs the USB timing, the result should be an occasional dropped transmission.

Furthermore, I turned down the speed of the servo thread to 500 Hz and the problem didn't change.

And the answer is ... I'm an idiot.

Thinking I was slowing down the servo thread from 1,000 Hz to 100 Hz, I changed the period from 1,000,000 nS to 100,000 nS.

It's now working fine at 200 Hz aka 5,000,000 nS, Any faster and it crashes. So it's either blocking too long in a USB call or just plain taking too long overall. I don't think I believe the latter.

Anyway, I will be able to find it now.