How to debug a real-time component?

Hi comunity!

I want to develop a real-time component. But, for a serious real-time component I think, it needs to be debugged. So that way I could "really see" what happens inside it. That is why I am writing this post (sorry about this is not the right place).

Ok, after some readings I realize that I have to use halcompile command to compile a real-time component. I did one example and it seems to works fine. AFAIK this halcompile command seems to resolve several dependencies that are hard to resolve with the normal gcc compiler. I tried to compile that example with gcc but I have not any success.

Anyway, I still think any serious software development needs to be debugged so here are my questions to the Gurus:

- Is there any way to debug a real-time component? I mean, is there any kind of "haldebug"?
- If the "haldebug" commnand does not exist, is there any way to create all the needed dependencies to compile the real-time component with gcc command so then I can debug it with gdb?
- Or in another way, is there any script to resolve all the dependencies needed in an easy way?

I really appreciate very much any suggestion resolving dependencies about HAL, real-time component, software core and the like.

Thanks in advance!
Regards,
Alejandro

Welcome to the world of real time coding an interrupt service routine.

The easiest solution is not to make a mistake in your code and add plenty of debugging pins along the way which you can delete afterwards.

Keep components small and short. break complex stuff up into small discrete pieces.
Read the docs!!!! use halcompile!!!

Thanks for your replay rodw!

I will keep close to your suggestions, but I still think such a great piece of software deserve a better debugger. It is the "eclipse era"

BTW, I read the docs and I found this:

wiki.linuxcnc.org/cgi-bin/wiki.pl?DebuggingRtapi

A kind of procedure to debug a remote process. I think it could be useful so I drop the URL to the thread

Thanks for your replay andypugh!

I found your link very interesting!

With the preempt-rt kernel it has been suggested that there might be a way ti use a debugger.

And I think yes! It could be a way (please see my replay to rodw).

Its so long since I have played around with any debugging tools, and never on Linux. Remember that the servo thread is really a Timer Interrupt Service Routine firing every millisecond. Its not possible to step though this kind of code with a software debugger as bad things will happen. Just starting the debugger is likely to exceed the timer period so lockups will certainly occur as the CPU can't service all of the other interrupts flying around doing general housekeeping. I suspect you'd need a hardware debugger to have much luck.

I think there is a print command you can use (rtapi_print or something) that will print to the console but don't put it in the main function where its called every pass through the component as it will likely flood the system as well. Just use Halsshow or halscope to monitor values externally.

You can certainly slow the servo thread down to 1 Hz or slower for
debugging purposes, allowing any amount of printfs in debug breakpoints

Even better, Its probably possible with a little work to make a conditional
thread invocation option in the developed hal component function which
would allow single stepping a hal component all in hal, independent of the
thread rate. (the do-nothing hal pin)

Hi PCW, thanks for your reply!

It looks very interesting but I didn't get the whole picture. Sorry.
Could you bring some basic example please?

Thanks in advance!
Alejandro

Hi rodw, thanks for your reply!

Yes, you are certainly right about the timing issues but, the programmer's life it is not all about realtime and performance matters. Sometimes I need to be sure what I’m doing before take care of speed. That is why I insist with gdb

Anyway, I have been doing as you have been told! But some new issues arose, so I’m here shooting:

I think it’s time to tell the whole history. I’m trying to install LinuxCNC from ground-zero on a RPi3B+

What I have been doing:

- I compile from source with uspace parameters as instructed in the compile tutorial. I think is up and running (I still keep an eye on it).

linuxcnc.org/docs/master/html/code/build...uxcnc.html#_realtime

- I installed a Preempt-RT kernel on a RPi3B+:
uname –r
4.14.74-rt44-v7+

- I programmed the RPi parallel port with the HelloWorld blinking led tutorial. It works fine. Some issues arose concerning authorities. Running with sudo solve the problem (I keep an eye on it).

- Then I started developing a driver for an RPi board to direct access the GPIO hardware with the same concept. I started this thread at this point.

- I successfully “halcompile” my first approach but some (apparently) authorities issues arose when trying to load the realtime module with halrun (as normal user, not root, not sudo):

halcmd: loadrt hal_rpi3_gpio
Note: Using POSIX realtime
Failed to open /dev/mem, try checking permissions. Error: -1
HAL: ERROR: component 0 not found
hal_rpi3_gpio: rtapi_app_main: Unknown error 1001 (-1001)
<stdin>:6: waitpid failed /home/pi/worktemp-rp3/LinuxCNC/linuxcnc-dev/bin/rtapi_app hal_rpi3_gpio
<stdin>:6: /home/pi/worktemp-rp3/LinuxCNC/linuxcnc-dev/bin/rtapi_app exited without becoming ready
<stdin>:6: insmod for hal_rpi3_gpio failed, returned -1
halcmd:

I think the relevant message from this log is:

Failed to open /dev/mem, try checking permissions. Error: -1

And it arises when I do this in the program:

// Open /dev/mem
if ((p->mem_fd = open("/dev/mem", O_RDWR|O_SYNC) ) < 0) {
printf("Failed to open /dev/mem, try checking permissions. Error: %i\n", p->mem_fd);
return -1;
}

I think the text of the message is not so relevant because the general explanation about the return value of the c open function explains: first try check permissions! I think could be a lot of things considering the complex environment of LinuxCNC . Another thing I think, could be the way I access the GPIO hardware through memory-mapped device (I keep an eye on it too).

The things I did to try to solve this with no luck:

- Try to use adopted authorities over the module in the …/LinuxCNC/linuxcnc-dev/rtlib folder using:
sudo chmod u+s hal_rpi3_gpio.so

- Change owner:
sudo chown root:root hal_rpi3_gpio.so

- Running halrun as superuser threw this:

halcmd: loadrt hal_rpi3_gpio
Refusing to run as root without fallback UID specified
To run under a debugger with I/O, use e.g.,
sudo env RTAPI_UID=`id -u` RTAPI_FIFO_PATH=$HOME/.rtapi_fifo gdb /home/pi/worktemp-rp3/LinuxCNC/linuxcnc-dev/bin/rtapi_app
<stdin>:10: waitpid failed /home/pi/worktemp-rp3/LinuxCNC/linuxcnc-dev/bin/rtapi_app hal_rpi3_gpio
<stdin>:10: /home/pi/worktemp-rp3/LinuxCNC/linuxcnc-dev/bin/rtapi_app exited without becoming ready
<stdin>:10: insmod for hal_rpi3_gpio failed, returned -1
halcmd:

Before rodw complain: I’m not using any debugger! But halrun encourage me! (See the log )

One interesting thing I found while googling was this:

wiki.linuxcnc.org/cgi-bin/wiki.pl?Uspace

From the Comparison RTAPI implementation table I could see:

- With uspace RTAPI implementation, I cannot access any memory-mapped device?
Is that right?
Could that be the problem?

- If all the above are right. Should I implement rtai implementation instead of the uspace one?

- But this link address to use uspace with Preempt-RT:

github.com/KurtJacobson/RF45-CNC/wiki/Bu...LinuxCNC-From-Source

I’m so confused…
Any idea?

Thanks in advance!

ajsinfotech wrote:

Hi PCW, thanks for your reply!

It looks very interesting but I didn't get the whole picture. Sorry.
Could you bring some basic example please?

Thanks in advance!
Alejandro

Hal real time components export 1 (or more) functions
The functions get called periodically by LinuxCNC based on the thread they are bound to by the addf line in the hal file

Imagine that for debugging, you add a hal pin of type bit to your new component call "step"
and further that if "step" is true you execute the components function and if "step" is false you do nothing
Then if you clear step in the components function, all you have to do in hal is setp "step" for you component
to single step (run only one time)