Python Interface makes race conditions - mayby

Hello,

i think i found one of the biggest bugs that exist in LCNC.
github.com/LinuxCNC/linuxcnc/issues/2586
Unfortunately, this issue is downplayed and the developers think it's just my problem.

Maybe the developers are right and I'm wrong, but I'll try to explain my thought process here.
I will try to explain how I think this problem arises.

The LCNC core (EMC2) was written in C and C++.
Then an interface for python was created.
linuxcnc.org/docs/stable/html/config/python-interface.html

The Python interface can do a lot of things, but it can't do everything that G-code can do. So there is a way to run G-code with Python.
linuxcnc.org/docs/stable/html/config/pyt...ing_to_send_commands

The main g-code execution command looks like this:
If we are at point 0,0,0 , the execution of this command will look like this:
 

The Python Interface has an important command that ensures that everything is not read at once:
I'll show you what the result of the previous case will look like when split using c.wait_complete(): 
The start and end coordinates are the same, but the progression is completely different.

Another condition for running G-code in the Python Interface is that the LCNC must be in MDI mode. This can be easily implemented:
Now let's look at the sample example again:The way this example works is that it checks that everything is OK to execute the G-code and then executes the G-code.
Unfortunately, in the real world, we want the G-code to execute whenever we need it to. Not that the G-code is executed only if the LCNC is suitable.

Unfortunately, I haven't found anyone checking for this condition in real-world code using the Python interface:
If this condition is not checked and we don't know if some other statement is finishing, we should always start with wait_complete(). (I think)

I made a code modification for Gmoccapy.
github.com/LinuxCNC/linuxcnc/pull/2552
For now, this Pull Request is awaiting Norbert's edits.

Now I will explain bigger problem.
The only way to wait for commands to complete in the python interface is wait_complete().
(If exists another way, tell me how and you dont need read next text.)
If we need to wait for our own command, which we created in the Python interface, then there is no problem here.

The problem is that commands in LCNC can be executed in multiple ways.
1)In AUTO mode:
 

2)In MDI mode:
 
Gmoccapy doesnt run mdi commands in its python code, but in mdi Widgets
So wait_complete() in Gmoccapy doesnt wait for mdi commands started in mdi Widgets
Proof that it is:
github.com/LinuxCNC/linuxcnc/issues/2489

3)In HALUI MDI_commands
 
HALUI MDI_commands also have the peculiarity that they consist of several commands.
- switching from some mode to MDI mode
- execution of own g-codes
- return to MDI
The problem is that even if the programmer checks linuxcnc.INTERP_IDLE, he still gets a race condition during return to MDI

4) In remaps
I dont know, how it works in remap.

This situation does not cause any problem in most cases, because everything happens gradually. For example:
1) Starting Gmoccapy (Python Interfaces commands)
2) Waiting for operator
3) The operator starts HALUI MDI commands for move to some specific coordinates
4) Waiting for operator
5) Using MDI commands in widgets in Gmoccapy, the operator starts the spindle to warm up
6) Waiting for operator
7) Using MDI commands in widgets in Gmoccapy, the operator stops the spindle
8) Waiting for operator
9) The operator starts the program in AUTO mode
Here in this case there can be no problem, because everything happens gradually. A race condition cannot occur.

The problem arises when the G-code contains tool change commands.
Gmoccapy uses signals that occur at the same time as the main G-code command is executed.github.com/LinuxCNC/linuxcnc/blob/9879ba...py/gmoccapy.py#L2635
It is okay that the response to this signal is immediate and nothing is waited for. We want to see changes in the GUI immediately.
Unfortunately, based on this signal, other G-code commands are triggered.github.com/LinuxCNC/linuxcnc/blob/9879ba...ccapy.py#L3536-L3540
At least it is handled here that this code does not run during the program.Unfortunately, this code can be executed during manual entry of MDI command with M6 T M61, or during execution of HALUI MDI_command. These two conditions are not treated.
So there is a situation where the main G-code has not yet been completed and you are already tearing another G-code for it.
A normal operator is not affected by this problem, but if you are a developer debugging ATC, I recommend deleting these lines.

If c.wait complete() could wait for MDI commands, there would be no random and unexpected behavior of LCNC.

I solved the G43 auto-on problem. I deleted it. It's not an ideal solution, but it works. Unfortunately now I have another problem and will have to look for another race condition source. It will take me a lot of time.

I think it would be better to remove the source of the problems than to mutilate the code already written.

Unfortunately, the Python Interface has nothing to allow waiting for HALUI MDI_commands and "widgets MDI_commands" to finish.

I would like to ask for your feedback.
Am I explaining this problem well enough to be understood?
Do you also think that Python Interface should be able to wait for HALUI MDI_commands and "widgets MDI_commands" to finish?
 

Qtvcp checks for idle state for MDi commands - it's actually in hal_glib function: check_for_modes().
I didn't check but i bet gladvcp uses it too.

wait_complete() checks for motion message ids so must compare that to a message id it made.
There is no way currently to wait for commands a process didn't make it's self.
This is a very big problem to try to solve.

How do you think it should work?
keep in mind we allow 'stacking' of MDI commands - you can type one in and run it and type in another (while the first is still running) and it will be run after the first.
Now add two other independant programs adding commands, both needing to finish some of their commands before calculating and adding another command. (wow)

It's fundamentally a bad process design.
But it's what we got, it works most of the time and we are short on devs with experience in that code.
Alot of the short comings we have are because we are working around fundamental problems with code too complicate for mortals to modify I tell you not to discourage your discussion only to explain why some things aren't fixed right away.
It seems you need to add an idle check in this function.

I would also suggest not using HALUI MDI commands.
Ask the Gmoccapy devs to add INI MDI command ability to Gmoccapy.

Chris

wait_complete() checks for motion message ids so must compare that to a message id it made.

Could you tell me more about "message id"? Or send me, what I should study.

keep in mind we allow 'stacking' of MDI commands - you can type one in and run it and type in another (while the first is still running) and it will be run after the first.

My question is Why? If it were up to me, I would only allow one MDI_commands to be sent. It would simplify many things. On the contrary, I try to prohibit "stacking" in my applications. Specifically, with my ATC application, it would be annoying if the operator pressed the exchange tool button 10 times and then had to wait for the ATC to take place 10 times.

How do you think it should work?

First of all, we need to think about how we want to implement it.

1st variant - we will extend the functionality of .wait_complete()
The advantage would be that there would be no need to modify ready-made codes.
The disadvantage would be that unexpected situations could arise.

2nd variant - create new functions:
.wait_halui_commands()
.wait_mdi_commands()
.wait_remaps_commands() - maybe this function won't be needed
Existing codes would need to be modified:
3rd variant - create new functions:
.halui_commands_is_on()
.mdi_commands_is_on()
.remaps_commands_is_on() - maybe this function won't be needed
These functions would have to be in periodic.
github.com/LinuxCNC/linuxcnc/blob/ed3b29...py/gmoccapy.py#L2376

For realizations, we need 3 variables in RAM memory:
int halui_commands_is_on = 0;
int mdi_commands_is_on = 0;
int remaps_commands_is_on = 0;

Python interfaces will only read this values.

4th variant- components (HALUI, MDI widget, Remap)should have HAL pins
.command_is_on OUT
.command_on IN

I already tried to implement this for HALUI commands, but I couldn't manage to write and read the variable from multiple files. Moreover, if halui is c++ and Gmoccapy is python.

For halui commands, I could write it via HAL pins, but not for normal MDI commands.

I appreciate you discussing this with me. It seems much simpler to me than what you program, for example.

 

wait complete:
github.com/LinuxCNC/linuxcnc/blob/master...ns/emcmodule.cc#L213

Stacking commands in important if you are machining by MDI commands (for instance), you don't want a pause of motion. It would mark the work piece.

Once you see how wait complete works you will see how hard it is to make it work perfectly.
It really was a hack added for AXIS back when axis couldn't embed anything and gladevcp, qtvcp and pyvcp were not even thought of yet.

I meant how do you think it should work with multiple programs sending MDI commands at the same time?
Does the first program get to run commands till it says it's finished (no stacking)?
Is it first come first serve?

Chris

In a real world the MDI is just that, for 'adjustments' and rarely used by the operator. Aren't there several ways to programmatically control the behaviour though? For instance;

Or better yet, block a condition to be met before issuing another mdi:
 

To Aciera:

HALUI MDI Commands = INI MDI Commands
This makes problems in Gmoccapy in specific situation.

c.mdi("G1 X10 Y20 Z30 F200")
This is Python Interface command

Mecanix wrote:

pos = 1.234
while (1):
    if(round(s.joint[jt]["output"], 3) == pos):
        break
    time.sleep(.01)
    s.poll()
return 1

 

I would like ask you for explaining this code.

Irrelevant to your issue, however to demonstrate how to programmatically block things up so to prevent losing control. Code is a simple motion cmd issued via MDI and blocked until it completes. Same concept can be applied to practically anything else that requires controlled logic. 

If I may add; there ain't need to clutter linuxcnc with fancy overheads designed to potentially introduce ten million bugs. AXIS(tcl), consistent HAL and a gtk+3 gui side panel with ultra-lightweight python logic is sufficient for 99.9% of the cases (I think). Lastly, MDI is an operator feature, not a software engineer/programmer one. I would avoid that at all cost.

To cmorley:

"Stacking commands in important if you are machining by MDI commands (for instance), you don't want a pause of motion. It would mark the work piece."
If someone sends commands in some non-standard application to the Python Interface c.mdi(commands), I understand that he wants it to have continuous movement. However, with machine tools such as milling machines and lathes, the ISO standard even requires that only one movement be implemented. I don't have access to a PC with LinuxCNC right now, so I don't know how MDI commands widgets behave when entering multiple commands. I will open a new thread for this issue.
forum.linuxcnc.org/gmoccapy/51209-we-all...di-commands-is-it-ok

"I meant how do you think it should work with multiple programs sending MDI commands at the same time?"
I want to prevent multiple MDIs from being sent at once.

"Does the first program get to run commands till it says it's finished (no stacking)?"
Yes, the information about finishing is what I'm missing.

"Is it first come first serve?"
No for everything.
Main program disable every other commands. This is already happening and is simply implemented using check the AUTO mode.

For widgets MDI commands and HALUI MDI commands, first come, first served applies.

For internal Python Interface c.mdi(commands) define rules GUI programer. Today is problem, that Python Interface does not give information about other MDI commands. At the same time, the GUI programmer should be able to disable the launch of widgets MDI commands and HALUI MDI commands.