Trajectory Planner using Ruckig Lib

unless the minimum radius setting would not allow for the transition to move from the previous line plane to the new line target plane maybe?

Yes, it's likely that the vertical move from the G0 traverse to the G1 contour depth is too short to create a fillet that satisfies the limit of curvature so it basically decides to take the longer route. Not sure how easy it is to detect such cases in the algorithm but it is going to be absolutely crucial to get this rock solid.

One solution: In cam programs, the travelheight has to be at least the G64 P..

I don't think this is practical as the operator would have to keep this in mind for every possible machining operation. Checking the filleting result in a GCODE file with thousands of lines is simply not possible and even if this were a settable global cam option it would mean that a change in G64 P value required running everything through the CAM again.

The G0 z line from z1 to z0 is 1mm. This line is then trimmed by the algo on both sides
back to minimal lenght ~0.001mm, as it can not be trimmed back 20mm.

So, if the algorithm realizes that the segment cannot be trimmed by the required distance it would seem possible to handle these cases separately. Either just don't create a fillet at all or create a fillet with a bigger curvature.

Something else I would like to ask is the meaning of the P value, currently it seems to be the distance a segment is trimmed back but really what the operator wants to know is the maximum deviation from the end/start point this is going to create. In the papers I've read about filleting algorithms the process is actually iterative. After a fillet is created the maximum deviation is calculated. If the deviation is over the acceptable limit the filleting parameters are changed, the fillet is created again and the deviation is again checked. This runs until the maximum deviation is below the limit (ie below the P value).
To be honest, I have no idea how difficult it is to calculate the maximum deviation a clothoid fillet creates from the original line segments but without the ability to precisely define a maximum value for path deviation I'd be _very_ reluctant to use a filleting algorithm.

Regarding the calculation of maximum path deviation, I wonder if it were enough to calculate the 3d coordinates at say 25%, 50% and 75% of the total length of the trimmed segments and compare those to the 3d coordinates at the same percentages along the replacement clothoid/spline. Then calculating the 3d distances between the resulting pairs of points would be easy.

Also I wonder if the trim-back length could be the variable driven by the max-deviation calculation result:

Start the iteration with a very small trim-back value and increase until either
a) the deviation exceeds the maximum set by the P value
b) one to the original path segments gets too short

This should also solve the current problem that the angle between two path segments will greatly influence the deviation of the fillet to the original path. With a constant trim-back value and a decreasing angle the fillet moves further and further away from the intersection point.
While a constant maximum deviation will create a shorter fillet with higher curvature. This will of course mean a decrease in tool velocity but brings the necessary certainty about maximum path deviation.

Regarding spline roughing.
I'm not sure if you envisioned using the filleting algorithm to optionally create feed optimized roughing paths but if you did this we could then abandon the max-deviation value and use a minmal-curvature value instead. The algorithm would then increase the trim-back until either
a) minimal-curvature value is reached
b) one of the original path segments gets too short.

It's done this way for now, to keep it simple and get things to work. Adding loops to the code to get as close
to the real path deviation is possible for sure. This code change is on the todo list.

I get it, but keep in mind that details like this will make all the difference between success and failure for a project like this. I just make suggestions and since you are doing all the hard work it is of course up to you how to proceed.

Current implementation uses the P value as a trim value for corners and then
create fillet. This results in a deviation less then should be.

See my post about dependency of deviation on the angle between two line segments

Path deviation check at progress t=0.5 of the fillet segment to original path is ok to find closest value of P..
If that is what you mean?

Yes

Yes, and on the other hand, i showed an exceptional case. Normally a travel height would be like 1 up to 50mm and
a G64 P0.01 for the fillets. This will not be a problem at all.

In my opinion it is usually much better to handle the exceptional cases from the beginning. Once these things go out into the wild there are going to be enough cases left that nobody ever thought about.

Here we tested a velocity profile based on current curvature along the path.
The red line is a interpolated velocity offset along the path based on the shape's curvature input.
This is interesting.

Indeed, very good idea to plot the velocity on top of the tool path! Thanks for sharing.