Remora - ethernet NVEM / EC300 / EC500 cnc board

Hi, the threadFreq is passed in the stepgen constructor which is set in main.cpp configThreads() function.

Those conversions are possible with the compiler handling this with the appropriate loss of precision along the way. Maybe these should all be doubles or floats until the last conversion step to int32_t.

The dds accumulator is a little mystery with the logic copied from the standard linuxCNC stepgen component with the implementation split between linuxCNC and the controller board.

I modified the remora-eth.c to ignore and log any delta larger than 40 counts. I ran the controller through a simple program and collected several hundred error events. Occasionally such an event would cause the controller to become slow and unresponsive for 0.3-0.4 seconds followed by a reset. This usually results in a following error, turning off the machine. The rest of the time the errors were single jump events which were successfully ignored and logged by my modified remora-eth in the raspberry pi. 

263 out of 310 logged events involved a jump ~419 counts. 228 of those were near a stop or direction change. The rest occurred at low speeds. The other 47 jumps were generally 50-100 counts. They generally occurred while the axis was moving. 

I added a whole bunch printfs inside if statements and found that every once in a while frequencyCommand within the stepgen module has a value close to 2^24. It seems as though it is losing a byte somewhere. 

Interestingly: 2^24 Hz * frequencyScale (104.8576) = 1759218604 which is 419.43 x 2^22

This appears to be where the jumps of ~419 counts is coming from. On a direction change the high bytes go from all 0 to all f or back, if the highest byte is left out this results in a value of 2^24. I have also recorded values of ~2^16 suggesting 2 lost bytes. Without a direction change the issue does not seem to be as bad. Possibly the byte is retaining the old value, so if the values don't change too much it isn't as large of a jump if it jumps at all. This suggests that these errors are going on more frequently and going undetected until the value changes by a large enough to cause an issue. 

On the plus side this means the accumulator is working, its just getting faulted input for some reason. I wondered if this was coming from the copy from rxBuffer to rxData being interrupted. but the same check that catches frequencyCommand being out of range in stepgen does not detect an error when implemented immediately after the copy. I tried making multiple reads, but it consistently reads the wrong value. I confirmed that at startup the pointers are configured correctly.

I am not very familiar with the debugging tools or 32 bit microcontrollers in general. I have been reading about the debugger and how to get more data about the point of failure so hopefully I can drill down further to the root of the problem.

I added more printfs and recorded this:
joint 2 frequency command: 16777215
frequency command pointer: 16777215
frequency command pointer: 536872064

rxData frequency command: 16777215
rxData frequency command pointer: 536872064
rxBuffer frequency command: -1
rxBuffer frequency command pointer: 536871996

This suggests that the copy from rxBuffer to rxData is being interrupted by a read event.

Some really interesting and useful observations and fault finding!!! 

The udp_data_callback() is running in the main loop, with the threads running in interrupt context. If a thread ISR is triggered during the call to the callback then it would be possible for the data move to be interrupted by a memory read.

Implementing a mutex for the data buffer would be the solution if your theory is correct. And probably a good feature anyway.

The may also explain why the SPI version of Remora has not seen this issue as DMA is used to move data with the memory management ensuring that only DMA or the CPU accesses memory at a time.

IT'S FIXED!! .... well it looks like it.

A huge thanks to spdbmp927 for working out that it was a corrupted rxBuffer. I have pushed a bug fix to the develop branch. I've run Roy's gcode file, which before would cause lots of following error spikes, successfully with a very small follower error setting. And this is at +400% normal speed.

The fix was as simple as disabling interrupts when moving the rxData into the rxBuffer.

Would be great to get feedback from others if this issue is now resolved.

Thanks again spdmbp927!!

Yes I can confirm that it's fixed. I'm really glad you both found and solved the problem!