6i25 custom firmware and weird latency

I have a 6i25 PCIe card connected to a dual core Intel Core i3 system.
No add-on cards, I'm connecting from the DB25 connectors directly to custom opto-isolated brushed DC drivers & encoders.
I have built a custom firmware for the FPGA to add the number of PWM generators and encoder inputs I need.

The basic setup works and I can control the motors, but I'm getting "Unexpected realtime delay on task 0 with period 1000000".
I have gone through the usual debugging, latency-test by itself shows under 20 us latency.
I have also tried the "intel_idle.max_cstate=0 processor.max_cstate=0 idle=poll" and "isolcpus=1" settings, to no effect.

After more debugging, I have discovered that the hm2_6i25.0.read.tmax jumps to over 14 ms values every few seconds. Otherwise it stays well under 100 us.

I created a very basic setup that reproduces the problem as soon as I set hm2_[HOSTMOT2](BOARD).0.pwmgen.00.enable to 1. As long as the pwm generators are off, everything is fine. The problem also happens if I don't have any cables connected to the FPGA card.

Then I tested with manufacturer provided 5i25_g540x2.bit firmware, and it does not have the problem. I also recompiled that configuration with the Xilinx ISE version I have installed, and it also works fine. Then I thought it was an issue with 6i25 version firmware, so I rebuilt the custom firmware with 5i25 configuration. It still does not work.

The only changes I made in the firmware are in the pin description file: custom vs. original .

I'm using LinuxCNC version 2.8.2-11-g6a3d0a434 installed from the official .iso file.
I have attached a package with configuration that reproduces the problem by running:

I guess next thing I have to try is to do the pin file changes one by one and see when it breaks. But maybe someone can see some obvious reason why this is happening?

It sounds like maybe the configuration is not meeting timing

I would try a 5i25 configuration and make sure the 6I25 is
jumpered for 5I25 emulation

I have the jumpers like this:


And I did try with the custom config compiled for 5i25.
The timing analyzer appears to pass.
The worst path is indeed from bus command to pwmgen, but it still appears to have enough slack for even 66 MHz operation (but I'd be fine with 33 MHz):

 

Strange, if you get similar symptoms on a 66 MHz config then
its not likely timing related.

Could possible be a bad card or a PC issue, do you have another 6I25
card or PC to try?

I only have this 6i25 card, but I can possibly arrange another PC.

With the config built for 6i25 66 MHz mode: When jumpered for 33 MHz, I got the same symptoms and behavior as with the 5i25 33 MHz build. When jumpered for 66 MHz, the whole PC froze as soon as PWM generator was enabled.

The fact that it only happens for the custom config and not for the quite similar G540x2 config suggests that it may be something in my config after all. It'll be a week until I'm at the hacklab again, but I'll try to figure out what change causes the problem.

Ok, I am back at debugging this.

By bisecting, I found that the problem starts occurring as soon as I add a PWMBDirPin to pinout configuration.

I then checked a factory built configuration that has PWMBDirPin, and I get the same problem with prebuilt 5i25_r990x2.bit (md5sum ab6c10fb701025588f68b7ba527bb4dc), as soon as I write hm2_5i25.0.pwmgen.00.enable to 1.
 

I'll see if I can duplicate this here. Still sounds like bad hardware (Either 6I25 or PC)

Actually it is not related to PWMBDirPin. I just had pwmgen.output-type 2, so it didn't output anything until B pin was present. If I set output-type to 1 or value to -0.5, the latency problem starts as soon as I have a PWM output pin.

Seems like it may be some kind of EMI problem caused by the PWM output. Weird though, considering I don't even have any cables connected.

Definite correlation with PWM frequency, if I raise it from 40 kHz to the 390 kHz maximum, the latency effects start occurring multiple times per second.

I guess I'll try another PC next, and maybe try to scope the power lines or something. Though it feels weird that toggling an IO pin that is just connected to a few centimeters of PCB trace and has slew rate set to slow would cause so much EMI.

A shorted GPIO might cause that kind of an issue. You might verify that all GPIO
are at 5V when linuxCNC is not running