can't get any signals from 7i96s I/O card

I’m having difficulty making the output pins on my Mesanet 7i96s card do anything at all. I’m looking at the outputs with an oscilloscope. 

System configuration: 
- computer hardware: n3160 CPU, 4 cores, 1.6GHz; 8GB RAM; 60 GB SATA SSD.
- OS: Xubuntu 22.01.1 LTS; Linux 5.15.55-rt48 #2 SMP PREEMPT_RT (real time kernel compiled by me).
- LinuxCNC: 2.9.0~pre0, compiled by me; run in place configuration.
- 7i96s firmware: 7i96s_d.bin; Mesaflash programs and verifies the firmware correctly.
- ethernet link: enp1s0 configured with static IP address 10.10.10.9; direct cable connection between computer and 7i96s card. 

Observations: 
- 7i96s firmware seems to load normally at power-on, including the LED flash sequence that seems to indicate that everything is normal.
- Mesaflash says:
mesaflash --addr 10.10.10.10 --device 7i96s --verify 7i96s_d.bin
Checking file... OK
File type: Efinix bin file
Verifying FLASH memory sectors starting from 0x100000...
|VVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVV
Board configuration verified successfully.
- I’m trying to configure the system with pncconf.
- Maximum servo interval is 2,703,317ns. Maximum servo jitter is 1,703,317ns.
- I have put a copy of the 7i96s binary and pin files at /lib/firmware/hm2/7i96s, following the strategy from my previous 5i25 setup. However, this seems unnecessary.
- On the MesaCard 0 configuration page, I set up the following:
-- Board name: 7i96s-Internal Data
-- Firmware: 7i96s_d
-- Card address: 10.10.10.10
-- I leave everything else unchanged and accept components changes.
- On the MesaCard 0 TB1/TB2 page, I set up the following:
-- Num 0: X-axis StepGen
-- Num 1: Y-axis StepGen
-- Num 2: Z-axis StepGen
-- everything else as default
- On the MesaCard 0 TB3 page, I set up the following:
-- input 00: ESTOP In
-- SSR output 01: Machine Is Enabled
-- everything else as default
- On the MesaCard 0 P1 page, I touch nothing.
- On the X motor page, the setup and hold periods are all set to 5000. The stepper scale is set to 16000. 

Symptoms:
- The problems start when I try to Test/Tune Axis X.- Pushing the enable button on the X-axis tune window does nothing. No signal changes state.
- Running the automatic jog program (with the delay set to zero) causes no output signals to change. I expect to see pulses and an occasional change in the dir signal.
- The LED’s change, but not in any pattern that I can detect. From time to time I see CR6 (watchdog timer overflow?) turn on, but not always. It seems to be reset upon pushing the Test/Tune Axis button.

Possible Diagnoses:
- Bad firmware on the 7i96s card. I doubt this. Most of it is built from earlier, proven iterations. It seems to correctly boot up and works with Mesaflash (which depends upon the internal FPGA based CPU functioning). It DOES seem to respond to pncconf, as in the LED’s change, but not in any predictable way.
- Bad ethernet link. I doubt this, but it is possible that the link between hm2_eth and the FPGA isn’t quite working correctly due to an improper configuration somewhere.
- Incorrect functioning of hostmot2. I doubt this, but leave it as an open option. This is pretty well tested software. BTW, following an example on another post, using halrun, I loaded hostmost2 and hm2_eth. It found the 7i96s card and listed its firmware capabilities correctly.
- Incorrect functioning of pncconf. This is likewise mature and well tested, but the interface with the 7i96s card is new, so it seems a higher probability than the previous diagnostic options.
- A configuration error on my part. I assume that this is the highest probability. However, in principle at least, the software prevents me from doing this. Outside of the new I/O board, I’m not doing anything different from my older computer/configurations. It’s not a complicated configuration.
- An incompatibility with LinuxCNC and the OS. I hope not. 

Comments:
- Please don’t chide me for wanting to use a current OS, a self compiled RT kernel and the latest LinuxCNC version. I’m just following a strategy that has worked before with my previous 32 bit computer (which ran Lubuntu 18.02) and 5i25 PCI based I/O board. Alas, there’s no more 32 bit support and PCI is long obsolete. I need LinuxCNC 2.9 in any case to use the 7i96s, which seems to be the only (sometimes) available I/O board available from Mesanet today. This basement computer will always be used for other things, although not at the same time as I am machining.
- The computer is a fanless NUC style with an almost sealed housing, perfect for the dirty environment that it will be used in. It seems to follow the KISS recommendations for computers from the developers of LinuxCNC, as much as that is possible these days. Internet connection is via WiFi. There is only one hardwired ethernet connection, directly between the computer and the 7i96s card, and my only option to connect an I/O board to this computer. 

Let me know if you need further information. 

And, thanks for your help. 

- Maximum servo interval is 2,703,317ns. Maximum servo jitter is 1,703,317ns.

This is pretty bad and likely explains the watch dog bite (no outputs will work after a watchdog bite)

I would make sure you have all power management options and speed changing options disabled
in the BIOS and also make sure you have irq coalescing disabled if you have a Intel Ethernet chip
in the PC

I am also not sure if the pncconf tune program works currently so a better test woud be to create a
LinuxCNC configuration with pncconf  ( or mesact ) and run that, as LinuxCNC has better run time diagnostics

Thanks for your very fast response!

It took me a bit to figure out whether irq coalescing was enabled. I used ethtool -c to find out. Near as I can tell, it's not.

When I've run the latency-test software before, it always seems to yield a large delay, but that seems to be related to startup. If I reset the statistics and then monitor the output, the jitter in the servo loop always seems to be below 125us.

I've yet to look into the speed changing and power management options in the BIOS, but in the past I've always had them turned off. TBD.

However, I have a question related to your comment about setting up a configuration and then running it without checking in pncconf. Are you suggesting running a program (i.e., a cutting operation) or something else that I can do in axis. Since this is my first time trying to do things this way, I'd like to be sure what you're suggesting.

Thanks again.

Yes, I was suggesting running a LinuxCNC configuration
(you could jog axis to test)

I had to take yesterday off to attend to other things. Here are the results as of now.

I configured everything with pncconf as I had done with a previous incarnation of this machine with the exception of the parameters associated with the new I/O card (which are explained above).

I then tried to run Axis. It opens with two errors. 1) Unexpected delay on task 0 with period 1000000. This message will display only once per session. Run the Latency Test and resolve before continuing. 2) hm2/hm2_7i96s.0: error finishing read! iter=445 (or other large number).

I assume that the second error is related to the first.

Axis won't let me enable any of the motor drivers (the red circle X icon at the top left is permanently enabled). Presumably this is because of a communication failure with the 7i96s card.

Meanwhile, I ran the latency test over nearly 48 hours, simultaneously running glxgears, a web browsers and hardinfo's benchmarks. This was after clearing the first set of statistics after it initializes which shows numbers similar to what I wrote above. The results of this test are that, on average, latency errors are on the order of 120us and the peak latency error is 200us. So, if the problem I'm having is related to latency, it's always the first (and it seems only) long delay that is causing the problem.

In any case, I can't move forward and am looking for more suggestions. FWIW, my nics are Realtek RTL8111/8168/8411 PCI express devices. I have no option to change these.

Thanks in advance.

That NIC is the one I am having problems with.
For some reason, it has excessive latency on the ethernet connection.
This is not picked up by a normal latency test
I have been meaning to try another version of Debian to see if it improves. But I am only configuring a machine now not using it so have other priorities.
It seems to be a problem reported in recent history, not in the past.
There is a specific debian driver for it you could try.. Search for it on the debian package search on their web site. It may help...

Yes, the errors are Ethernet latency related

What are the ping times if you ping the 7I96S card?

I would make sure that you have all power saving options disabled
in the PC BIOS, especially any CPU speed changing options, C1 states > 1
etc.

You get the read error (with default settings) when you have 5 timeouts in a a row ( the default timeout at a 1 ms servo thread rate is 800 usec or 80%
of the servo thread period )

OK folks. RodW is the wackamol winner! He hit the nail on the head!

The solution (in my case) is to install the r8168-dkms driver following the directions given by unixblogger here: unixblogger.wordpress.com/2016/08/11/how...rking-updated-guide/.

The result is that I can communicate with the 7i96s card without any problems, with no indication of being bitten by the watchdog, and with pulse and dir signals coming from the card exactly as one would expect.  Jitter is considerably reduced as well, typically to < 100us, worst case 200us, and no delay at startup.

I still have a problem with getting the enable signal to work, both as an output and with axis, but I'll post that as another issue. The issue related to this post has been solved. It's important enough for other potential users to keep separate from everything else.

Hooray!

That's interesting, looks like the default RTK driver has gotten broken recently
as this does seem like a relatively new issue.