FWIW, here’s some more info on this problem. Recently, I’ve been tossing out old computers due to a pending move, so I’ve really got no spares. But, I realized that my old 32 bit LinuxCNC computer was still a viable candidate for installing the latest version of LinuxCNC, which I did. The results were unexpected: exactly the same. pncconf works just fine; axis doesn’t at all. These results don’t help isolate the problem much, but they tend to rule out the probability of a problem with the computer hardware or the OS (the old one is Lubuntu 18.04).
What more then could I do to triangulate the problem? Well, I had additional data from tcpdump, so I decided to analyze it. Each analysis included about 100,000 packets. Here are my results.
1) On average, there are two packets transmitted by the computer for each one transmitted by the card. The standard deviation of the times between packets is about 35us and the maximum period between a received packet and the next transmitted one in successive packets, i.e, the two that come before the response packet ,is 880us typically, and exceeds 950us once (954us). There are zero dropped packets.
2) I reverse engineered the communication protocol within the udp packets in accordance with the documentation for the 7i96s card. All packets are being transmitted are within the bounds of the documented protocol (although I can’t dig deeper than this because it requires a deeper understanding of how the card works than I’m willing to try to get). All of the response packets return the correct number of bytes in accordance with the read commands.
3) This true for both axis and pncconf. The differences between the two sets of results are essentially zero.
What this tells me is that the ethernet hardware, the handlers within the OS, and the parts of LinuxCNC that are involved with setting up packets, transmitting and receiving them, and scheduling their transmissions are working correctly. It appears that the firmware in the card is working correctly too, at least as it relates to the reception, interpretation and response to the packets it receives. Though I don’t know for sure, I assume that the handlers for these operations are identical for pncconf and axis.
From arvidb’s documentation of the communications call stack and from my further investigation into the hm2eth software, it seems improbable that something is mangled along the way. The message handling seems straightforward, although how it gets from level to level is not always obvious (to me).
This suggests some sort of incompatibility between axis and the 7i96s card that I have. Given that pncconf moves the axes on my mill correctly with the 7i96s, I have doubts about whether the problem is with the card, but can’t entirely eliminate it. Could it be, for example, a new “feature,” needed by axis, was added to the card at the last moment but somehow the firmware version that I have is one that precedes that? There is no way that I can assuredly make that diagnosis, nor eliminate the possibility. Otherwise, axis and the software upon which it uniquely depends remains the primary likely cause for the problem.
Because of the move I’ve been in a race against time. At this point it looks like I’ll assuredly lose. If I return the card, then, if things go well for the move (and if the card arrives on time), it will be packed in a box apart from my mill which will ship only partially assembled, and I’ll have to discover its location six months or more down the road when I’m ready to try to bring it up. Since my move is overseas, I’ll have to explain to the customs agent why I need yet another computer, this one 15 years old, because it’s the only one that has a PCI bus which I need to keep the mill functional until I can reintegrate it will the 7i96s. Making it work with my 5i25 PCI card is / will be harder than it seems because of the fact that I’ve had to repackage the mill in order to make it ready to move and I designed the package around the 7i96s. Hence, a rather intense motivation to get things right before I move.
You should know that I’m an electronics engineer, with 45+ years of experience, much of it managing others, and a good deal of it associated with writing software/firmware for medical devices. I know full well the impossibility of proving the negative, i.e., that there are no bugs, especially in systems as complex as LinuxCNC. I also am well familiar with the problem of having obstinate engineers who won’t investigate their own work to discover why things won’t work, even if the evidence points there. I’ve got about a 90% hit rate when I follow the evidence without prejudice. It’s been my experience that following the evidence is a much faster way getting to and fixing problems than any other way, and right now time is of the essence. If my style has been abrasive because of my insistence on it, I hope you’ll understand why.
Of course, that leaves the other 10% where I’m wrong, and this could yet be one of those cases. I’m open to other troubleshooting suggestions and other potential failure modes if you’ll let me know what they are and provide justification as to how they can induce the current behavior. If there are none, I guess I’ll return the card, take the hit, and hope for the best.
