watchdog and network connection problems

Do you get a red /INIT LED illuminated  when simply powering the 7I96S?
When cycling the 7I96S power, both the /DONE and /INIT
red LEDs will blink briefly but then should go out. If they remain
illuminated after power up, this indicates corrupted flash memory.
(with one exception (W4,W5 up,up) shown below.

An illuminated  red /INIT LED can occur after running LinuxCNC as
this indicates a watchdog fault (a normal occurrence when linuxCNC is exited)
You can reset this by cycling the 7I96S power

If so, can you try changing the IP address jumpers (W4 and W5)
to the "up,up" positions and cycling the 7I96S power?

If that gives you a blinking red light, you can try and reflash the card
 

Error report created by /usr/lib/tcltk/linuxcnc/show_errors.tcl:

Print file information:
RUN_IN_PLACE=no
LINUXCNC_DIR=
LINUXCNC_BIN_DIR=/usr/bin
LINUXCNC_TCL_DIR=/usr/lib/tcltk/linuxcnc
LINUXCNC_SCRIPT_DIR=
LINUXCNC_RTLIB_DIR=/usr/lib/linuxcnc/modules
LINUXCNC_CONFIG_DIR=
LINUXCNC_LANG_DIR=/usr/lib/tcltk/linuxcnc/msgs
INIVAR=inivar
HALCMD=halcmd
LINUXCNC_EMCSH=/usr/bin/wish8.6
LINUXCNC - 2.9.3
Machine configuration directory is '/home/mill/linuxcnc/configs/mill'
Machine configuration file is 'mill.ini'
INIFILE=/home/mill/linuxcnc/configs/mill/mill.ini
VERSION=1.1
PARAMETER_FILE=linuxcnc.var
TPMOD=
HOMEMOD=
TASK=milltask
HALUI=halui
DISPLAY=qtvcp qtdragon
COORDINATES=XYZ
KINEMATICS=trivkins coordinates=XYZ
Starting LinuxCNC...
Starting LinuxCNC server program: linuxcncsvr
Loading Real Time OS, RTAPI, and HAL_LIB modules
Starting LinuxCNC IO program: io
libnml/buffer/shmem.cc 320: Shared memory buffers toolCmd and toolCmd may conflict. (key=1004(0x3EC))
libnml/buffer/shmem.cc 320: Shared memory buffers toolSts and toolSts may conflict. (key=1005(0x3ED))
libnml/buffer/shmem.cc 320: Shared memory buffers emcError and emcError may conflict. (key=1003(0x3EB))
Starting HAL User Interface program: halui
linuxcnc TPMOD=tpmod HOMEMOD=homemod EMCMOT=motmod
Found file(REL): ./mill.hal
Found file(REL): ./custom.hal
Starting TASK program: milltask
Starting DISPLAY program: qtvcp
hm2/hm2_7i96s.0: Watchdog has bit! (set the .has-bit pin to False to resume)

Unexpected realtime delay on task 0 with period 1600000
This Message will only display once per session.
Run the Latency Test and resolve before continuing.

hm2/hm2_7i96s.0: error finishing read! iter=14604

Shutting down and cleaning up LinuxCNC...
Running HAL shutdown script
task: 4597 cycles, min=0.000007, max=0.014843, avg=0.010020, 0 latency excursions (> 10x expected cycle time of 0.010000s)
hm2: loading Mesa HostMot2 driver version 0.15
hm2_eth: loading Mesa AnyIO HostMot2 ethernet driver version 0.2
hm2_eth: 10.10.10.10: INFO: Hardware address (MAC): 00:60:1b:16:8a:7b
hm2_eth: discovered 7I96S
hm2/hm2_7i96s.0: Low Level init 0.15
hm2/hm2_7i96s.0: Smart Serial Firmware Version 43
hm2/hm2_7i96s.0: 51 I/O Pins used:
hm2/hm2_7i96s.0: IO Pin 000 (TB3-01): InM Input Module #0, pin in0 (Input)
hm2/hm2_7i96s.0: IO Pin 001 (TB3-02): InM Input Module #0, pin in1 (Input)
hm2/hm2_7i96s.0: IO Pin 002 (TB3-03): InM Input Module #0, pin in2 (Input)
hm2/hm2_7i96s.0: IO Pin 003 (TB3-04): InM Input Module #0, pin in3 (Input)
hm2/hm2_7i96s.0: IO Pin 004 (TB3-05): InM Input Module #0, pin in4 (Input)
hm2/hm2_7i96s.0: IO Pin 005 (TB3-06): InM Input Module #0, pin in5 (Input)
hm2/hm2_7i96s.0: IO Pin 006 (TB3-07): InM Input Module #0, pin in6 (Input)
hm2/hm2_7i96s.0: IO Pin 007 (TB3-08): InM Input Module #0, pin in7 (Input)
hm2/hm2_7i96s.0: IO Pin 008 (TB3-09): InM Input Module #0, pin in8 (Input)
hm2/hm2_7i96s.0: IO Pin 009 (TB3-10): InM Input Module #0, pin in9 (Input)
hm2/hm2_7i96s.0: IO Pin 010 (TB3-11): InM Input Module #0, pin in10 (Input)
hm2/hm2_7i96s.0: IO Pin 011 (TB3-13/TB3-14): SSR #0, pin Out-00 (Output)
hm2/hm2_7i96s.0: IO Pin 012 (TB3-15/TB3-16): SSR #0, pin Out-01 (Output)
hm2/hm2_7i96s.0: IO Pin 013 (TB3-17/TB3-18): SSR #0, pin Out-02 (Output)
hm2/hm2_7i96s.0: IO Pin 014 (TB3-19/TB3-20): SSR #0, pin Out-03 (Output)
hm2/hm2_7i96s.0: IO Pin 015 (TB3-21/TB3-22): OutM Output Module #0, pin Out-04 (Output)
hm2/hm2_7i96s.0: IO Pin 016 (TB3-23/TB3-24): OutM Output Module #0, pin Out-05 (Output)
hm2/hm2_7i96s.0: IO Pin 017 (TB1-02/TB1-03): StepGen #0, pin Step (Output)
hm2/hm2_7i96s.0: IO Pin 018 (TB1-04/TB1-05): StepGen #0, pin Direction (Output)
hm2/hm2_7i96s.0: IO Pin 019 (TB1-08/TB1-09): StepGen #1, pin Step (Output)
hm2/hm2_7i96s.0: IO Pin 020 (TB1-10/TB1-11): StepGen #1, pin Direction (Output)
hm2/hm2_7i96s.0: IO Pin 021 (TB1-14/TB1-15): StepGen #2, pin Step (Output)
hm2/hm2_7i96s.0: IO Pin 022 (TB1-16/TB1-17): StepGen #2, pin Direction (Output)
hm2/hm2_7i96s.0: IO Pin 023 (TB1-20/TB1-21): StepGen #3, pin Step (Output)
hm2/hm2_7i96s.0: IO Pin 024 (TB1-22-TB1-23): StepGen #3, pin Direction (Output)
hm2/hm2_7i96s.0: IO Pin 025 (TB2-02/TB2-03): StepGen #4, pin Step (Output)
hm2/hm2_7i96s.0: IO Pin 026 (TB2-04/TB2-05): StepGen #4, pin Direction (Output)
hm2/hm2_7i96s.0: IO Pin 027 (TB2-07/TB2-08): Encoder #0, pin A (Input)
hm2/hm2_7i96s.0: IO Pin 028 (TB2-10/TB2-11): Encoder #0, pin B (Input)
hm2/hm2_7i96s.0: IO Pin 029 (TB2-13/TB2-14): Encoder #0, pin Index (Input)
hm2/hm2_7i96s.0: IO Pin 030 (TB2-16/TB2-17): IOPort
hm2/hm2_7i96s.0: IO Pin 031 (TB2-18/TB2-19): IOPort
hm2/hm2_7i96s.0: IO Pin 032 (internal): IOPort
hm2/hm2_7i96s.0: IO Pin 033 (internal): SSR #0, pin AC Ref (internal) (Output)
hm2/hm2_7i96s.0: IO Pin 034 (P1-01/DB25-01): IOPort
hm2/hm2_7i96s.0: IO Pin 035 (P1-02/DB25-14): IOPort
hm2/hm2_7i96s.0: IO Pin 036 (P1-03/DB25-02): IOPort
hm2/hm2_7i96s.0: IO Pin 037 (P1-04/DB25-15): IOPort
hm2/hm2_7i96s.0: IO Pin 038 (P1-05/DB25-03): IOPort
hm2/hm2_7i96s.0: IO Pin 039 (P1-06/DB25-16): IOPort
hm2/hm2_7i96s.0: IO Pin 040 (P1-07/DB25-04): IOPort
hm2/hm2_7i96s.0: IO Pin 041 (P1-08/DB25-17): IOPort
hm2/hm2_7i96s.0: IO Pin 042 (P1-09/DB25-05): IOPort
hm2/hm2_7i96s.0: IO Pin 043 (P1-11/DB25-06): IOPort
hm2/hm2_7i96s.0: IO Pin 044 (P1-13/DB25-07): IOPort
hm2/hm2_7i96s.0: IO Pin 045 (P1-15/DB25-08): IOPort
hm2/hm2_7i96s.0: IO Pin 046 (P1-17/DB25-09): IOPort
hm2/hm2_7i96s.0: IO Pin 047 (P1-19/DB25-10): IOPort
hm2/hm2_7i96s.0: IO Pin 048 (P1-21/DB25-11): IOPort
hm2/hm2_7i96s.0: IO Pin 049 (P1-23/DB25-12): IOPort
hm2/hm2_7i96s.0: IO Pin 050 (P1-25/DB25-13): IOPort
hm2/hm2_7i96s.0: registered
hm2_eth: in hm2_eth_reset
hm2_eth: HostMot2 ethernet driver unloaded
hm2: unloading
Removing HAL_LIB, RTAPI, and Real Time OS modules
Removing NML shared memory segments

Debug file information:
Note: Using POSIX realtime
[QTvcp.QTDRAGON_HANDLER][WARNING] QtDragon Warning with loading QtWebEngineWidget - is python3-pyqt5.qtwebengine installed? (qtdragon_handler.py:31)
[QTvcp.QTVCP.QT_ISTAT][WARNING] INI Parsing Error, No USE_PROBE Entry in PROBE, Using: none (qt_istat.py:532)
hm2/hm2_7i96s.0: Watchdog has bit! (set the .has-bit pin to False to resume)
Unexpected realtime delay on task 0 with period 1600000
This Message will only display once per session.
Run the Latency Test and resolve before continuing.
hm2/hm2_7i96s.0: error finishing read! iter=14604
[QTvcp][CRITICAL] Retry from Error Dialog
Qtvcp encountered an error. The following information may be useful in troubleshooting:
LinuxCNC Version : 2.9.3

Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/qtvcp/widgets/origin_offsetview.py", line 383, in periodic_check
self.reload_offsets()
File "/usr/lib/python3/dist-packages/qtvcp/widgets/origin_offsetview.py", line 270, in reload_offsets
degree_tmpl = "%{}.2f".format(len(locale.format(tmpl, 0)))
^^^^^^^^^^^^^
AttributeError: module 'locale' has no attribute 'format'. Did you mean: '_format'?

(qtvcp:515)
[QTvcp][CRITICAL] Retry from Error Dialog
Qtvcp encountered an error. The following information may be useful in troubleshooting:
LinuxCNC Version : 2.9.3

Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/qtvcp/widgets/origin_offsetview.py", line 383, in periodic_check
self.reload_offsets()
File "/usr/lib/python3/dist-packages/qtvcp/widgets/origin_offsetview.py", line 270, in reload_offsets
degree_tmpl = "%{}.2f".format(len(locale.format(tmpl, 0)))
^^^^^^^^^^^^^
AttributeError: module 'locale' has no attribute 'format'. Did you mean: '_format'?

(qtvcp:515)
[QTvcp][CRITICAL] Aborted from Error Dialog
Qtvcp encountered an error. The following information may be useful in troubleshooting:
LinuxCNC Version : 2.9.3

Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/qtvcp/widgets/origin_offsetview.py", line 383, in periodic_check
self.reload_offsets()
File "/usr/lib/python3/dist-packages/qtvcp/widgets/origin_offsetview.py", line 270, in reload_offsets
degree_tmpl = "%{}.2f".format(len(locale.format(tmpl, 0)))
^^^^^^^^^^^^^
AttributeError: module 'locale' has no attribute 'format'. Did you mean: '_format'?

(qtvcp:511)
/usr/bin/linuxcnc: line 977: 22836 Segmentation fault $EMCDISPLAY -ini "$INIFILE" $EMCDISPLAYARGS $EXTRA_ARGS
22785
22833
Stopping realtime threads
Unloading hal components
Note: Using POSIX realtime

---

Info report created by linuxcnc_info:
The file: /tmp/linuxcnc_info.txt
can be posted to a forum or a web site like:
pastebin.com
in order to provide information about the linuxcnc
system and configuration.

Date: Mon Jul 15 17:24:06 UTC 2024
UTC Date: Mon Jul 15 17:24:06 UTC 2024
this program: /usr/bin/linuxcnc_info
uptime: 17:24:06 up 53 min, 2 users, load average: 2.60, 2.54, 2.86
lsb_release -sa: Debian Debian GNU/Linux trixie/sid n/a trixie
linuxcnc: /usr/bin/linuxcnc
pwd: /home/mill/linuxcnc/configs/mill
USER: mill
LOGNAME: mill
HOME: /home/mill
EDITOR:
VISUAL:
LANGUAGE:
TERM: dumb
COLORTERM:
DISPLAY: :0.0
DESKTOP: lightdm-xsession
display size: 3200x1080 pixels (847x286 millimeters)
PATH: /usr/bin:/home/mill/linuxcnc/configs/mill/bin:/usr/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games

uname items:
nodename -n: 4axis
kernel-name -s: Linux
kernel-vers -v: #1 SMP PREEMPT_RT Debian 6.9.8-1 (2024-07-07)
machine -m: x86_64
processor -p: unknown
platform -i: unknown
oper system -o: GNU/Linux

/proc items:
cmdline: BOOT_IMAGE=/boot/vmlinuz-6.9.8-rt-amd64 root=UUID=49de45c3-2dc5-457b-ace0-1f60b3cb03d3 ro quiet
model name: Intel(R) Core(TM) i5-3570 CPU @ 3.40GHz
cores: 4
cpu MHz: 3393.341
parport:
serial:

Versions:
gcc: gcc (Debian 13.3.0-1) 13.3.0
python: Python 3.12.4
git: git version 2.43.0
git commit: NA
tcl: 8.6
tk: 8.6
glade: not_in_PATH

linuxcnc_var all:

LINUXCNCVERSION: 2.9.3
LINUXCNC_AUX_GLADEVCP: /usr/share/linuxcnc/aux_gladevcp
LINUXCNC_AUX_EXAMPLES: /usr/share/linuxcnc/aux_examples
REALTIME: /usr/lib/linuxcnc/realtime
RTS: uspace
HALLIB_DIR: /usr/share/linuxcnc/hallib
PYTHON: /usr/bin/python3

dpkg -l '*linuxcnc*':
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-===================-============-============-=====================================================================
un linuxcnc <none> <none> (no description available)
un linuxcnc-dev <none> <none> (no description available)
un linuxcnc-doc <none> <none> (no description available)
ii linuxcnc-doc-en 2.9.3-1 all motion controller for CNC machines and robots (English documentation)
un linuxcnc-doc-es <none> <none> (no description available)
un linuxcnc-doc-fr <none> <none> (no description available)
un linuxcnc-sim <none> <none> (no description available)
un linuxcnc-sim-dev <none> <none> (no description available)
ii linuxcnc-uspace 2.9.3-1+b2 amd64 motion controller for CNC machines and robots
ii linuxcnc-uspace-dev 2.9.3-1+b2 amd64 PC based motion controller for real-time Linux

here is the error report
i installed debian 13 according to docs.google.com/document/d/1jeV_4VKzVmOI...m1x7oiw8VcLFdiY/edit.
i was getting the red light when i powered on the board with debian 12 but since i installed debian 13, i can go through pncconf and test each axis i just cant run linuxcnc. and i was haviing trouble testing before. and forget about adding another ethernet card. i had it working for a few hours on debian 12 but i would shut it down and go back the next day and it wound give me the wathcdog errror. now i just get the red light when i try to run linuxcnc. and i can test each axis in pncconf.

"i was getting the red light when i powered on the board with debian 12"

This really does not make any sense as there is nothing the Ethernet/host connection
can do to illuminate a red LED on the 7I96S other than by running LinuxCNC.
(exiting a LinuxCNC run _will_ result in a red /INIT LED being illuminated)

Can you run this command:

sudo chrt 99 ping -i .001 -q 10.10.10.10
 hit control C
then run it again
sudo chrt 99 ping -i .001 -q 10.10.10.10
wait about a minute and hit control C again
 and copy paste the results here
 

i will run sudo chrt 99 ping -i .001 -q 10.10.10.10, when i am in front of it again tomorrow. i have run that command before and let it run for several minutes. the ping times don't seem too bad with ping 10.10.10.10 and ping 10.10.10.1. > ~ .100 and ~.037 respectively. i'll get an accurate measurement in the morning.

what would be the drawbacks of reflashing the mesa 7i96s? a clean slate if you know what i mean. but i don't want to do something for no reason. if you think a re-flash would do anything or not.............

the reason i am questioning the mesa hardware is, i have an intel and an amd machine and i get very similar errors with both machines. im currently running on the dell vostro 270 i5. an extra ethernet card in both machines conflict and it is unable to start. but not right away. on either machine. if i install the OS with the card in, it conflicts. i have to install debian without the additional card in to get linuxcnc installed, run pncconf, setup linuxcnc, and then install additional ethernet card and it will run for a little while, but after i shut it off, it won't start back up inless i reinstall the OS.

at this point, even with that extra card out, i can go through pncconf and test each axis, but when in qtdragon, i get a watchdog error in the gui and the red light on the mesa 7i96s comes on and it makes me exit.
i

I don't think re-flashing the card will do anything useful

You should never have to re-install the OS to fix a networking issue

Try the ping test and I will have a list of other tests to try and lock
down what is happening.

10.10.10.10 ping statistics ---
572314 packets transmitted, 572232 received, +33 errors, 0.0143278% packet loss, time 582100ms
rtt min/avg/max/mdev = 0.051/0.059/7.494/0.045 ms, pipe 12
mill@4axis:~$ sudo chrt 99 ping -i .001 -q 10.10.10.10
PING 10.10.10.10 (10.10.10.10) 56(84) bytes of data.
^C
--- 10.10.10.10 ping statistics ---
104114 packets transmitted, 104114 received, 0% packet loss, time 104127ms
rtt min/avg/max/mdev = 0.052/0.060/7.454/0.033 ms

actually, this works in axis. its just qtdragon im having trouble with. thank you for all the help. as long as i have a working system, i can figure it out from here. thanks again.

i would like to install another ethernet card but i want to see how this is going to be for reliability over the next few days and if i don't have any issues, i will see about getting another port in there. it seems those realtek devices are in everything. only time will tell.

to recap; i was having network issues with debian 12 and network connection issues with Kernel up to 6.3. when i installed debian "13" trixie, the system seems to be working for now. .....

Traceback (most recent call last):
File "/usr/bin/qtvcp", line 555, in <module>
_qtvcp = QTVCP()
^^^^^^^
File "/usr/bin/qtvcp", line 261, in __init__
window.load_extension(opts.usermod)
File "/usr/lib/python3/dist-packages/qtvcp/qt_makegui.py", line 269, in load_extension
methods, self[obj].handler_module, self[obj].handler_instance = self._load_handlers([handlerpath], self.halcomp, self[obj])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/qtvcp/qt_makegui.py", line 297, in _load_handlers
mod = __import__(basename)
^^^^^^^^^^^^^^^^^^^^
File "/usr/share/qtvcp/panels/test_panel/test_panel_handler.py", line 5, in <module>
from distutils.spawn import find_executable
ModuleNotFoundError: No module named 'distutils'

is the error i am getting with qtdragon

this machine will work for a few days in Axis gui and then stops communicating. Resource temporarily unavailable. no communication lights on the rj45 connector on the mesa 7i96 and unable to ping 10.10.10.10. i didn't change anything. im also having some other communication problems within Axis when it was running. my servo spindle will have a long delay before linuxcnc tells it to turn on. minutes agter i turn it on, it comes on. i checked the wiring. its the same issue ive been having. its not new. a reinstall of the os would be a work around but ultimatley, that doesn't fix anything. i am not getting the watchdog error light(red light on the mesa 7i96s. )

here is ip a;
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host noprefixroute
valid_lft forever preferred_lft forever
2: enp4s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
link/ether a4:1f:72:67:47:1f brd ff:ff:ff:ff:ff:ff
inet 10.10.10.1/8 brd 10.255.255.255 scope global enp4s0
valid_lft forever preferred_lft forever
inet6 fe80::a61f:72ff:fe67:471f/64 scope link proto kernel_ll
valid_lft forever preferred_lft forever
3: wlp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 9c:2a:70:32:72:49 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.106/24 brd 192.168.1.255 scope global dynamic noprefixroute wlp3s0
valid_lft 42363sec preferred_lft 42363sec
inet6 2603:7080:e936:2933::12d8/128 scope global dynamic noprefixroute
valid_lft 603964sec preferred_lft 603964sec
inet6 fd00:f405:95cf:605a::12d8/128 scope global dynamic noprefixroute
valid_lft 603964sec preferred_lft 603964sec
inet6 fd00:f405:95cf:605a:bc86:6ca6:e17a:1d65/64 scope global temporary dynamic
valid_lft 603965sec preferred_lft 85084sec
inet6 fd00:f405:95cf:605a:9e2a:70ff:fe32:7249/64 scope global dynamic mngtmpaddr noprefixroute
valid_lft 604537sec preferred_lft 604537sec
inet6 2603:7080:e936:2933:6b4b:639:ebcd:c3eb/64 scope global temporary dynamic
valid_lft 603965sec preferred_lft 85084sec
inet6 2603:7080:e936:2933:9e2a:70ff:fe32:7249/64 scope global dynamic mngtmpaddr noprefixroute
valid_lft 604537sec preferred_lft 604537sec
inet6 fe80::9e2a:70ff:fe32:7249/64 scope link noprefixroute
valid_lft forever preferred_lft forever
destination unreachable with ping.