Linux Mint / LinuxCNC 2.8.0 freezing - SOLVED?

The last freezes were almost 2 days ago during and following an update using Update Manager. Since then it's been running continuously without issue. It's too early to say but it's possible that fixed it. Dangerous talk, I know.

I've got Axis Sim running in a continuous loop plus Youtube streaming endless music video, Visual Studio code sitting in the background, Thunderbird email and VNC server running.

It has a Sunon fan in the case running all the time and I've not been able to get the core temperature above 59C no matter what I do.

About to go for a few hours walk in the hills. If it's still running on our return, I'll be tempted to consider itself fixed. Fingers crossed....

I hope this isn't premature but I'm becoming more confident that I've fixed the issue here. After an unplanned excursion into Linux crash logs and bug fixes, I appear to have a solution. If so, it's possible someone else may find this useful.

The error messages in the crash log seemed to feature 2 main themes:

• PAM adding faulty module: pam_kwallet5.so
• [drm:intel_pipe_update_end [i915}} *ERROR* Atomic update failure on pipe B .........etc.

The first seems to refer to a password wallet (KWallet), so was easy to disable.  forums.linuxmint.com/viewtopic.php?t=286950

The second seems to be caused by an "open bug" in the Intel HD graphics device. Something about a "PSR" feature that doesn't appear to be critically required yet can cause an "Atomic failure" (aka the processor won't continue until it's finished).  www.dedoimedo.com/computers/intel-microcode-atomic-update.html . This sounded like a strong candidate.

My first step to fixing the i915 error was to run Update Manager. I was 99% certain I'd done this previously but when I ran it late last week, I recall it reported that it planned to install the "Intel microcode", which is the firmware for the chipset to run Linux. Perhaps I hadn't run it previously after all. This almost fixed it - but not quite. I got a dark screen lockup shortly after the update which at least represented a change from the previous freeze (a change is as good as a rest?). The second and hopefully final step was to disable the PSR feature and after something like 4 days of continuous stress testing, I haven't seen any issues. That could change at any moment of course but I seem to have found a plausible root case and a credible corrective action.

If I've spoken too soon, I'll report back....

Hmmph. That didn't stick. I started to get freezes again and in the end they were as little as 10 mins apart. No issues shown through the memory tests. I even did a fresh install of Debian on the same SSD without LinuxCNC and it froze too. Enough.

I made a complete new install of Mint 20 on a different (Samsung) SSD at the weekend and just installed 2.8.2. So far it seems to be stable. There's always the possibility it's the motherboard, so I can never be 100% certain if a freeze is seconds away. Fingers remain tightly crossed again.....

Quick update on this. It's still randomly freezing, although it typically runs for several days between freezes. None of the system test tools find anything of concern and this is a new SSD with a fresh install of Mint 20 and LCNC 2.8.2-39. The mobo and memory were brand new at the start of this recent  build.

Perhaps my mistake is expecting it to run for days on end. I'm guessing most people will power down after each session, so perhaps the underlying instability isn't confined to my system but wouldn't make itself widely apparent. Does anyone else actually leave their PCs running for days at a time? Although I will likely be powering down myself once I'm done fiddling, it feels to me that the underlying OS for a machine tool should be reasonably robust - more so than I'm seeing.

I don't need to be actually running LinuxCNC for this problem to show itself. However, it sounds as if the RT kernel is always running, which may be a key difference between the std Mint installation and Mint after LinuxCNC / RT has been installed.

I have a life to live and metal to cut, so perhaps I should leave it there and focus on making stuff now!

I would expect that it should just run. I have had instances of linucCNC running for many months without issue. I have had issues with some kernels and they are very frustrating to debug

You might try another kernel

Could you point me at some info about how I might do that? I'd give it a go but at the moment I wouldn't have a clue where to start, so I'd need to read up on it.

Many thanks

Looks like PREEMPT:

muzzer_linux@LinuxCNC:~$ uname -a
Linux LinuxCNC 4.9.0-13-rt-amd64 #1 SMP PREEMPT RT Debian 4.9.228-1 (2020-07-05) x86_64 x86_64 x86_64 GNU/Linux

What should I try? I seem to recall my first install (also prone to freezing, top of this thread) used RTAI(?)

Many thanks, I'll look out for that.

The PC has just frozen again, some time in the last hour or so.....