VistaCNC P4-SE pendant stopped working

Last year, I took a break from my Hurco CNC conversion because I got frustrated trying to figure out how to engrave a front panel. That is a different story (I think tools have progressed now), but prior to that break, I had integrated my VistaCNC P4-SE pendant. It was working great! In fact, it exceeded my expectations.... One of the last things I did was to permanently mount a waterproof connector through the enclosure to tidy everything up.

OK, fast forward to this last week. I have decided to go back to my project (yay!). I performed the system upgrades (not a full install, just the upgrade) and then tried out the machine. It still does what I want, as expected. However, I moved to the pendant and it was displaying "LinuxCNC" and was otherwise dead to the world (no LEDs, no response to keypresses, etc). I thought it might be the cable or wiring, but I had an epiphany and I decided to bring everything up in simulation mode (first on a laptop and then on my real, traditional motherboard, conversion machine). The pendant works perfectly. With the pendant, I can "jog" the simulator, it shows axis position on the LCD, etc. That can only happen if the pendant has good wiring (my belief, but I'm always open to disagreement on that, or any other, point).

I talked with Lee at VistaCNC. He has been great, but I get the feeling even he is starting to be challenged by this problem.... He did say that after startup, if there isn't a handshaking that should occur between LinuxCNC and the pendant, then the pendant will go into the mode I'm seeing (no LEDs, no button response, "LinuxCNC" on the LCD, etc). At one point in time, I thought that *maybe* it might be related to the Mesa 7i77 watchdog not being pet. Yeah, I understand that this is refuted by the fact that the machine remains in perfect state (can home, jog, run programs, etc) when controlled with a mouse and the computer screen; a watchdog fault should shut down the servo amplifiers (yes, their enable is tied to the Mesa card). But, to cover myself, I did verify that the watchdog LED on the 7i77 continues with its on/off sequence after the pendant has frozen.

My strong suspicion is that there is a configuration problem somehow (strange though because the current INI/HAL worked before, and the backups I had been making suffer from the same fault now). But, a HAL/INI failure make the most sense, so the question being posed in this thread, unless someone has a different suggestion, is "how does the handshaking with the pendant work, and what HAL/INI settings could interfer with this process?" And, yes, the pendant settings have been inserted into the files; in fact, I went through that process a second time, although I am not infallible; I could make the same mistake 10 times!

The second question is... I know there is a debug mode in LinuxCNC's config files, but I haven't seen much on how to access a console output. I'm hoping there might be something I can see, like a "lost communication with P4" message, etc. So, assuming a console exists, how do I access it.

Any suggestions on how to diagnose the problem, or ideas on what the problem could be, are always welcomed.

Thanks,
Alan

P.S. I'm on a different computer right now, but I'll try to post the INI and HAL files in a bit. I'm sure those will be requested.

And the remaining files (I've included everything in the config directory except the vista executable itself, although that was copied from the sim version, so I know it is good), excepting the VAR and TBL files.

Note that the date was appended to hurco_kbm1_3axis INI and HAL files in the original post. I am not sure that is the case. On my system the filenames are simply hurco_kbm1_3axis.ini and hurco_kbm1_3axis.hal

Basic information on the machine:

• Hurco KMB-1 metal
• LinuxCNC stock distro, version 2.7.14
• motherboard, not laptop
• Mesa 7i77 board
• Granite Devices VSD-E servo amplifiers
• Huanyang VFD

Note:
I discovered the launching of Linuxcnc by simply opening the opening of a command window and executing 'linuxcnc'. That is not the console I was hoping would provide more information. However, it did show that I had DOS formatted files. I didn't expect any change after fixing this, and that was the experience--no change.

Can you copy the output on the terminal window when you start LinuxCNC (copy and paste to text file).?
When debugging, always start linuxcnc from a terminal window as errors will be displayed. It might take a bit to spot them if you are not experienced.

Sure... Please see below. Sadly there isn't much. It is so small, I don't even need to add it as an attachment. The main loop keeps outputting a message with a message, usually with a time of about 0.1 seconds or so: Correction, while the pendant is flashing ESTOP, I get a maximum of 10 loop outputs. If I release estop on the pendant, then the outputs on the screen stop pretty quickly too.

I should point out that right now I'm not powering the axis, so they are faulted. I did that because it is quieter and because I don't think it should affect whether or not the pendant bricks on me. But, I wanted to disclose what I'm doing in case it is a no-no or flags a memory in you.

Rationalizing that the problem sure seems like a configuration one, I decided to do some exploratory surgery with the HAL and INI files. Because I don't have any power to the servo amps, this is perfectly safe. The first thing I did was 'not' the air pressure so that it is OK with no pressure (my regulator needs a rebuild and leaks like a sieve, so I keep it unconnected when I'm not actually running the machine).

Great... So, this isn't written in stone yet, but I've definitely changed (possibly fixed) the behavior of the system. In my HAL file, I have the three similar sections that follow the same pattern:

The intent is for me to determine if the servo amplifiers are in a faulted state. For some reason, however, if I comment the last line for each of the 3 servo amplifiers (X, Y, and Z), the pendant shows the position, the buttons and LEDs are 'live', etc. Of course, this removes the feedback I have for the servo amplifier, but the fact that it gets me closer to a source of the problem is promising. Why this all worked last year remains a mystery, but I'm not concentrating on that.

I think I've stumbled on something, although I still don't have a clue how a servo fault state could cause the pendant to reset.

Yes, that was my intent.... A servo fault would indicate a servo no longer capable of moving, of course, and that is a mode where I want to shut the machine off (estop or machine off seem to have the same net result to me... the relay providing power to the amps will drop out, etc)

OK, good... I think we're onto something. I want to do some more experimenting before I feel comfortable stating that the commenting of the three fault sense signals (X, Y, and Z) directly correlates with the VistaCNC pendant losing sync (BTW, shutting down LinuxCNC and then restarting it re-enables the pendant... I don't need to disconnect it or anything).

I was integrating the fault lines into my HAL as one of the later steps before I mothballed everything last year. It is possible that I didn't detect that the pendant had stopped working last year. My primary focus was on adding the three state lights to my control panel (fault, neg limit, pos limit). As such, I could have been working entirely with the mouse and computer display.

I think my next step will be to (figure out and then) add a pushbutton switch to my UI and reconfigure my HAL so that I can manually "trigger" a fault; this would replace my comparator and sense of the real amplifier output. If that condition results in the pendant going into its lost communication mode, then I think we've nailed the cause of my problem. Sound like a reasonable plan?

Is it possible that my LinuxCNC is corrupted? That seems awfully unlikely in my mind, but does it happen often? I guess I should look at how one upgrades the application. I doubt (hope) that I need to reinstall the whole image. Getting to the latest released version probably makes sense though. Never mind.... I guess I *am* on the latest version!

This has to be one of the most frustrating problems I've ever diagnosed. Just when you start to think you have something predictable, things start going south, and I have to reset. I'm trying to find a common denominator, and I think I may have one....

I've hatched out a lot of stuff. But even with that, I think that the following behavior should not occur.... Again, using some of the modified HAL files, I can virtually start the system. I say virtual because I really don't have any power going to the servo amplifiers, but the system doesn't know that (my disclosure, in case I'm violating something).

Anyhow, I start LinuxCNC. Then I press and release the pendant estop key over and over again. At this point, I've commented out software interpretation of this switch. It is literally just in the electrical estop chain. OK, so everything works fine.

Then I leave the pendant estop in the released (run) state. After that I toggle the EStop (F1) on the computer.

If I then press the ESTOP key on the pendant, the pendant will timeout and go to the "LinuxCNC" screen again. That seems wrong to me, and it is kind of the minimalist error state I can come up with. Ideas?

A note about my wiring....
My original intent was to have a relay chain that would require me to press a physical (momentary) start button. If there were no open switches in the estop chain, this would allow the estop relay to latch. One of the poles of the ESTOP relay is parallel to the start button, which would allow the relay to remain latched even after the START button was released. I did this for safety. However, it seems like it might be in conflict with the F1 button. Therefore, at the present time, the "start" button has a jumper across it. That means that the ESTOP (F1) on the screen and the physical ESTOP are the only two human controllable "switches" in the estop chain (air pressure and oil level would be other physical one). Because software identification of the pendant estop is presently disabled, the result is that if I press and release the pendant estop relay (while the LinuxCNC estop is in RUN mode), the physical relay will energize and release as a function of the pendant switch state. I don't believe this is pertinent to the problem I'm having but I want to be sure anyone reading this understands how my estop chain is setup. I intend to eventually have a relay drop out because I believe it is safer;

Good question... In my main INI file (hurco_kmb1-3axis.ini) there are a few new entries:

[HAL]
HALUI = halui
HALFILE = hurco_kmb1-3axis.hal
HALFILE = custom.hal
HALFILE = vc-p4s.hal
POSTGUI_HALFILE = postgui_call_list.hal
POSTGUI_HALFILE = vc-postgui.hal
HALUI = halui
SHUTDOWN = shutdown.hal
........
[HALUI]
MDI_COMMAND=...
MDI_COMMAND=...
MDI_COMMAND=...
MDI_COMMAND=...
MDI_COMMAND=...
MDI_COMMAND=G10 L20 P1 X0
MDI_COMMAND=G10 L20 P1 Y0
MDI_COMMAND=G10 L20 P1 Z0
MDI_COMMAND=G10 X0
MDI_COMMAND=G10 Y0
MDI_COMMAND=G10 Z0

From there, it is standard HAL and INI configuration. Everything was included in the first two posts except for the actual executable driver for the pendant. A better explanation is attached (from VistaCNC, describing the installation process).

I decided to experiment with the simulator I had previously modified to use the VistaCNC pendant. I went through the same process as I did with my real machine, pushing and releasing the estop button multiple times. Everything worked flawlessly. I guess that is good, although it doesn't help me narrow down the cause of the failure.