performance improvement suggestions

robertspark
Topic Author
Offline
Platinum Member

5 years 8 months ago #198864 by robertspark

Replied by robertspark on topic performance improvement suggestions

hmm..... learnt something else.... thanks gents

en.wikipedia.org/wiki/Private_network

my error has come back....

PCW can you elaborate

PCW wrote: Also are you running both cards at once? this requires special hal file setup
using read-request to allow overlapping response times rather than adding them

Please Log in to join the conversation.

PCW
Offline
Platinum Member

5 years 8 months ago #198874 by PCW

Replied by PCW on topic performance improvement suggestions

If you are running multiple cards at the same time you need to overlap the
read parts of the cycle to avoid simply adding all the response times.
This is done by using the read-request function in addition to the read function:

That is, rather than this:

addf hm2_7i96.0.read servo-thread # this asks for data and waits for return data before continuing
addf hm2_7i96.1.read servo-thread # delaying this request/wait
addf motion-command-handler servo-thread
addf motion-controller servo-thread
addf pid.0.do-pid-calcs servo-thread
addf pid.1.do-pid-calcs servo-thread
addf pid.2.do-pid-calcs servo-thread
addf pid.3.do-pid-calcs servo-thread
addf hm2_7i96.0.write servo-thread
addf hm2_7i96.1.write servo-thread

You do this:

addf hm2_7i96.0.read-request servo-thread # these don't wait
addf hm2_7i96.1.read-request servo-thread
addf hm2_7i96.0.read servo-thread
addf hm2_7i96.1.read servo-thread
addf motion-command-handler servo-thread
addf motion-controller servo-thread
addf pid.0.do-pid-calcs servo-thread
addf pid.1.do-pid-calcs servo-thread
addf pid.2.do-pid-calcs servo-thread
addf pid.3.do-pid-calcs servo-thread
addf hm2_7i96.0.write servo-thread
addf hm2_7i96.1.write servo-thread

But, I don't think this is your situation

Note for debugging real time issues you should watch (with halshow) or dump:

halcmd show param *.tmax

the tmax values

Please Log in to join the conversation.

robertspark
Topic Author
Offline
Platinum Member

5 years 8 months ago #198882 by robertspark

Replied by robertspark on topic performance improvement suggestions

Thanks,

Code:

Error report created by /usr/lib/tcltk/linuxcnc/show_errors.tcl:

Print file information:
RUN_IN_PLACE=no
LINUXCNC_DIR=
LINUXCNC_BIN_DIR=/usr/bin
LINUXCNC_TCL_DIR=/usr/lib/tcltk/linuxcnc
LINUXCNC_SCRIPT_DIR=
LINUXCNC_RTLIB_DIR=/usr/lib/linuxcnc/modules
LINUXCNC_CONFIG_DIR=
LINUXCNC_LANG_DIR=/usr/lib/tcltk/linuxcnc/msgs
INIVAR=inivar
HALCMD=halcmd
LINUXCNC_EMCSH=/usr/bin/wish8.6
LINUXCNC - 2.9.0-pre0-3542-g0e4c7c02b
Machine configuration directory is '/home/rob/linuxcnc/configs/RobsMill'
Machine configuration file is 'RobsMill.ini'
INIFILE=/home/rob/linuxcnc/configs/RobsMill/RobsMill.ini
VERSION=1.1
PARAMETER_FILE=linuxcnc.var
TASK=milltask
HALUI=halui
DISPLAY=axis
COORDINATES=XYZ
KINEMATICS=trivkins coordinates=XYZ
Starting LinuxCNC...
Starting LinuxCNC server program: linuxcncsvr
Loading Real Time OS, RTAPI, and HAL_LIB modules
Starting LinuxCNC IO program: io
Starting HAL User Interface program: halui
Found file(REL): ./RobsMill.hal
Found file(REL): ./custom.hal
Starting TASK program: milltask
Starting DISPLAY program: axis
hm2: loading Mesa HostMot2 driver version 0.15
hm2_eth: loading Mesa AnyIO HostMot2 ethernet driver version 0.2
hm2_eth: 10.10.10.10: INFO: Hardware address (MAC): 00:60:1b:10:41:50
hm2_eth: discovered 7I76E-16
hm2/hm2_7i76e.0: Low Level init 0.15
hm2/hm2_7i76e.0: Smart Serial Firmware Version 43
Board hm2_7i76e.0.7i76.0.0 Hardware Mode 0 = standard
Board hm2_7i76e.0.7i76.0.0 Software Mode 0 = io_spin
Board hm2_7i76e.0.7i76.0.0 Software Mode 1 = io_ana_spin
Board hm2_7i76e.0.7i76.0.0 Software Mode 2 = io_enc_ana_spin_fv
hm2/hm2_7i76e.0: 51 I/O Pins used:
hm2/hm2_7i76e.0:     IO Pin 000 (P1-01): StepGen #0, pin Direction (Output)
hm2/hm2_7i76e.0:     IO Pin 001 (P1-14): StepGen #0, pin Step (Output)
hm2/hm2_7i76e.0:     IO Pin 002 (P1-02): StepGen #1, pin Direction (Output)
hm2/hm2_7i76e.0:     IO Pin 003 (P1-15): StepGen #1, pin Step (Output)
hm2/hm2_7i76e.0:     IO Pin 004 (P1-03): StepGen #2, pin Direction (Output)
hm2/hm2_7i76e.0:     IO Pin 005 (P1-16): StepGen #2, pin Step (Output)
hm2/hm2_7i76e.0:     IO Pin 006 (P1-04): StepGen #3, pin Direction (Output)
hm2/hm2_7i76e.0:     IO Pin 007 (P1-17): StepGen #3, pin Step (Output)
hm2/hm2_7i76e.0:     IO Pin 008 (P1-05): StepGen #4, pin Direction (Output)
hm2/hm2_7i76e.0:     IO Pin 009 (P1-06): StepGen #4, pin Step (Output)
hm2/hm2_7i76e.0:     IO Pin 010 (P1-07): Smart Serial Interface #0, pin tx0 (Output)
hm2/hm2_7i76e.0:     IO Pin 011 (P1-08): Smart Serial Interface #0, pin rx0 (Input)
hm2/hm2_7i76e.0:     IO Pin 012 (P1-09): IOPort
hm2/hm2_7i76e.0:     IO Pin 013 (P1-10): IOPort
hm2/hm2_7i76e.0:     IO Pin 014 (P1-11): Encoder #0, pin Index (Input)
hm2/hm2_7i76e.0:     IO Pin 015 (P1-12): Encoder #0, pin B (Input)
hm2/hm2_7i76e.0:     IO Pin 016 (P1-13): Encoder #0, pin A (Input)
hm2/hm2_7i76e.0:     IO Pin 017 (P2-01): IOPort
hm2/hm2_7i76e.0:     IO Pin 018 (P2-14): IOPort
hm2/hm2_7i76e.0:     IO Pin 019 (P2-02): IOPort
hm2/hm2_7i76e.0:     IO Pin 020 (P2-15): IOPort
hm2/hm2_7i76e.0:     IO Pin 021 (P2-03): IOPort
hm2/hm2_7i76e.0:     IO Pin 022 (P2-16): IOPort
hm2/hm2_7i76e.0:     IO Pin 023 (P2-04): IOPort
hm2/hm2_7i76e.0:     IO Pin 024 (P2-17): IOPort
hm2/hm2_7i76e.0:     IO Pin 025 (P2-05): IOPort
hm2/hm2_7i76e.0:     IO Pin 026 (P2-06): IOPort
hm2/hm2_7i76e.0:     IO Pin 027 (P2-07): IOPort
hm2/hm2_7i76e.0:     IO Pin 028 (P2-08): IOPort
hm2/hm2_7i76e.0:     IO Pin 029 (P2-09): IOPort
hm2/hm2_7i76e.0:     IO Pin 030 (P2-10): IOPort
hm2/hm2_7i76e.0:     IO Pin 031 (P2-11): IOPort
hm2/hm2_7i76e.0:     IO Pin 032 (P2-12): IOPort
hm2/hm2_7i76e.0:     IO Pin 033 (P2-13): IOPort
hm2/hm2_7i76e.0:     IO Pin 034 (P3-01): IOPort
hm2/hm2_7i76e.0:     IO Pin 035 (P3-14): IOPort
hm2/hm2_7i76e.0:     IO Pin 036 (P3-02): IOPort
hm2/hm2_7i76e.0:     IO Pin 037 (P3-15): IOPort
hm2/hm2_7i76e.0:     IO Pin 038 (P3-03): IOPort
hm2/hm2_7i76e.0:     IO Pin 039 (P3-16): IOPort
hm2/hm2_7i76e.0:     IO Pin 040 (P3-04): IOPort
hm2/hm2_7i76e.0:     IO Pin 041 (P3-17): IOPort
hm2/hm2_7i76e.0:     IO Pin 042 (P3-05): IOPort
hm2/hm2_7i76e.0:     IO Pin 043 (P3-06): IOPort
hm2/hm2_7i76e.0:     IO Pin 044 (P3-07): IOPort
hm2/hm2_7i76e.0:     IO Pin 045 (P3-08): IOPort
hm2/hm2_7i76e.0:     IO Pin 046 (P3-09): IOPort
hm2/hm2_7i76e.0:     IO Pin 047 (P3-10): IOPort
hm2/hm2_7i76e.0:     IO Pin 048 (P3-11): IOPort
hm2/hm2_7i76e.0:     IO Pin 049 (P3-12): IOPort
hm2/hm2_7i76e.0:     IO Pin 050 (P3-13): IOPort
hm2/hm2_7i76e.0: registered
USRMOT: ERROR: command timeout
emcMotionInit: emcJointInit(0) failed
waiting for s.joints<0>, s.kinematics_type<0>
waiting for s.joints<0>, s.kinematics_type<0>
waiting for s.joints<0>, s.kinematics_type<0>
waiting for s.joints<0>, s.kinematics_type<0>
waiting for s.joints<0>, s.kinematics_type<0>
waiting for s.joints<0>, s.kinematics_type<0>
waiting for s.joints<0>, s.kinematics_type<0>
waiting for s.joints<0>, s.kinematics_type<0>
waiting for s.joints<0>, s.kinematics_type<0>
Shutting down and cleaning up LinuxCNC...
Running HAL shutdown script
Timeout, trying kill -9
Removing HAL_LIB, RTAPI, and Real Time OS modules
Removing NML shared memory segments

Debug file information:
Note: Using POSIX realtime
Waiting for component 'hm2_eth' to become ready...........
rtapi_app: caught signal 11 - dumping core
note: MAXV     max: 25.000 units/sec 1500.000 units/min
note: LJOG     max: 25.000 units/sec 1500.000 units/min
note: LJOG default: 12.500 units/sec 750.000 units/min
Waiting for component 'inihal' to become ready..................................A configuration error is preventing LinuxCNC from starting.
More information may be available when running from a terminal.
..1342
1386
.....................
<commandline>:0: milltask exited without becoming ready
Stopping realtime threads
Unloading hal components
Waited 3 seconds for master.  giving up.
Note: Using POSIX realtime
pid: not loaded
<commandline>:0: exit value: 255
<commandline>:0: rmmod failed, returned -1
Note: Using POSIX realtime
hm2_eth: not loaded
<commandline>:0: exit value: 255
<commandline>:0: rmmod failed, returned -1
Note: Using POSIX realtime
hostmot2: not loaded
<commandline>:0: exit value: 255
<commandline>:0: rmmod failed, returned -1
Note: Using POSIX realtime
motmod: not loaded
<commandline>:0: exit value: 255
<commandline>:0: rmmod failed, returned -1
Note: Using POSIX realtime
trivkins: not loaded
<commandline>:0: exit value: 255
<commandline>:0: rmmod failed, returned -1
<commandline>:0: unloadrt failed
Note: Using POSIX realtime



-----------------------------------------------------------------------
Info report created by linuxcnc_info:
The file:    /tmp/linuxcnc_info.txt
can be posted to a forum or a web site like:
     http://pastebin.com
in order to provide information about the linuxcnc
system and configuration.

                Date: Sun 14 Feb 23:21:18 GMT 2021
            UTC Date: Sun 14 Feb 23:21:18 UTC 2021
        this program: /usr/bin/linuxcnc_info
              uptime: 23:21:18 up 3 min, 1 user, load average: 0.74, 0.48, 0.21
     lsb_release -sa: Debian Debian GNU/Linux 10 (buster) 10 buster
      which linuxcnc: /usr/bin/linuxcnc
                 pwd: /home/rob/linuxcnc/configs/RobsMill
                USER: rob
             LOGNAME: rob
                HOME: /home/rob
              EDITOR: 
              VISUAL: 
            LANGUAGE: en_GB:en
                TERM: dumb
           COLORTERM: 
             DISPLAY: :0.0
             DESKTOP: lightdm-xsession
        display size: 1280x1024 pixels (338x270 millimeters)
                PATH: /usr/bin:/home/rob/linuxcnc/configs/RobsMill/bin:/usr/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games

uname items:
         nodename -n: RobsCNC
      kernel-name -s: Linux
      kernel-vers -v: #1 SMP PREEMPT RT Debian 4.19.171-2 (2021-01-30)
          machine -m: x86_64
        processor -p: unknown
         platform -i: unknown
      oper system -o: GNU/Linux

/proc items:
             cmdline: BOOT_IMAGE=/boot/vmlinuz-4.19.0-14-rt-amd64 root=UUID=9ccf0ebd-3d89-4339-aa28-1f5bdb2a3e28 ro initrd=/install/gtk/initrd.gz quiet
          model name: Intel(R) Core(TM) i7-5500U CPU @ 2.40GHz
               cores: 2
             cpu MHz: 2394.422
             parport: 
              serial: 

Versions:
                 gcc: gcc (Debian 8.3.0-6) 8.3.0
              python: Python 2.7.16
                 git: not_in_PATH
          git commit: NA
                 tcl: 8.6
                  tk: 8.6
               glade: not_in_PATH
          glade-gtk2: not_in_PATH

linuxcnc_var all:

     LINUXCNCVERSION: 2.9.0-pre0-3542-g0e4c7c02b
LINUXCNC_AUX_GLADEVCP: /usr/share/linuxcnc/aux_gladevcp
LINUXCNC_AUX_EXAMPLES: /usr/share/linuxcnc/aux_examples
            REALTIME: /etc/init.d/realtime
                 RTS: uspace
          HALLIB_DIR: /usr/share/linuxcnc/hallib
              PYTHON: /usr/bin/python2.7

dpkg -l '*linuxcnc*':
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                Version                      Architecture Description
+++-===================-============================-============-=====================================================================
un  linuxcnc            <none>                       <none>       (no description available)
un  linuxcnc-dev        <none>                       <none>       (no description available)
un  linuxcnc-doc        <none>                       <none>       (no description available)
ii  linuxcnc-doc-en     1:2.9.0~pre0.3542.g0e4c7c02b all          motion controller for CNC machines and robots (English documentation)
un  linuxcnc-sim        <none>                       <none>       (no description available)
un  linuxcnc-sim-dev    <none>                       <none>       (no description available)
ii  linuxcnc-uspace     1:2.9.0~pre0.3542.g0e4c7c02b amd64        motion controller for CNC machines and robots
ii  linuxcnc-uspace-dev 1:2.9.0~pre0.3542.g0e4c7c02b amd64        PC based motion controller for real-time Linux

Code:

rob@RobsCNC:~$ halcmd show param *.tmax
Parameters:
Owner   Type  Dir         Value  Name
s32   RW              0  hm2_7i76e.0.read-request.tmax
s32   RW        2817480  hm2_7i76e.0.read.tmax
s32   RW         268648  hm2_7i76e.0.write.tmax
s32   RW          42396  motion-command-handler.tmax
s32   RW         200084  motion-controller.tmax
s32   RW          13928  pid.s.do-pid-calcs.tmax
s32   RW          31212  pid.x.do-pid-calcs.tmax
s32   RW          33332  pid.y.do-pid-calcs.tmax
s32   RW           9896  pid.z.do-pid-calcs.tmax
s32   RW        3266052  servo-thread.tmax

Please Log in to join the conversation.

andypugh
Offline
Administrator

5 years 8 months ago #198883 by andypugh

Replied by andypugh on topic performance improvement suggestions

tommylight wrote: missed the 11, sorry for that, but that makes things even worse as addresses starting with 11.n.n.n are routable and public

... and historically allocated to the US DoD.

xkcd.com/195/

Though that map is now out of date, and there are few /8s left.
en.wikipedia.org/wiki/List_of_assigned_/8_IPv4_address_blocks
Apple, Ford and AT&T amongst a few others.

Please Log in to join the conversation.

tommylight
Offline
Administrator

5 years 8 months ago #198889 by tommylight

Replied by tommylight on topic performance improvement suggestions

andypugh wrote:
tommylight wrote: missed the 11, sorry for that, but that makes things even worse as addresses starting with 11.n.n.n are routable and public

... and historically allocated to the US DoD.

xkcd.com/195/

Though that map is now out of date, and there are few /8s left.
en.wikipedia.org/wiki/List_of_assigned_/8_IPv4_address_blocks
Apple, Ford and AT&T amongst a few others.

Dang, i did not know that, thank you.

Please Log in to join the conversation.

tommylight
Offline
Administrator

5 years 8 months ago #198890 by tommylight

Replied by tommylight on topic performance improvement suggestions

Ooooopssss !

Code:

NetRange:       11.0.0.0 - 11.255.255.255
CIDR:           11.0.0.0/8
NetName:        DODIIS
NetHandle:      NET-11-0-0-0-1
Parent:          ()
NetType:        Direct Allocation
OriginAS:       
Organization:   DoD Network Information Center (DNIC)
RegDate:        1984-01-19
Updated:        2007-08-22
Ref:            https://rdap.arin.net/registry/ip/11.0.0.0

Please Log in to join the conversation.

PCW
Offline
Platinum Member

5 years 8 months ago #198893 by PCW

Replied by PCW on topic performance improvement suggestions

rob@RobsCNC:~$ halcmd show param *.tmax

Code:

Parameters:
Owner   Type  Dir         Value  Name
s32   RW              0  hm2_7i76e.0.read-request.tmax
s32   RW        2817480  hm2_7i76e.0.read.tmax  <<<<<<<<<<<<<<<<<<<  ~1.17 ms
s32   RW         268648  hm2_7i76e.0.write.tmax
s32   RW          42396  motion-command-handler.tmax
s32   RW         200084  motion-controller.tmax
s32   RW          13928  pid.s.do-pid-calcs.tmax
s32   RW          31212  pid.x.do-pid-calcs.tmax
s32   RW          33332  pid.y.do-pid-calcs.tmax
s32   RW           9896  pid.z.do-pid-calcs.tmax
s32   RW        3266052  servo-thread.tmax <<<<<<<<<<<<<<<<<  ~1.36 ms

So a 1.5 or 2 ms servo thread may be needed with that hardware
unless you can find the cause and eliminate it.

Please Log in to join the conversation.

robertspark
Topic Author
Offline
Platinum Member

5 years 8 months ago #198926 by robertspark

Replied by robertspark on topic performance improvement suggestions

thanks PCW, I've got it running on a 2ms servo thread, hoping to give it a big file test and some air cutting on the plasma for a couple of hours to see what gives.

I may also start again with the bios settings from default to see where the starting point is

thanks for your help on that one

Please Log in to join the conversation.

robertspark
Topic Author
Offline
Platinum Member

5 years 8 months ago #198933 by robertspark

Replied by robertspark on topic performance improvement suggestions

photos of the bios settings (I am going to have anther play around with these starting from resetting it to default and working my way through turning off the recommended c-states etc as listed in other posts on the forum.

Attachment not found

Please Log in to join the conversation.

robertspark
Topic Author
Offline
Platinum Member

5 years 8 months ago #199987 by robertspark

Replied by robertspark on topic performance improvement suggestions

Well, I have continued to dabble and headscratch.....

I thought that I should be able to get better performance out of an i7 5500u CPU than other offerings (I kind of have avoided PC's for years now as I hate the dust they collect in the garage and space that they take up, hence I've gone fanless for a while..... )

The other fanless PC that I have is a Celeron N3150

At one point I was worried that I'd killed the i7 5500u having fiddled that much with settings that it got to the point that I was booting it but unable to get into the bios, however after a while of a blank screen it went straight into Linux login screen.

I then gave an HDMI screen a go as I was using a DVI input screen with an HDMI > DVI cable, and was able to get back into the bios and change things. I am now using my HDMI screen to dabble as it works (detects) the screen from boot and have also added a bios boot delay while I dabble.

Yes I even removed the bios battery (several times, and left it unpowered for increasing amounts of time, in the hope that the bios would reset and I would be able to see the bios screen from time of boot..... but to no avail ..... LESSON is probably if you only have HDMI outputs.... use an HDMI screen / TV with an HDMI as you may be able to get back in). No idea, but it now works, learnt something new.....

Anyway back sort of ontopic....

I then decided to have a play around with isolcpus so see if I could improve the latency numbers..... and to my surprise I managed to get better stability (sharp triangle on the latency histogram) + I also found the latency test script too to play with....

In the end I believe I have around 20,000uS latency using isolcpus 1,3 (I have a dual core i7 5500u) and have reenabled hyperthreading ..... which seems to have provided me with better results given it goes from 2 threads without to 4 threads with hyperthreading enabled, and used isolcpus 1,3 to isolate one of the cores (and its associated second thread). All the power saving is turned off (c-states, speedstepping etc etc etc).

I thought (from previous reads across the net) that isolcpus was depricated and no longer the way to improve latency on multicore CPU's but I guess it seems to work. I have not tried the latency test for 2 hours yet, {I fiddled with the script a bit...., removed the base thread, and changed it to a 10min run} and the latency using that script went from 70,000+ variable (plus higher) to a rock solid 36,000.

I am using Mesa hardware so yes I know the latency should not matter that much..... I just kind of "thought" that I'd get better performance out of an 5th Gen i7 than appeared to be the case.

I am hoping to try the 3150N at some point (just can't seem to get it to boot.... suspect a memory issue.... I can't find the chip that I had in it and I bought a replacement I thought but no dice (yet!)....)

It's all just a learning curve to me and I am trying to get use to understanding Linux + RT a bit better + latency.

I am also dabbling with my simple 3-axis mill setup via 7i76E hardware (its simple because it doesn't need anything but axis motion..... but it has prox limit switches and also a VFD that I'm adding to the profile and am working my way through the HAL bits as a learning curve {it may be simple for some but this is the stuff that has put me off LinuxCNC for years every time I dabbled with it so this time I am making a concerted effort to understand the stuff I am changing and how it works too).

I have also been trying to understand application priority and RT_PREEMPT (SCHED_FIFO, and SCHED_RR), and am getting my head into it a little better + using HTOP.

Always learning.... and enjoying it.

Please Log in to join the conversation.

Time to create page: 0.802 seconds