AXIS memory leak == 2.8.0-pre1

jCandlish
Topic Author
Offline
Premium Member

12 Mar 2017 16:47 - 12 Mar 2017 18:25 #89472 by jCandlish

AXIS memory leak == 2.8.0-pre1 was created by jCandlish

I woke this morning to a hung machine due to thrashing caused by a memory leak.

I see now with 'top' that the axis process is consuming memory.

This is with 2.8.0-pre1-2910-g2aaae01.

I tried going back to 2.7 but that is dumping core with:

rtapi_app: caught signal 11 - dumping core

How to proceed?

=======edit========

The leak consumes 10% of memory per hour on a 4G machine.

Last edit: 12 Mar 2017 18:25 by jCandlish.

Please Log in or Create an account to join the conversation.

PCW
Offline
Moderator

12 Mar 2017 18:24 - 12 Mar 2017 18:25 #89477 by PCW

Replied by PCW on topic AXIS memory leak == 2.8.0-pre1

I would say that axis leaking major amounts of memory is unusual
( i have 2.8 test setups running axis for months on end 24/7 with no issues )

Do you have a very large gcode file loaded?
What is your system like otherwise (hardware/memory/OS etc) ?

Last edit: 12 Mar 2017 18:25 by PCW.

Please Log in or Create an account to join the conversation.

jCandlish
Topic Author
Offline
Premium Member

12 Mar 2017 18:32 - 12 Mar 2017 18:36 #89478 by jCandlish

Replied by jCandlish on topic AXIS memory leak == 2.8.0-pre1

Our posts crossed.

AXIS leaks at idle, with only the banner g-code loaded.

Debian Stretch.
Linux 125cnc 4.9.0-2-rt-amd64 #1 SMP PREEMPT RT Debian 4.9.13-1 (2017-02-27) x86_64 GNU/Linux

7i80 => 7i47 => (7i84 + 7i73)

latheoperator@125cnc:~$  cat /proc/cpuinfo 
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 78
model name	: Intel(R) Celeron(R) CPU 3855U @ 1.60GHz
stepping	: 3
microcode	: 0x9e
cpu MHz		: 628.125
cache size	: 2048 KB
physical id	: 0
siblings	: 2
core id		: 0
cpu cores	: 2
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 22
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave rdrand lahf_lm abm 3dnowprefetch epb intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust erms invpcid rdseed smap clflushopt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp
bugs		:
bogomips	: 3216.00
clflush size	: 64
cache_alignment	: 64
address sizes	: 39 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 78
model name	: Intel(R) Celeron(R) CPU 3855U @ 1.60GHz
stepping	: 3
microcode	: 0x9e
cpu MHz		: 720.117
cache size	: 2048 KB
physical id	: 0
siblings	: 2
core id		: 1
cpu cores	: 2
apicid		: 2
initial apicid	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 22
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave rdrand lahf_lm abm 3dnowprefetch epb intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust erms invpcid rdseed smap clflushopt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp
bugs		:
bogomips	: 3217.15
clflush size	: 64
cache_alignment	: 64
address sizes	: 39 bits physical, 48 bits virtual
power management:

latheoperator@125cnc:~/linuxcnc/configs/work/CNC_003$ free
              total        used        free      shared  buff/cache   available
Mem:        3928812      251580     3260956       52740      416276     3563052
Swap:       6369276           0     6369276
latheoperator@125cnc:~/linuxcnc/configs/work/CNC_003$  mesaflash --verbose --device 7i80 --addr 10.100.10.100 --readhmid
Configuration Name: HOSTMOT2

General configuration information:

  BoardName : MESA7I80
  FPGA Size: 16 KGates
  FPGA Pins: 256
  Number of IO Ports: 3
  Width of one I/O port: 24
  Clock Low frequency: 100.0000 MHz
  Clock High frequency: 200.0000 MHz
  IDROM Type: 3
  Instance Stride 0: 4
  Instance Stride 1: 64
  Register Stride 0: 256
  Register Stride 1: 256

Modules in configuration:

  Module: DPLL
  There are 1 of DPLL in configuration
  Version: 0
  Registers: 7
  BaseAddress: 7000
  ClockFrequency: 100.000 MHz
  Register Stride: 256 bytes
  Instance Stride: 4 bytes

  Module: WatchDog
  There are 1 of WatchDog in configuration
  Version: 0
  Registers: 3
  BaseAddress: 0C00
  ClockFrequency: 100.000 MHz
  Register Stride: 256 bytes
  Instance Stride: 4 bytes

  Module: MuxedQCount
  There are 12 of MuxedQCount in configuration
  Version: 3
  Registers: 5
  BaseAddress: 3500
  ClockFrequency: 100.000 MHz
  Register Stride: 256 bytes
  Instance Stride: 4 bytes

  Module: SSerial
  There are 1 of SSerial in configuration
  Version: 0
  Registers: 6
  BaseAddress: 5A00
  ClockFrequency: 100.000 MHz
  Register Stride: 256 bytes
  Instance Stride: 64 bytes

  Module: LED
  There are 1 of LED in configuration
  Version: 0
  Registers: 1
  BaseAddress: 0200
  ClockFrequency: 100.000 MHz
  Register Stride: 256 bytes
  Instance Stride: 4 bytes

  Module: IOPort
  There are 3 of IOPort in configuration
  Version: 0
  Registers: 5
  BaseAddress: 1000
  ClockFrequency: 100.000 MHz
  Register Stride: 256 bytes
  Instance Stride: 4 bytes

  Module: StepGen
  There are 5 of StepGen in configuration
  Version: 2
  Registers: 10
  BaseAddress: 2000
  ClockFrequency: 100.000 MHz
  Register Stride: 256 bytes
  Instance Stride: 4 bytes

Configuration pin-out:

IO Connections for P1
Pin#  I/O   Pri. func    Sec. func       Chan      Pin func        Pin Dir

 1      0   IOPort       StepGen          1        Step/Table1     (Out)
 3      1   IOPort       StepGen          1        Dir/Table2      (Out)
 5      2   IOPort       StepGen          2        Step/Table1     (Out)
 7      3   IOPort       StepGen          2        Dir/Table2      (Out)
 9      4   IOPort       SSerial          0        RXData1         (In)
11      5   IOPort       MuxedQCount      1        MuxQ-A          (In)
13      6   IOPort       SSerial          0        RXData2         (In)
15      7   IOPort       MuxedQCount      1        MuxQ-B          (In)
17      8   IOPort       None           
19      9   IOPort       MuxedQCount      1        MuxQ-IDX        (In)
21     10   IOPort       MuxedQCount      0        MuxQ-A          (In)
23     11   IOPort       MuxedQCount      2        MuxQ-A          (In)
25     12   IOPort       MuxedQCount      0        MuxQ-B          (In)
27     13   IOPort       MuxedQCount      2        MuxQ-B          (In)
29     14   IOPort       MuxedQCount      0        MuxQ-IDX        (In)
31     15   IOPort       MuxedQCount      2        MuxQ-IDX        (In)
33     16   IOPort       StepGen          3        Step/Table1     (Out)
35     17   IOPort       StepGen          3        Dir/Table2      (Out)
37     18   IOPort       StepGen          4        Step/Table1     (Out)
39     19   IOPort       StepGen          4        Dir/Table2      (Out)
41     20   IOPort       SSerial          0        TXData1         (Out)
43     21   IOPort       SSerial          0        TXData2         (Out)
45     22   IOPort       StepGen          0        Step/Table1     (Out)
47     23   IOPort       StepGen          0        Dir/Table2      (Out)

IO Connections for P2
Pin#  I/O   Pri. func    Sec. func       Chan      Pin func        Pin Dir

 1     24   IOPort       None           
 3     25   IOPort       None           
 5     26   IOPort       None           
 7     27   IOPort       None           
 9     28   IOPort       None           
11     29   IOPort       None           
13     30   IOPort       None           
15     31   IOPort       None           
17     32   IOPort       None           
19     33   IOPort       None           
21     34   IOPort       None           
23     35   IOPort       None           
25     36   IOPort       None           
27     37   IOPort       None           
29     38   IOPort       None           
31     39   IOPort       None           
33     40   IOPort       None           
35     41   IOPort       None           
37     42   IOPort       None           
39     43   IOPort       None           
41     44   IOPort       None           
43     45   IOPort       None           
45     46   IOPort       None           
47     47   IOPort       None           

IO Connections for P3
Pin#  I/O   Pri. func    Sec. func       Chan      Pin func        Pin Dir

 1     48   IOPort       None           
 3     49   IOPort       None           
 5     50   IOPort       None           
 7     51   IOPort       None           
 9     52   IOPort       None           
11     53   IOPort       None           
13     54   IOPort       None           
15     55   IOPort       None           
17     56   IOPort       None           
19     57   IOPort       None           
21     58   IOPort       None           
23     59   IOPort       None           
25     60   IOPort       None           
27     61   IOPort       None           
29     62   IOPort       None           
31     63   IOPort       None           
33     64   IOPort       None           
35     65   IOPort       None           
37     66   IOPort       None           
39     67   IOPort       None           
41     68   IOPort       None           
43     69   IOPort       None           
45     70   IOPort       None           
47     71   IOPort       None           

latheoperator@125cnc:~$  mesaflash --verbose --device 7i80 --addr 10.100.10.100 --sserial
SSLBP port 0:
  SSLBP Version: 1.43
  SSLBP Channels: 2
  SSLBP Baud Rate: 2500000
  interface type: 12
  interface width: 8
  SSLBP Channel Start: 112
  SSLBP Channel Stride: 50
  SSLBP Processor Type: d8
  SSLBP Clock: 100 MHz
  sserial device at channel 0: 7I84
    SwRevision = 15
    HwRevision = 1
    NVBaudRate = 2.5Mb
    NVUnitNumber = 0x180000F3
    NVWatchDogTimeout = 50ms
    EncMode0 = ffff [16 bits UNSIGNED IN | UNIT: None | RANGE: 0.00 - 0.00 | ADDR: 0998]
    EncMode1 = ffff [16 bits UNSIGNED IN | UNIT: None | RANGE: 0.00 - 0.00 | ADDR: 099A]
    NVEncMode0 = 0 [16 bits NV UNSIGNED IN | UNIT: None | RANGE: 0.00 - 0.00 | ADDR: 0066]
    NVEncMode1 = 0 [16 bits NV UNSIGNED IN | UNIT: None | RANGE: 0.00 - 0.00 | ADDR: 0068]
    SOFTWARE MODE Input_Output [index 00]
    SOFTWARE MODE IO_Analog_FieldVoltage [index 01]
    SOFTWARE MODE IO_Encoder_Analog [index 02]
  sserial device at channel 1: 7I73
    Contrast = ffff [16 bits UNSIGNED OUT | UNIT: None | RANGE: 0.00 - 0.00 | ADDR: 09C6]
    EncMode0 = ffff [16 bits UNSIGNED OUT | UNIT: None | RANGE: 0.00 - 0.00 | ADDR: 0998]
    EncMode1 = ffff [16 bits UNSIGNED OUT | UNIT: None | RANGE: 0.00 - 0.00 | ADDR: 099A]
    EncMode2 = ffff [16 bits UNSIGNED OUT | UNIT: None | RANGE: 0.00 - 0.00 | ADDR: 099C]
    EncMode3 = ffff [16 bits UNSIGNED OUT | UNIT: None | RANGE: 0.00 - 0.00 | ADDR: 099E]
    SwRevision = 14
    HwRevision = 1
    KeyMode = ffff [16 bits UNSIGNED OUT | UNIT: None | RANGE: 0.00 - 0.00 | ADDR: 09DE]
    NVContrast = 2710 [16 bits NV UNSIGNED OUT | UNIT: None | RANGE: 0.00 - 0.00 | ADDR: 0044]
    NVEncMode0 = 0 [16 bits NV UNSIGNED OUT | UNIT: None | RANGE: 0.00 - 0.00 | ADDR: 003A]
    NVEncMode1 = 0 [16 bits NV UNSIGNED OUT | UNIT: None | RANGE: 0.00 - 0.00 | ADDR: 003C]
    NVEncMode2 = 0 [16 bits NV UNSIGNED OUT | UNIT: None | RANGE: 0.00 - 0.00 | ADDR: 003E]
    NVEncMode3 = 0 [16 bits NV UNSIGNED OUT | UNIT: None | RANGE: 0.00 - 0.00 | ADDR: 0040]
    NVDispMode = 414 [16 bits NV UNSIGNED OUT | UNIT: None | RANGE: 0.00 - 0.00 | ADDR: 0006]
    NVKeyTimer = 1e [16 bits NV UNSIGNED OUT | UNIT: None | RANGE: 0.00 - 0.00 | ADDR: 0046]
    NVAnalogFilter = 8000 [16 bits NV UNSIGNED OUT | UNIT: None | RANGE: 0.00 - 0.00 | ADDR: 0042]
    NVBaudRate = 2.5Mb
    NVUnitNumber = 0x130000E2
    NVWatchDogTimeout = 50ms
    SOFTWARE MODE InputOutputEncoder [index 00]
    SOFTWARE MODE InputOutputEncoderAnalog [index 01]
    SOFTWARE MODE InputOutputEncoderAnalog [index 02]
    SOFTWARE MODE InputOutputEncoderDisplay [index 00]
    SOFTWARE MODE InputOutputEncoderAnalogDisplay [index 01]
    SOFTWARE MODE InputOutputEncoderAnalogWideDisplay [index 02]
    SOFTWARE MODE InputOutputEncoderKeycode4by8 [index 00]
    SOFTWARE MODE InputOutputEncoderAnalogKeycode4by8 [index 01]
    SOFTWARE MODE InputOutputEncoderAnalogKeycode4by8 [index 02]
    SOFTWARE MODE InputOutputEncoderDisplayKeycode4by8 [index 00]
    SOFTWARE MODE InputOutputEncoderAnalogDisplayKeyCode4by8 [index 01]
    SOFTWARE MODE InputOutputEncoderAnalogWideDisplayKeyCode4by8 [index 02]
    SOFTWARE MODE InputOutputEncoderKeycode8by8 [index 00]
    SOFTWARE MODE InputOutputEncoderAnalogKeyCode8by8 [index 01]
    SOFTWARE MODE InputOutputEncoderAnalogKeyCode8by8 [index 02]
    SOFTWARE MODE InputOutputEncoderDisplayKeycode8by8 [index 00]
    SOFTWARE MODE InputOutputEncoderAnalogDisplayKeyCode8by8 [index 01]
    SOFTWARE MODE InputOutputEncoderAnalogWideDisplayKeyCode8by8 [index 02]
latheoperator@125cnc:~$

File Attachment:

File Name: CNC_003.ini
File Size:3 KB

File Attachment:

File Name: CNC_003.hal
File Size:9 KB

Last edit: 12 Mar 2017 18:36 by jCandlish. Reason: added attachments

Please Log in or Create an account to join the conversation.

PCW
Offline
Moderator

12 Mar 2017 19:00 - 12 Mar 2017 19:03 #89480 by PCW

Replied by PCW on topic AXIS memory leak == 2.8.0-pre1

Weird, possible stretch or dependencies related?
( I notice Stretch is not officially supported so there could definitely be issues there )

I have 4 systems running axis 24/7, never noticed a memory leak
but I'm not running stretch or a 4.9 kernel

systems:

CoreDuo E8500 Ubuntu 16.04 4.1.xx kernel
CoreDuo E8500 Mint18.1 4.1.xx kernel
I5-3770 Mint18.1 4.4.xx kernel
N3355 Wheezy 4.1.xx kernel

also have run but not running currently
n3150 Mint 17 4.4.xx kernel
G3258_H97 Wheezy 4.1.xx kernel

I've run both 2.7 and 2.8 pre on all, never a axis memory leak issue

Last edit: 12 Mar 2017 19:03 by PCW.

Please Log in or Create an account to join the conversation.

tommylight
Offline
Moderator

12 Mar 2017 19:11 #89481 by tommylight

Replied by tommylight on topic AXIS memory leak == 2.8.0-pre1

Debian wheezy, linux mint 17.3 64bit, linux mint 18.1 32bit.
No such problems, weeks at a time.
Those are all on production machines running all day every day.

Please Log in or Create an account to join the conversation.

jCandlish
Topic Author
Offline
Premium Member

12 Mar 2017 19:18 #89484 by jCandlish

Replied by jCandlish on topic AXIS memory leak == 2.8.0-pre1

All my other systems are Strech so going back to Jessie is a bit clunky.

I'll create a Jessie partition on this test machine tomorrow and see if the leak persists.

I do note that there have been leaks before. What has been the effective approach for squashing them?

Please Log in or Create an account to join the conversation.

grump
Offline
Premium Member

12 Mar 2017 19:20 #89485 by grump

Replied by grump on topic AXIS memory leak == 2.8.0-pre1

I am running on the same Stretch Linux_image 4.9.0-2-rt-amd64 SMP PREEMPT RT Debian 4.9.13-1.
LXDE, Google Chrome constantly connected to my drive and using cloud cad.
Running a four axis machine via parport on an old laptop with no issues to report as yet been on since 02-03-17.
I can't fault the software or the laptop only my skills in using them with the cnc.

Good luck, hope you get it sorted soon.

Please Log in or Create an account to join the conversation.

jCandlish
Topic Author
Offline
Premium Member

12 Mar 2017 19:20 #89486 by jCandlish

Replied by jCandlish on topic AXIS memory leak == 2.8.0-pre1

tommylight wrote:
Debian wheezy, linux mint 17.3 64bit, linux mint 18.1 32bit.
No such problems, weeks at a time.
Those are all on production machines running all day every day.

When you follow the beaten path the terrain is familiar.

Please Log in or Create an account to join the conversation.

tommylight
Offline
Moderator

12 Mar 2017 19:32 #89487 by tommylight

Replied by tommylight on topic AXIS memory leak == 2.8.0-pre1

Yes it is, although i have plenty more but not used daily. I still have 2 machimes that use ubuntu 10.04 with Linuxcnc in use for over 4 years, never fail, used daily but not all day and they are turned off every evening.
From experience, loooooong time ago, that type of "leak" can be caused by a bad memory controller.
Linux has processor load and system load, check the latter. If it goes often to 200, that is a sign of something wrong with sot or hardware.
Regards,
Tom

Please Log in or Create an account to join the conversation.

jCandlish
Topic Author
Offline
Premium Member

13 Mar 2017 10:46 #89509 by jCandlish

Replied by jCandlish on topic AXIS memory leak == 2.8.0-pre1

Installing on Jessie appears to have resolved the memory leak. It must have been caused subtle dependency of package versions.

Running master-rt on Jessie requires a backported PREEMPT RT kernel, and backported glx and intel firmwares if your motherboard has an integrated intel graphics chipset.

backports snipped for /etc/apt/sources.list

# backports
deb http://ftp.ch.debian.org/debian/ jessie-backports main contrib non-free
deb-src http://ftp.ch.debian.org/debian/ jessie-backports main contrib non-free

needful backported packages

latheoperator@125cnc:~$  dpkg-query -l | grep ~bpo8 | awk '{print $2}'
firmware-misc-nonfree
firmware-realtek
intel-gpu-tools
iucode-tool
libdrm-amdgpu1:amd64
libdrm-dev:amd64
libdrm-intel1:amd64
libdrm-nouveau2:amd64
libdrm-radeon1:amd64
libdrm2:amd64
libgl1-mesa-dri:amd64
libglib2.0-0:amd64
libglib2.0-bin
libglib2.0-dev
libllvm3.8:amd64
libva1:amd64
linux-base
linux-compiler-gcc-4.9-x86
linux-headers-4.9.0-0.bpo.2-common-rt
linux-headers-4.9.0-0.bpo.2-rt-amd64
linux-headers-rt-amd64
linux-image-4.9.0-0.bpo.2-rt-amd64
linux-image-rt-amd64
linux-kbuild-4.9
xserver-xorg-video-intel
latheoperator@125cnc:~$ uname -a
Linux 125cnc 4.9.0-0.bpo.2-rt-amd64 #1 SMP PREEMPT RT Debian 4.9.13-1~bpo8+1 (2017-02-27) x86_64 GNU/Linux
latheoperator@125cnc:~$

latheoperator@125cnc:~/linuxcnc$ linuxcnc
LINUXCNC - 2.8.0-pre1-2912-g2636009
Machine configuration directory is '/home/latheoperator/linuxcnc/configs/work/CNC_003'
Machine configuration file is 'CNC_003.ini'
Starting LinuxCNC...
(time=1489400763.016967,pid=7358): Registering server on TCP port 5005.
(time=1489400763.017573,pid=7358): running server for TCP port 5005 (connection_socket = 3).
iocontrol: machine: 'CNC_003'  version 'unknown'
Found file(REL): ./CNC_003.hal
Note: Using POSIX realtime
hm2: loading Mesa HostMot2 driver version 0.15
hm2_eth: loading Mesa AnyIO HostMot2 ethernet driver version 0.2
...

The following user(s) said Thank You: Nico2017

Please Log in or Create an account to join the conversation.

Time to create page: 0.160 seconds

AXIS memory leak == 2.8.0-pre1

File Attachment:

File Attachment:

Attachments: