AXIS memory leak == 2.8.0-pre1

More
9 years 7 months ago #89472 by jCandlish
I woke this morning to a hung machine due to thrashing caused by a memory leak.

I see now with 'top' that the axis process is consuming memory.

This is with 2.8.0-pre1-2910-g2aaae01.

I tried going back to 2.7 but that is dumping core with:
Code:
rtapi_app: caught signal 11 - dumping core

How to proceed?

=======edit========

The leak consumes 10% of memory per hour on a 4G machine.

Please Log in to join the conversation.

More
9 years 7 months ago #89477 by PCW
Replied by PCW on topic AXIS memory leak == 2.8.0-pre1
I would say that axis leaking major amounts of memory is unusual
( i have 2.8 test setups running axis for months on end 24/7 with no issues )

Do you have a very large gcode file loaded?
What is your system like otherwise (hardware/memory/OS etc) ?

Please Log in to join the conversation.

More
9 years 7 months ago #89478 by jCandlish
Our posts crossed.

AXIS leaks at idle, with only the banner g-code loaded.

Debian Stretch.
Linux 125cnc 4.9.0-2-rt-amd64 #1 SMP PREEMPT RT Debian 4.9.13-1 (2017-02-27) x86_64 GNU/Linux

7i80 => 7i47 => (7i84 + 7i73)
Code:
latheoperator@125cnc:~$ cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 78 model name : Intel(R) Celeron(R) CPU 3855U @ 1.60GHz stepping : 3 microcode : 0x9e cpu MHz : 628.125 cache size : 2048 KB physical id : 0 siblings : 2 core id : 0 cpu cores : 2 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 22 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave rdrand lahf_lm abm 3dnowprefetch epb intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust erms invpcid rdseed smap clflushopt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp bugs : bogomips : 3216.00 clflush size : 64 cache_alignment : 64 address sizes : 39 bits physical, 48 bits virtual power management: processor : 1 vendor_id : GenuineIntel cpu family : 6 model : 78 model name : Intel(R) Celeron(R) CPU 3855U @ 1.60GHz stepping : 3 microcode : 0x9e cpu MHz : 720.117 cache size : 2048 KB physical id : 0 siblings : 2 core id : 1 cpu cores : 2 apicid : 2 initial apicid : 2 fpu : yes fpu_exception : yes cpuid level : 22 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave rdrand lahf_lm abm 3dnowprefetch epb intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust erms invpcid rdseed smap clflushopt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp bugs : bogomips : 3217.15 clflush size : 64 cache_alignment : 64 address sizes : 39 bits physical, 48 bits virtual power management: latheoperator@125cnc:~/linuxcnc/configs/work/CNC_003$ free total used free shared buff/cache available Mem: 3928812 251580 3260956 52740 416276 3563052 Swap: 6369276 0 6369276 latheoperator@125cnc:~/linuxcnc/configs/work/CNC_003$ mesaflash --verbose --device 7i80 --addr 10.100.10.100 --readhmid Configuration Name: HOSTMOT2 General configuration information: BoardName : MESA7I80 FPGA Size: 16 KGates FPGA Pins: 256 Number of IO Ports: 3 Width of one I/O port: 24 Clock Low frequency: 100.0000 MHz Clock High frequency: 200.0000 MHz IDROM Type: 3 Instance Stride 0: 4 Instance Stride 1: 64 Register Stride 0: 256 Register Stride 1: 256 Modules in configuration: Module: DPLL There are 1 of DPLL in configuration Version: 0 Registers: 7 BaseAddress: 7000 ClockFrequency: 100.000 MHz Register Stride: 256 bytes Instance Stride: 4 bytes Module: WatchDog There are 1 of WatchDog in configuration Version: 0 Registers: 3 BaseAddress: 0C00 ClockFrequency: 100.000 MHz Register Stride: 256 bytes Instance Stride: 4 bytes Module: MuxedQCount There are 12 of MuxedQCount in configuration Version: 3 Registers: 5 BaseAddress: 3500 ClockFrequency: 100.000 MHz Register Stride: 256 bytes Instance Stride: 4 bytes Module: SSerial There are 1 of SSerial in configuration Version: 0 Registers: 6 BaseAddress: 5A00 ClockFrequency: 100.000 MHz Register Stride: 256 bytes Instance Stride: 64 bytes Module: LED There are 1 of LED in configuration Version: 0 Registers: 1 BaseAddress: 0200 ClockFrequency: 100.000 MHz Register Stride: 256 bytes Instance Stride: 4 bytes Module: IOPort There are 3 of IOPort in configuration Version: 0 Registers: 5 BaseAddress: 1000 ClockFrequency: 100.000 MHz Register Stride: 256 bytes Instance Stride: 4 bytes Module: StepGen There are 5 of StepGen in configuration Version: 2 Registers: 10 BaseAddress: 2000 ClockFrequency: 100.000 MHz Register Stride: 256 bytes Instance Stride: 4 bytes Configuration pin-out: IO Connections for P1 Pin# I/O Pri. func Sec. func Chan Pin func Pin Dir 1 0 IOPort StepGen 1 Step/Table1 (Out) 3 1 IOPort StepGen 1 Dir/Table2 (Out) 5 2 IOPort StepGen 2 Step/Table1 (Out) 7 3 IOPort StepGen 2 Dir/Table2 (Out) 9 4 IOPort SSerial 0 RXData1 (In) 11 5 IOPort MuxedQCount 1 MuxQ-A (In) 13 6 IOPort SSerial 0 RXData2 (In) 15 7 IOPort MuxedQCount 1 MuxQ-B (In) 17 8 IOPort None 19 9 IOPort MuxedQCount 1 MuxQ-IDX (In) 21 10 IOPort MuxedQCount 0 MuxQ-A (In) 23 11 IOPort MuxedQCount 2 MuxQ-A (In) 25 12 IOPort MuxedQCount 0 MuxQ-B (In) 27 13 IOPort MuxedQCount 2 MuxQ-B (In) 29 14 IOPort MuxedQCount 0 MuxQ-IDX (In) 31 15 IOPort MuxedQCount 2 MuxQ-IDX (In) 33 16 IOPort StepGen 3 Step/Table1 (Out) 35 17 IOPort StepGen 3 Dir/Table2 (Out) 37 18 IOPort StepGen 4 Step/Table1 (Out) 39 19 IOPort StepGen 4 Dir/Table2 (Out) 41 20 IOPort SSerial 0 TXData1 (Out) 43 21 IOPort SSerial 0 TXData2 (Out) 45 22 IOPort StepGen 0 Step/Table1 (Out) 47 23 IOPort StepGen 0 Dir/Table2 (Out) IO Connections for P2 Pin# I/O Pri. func Sec. func Chan Pin func Pin Dir 1 24 IOPort None 3 25 IOPort None 5 26 IOPort None 7 27 IOPort None 9 28 IOPort None 11 29 IOPort None 13 30 IOPort None 15 31 IOPort None 17 32 IOPort None 19 33 IOPort None 21 34 IOPort None 23 35 IOPort None 25 36 IOPort None 27 37 IOPort None 29 38 IOPort None 31 39 IOPort None 33 40 IOPort None 35 41 IOPort None 37 42 IOPort None 39 43 IOPort None 41 44 IOPort None 43 45 IOPort None 45 46 IOPort None 47 47 IOPort None IO Connections for P3 Pin# I/O Pri. func Sec. func Chan Pin func Pin Dir 1 48 IOPort None 3 49 IOPort None 5 50 IOPort None 7 51 IOPort None 9 52 IOPort None 11 53 IOPort None 13 54 IOPort None 15 55 IOPort None 17 56 IOPort None 19 57 IOPort None 21 58 IOPort None 23 59 IOPort None 25 60 IOPort None 27 61 IOPort None 29 62 IOPort None 31 63 IOPort None 33 64 IOPort None 35 65 IOPort None 37 66 IOPort None 39 67 IOPort None 41 68 IOPort None 43 69 IOPort None 45 70 IOPort None 47 71 IOPort None latheoperator@125cnc:~$ mesaflash --verbose --device 7i80 --addr 10.100.10.100 --sserial SSLBP port 0: SSLBP Version: 1.43 SSLBP Channels: 2 SSLBP Baud Rate: 2500000 interface type: 12 interface width: 8 SSLBP Channel Start: 112 SSLBP Channel Stride: 50 SSLBP Processor Type: d8 SSLBP Clock: 100 MHz sserial device at channel 0: 7I84 SwRevision = 15 HwRevision = 1 NVBaudRate = 2.5Mb NVUnitNumber = 0x180000F3 NVWatchDogTimeout = 50ms EncMode0 = ffff [16 bits UNSIGNED IN | UNIT: None | RANGE: 0.00 - 0.00 | ADDR: 0998] EncMode1 = ffff [16 bits UNSIGNED IN | UNIT: None | RANGE: 0.00 - 0.00 | ADDR: 099A] NVEncMode0 = 0 [16 bits NV UNSIGNED IN | UNIT: None | RANGE: 0.00 - 0.00 | ADDR: 0066] NVEncMode1 = 0 [16 bits NV UNSIGNED IN | UNIT: None | RANGE: 0.00 - 0.00 | ADDR: 0068] SOFTWARE MODE Input_Output [index 00] SOFTWARE MODE IO_Analog_FieldVoltage [index 01] SOFTWARE MODE IO_Encoder_Analog [index 02] sserial device at channel 1: 7I73 Contrast = ffff [16 bits UNSIGNED OUT | UNIT: None | RANGE: 0.00 - 0.00 | ADDR: 09C6] EncMode0 = ffff [16 bits UNSIGNED OUT | UNIT: None | RANGE: 0.00 - 0.00 | ADDR: 0998] EncMode1 = ffff [16 bits UNSIGNED OUT | UNIT: None | RANGE: 0.00 - 0.00 | ADDR: 099A] EncMode2 = ffff [16 bits UNSIGNED OUT | UNIT: None | RANGE: 0.00 - 0.00 | ADDR: 099C] EncMode3 = ffff [16 bits UNSIGNED OUT | UNIT: None | RANGE: 0.00 - 0.00 | ADDR: 099E] SwRevision = 14 HwRevision = 1 KeyMode = ffff [16 bits UNSIGNED OUT | UNIT: None | RANGE: 0.00 - 0.00 | ADDR: 09DE] NVContrast = 2710 [16 bits NV UNSIGNED OUT | UNIT: None | RANGE: 0.00 - 0.00 | ADDR: 0044] NVEncMode0 = 0 [16 bits NV UNSIGNED OUT | UNIT: None | RANGE: 0.00 - 0.00 | ADDR: 003A] NVEncMode1 = 0 [16 bits NV UNSIGNED OUT | UNIT: None | RANGE: 0.00 - 0.00 | ADDR: 003C] NVEncMode2 = 0 [16 bits NV UNSIGNED OUT | UNIT: None | RANGE: 0.00 - 0.00 | ADDR: 003E] NVEncMode3 = 0 [16 bits NV UNSIGNED OUT | UNIT: None | RANGE: 0.00 - 0.00 | ADDR: 0040] NVDispMode = 414 [16 bits NV UNSIGNED OUT | UNIT: None | RANGE: 0.00 - 0.00 | ADDR: 0006] NVKeyTimer = 1e [16 bits NV UNSIGNED OUT | UNIT: None | RANGE: 0.00 - 0.00 | ADDR: 0046] NVAnalogFilter = 8000 [16 bits NV UNSIGNED OUT | UNIT: None | RANGE: 0.00 - 0.00 | ADDR: 0042] NVBaudRate = 2.5Mb NVUnitNumber = 0x130000E2 NVWatchDogTimeout = 50ms SOFTWARE MODE InputOutputEncoder [index 00] SOFTWARE MODE InputOutputEncoderAnalog [index 01] SOFTWARE MODE InputOutputEncoderAnalog [index 02] SOFTWARE MODE InputOutputEncoderDisplay [index 00] SOFTWARE MODE InputOutputEncoderAnalogDisplay [index 01] SOFTWARE MODE InputOutputEncoderAnalogWideDisplay [index 02] SOFTWARE MODE InputOutputEncoderKeycode4by8 [index 00] SOFTWARE MODE InputOutputEncoderAnalogKeycode4by8 [index 01] SOFTWARE MODE InputOutputEncoderAnalogKeycode4by8 [index 02] SOFTWARE MODE InputOutputEncoderDisplayKeycode4by8 [index 00] SOFTWARE MODE InputOutputEncoderAnalogDisplayKeyCode4by8 [index 01] SOFTWARE MODE InputOutputEncoderAnalogWideDisplayKeyCode4by8 [index 02] SOFTWARE MODE InputOutputEncoderKeycode8by8 [index 00] SOFTWARE MODE InputOutputEncoderAnalogKeyCode8by8 [index 01] SOFTWARE MODE InputOutputEncoderAnalogKeyCode8by8 [index 02] SOFTWARE MODE InputOutputEncoderDisplayKeycode8by8 [index 00] SOFTWARE MODE InputOutputEncoderAnalogDisplayKeyCode8by8 [index 01] SOFTWARE MODE InputOutputEncoderAnalogWideDisplayKeyCode8by8 [index 02] latheoperator@125cnc:~$

Attachment not found

Attachment not found

Please Log in to join the conversation.

More
9 years 7 months ago #89480 by PCW
Replied by PCW on topic AXIS memory leak == 2.8.0-pre1
Weird, possible stretch or dependencies related?
( I notice Stretch is not officially supported so there could definitely be issues there )

I have 4 systems running axis 24/7, never noticed a memory leak
but I'm not running stretch or a 4.9 kernel

systems:

CoreDuo E8500 Ubuntu 16.04 4.1.xx kernel
CoreDuo E8500 Mint18.1 4.1.xx kernel
I5-3770 Mint18.1 4.4.xx kernel
N3355 Wheezy 4.1.xx kernel

also have run but not running currently
n3150 Mint 17 4.4.xx kernel
G3258_H97 Wheezy 4.1.xx kernel

I've run both 2.7 and 2.8 pre on all, never a axis memory leak issue

Please Log in to join the conversation.

More
9 years 7 months ago #89481 by tommylight
Debian wheezy, linux mint 17.3 64bit, linux mint 18.1 32bit.
No such problems, weeks at a time.
Those are all on production machines running all day every day.

Please Log in to join the conversation.

More
9 years 7 months ago #89484 by jCandlish
All my other systems are Strech so going back to Jessie is a bit clunky.

I'll create a Jessie partition on this test machine tomorrow and see if the leak persists.

I do note that there have been leaks before. What has been the effective approach for squashing them?

Please Log in to join the conversation.

More
9 years 7 months ago #89485 by grump
I am running on the same Stretch Linux_image 4.9.0-2-rt-amd64 SMP PREEMPT RT Debian 4.9.13-1.
LXDE, Google Chrome constantly connected to my drive and using cloud cad.
Running a four axis machine via parport on an old laptop with no issues to report as yet been on since 02-03-17.
I can't fault the software or the laptop only my skills in using them with the cnc.

Good luck, hope you get it sorted soon.

Please Log in to join the conversation.

More
9 years 7 months ago #89486 by jCandlish

tommylight wrote: Debian wheezy, linux mint 17.3 64bit, linux mint 18.1 32bit.
No such problems, weeks at a time.
Those are all on production machines running all day every day.


When you follow the beaten path the terrain is familiar.

Please Log in to join the conversation.

More
9 years 7 months ago #89487 by tommylight
Yes it is, although i have plenty more but not used daily. I still have 2 machimes that use ubuntu 10.04 with Linuxcnc in use for over 4 years, never fail, used daily but not all day and they are turned off every evening.
From experience, loooooong time ago, that type of "leak" can be caused by a bad memory controller.
Linux has processor load and system load, check the latter. If it goes often to 200, that is a sign of something wrong with sot or hardware.
Regards,
Tom

Please Log in to join the conversation.

More
9 years 7 months ago #89509 by jCandlish
Installing on Jessie appears to have resolved the memory leak. It must have been caused subtle dependency of package versions.

Running master-rt on Jessie requires a backported PREEMPT RT kernel, and backported glx and intel firmwares if your motherboard has an integrated intel graphics chipset.

backports snipped for /etc/apt/sources.list
Code:
# backports deb http://ftp.ch.debian.org/debian/ jessie-backports main contrib non-free deb-src http://ftp.ch.debian.org/debian/ jessie-backports main contrib non-free

needful backported packages
Code:
latheoperator@125cnc:~$ dpkg-query -l | grep ~bpo8 | awk '{print $2}' firmware-misc-nonfree firmware-realtek intel-gpu-tools iucode-tool libdrm-amdgpu1:amd64 libdrm-dev:amd64 libdrm-intel1:amd64 libdrm-nouveau2:amd64 libdrm-radeon1:amd64 libdrm2:amd64 libgl1-mesa-dri:amd64 libglib2.0-0:amd64 libglib2.0-bin libglib2.0-dev libllvm3.8:amd64 libva1:amd64 linux-base linux-compiler-gcc-4.9-x86 linux-headers-4.9.0-0.bpo.2-common-rt linux-headers-4.9.0-0.bpo.2-rt-amd64 linux-headers-rt-amd64 linux-image-4.9.0-0.bpo.2-rt-amd64 linux-image-rt-amd64 linux-kbuild-4.9 xserver-xorg-video-intel latheoperator@125cnc:~$ uname -a Linux 125cnc 4.9.0-0.bpo.2-rt-amd64 #1 SMP PREEMPT RT Debian 4.9.13-1~bpo8+1 (2017-02-27) x86_64 GNU/Linux latheoperator@125cnc:~$
Code:
latheoperator@125cnc:~/linuxcnc$ linuxcnc LINUXCNC - 2.8.0-pre1-2912-g2636009 Machine configuration directory is '/home/latheoperator/linuxcnc/configs/work/CNC_003' Machine configuration file is 'CNC_003.ini' Starting LinuxCNC... (time=1489400763.016967,pid=7358): Registering server on TCP port 5005. (time=1489400763.017573,pid=7358): running server for TCP port 5005 (connection_socket = 3). iocontrol: machine: 'CNC_003' version 'unknown' Found file(REL): ./CNC_003.hal Note: Using POSIX realtime hm2: loading Mesa HostMot2 driver version 0.15 hm2_eth: loading Mesa AnyIO HostMot2 ethernet driver version 0.2 ...

Please Log in to join the conversation.

Time to create page: 0.114 seconds
Powered by Kunena Forum