System freezes above certain pulse rate
18 Sep 2015 02:14 - 18 Sep 2015 02:19 #62728
by staviq
System freezes above certain pulse rate was created by staviq
Hello everyone.
I have finished my xyz cnc about two weeks ago, did some initial feeds and speeds testing and decided to increase stepper speeds so i went up from 40mm/s to about 80mm/s in stepconf.
And that's when problems begun.
Everything works fine, until i get steppers to max speed. I can jog slowly, can run gcode as long as jog speed is reduced in axis, but when steppers reach maximum defined speed, system becomes completly unresponsive, mouse cursor freezes, capslock/numlock wont light up on the keyboard, ssh session disconnects, systems stops to respond to ping. A few seconds to a few minutes later screen goes black and gives me kernel panic, not a specific one, seems to diffrent each time.
I gave it a try today, fresh linuxcnc 2.7 install, stepconf, try axis, run, and system freezes almost immediately, stepper however did not freeze, it went nicely left to right and back, so i guess step planner thread was fine, somewhere in the depths of the cpu.
Before you suggest hardware problem, i have ruled that out by bringing completely different pc from work, twice the ram, twice the Ghz, fresh install, same problem.
So i thought, i'm pushing poor linux too hard, changed microstepping on the driver from 1/16 to 1/8, adjusted microstepping and changed speed in stepconf from 40 to 80 mm/s, this should get me to the exactly same pulse rate as it was working fine, but twice the speed. Nope. same problem.
LPT port is fully optoisolated, so there is no chance of any inductive loads getting back into the pc.
What am i doing wrong ?
EDIT: THIS ISSUE OCCURS BOTH IN 2.6 AND 2.7 LINUXCNC VERSIONS.
I have finished my xyz cnc about two weeks ago, did some initial feeds and speeds testing and decided to increase stepper speeds so i went up from 40mm/s to about 80mm/s in stepconf.
And that's when problems begun.
Everything works fine, until i get steppers to max speed. I can jog slowly, can run gcode as long as jog speed is reduced in axis, but when steppers reach maximum defined speed, system becomes completly unresponsive, mouse cursor freezes, capslock/numlock wont light up on the keyboard, ssh session disconnects, systems stops to respond to ping. A few seconds to a few minutes later screen goes black and gives me kernel panic, not a specific one, seems to diffrent each time.
I gave it a try today, fresh linuxcnc 2.7 install, stepconf, try axis, run, and system freezes almost immediately, stepper however did not freeze, it went nicely left to right and back, so i guess step planner thread was fine, somewhere in the depths of the cpu.
Before you suggest hardware problem, i have ruled that out by bringing completely different pc from work, twice the ram, twice the Ghz, fresh install, same problem.
So i thought, i'm pushing poor linux too hard, changed microstepping on the driver from 1/16 to 1/8, adjusted microstepping and changed speed in stepconf from 40 to 80 mm/s, this should get me to the exactly same pulse rate as it was working fine, but twice the speed. Nope. same problem.
LPT port is fully optoisolated, so there is no chance of any inductive loads getting back into the pc.
What am i doing wrong ?
EDIT: THIS ISSUE OCCURS BOTH IN 2.6 AND 2.7 LINUXCNC VERSIONS.
user@debian:~/linuxcnc/configs/gst$ cat gst.ini
# Generated by stepconf 1.1 at Wed Sep 16 23:19:12 2015
# If you make changes to this file, they will be
# overwritten when you run stepconf again
[EMC]
MACHINE = gst
DEBUG = 0
[DISPLAY]
DISPLAY = axis
EDITOR = gedit
POSITION_OFFSET = RELATIVE
POSITION_FEEDBACK = ACTUAL
ARCDIVISION = 64
GRIDS = 10mm 20mm 50mm 100mm 1in 2in 5in 10in
MAX_FEED_OVERRIDE = 1.2
MIN_SPINDLE_OVERRIDE = 0.5
MAX_SPINDLE_OVERRIDE = 1.2
DEFAULT_LINEAR_VELOCITY = 8.00
MIN_LINEAR_VELOCITY = 0
MAX_LINEAR_VELOCITY = 80.00
INTRO_GRAPHIC = linuxcnc.gif
INTRO_TIME = 5
PROGRAM_PREFIX = /home/user/linuxcnc/nc_files
INCREMENTS = 5mm 1mm .5mm .1mm .05mm .01mm .005mm
[FILTER]
PROGRAM_EXTENSION = .png,.gif,.jpg Greyscale Depth Image
PROGRAM_EXTENSION = .py Python Script
png = image-to-gcode
gif = image-to-gcode
jpg = image-to-gcode
py = python
[TASK]
TASK = milltask
CYCLE_TIME = 0.010
[RS274NGC]
PARAMETER_FILE = linuxcnc.var
[EMCMOT]
EMCMOT = motmod
COMM_TIMEOUT = 1.0
COMM_WAIT = 0.010
BASE_PERIOD = 13671
SERVO_PERIOD = 1000000
[HAL]
HALFILE = gst.hal
HALFILE = custom.hal
POSTGUI_HALFILE = custom_postgui.hal
[TRAJ]
AXES = 3
COORDINATES = X Y Z
LINEAR_UNITS = mm
ANGULAR_UNITS = degree
CYCLE_TIME = 0.010
DEFAULT_VELOCITY = 8.00
MAX_VELOCITY = 80.00
[EMCIO]
EMCIO = io
CYCLE_TIME = 0.100
TOOL_TABLE = tool.tbl
[AXIS_0]
TYPE = LINEAR
HOME = 0.0
MAX_VELOCITY = 76.0048643113
MAX_ACCELERATION = 250.0
STEPGEN_MAXACCEL = 312.5
SCALE = 914.285714286
FERROR = 1
MIN_FERROR = .25
MIN_LIMIT = -0.001
MAX_LIMIT = 200.0
HOME_OFFSET = -10.000000
HOME_SEARCH_VEL = 20.000000
HOME_LATCH_VEL = 0.546875
[AXIS_1]
TYPE = LINEAR
HOME = 0.0
MAX_VELOCITY = 76.0048643113
MAX_ACCELERATION = 250.0
STEPGEN_MAXACCEL = 312.5
SCALE = 914.285714286
FERROR = 1
MIN_FERROR = .25
MIN_LIMIT = -0.001
MAX_LIMIT = 325.0
HOME_OFFSET = -10.000000
HOME_SEARCH_VEL = 20.000000
HOME_LATCH_VEL = 0.546875
[AXIS_2]
TYPE = LINEAR
HOME = 0.0
MAX_VELOCITY = 38.0024321557
MAX_ACCELERATION = 150.0
STEPGEN_MAXACCEL = 187.5
SCALE = 1828.57142857
FERROR = 1
MIN_FERROR = .25
MIN_LIMIT = -100.0
MAX_LIMIT = 100.0
HOME_OFFSET = 0.0
user@debian:~/linuxcnc/configs/gst$
Last edit: 18 Sep 2015 02:19 by staviq.
Please Log in or Create an account to join the conversation.
18 Sep 2015 14:56 #62741
by Rick G
Replied by Rick G on topic System freezes above certain pulse rate
I assume you looked here...
wiki.linuxcnc.org/cgi-bin/wiki.pl?TroubleShooting
Did you run a latency test on both computers? How did they compare?
Any error messages?
Rick G
wiki.linuxcnc.org/cgi-bin/wiki.pl?TroubleShooting
Did you run a latency test on both computers? How did they compare?
Any error messages?
Rick G
Please Log in or Create an account to join the conversation.
18 Sep 2015 18:48 #62743
by staviq
Replied by staviq on topic System freezes above certain pulse rate
No error messages unfortunately, when system freezes it gets into a state where it's unable to flush any data to disk.
Latencies on both computers are about the same and quite stable ( i can give you values when i go back home today ).
I'm thinking, maybe my whole approach is wrong, what is a reasonable value for max step rate, because i'm trying to push it to over 100kHz.
Lpt port in epp mode with hardware dma should do 2Mb/s fine, stepper drivers are rated for 200kHz step rate, but i have no idea what's the overhead on step generation.
On the other hand, steppers run surprisingly smoothly, until system freezes, and then it seems like step planner just executes last trajectory infinitely ( y axis tried to escape my room once, it did not go well... ).
I can't wrap my head around why it ends up in kernel panic state...
Latencies on both computers are about the same and quite stable ( i can give you values when i go back home today ).
I'm thinking, maybe my whole approach is wrong, what is a reasonable value for max step rate, because i'm trying to push it to over 100kHz.
Lpt port in epp mode with hardware dma should do 2Mb/s fine, stepper drivers are rated for 200kHz step rate, but i have no idea what's the overhead on step generation.
On the other hand, steppers run surprisingly smoothly, until system freezes, and then it seems like step planner just executes last trajectory infinitely ( y axis tried to escape my room once, it did not go well... ).
I can't wrap my head around why it ends up in kernel panic state...
Please Log in or Create an account to join the conversation.
18 Sep 2015 20:16 #62746
by ArcEye
Replied by ArcEye on topic System freezes above certain pulse rate
Hi
I think you answered your own question in your first post
Have you read the docs www.linuxcnc.org/docs/devel/html/motion/tweaking-steppers.html and done real calculations - not theoretical wishful thinking ones?
Your base thread figure is very low, no idea what your latency figure was, but if that is it, then you have no stepper driver overhead or leeway for the slightest hiccup.
Add to that you were using 16x microstepping, so reducing the distance covered per step by a factor of 16, for no good payback, whilst expecting a stepper machine to travel at 4.8metres / min
Problems with excessive velocity and acceleration settings will often only show themselves in longer moves, because the machine never gets to reach those figures in shorter ones.
Set your base thread to 30000
Set your max velocity to 25
Set your max acceleration to 100 - 150
Set microstepping to max 8x and adjust scale accordingly
I bet you, your machine will run happily without a hitch and you can tune 'up' from there if you wish.
Too often users view stepper tuning as a p*****g contest, trying to get as high as possible and beyond.
What you actually want is a machine that will run a program all day, without you wondering if it has missed steps and ruined another workpiece
regards
I think you answered your own question in your first post
................so i went up from 40mm/s to about 80mm/s in stepconf. And that's when problems begun.
Have you read the docs www.linuxcnc.org/docs/devel/html/motion/tweaking-steppers.html and done real calculations - not theoretical wishful thinking ones?
Your base thread figure is very low, no idea what your latency figure was, but if that is it, then you have no stepper driver overhead or leeway for the slightest hiccup.
Add to that you were using 16x microstepping, so reducing the distance covered per step by a factor of 16, for no good payback, whilst expecting a stepper machine to travel at 4.8metres / min
Problems with excessive velocity and acceleration settings will often only show themselves in longer moves, because the machine never gets to reach those figures in shorter ones.
I'm thinking, maybe my whole approach is wrong, what is a reasonable value for max step rate, because i'm trying to push it to over 100kHz.
Set your base thread to 30000
Set your max velocity to 25
Set your max acceleration to 100 - 150
Set microstepping to max 8x and adjust scale accordingly
I bet you, your machine will run happily without a hitch and you can tune 'up' from there if you wish.
Too often users view stepper tuning as a p*****g contest, trying to get as high as possible and beyond.
What you actually want is a machine that will run a program all day, without you wondering if it has missed steps and ruined another workpiece
regards
Please Log in or Create an account to join the conversation.
18 Sep 2015 20:59 #62751
by PCW
Replied by PCW on topic System freezes above certain pulse rate
My guess is that you are simply running out of time
(all CPU used for LPT access and at higher step rates there are more accesses)
This is especially true if you are using a motherboard LPT port
Just about all current MB LPTs ports use a LPC (serial) interface and
can have up to 2-3 usec overhead for _every_ access
(and each LPT port is accessed probably 3 times per thread)
A PCI port will typically be faster
(all CPU used for LPT access and at higher step rates there are more accesses)
This is especially true if you are using a motherboard LPT port
Just about all current MB LPTs ports use a LPC (serial) interface and
can have up to 2-3 usec overhead for _every_ access
(and each LPT port is accessed probably 3 times per thread)
A PCI port will typically be faster
Please Log in or Create an account to join the conversation.
18 Sep 2015 21:19 #62752
by staviq
That i think would be reasonable cause of kernel panic, as cpu is forced to spends too much time talking to hardware, and since some low level hardware talking parts of linux kernel take priority over software processing, and i have seen it cause problems before.
I think i have pci lpt card somewhere lying around, i'll give it a try.
Replied by staviq on topic System freezes above certain pulse rate
My guess is that you are simply running out of time
(all CPU used for LPT access and at higher step rates there are more accesses)
This is especially true if you are using a motherboard LPT port
Just about all current MB LPTs ports use a LPC (serial) interface and
can have up to 2-3 usec overhead for _every_ access
(and each LPT port is accessed probably 3 times per thread)
A PCI port will typically be faster
That i think would be reasonable cause of kernel panic, as cpu is forced to spends too much time talking to hardware, and since some low level hardware talking parts of linux kernel take priority over software processing, and i have seen it cause problems before.
I think i have pci lpt card somewhere lying around, i'll give it a try.
Please Log in or Create an account to join the conversation.
19 Sep 2015 03:19 #62768
by staviq
Replied by staviq on topic System freezes above certain pulse rate
Great Success ! ( you need to read this in Borat's voice. )
Found yet another old pc, c2d 2.2GHz, and:
1. Removed GPU completely ( no built in GPU either on this board, this required some fiddling with the bios to pull off ).
2. Disabled onboard everything ( except ethernet ).
3. Added PCI parport card.
4. Fresh install, added "intel_idle.max_cstate=0 processor.max_cstate=0 idle=poll" to kernel parameters.
5. Replaced default mesa with software opengl packages
6. Disabled X server completely
7. Latency stable(ish) at around 3500 ns
8. SSH + X Forwarding, Samba to exchange files.
Steppers work ok ( little hiccups, but that is latency problem to be solved another time ), no freezes, max 20% CPU usage while running job trough axis.
Thanks Guys !!!
Found yet another old pc, c2d 2.2GHz, and:
1. Removed GPU completely ( no built in GPU either on this board, this required some fiddling with the bios to pull off ).
2. Disabled onboard everything ( except ethernet ).
3. Added PCI parport card.
4. Fresh install, added "intel_idle.max_cstate=0 processor.max_cstate=0 idle=poll" to kernel parameters.
5. Replaced default mesa with software opengl packages
6. Disabled X server completely
7. Latency stable(ish) at around 3500 ns
8. SSH + X Forwarding, Samba to exchange files.
Steppers work ok ( little hiccups, but that is latency problem to be solved another time ), no freezes, max 20% CPU usage while running job trough axis.
Thanks Guys !!!
Please Log in or Create an account to join the conversation.
Time to create page: 0.179 seconds