Acceptable Load During Latency Testing

Is there an upper limit to the kind of workload one should impose during latency testing?

I picked up a used Intel D525MW and installed 2.7.4 on a small SSD. Configured based on information scattered around here and the wiki - Networking, USB, hyperthreading, serial port turned off in the BIOS, power saving features disabled, isolcpus=1 boot parameter in GRUB, and a CPU hog.

Been running the latency test for approx. 48 hours now, and the counter hit 100,000 ns of jitter at some point last night. Seems to be a far cry from what others have reported (20-30,000 worst-case).

I've got about 200 Iceweasel windows open, half a dozen instances of GIMP with 8-900mb images, 4 terminal windows withrunning, along as many screen saver settings windows open as I can fit onto the desktop, to fill the screen with some sort of graphics load. The spike happened at some point when the screen saver was running last night.

With a lesser load (that still shows a 100% CPU/RAM utilization), the latency seems to be stable at around 25,000 ns.

Fairly certain I'm just going to set the worse case latency to 25,000 and see if I hit any major roadblocks during the operation of the machine (stepper driven gantry router), but there seems to be a large enough discrepancy between my latency numbers and what I've seen posted previously to warrant asking whether or not I'm being overzealous with my stress testing.

Worst-case I'll pick up a Mesa Anything I/O card or a Smoothstepper and switch to hardware step generation, but I don't know what kind of rapids I'll be able to achieve mechanically at this point, so whether or not the machine will be the bottleneck is yet to be seen. This thing definitely isn't a VMC.

"Stressing the system" should be just putting a reasonable potentially expectable worst case scenario, not drowning the system till it grinds to a halt. All I usually do is a couple glxgears, a little web browsing and some system updates. That's usually good enough. Especially if the system will only be used as a control. You might want to do a little more if you plan on doing other stuff with it, while machining.

As to your grub setups. I think you only want to run one or the other, isocpus or a cpuhog, but not both. Especially with a week anemic CPU like a D525. While it seems D525 systems can post decent latency numbers, when worked hard, they can just plain run out of steam and end up not meeting latency deadlines. Still they often perform more than adequately to do a nice software stepper system. (provided that is all it's asked to do.)

isolcpus=1 boot parameter in GRUB, and a CPU hog.

They both do the same job, you don't need both.
If you have isolated one core there is nothing for the cpu-hog to do but disrupt things on the remaining core.

I've got about 200 Iceweasel windows open, half a dozen instances of GIMP with 8-900mb images, 4 terminal windows with
dd if=/dev/zero of=/dev/null
running, along as many screen saver settings windows open as I can fit onto the desktop, to fill the screen with some sort of graphics load.

That is just plain ridiculous.

You are trying to simulate the load of running Axis (largely) and a realtime system, not trying to grind the poor little Atom into the dirt.
Todd has covered it.

Thanks to both of you for the replies, and for confirming my suspicions.

I was approaching latency testing like stability testing is done in the overclocking community - Throw an absolute worst-case scenario at the PC, and see if it falls on its face. If not, one can be reasonably sure it's stable for any use case.

Sweet! Now I can focus on getting this control box wired up, and start making some test cuts!