How to configure for spindle motor - DC brushed servo w/encoder step/dir

From the documentation of INI. See BASE_PERIOD =

2.5. [EMCMOT] Section
This section is a custom section and is not used by LinuxCNC directly. Most configurations use values from this section to load the motion controller. For more information on the motion controller see the Motion Section.

EMCMOT = motmod - the motion controller name is typically used here.

BASE_PERIOD = 50000 - the Base task period in nanoseconds.

SERVO_PERIOD = 1000000 - This is the "Servo" task period in nanoseconds.

TRAJ_PERIOD = 100000 - This is the Trajectory Planner task period in nanoseconds.

COMM_TIMEOUT = 1.0 - Number of seconds to wait for Motion (the realtime part of the motion controller) to acknowledge receipt of messages from Task (the non-realtime part of the motion controller).

Sorry I guess that might come easier from the latency test.
According to that:
base thread is around 14000
servo thread is around 34000

but the values in the ini are
base 100000
servo 1000000

Maybe I can edit those values a bit to speed up steps in any case?

You know, I'm paddling upstream here. To Richard's earlier point, I'm likely kludging hardware together that won't be satisfying in the end. Basically I was thinking if I could get 3000rpm out of this motor, that matches the rpm of the single-speed motor that the mill came with but I could have UI (g-code) control of it, which for me is very desirable.

If 3000rpm is ultimately impossible going this route, then I can think of three options.
1) deal with the klunky AC motor & step pulley spindle arrangement it has as-is.
2) replace the encoder with something which has 10ppr or something, or even hack a 1ppr or whatever I guess, but that's a whole 'nuther project.
3) skip the encoder altogether and just run the servo as a DC brushed motor and buy a controller to suit.
I guess 4) would be to punt the whole thing and buy a motor/controller pair and replace the whole smash.

I'm just trying not to spend a bunch of money on this thing and would rather spend it on a more capable future system instead. My hidden agenda is to simply dip my toe in and become somewhat familiar on some basic level with LinuxCNC, so I guess I'm hacking my way through that one. With that in mind, I'd like to at least figure out how to make this motor run as best it can in velocity mode in a pure effort to learn something. So I guess I'm not quite throwing the towel in yet.

Long time ago I fidled with lpt but I'd try lower Base thread. Maybe 30000.

Yeah, I wouldn't mind getting into Mesa boards or the like, but that will make much more sense for a later phase on a much more capable machine. I have a reasonable means but not rolling in it enough to buy a full-on machining center, so I'm kinda caught in the middle for now.
For this little toy, lpt (and the accompanying hardware it already came for it) is as worth the effort as it's gonna get.

I'll try fiddling with Base Thread a bit tomorrow, just to see what happens.

don't change the base thread,
but you have to set the "gear-ratio" in the servo-driver.
For max rpm you also may have to increase the DC-supply of the servo driver

you may look here to find some answers

LinuxCNC is telling you the issue:

STEPGEN: Channel 3: The requested maximum velocity of 5006 steps/sec is too high.
STEPGEN: The maximum possible frequency is 5004 steps/sec.

This limit is because your base thread is too slow as Mike_Eitel mentioned

That is, you have a 10 KHz base thread (100000 ns), this can only generate a ~ 5 KHz step rate
It addition to reducing the base thread period, you can enable the stepgens reset mode (your axis stepgens have this enabled)
which will allow you to generate steps up to the base thread rate. These 2 steps should allow ~30 KHz step rates, that is, 6 times
higher than your current limit

6x the current rpm gets me up to 720, so that's still a good chunk slower than before and still a few times slower than I need.
So I plan to tinker with those settings just for the practice, but it's probably time to punt on this motor for driving the spindle.
I'll just connect a relay to LinuxCNC so I can switch on/off the old motor via the UI and vary speed via the step pulley.

It wasn't for nuthin' I guess. This servo & driver is proven to work properly at least so I can use it to run an axis on a future machine. This was unknown a couple days ago.
I want to experiment with it further and see what kind of top speed it can achieve with this servo on a signal generated by other hardware I guess. Honestly I didn't expect to bump up against this particular limit so easy with a PC but I'm sure there are valid reasons it. Maybe it's an lpt thing?
Thanks to all for the help. I learned some things.

Can you lower the resolution of the servo step input?

if 5000 steps/sec = 120 RPM that means you have ~2500 steps/turn

At 2500 steps/turn, 3000 RPM would be 125 KHz, outside of the range of most parallel port systems.