I changed the numbers to 0,0,1024,768 under "Start of window". I cycled through that option (which worked correctly as expected), and then back to "Start as fullscreen" (which showed the display correctly, as described above). I exited LinuxCNC and restarted. It still exhibits the same behavior where the screen is slightly too big. So, the size of the windowed screed does not seem to affect the strange behavior.
