So, I've run into the same problem.
I run on 15,6" (system see it as 16") touch screen which I've mount vertically, so my resolution i now 1080x1920.
After trying with screen physical size, screen DPI and few other things, nothing worked...
I've changed to 2.9.10 with debian 13 (i had 2.9.3 with debian 12) and for a brief moment it was ok, until i uploaded old configuration for linuxcnc and problem went back again.
In my case source of the problem is
probe_screen_v2.9
it requires resolution of 1280x1024 and I wasn't able to change it.
There is no fixed dimensions in glade file for whole window, it depends on sizes of single components. So it would be a hassle to change it.