You must not connect an analog output to drive ground. Isolating
the encoder would not help as you would get interference between
axis analog signals. If connecting the encoder lowers the analog output
you have a wiring error.
If you wish to reverse the analog output polarity (as Y seems to need),
you do this by changing the sign of the PWM scale (this might be set
by OUTPUT_SCALE depending on the hal.ini file structure), NOT by
swapping AOUT and GND