Sometimes not knowing when to quit pays off:
Section "Device"
Identifier "Device1"
Driver "nouveau"
Option "ShadowFB" "true" << Added this, and scrolling is
fine now.
Option "WrappedFB" "true" << This might help as well, tho not
as much.
BusID "PCI:1:0:0"
Screen 1
EndSection