Those settings look fine to me (I set the hardware acceleration to xv, but that shouldn't be causing your problem). That, also, doesn't necessarily mean that X is actually using those settings... although in theory it should. Can you post your xorg.conf file (/etc/X11/xorg.conf) and the Xorg.log file (/var/log/Xorg.0.log) ? Also, could you please post the result of the following command, as a dvd is playing. You will likely have to do this from an ssh session.
free -m
The reason I ask for the last command is because I have had difficulty in the past with skipping video when I was using most of my memory and swap (if it was even activated). I haven't had this issue in LinuxMCE (especially with specs like that), so this might not be it, but it wouldn't hurt to check...