The 7600GS doesn't support VDPAU acceleration. Just because hardware works in Windows doesn't mean that it will work in Linux, software drivers are very different and there are different levels of compatibility under Linux than Windows because of this.
Turn off the VDPAU, your system should be failing back to OpenGL anyways. You will have to remove the 'vo=...' and 'vc=...' section of the mplayer config file as well. The system will then rely on your CPU for the decoding and it will need to be hefty to do a good job with h264 media. Do some searching on the forums there have been discussions about non-vdpau requirements for 1080P playback.
My immediate recommendation would be to ditch the 7600GS and get an 8xxx or 9xxx card with VDPAU acceleration. Havn't looked myself but I remember posts where people have said a PCI-e 9xxx something or other could be had for around $30.
Check out nvidia's vdpau hardware compatibility list: http://us.download.nvidia.com/XFree86/Linux-x86/190.53/README/appendix-a.html