Given that a Xine lib based media player is the primary device for media playing on Media directors wouldn't your efforts be better placed rebuilding XinePlayer with a newer version of xinelib that supports VDPAU?
I don't have any real advice for your actual problem, sorry. Was the compile successful? Or any warnings to speak of?
I want first to check that my hardware works with VDPAU and new NVidia driver, and this process in unobtrusive to LinuxMCE playback and if I mess something it doesn't interfere with LinuxMCE playback. If I mess something up with xinelib then I need to reinstall the whole LinuxMCE and that I'm still not ready to do
So if anybody can replicate these easy steps so we can together find what is going on here, and the whole community would benefit from this.
I'm recompiling it again and will report any errors, but it is trivial for anybody with NVidia card to try this and you will not interfere with existing mplayer or xine binaries.
I have recompiled mplayer again and there are quite a few warnings but no fatal errors that are unusual... so I can't figure out why I still have no vdpau support with this patched mplayer. Could somebody please try it also.