OK, I'm not entirely sure this has anything to do with the problem as I see it probably has more to do with vdpau than xine but here is a conversation I had that may help a bit...
I asked: whats the recommend way to get mkv and other h264 codecs to work?
[Fri Jul 9 2010] [21:00:48] <TSCHAKeee>
first, a correction
what you're experiencing has nothing to do with h.264
we supply h.264 codecs already, as part of ffmpeg
and an appropriate GPU can decode them on-card via VDPAU
with that said, what you're experiencing
is the fact that you're downloading pirated movies off bittorrent
(don't deny it)
which are using a newer MKV encapsulation
MKV is a container format
we do have an updated xine, which niz23 has backported the demuxer from the upcoming xine release
however, we do not currently supply them as part of the build
because they break VDR
at least until we can rebuild vdr-sxfe to properly handle the new xine libs
IF you want the new xine, you will have to forcefully install the debs
they are located here: http://tschak.homelinux.com/deb/
install them all at the same time, with dpkg
keep in mind however, that you are on your own
we can't provide any support for these packages, as we are still working things out with it relating to what I mentioned above.
if you do not know how to install packages by hand with dpkg
much less how to install a batch of them at the same time
either (a) google it or (b) forget it
the mplayer player will be removed.
we will remove it from the build soon.
This got video and sound to work correctly.... kinda... I'm working on it; but it does seem to work