That CPU will definitely struggle decoding X264/H264 video at 720P and above! The problem is you have no hardware decode acceleration, so anything that is very high bit rate video and/or a high demand codec will struggle as it relies entirely on the CPU. As mentioned, with a supported nVidia GPU and the VDPAU configured to work feature, this ceases to be an issue. Its unlikely you can upgrade something as old as a GX270/P4 machine to be adequate in CPU alone. RAM will make no difference to this particular issue.
Perhaps look into the small/cheap nVidia ION platforms that are becoming popular? Like the Acer Revo, etc low power CPU, but have nVidia chipsets compatible with VDPAU, so apparently they work like a dream... just note that even the beta doesn't use VDPAU out of the box yet....