Your hardware assumption sound about right. We are working on a boot image for the VIA platform, which will make for very low cost media directors. Of course the safe bet for the moment is a P4 as you said.
Of course you can just control a system with a mouse and keyboard, although it admittedly even a wireless keyboard isn't something that comfortably sits on the sofa. The cheapest (small) way to control Pluto is just using a normal infrared remote control. The 28 version due to be released on Monday or Tuesday as much improved infrared functionality, and every screen is designed to be controllable with infrared as well. The problem is the infrared receivers we been using with the LIRC Project are not very reliable or accurate. We would really like to be able to use the same infrared transceiver of the Microsoft Windows XP media Center, but the problem is there are just no reliable linux drivers for it yet. So for the 28 release we have a new driver for the irtrans.de tranceiver. This does both infrared receive and transmit. And it seems to work flawlessly, and a 28 release includes an optimized template for her in the windows XP media Center remote controls, which are available everywhere very cheap. You could also get a DIGN/Dvine 5 case (see ahanix.com), which has an IR trans receiver and VFD display built-in. That's what I'm using, and it works great. Remember you can also get a Symbian series 60 phone on eBay for around 80 or $90, you don't need any service to use it, or you could get it cheap PDA from eBay as well.
Of course the Windows version of orbiter can run on any Windows PC, and any media director can be used as a remote control for any other media director.