Yes, i do believe there are problems with FAT32 drives, so you should certainly switch filesystems. However use XFS rather than EXT3, it has better performance for this type of usage. Just it being USB probably also has an impact though, so don't be surprised if changing filesystems doesnt clear all of it up.
Ok, I'll find a way to save my media and give xfs a shot. I do have another USB drive that is problem free and the little hack of cd'ing into a directory on that drive has made everything fine. I also noticed the load average dropped from 2.x to 0.x. I had been wondering why the load average was so high.. Anyway, I think the right USB drive with the right filesystem will work out, I just hope MY drive with xfs/ext3 will work.