(with a small SSD disk inside things could be very fast)
To make my life easier, when I first started designing the project specs, I was planning to install Linuxmce 810 final, but (on installation time) it would not see the IBM controller (serveraid M1015)..
As Thom said, you're I/O bound. Your issues are the result of 10 NFS clients using the NFS shares as their root file system, so you have all sorts of random I/O's happening, including lots of writes due to logging. I'm also presuming that the MySQL database which powers the application logic on the core is residing on the same spindle, so you're probably maxing out the IOPS of a single drive, and hitting wait states and timeouts as a result. 'top' and 'iotop' run on both the core and MD's will tell you for sure.
Things you can do to improve the situation:
- Install your core's filesystem to an SSD. You'll be going from 100 IOPS to 4K+ IOPS in one easy step. If you can't re-install or move it, then add an SSD to the system using LVM, and create logical volumes which get mounted at /var/lib/mysql and /usr/pluto/diskless to host the IO intensive workloads. Obviously, you'll need to move the contents of those directories while in single-user mode, then reboot.
- If you can't do SSD, then use the RAID card in RAID 10 mode for the core's file system. SSD would be better though for random I/O. Either way, set the I/O scheduler for that /dev/sdX to 'deadline'.
- If you have lots of memory on your MD's, consider making /tmp a tmpfs ramdisk
- Consider setting vm.swappiness=1 i sysctl.conf on your MD's, to reduce swapping, or install a local disk for swap if you don't have enough memory on the MD's.
- Tune the TCP stack for throughput on the core, and increase your memory buffers. http://www.cyberciti.biz/faq/linux-tcp-tuning/
- Use a really good network card for the internal network. Intel GigE cards are typically the best on Linux; Realtek and NVIDIA on-boards are crap under heavy loads. If you have a capable managed switch and a multi-port NIC (or multiple NIC's), you can look at port aggregation for increased bandwidth. Or go 10G between the core and the switch if you have money to spare. No matter what, you need a switch with a non-blocking fabric, capable of full wire-speeds for the number of ports available.
Hope that helps!