Messages

Any luck?

When you have a chance also check your memory usage.  I remember having a problem with the mdadm monitoring daemon where it was consuming memory until the system died.  I don't remember how exactly I fixed it, but if it appears to be your problem, I'll do some digging.

I used "top" to track memory usage and hit the "M" key to sort by memory.  mdadm should be using trivial amounts of memory (less than 1%) (0.1% on my system).

These should provide us with some clue.

Log messages generated during the boot process:

Code Select Expand

/var/log/dmesg

System log file:

Code Select Expand

/var/log/syslog

Well...if you know the approximate time frame we might be able to find something in the syslog regarding why mdadm removed the device (or if something else did).

Out of curiosity, what does "mdadm --detail /dev/md0" say after the reboot?

mdadm might have flagged your 3rd drive as a spare while it's rebuilding it.

My sincere condolences for your data loss.  Something like that is quite disconcerting, especially considering that the whole purpose behind a raid is to prevent such an occurrence. 

I will help in anyway that I can, though I will admit, I may not have the depth of knowledge you seek.  I will do my best. 

That being said, here's one of my previous posts concerning this issue:

http://ubuntuforums.org/showthread.php?p=5010533#post5010533

...again I'm not 100% sure this is exactly the issue you've faced.  I just know the road to software raid became a little more bumpy with 8.04.  If your device was removed after a reboot/power outage, then my theory holds a little more water.  Otherwise, we'll need to start looking at the syslog entries for more information.

Yes, your raid is currently running in degraded mode meaning if you were to lose either sdc or sdd right now, you'd lose all of your data.  Once sdb has been rebuilt, your raid should have the expected level of redundancy.

My speculation as to why the drive disappeared is because of the mdadm configuration.  I had an issue in the past with device names (e.g. sda, sdb) versus UUID's.  My raid was never safe.  mdadm would try to scan for superblocks on my system and would only sometimes succeed.  I needed to change my fstab to mount by UUID as well as tell mdadm which UUID to look for (mdadm.conf).  My theory is that your issue is related.  At the very least, explicitly telling mdadm where your raid is couldn't hurt.

I hope this helps.

One thing to consider is adding the following line to your mdadm.conf file.

Code Select Expand

ARRAY /dev/md0 level=raid5 num-devices=3 UUID=ef05e3dd:c2ec8d78:bd9f1658:0a1d2015

The resulting mdadm.conf file should look like this:

Code Select Expand

# mdadm.conf
#
# Please refer to mdadm.conf(5) for information about this file.
#

# by default, scan all partitions (/proc/partitions) for MD superblocks.
# alternatively, specify devices to scan, using wildcards if desired.
DEVICE partitions

# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes

# automatically tag new arrays as belonging to the local system
HOMEHOST <system>

# instruct the monitoring daemon where to send mail alerts
MAILADDR root

# definitions of existing MD arrays
ARRAY /dev/md0 level=raid5 num-devices=3 UUID=ef05e3dd:c2ec8d78:bd9f1658:0a1d2015

# This file was auto-generated on Mon, 21 Apr 2008 15:58:25 -0400
# by mkconf $Id: mkconf 324 2007-05-05 18:49:44Z madduck $
PROGRAM /usr/pluto/bin/monitoring_RAID.sh

This will give mdadm a big hint on where to find your raid devices and might help prevent future confusion on mdadm's part.

You'll need to re-add your removed device to get the raid back up and running in full mode:

Code Select Expand

mdadm -a /dev/md0 /dev/sdb

You should then start to see your raid re-build itself in the mdstat file in proc:

Code Select Expand

cat /proc/mdstat

Do not remove any of the drives while this process is in progress or data loss will likely occur.

Please post your mdadm.conf file as well as output from the following command:

Code Select Expand

mdadm -Ebsc partitions

This command will scan your partitions for the existence of any RAID superblock.  This is primarily important for the config lines it will spit out.  These should match what you have in your mdadm.conf file.

In my opinion, this is really just the first place to start.  Hopefully we'll figure something out from this.