[SOLVED]: URGENT 810 Software RAID failed after power outage

[Beeker]

Hey Crumble,
Tried formatting sde1 and got the following then tried to add and still get that same error and I also have included a fdisk -l report just as an fyi

dcerouter_1024641:/home/# mkfs.ext3 /dev/sde1
mke2fs 1.41.3 (12-Oct-2008)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
91578368 inodes, 366283520 blocks
18314176 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=0
11179 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
       4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
        102400000, 214990848

Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 33 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.
dcerouter_1024641:/home/#


dcerouter_1024641:/home/# mdadm --add /dev/md1 /dev/sde1
                  mdadm: /dev/sde1 not large enough to join array


dcerouter_1024641:/home/# fdisk -l

Disk /dev/sda: 750.1 GB, 750156374016 bytes
255 heads, 63 sectors/track, 91201 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x39686ed2

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1       90446   726507463+  83  Linux
/dev/sda2           90447       91201     6064537+   5  Extended
/dev/sda5           90447       91201     6064506   82  Linux swap / Solaris

Disk /dev/sdb: 1500.3 GB, 1500301910016 bytes
255 heads, 63 sectors/track, 182401 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x0006a5c8

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1      182401  1465136001   83  Linux

Disk /dev/sdc: 1500.3 GB, 1500301910016 bytes
255 heads, 63 sectors/track, 182401 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0xb217d64b

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1               1      182401  1465136001   83  Linux

Disk /dev/sdd: 1500.3 GB, 1500301910016 bytes
255 heads, 63 sectors/track, 182401 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x43b7a284

   Device Boot      Start         End      Blocks   Id  System
/dev/sdd1               1      182401  1465136001   83  Linux

Disk /dev/sde: 1500.3 GB, 1500301910016 bytes
255 heads, 63 sectors/track, 182401 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x2848762e

   Device Boot      Start         End      Blocks   Id  System
/dev/sde1               1      182401  1465134080   83  Linux

[mkbrown69]

Your disks appear to be identical sizes, but the number of blocks available on sde are less than the others, which is the cause of the error.  More than likely, your older disks were partitioned with a starting sector of 63, which was the old standard.  The user-land disk partitioning tools had their defaults changed to accommodate the newer disks, where you want to partition align on a different boundary (usually 2048).  change your units to sectors, and see what the starting sector is for your sde vs sdd.  You'll likely have to delete the partition again, and re-create while forcing the starting sector to 63 to match your existing disks.

Hope that helps!

/Mike

[Crumble]

mkbrown is right the block size is off by 4000 KB.  That was sdf, which was never part of the raid.  I checked the thread again and that has been the case all along for that drive.  Looks like we need to delete the partition with fdisk not just format it.  Follow these instructions to just delete the partition.
Do not create a new partition just save and exit once deleted.


http://www.howtogeek.com/106873/how-to-use-fdisk-to-manage-partitions-on-linux/

stop the raid mdadm --stop /dev/md1

then run this command, this is the easy way.

 sfdisk -d /dev/sdd | sfdisk /dev/sde

then format it using mkfs

then add

[Beeker]

Thanks guys will give it a go and let you know how it goes

Kind regards
Beeker

[mkbrown69]

then run this command, this is the easy way.

 sfdisk -d /dev/sdd | sfdisk /dev/sde

then format it using mkfs

then add

Actually, you don't need to format the individual members of the raid array.  They contain the striped blocks of the md device, not an actual file system.  It's the /dev/mdX that actually gets a file system, which in this case already exists so DON'T do a mkfs on /dev/md1.

Just copy the partition table as per the sfdisk command above, and then add /dev/sde1 into the array.

Things should work a lot better this time around.

HTH!

/Mike

[Beeker]

OK thanks Mike

[Beeker]

All fixed thanks to Crumble for a super human effort and avoid an early funeral after recovering the wedding photos

Thanks to Mike as well

Kind regards
Beeker

[mkbrown69]

Beeker,

Glad to hear you recovered your treasured memories!  RAID is meant for availability in the case of a hardware failure in one of the "spinning platters of rust", to quote a colleague.  It won't ever replace a good backup, which is something you may wish to do in the near future...

Take care!

/Mike

[Beeker]

Thanks Mike.............I do have a back up in place now to avoid sudden death in the future