Software RAID 1 Setup With Linux

Software RAID 1 Setup With Linux
(February 2006)

Table of contents

Introduction

Step 1: Find out the size of the array

Step 2: Create the array and set the size

Step 3: Populate /dev/md0

Step 4: Change the partition type of /dev/hdc1

Step 5: Setup of fstab and GRUB

Step 6: /dev/hda1 configuration

Step 7: GRUB configuration

Final Step : Change the partition type of /dev/hda1

Troubleshooting

Maintenance

Introduction

What we have

2 hard disks (/dev/hda1 and /dev/hdc1)
a Linux system - kernel 2.6.14 with RAID 1 support
the GRUB bootloader on /dev/hda1
/dev/hda1 mounted on /
/dev/hdc1 is not used
/dev/hdc1 has approximately the same size as /dev/hda1 (4GB)
mdadm - v1.9.0 - 04 February 2005
the BIOS boots on /dev/hda1

What we want

both disks /dev/hda1 and /dev/hdc1 mirrored (RAID 1 array)
booting directly on this RAID 1 array
both disks can be switched
if /dev/hda fails, we want to move the remaining disk (/dev/hdc) to the first IDE slot (/dev/hda) and boot as well

We give here our partition tables but it might be not useful at all:

shell# fdisk -l /dev/hda
Disk /dev/hda: 4325 MB, 4325529600 bytes
255 heads, 63 sectors/track, 525 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/hda1               1         525     4217031   83  Linux

shell# fdisk -l /dev/hdc
Disk /dev/hdc: 30.7 GB, 30735581184 bytes
255 heads, 63 sectors/track, 3736 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/hdc1               1         535     4297356   83  Linux
/dev/hdc2             536        3736    25712032+   f  W95 Ext'd (LBA)
/dev/hdc5             536        3736    25712001   83  Linux

Step 1: Find out the size of the array

As our two disks have not exactly the same size (around 4 GB each), we want to know the best size.
The only way I know (but certainly not an official way !) is to issue a "create" command, and abort when mdadm has given us the information:

shell# mdadm --create --verbose /dev/md0 --level=1 --raid-devices=2 /dev/hda1 /dev/hdc1
mdadm: /dev/hda1 appears to contain an ext2fs file system
    size=4217028K  mtime=Fri Dec 23 04:53:30 2005
mdadm: /dev/hdc1 appears to contain an ext2fs file system
    size=4297280K  mtime=Fri Dec 23 02:10:28 2005
mdadm: /dev/hdc1 appears to be part of a raid array:
    level=1 devices=2 ctime=Thu Dec 22 23:02:50 2005
mdadm: size set to 4216960K
mdadm: largest drive (/dev/hdc1) exceed size (4216960K) by more than 1%
Continue creating array?

Answer 'n', or enter ctrl-C (because we don't want to create the complete RAID array now, as /dev/hda1 is still mounted and in use).
The important information here is "size set to 4216960K". This is the size we shall use.

Step 2: Create the array and set the size

Our RAID 1 array is called /dev/md0.

shell# mdadm --create --verbose /dev/md0 --level=1 --raid-devices=2 missing /dev/hdc1

missing stands for /dev/hda1 which will be added later when it will not be mounted any longer.
Now we want to set the correct size to this array:

shell# mdadm --grow /dev/md0 -z 4216960

Step 3: Populate /dev/md0

Create an ext3 filesystem on /dev/md0:

shell# mkfs.ext3 /dev/md0

Copy all the files contained in / and its sub-diretories:

shell# mkdir /RAID1
shell# mount -t ext3 /dev/md0 /RAID1 
shell# for f in /*
do  
    if [ "$f" != "/RAID1" -a "$f" != "/proc" -a "$f" != "/sys" ]; then
        echo "$f" ...
        cp -Rp "$f" /RAID1/.
    fi
done
shell# mkdir /RAID1/proc

(remember that /dev/md0 must contain a complete system, on which we may want to boot later)
Some parts need not be copied as they are created at every boot (/proc, /sys - this depends on your Linux distribution).
/proc must be at least created.

Step 4: Change the partition type of /dev/hdc1

We use the fdisk command:

shell# fdisk /dev/hdc

In the fdisk interactive prompt, we choose to set the partition type to fd (raid autodetect).
The resulting partition table of our disk is:

Disk /dev/hdc: 30.7 GB, 30735581184 bytes
255 heads, 63 sectors/track, 3736 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/hdc1               1         535     4297356   fd  Linux raid autodetect
/dev/hdc2             536        3736    25712032+   f  W95 Ext'd (LBA)
/dev/hdc5             536        3736    25712001   83  Linux

Step 5: Setup of fstab and GRUB

We want now to prepare the booting on /dev/md0.
Add the following line in /RAID1/etc/fstab:

#/dev/hda1       /               ext3    defaults 0       1
/dev/md0        /               ext3    defaults 0       1

(comment out the hda1 line)

Add the following entry in the GRUB config file (/boot/GRUB/menu.lst, on /dev/hda1):

title           Linux kernel 2.6.14 (RAID as root)
kernel          (hd0,0)/boot/vmlinuz-2.6.14.4 root=/dev/md0
boot

Step 6: /dev/hda1 configuration

Reboot your computer and choose the "RAID as root" entry in the GRUB menu.
At this point /dev/hda1 should not be mounted anymore.

# mdadm -D /dev/md0
/dev/md0:
        Version : 00.90.02
  Creation Time : Thu Dec 22 23:02:50 2005
     Raid Level : raid1
     Array Size : 4216960 (4.02 GiB 4.32 GB)
    Device Size : 4216960 (4.02 GiB 4.32 GB)
   Raid Devices : 2
  Total Devices : 1
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Fri Dec 23 03:35:13 2005
          State : clean, degraded
 Active Devices : 1
Working Devices : 1
 Failed Devices : 0
  Spare Devices : 0

           UUID : af34a3b3:8d2eef00:73d43def:403507df
         Events : 0.997

    Number   Major   Minor   RaidDevice State
       0       0        0        -      removed
       1      22        1        1      active sync   /dev/hdc1

shell# mount
/dev/md0 on / type ext3 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
usbfs on /proc/bus/usb type usbfs (rw)

shell# cat /proc/mdstat
Personalities : [raid1] [multipath]
md0 : active raid1 hdc1[1]
      4297280 blocks [2/1] [_U]

unused devices: <none>

We can now insert /dev/hda1 into our RAID array.
WARNING: after the following command, the recovery process will begin and /dev/hda1 will be replaced with the contents of /dev/md0 (but no data will be lost as we have previously copied all the files from hda1 to md0):

shell# mdadm /dev/md0 -a /dev/hda1

We check that the recovery process is running:

shell# cat /proc/mdstat
Personalities : [raid1] [multipath]
md0 : active raid1 hda1[2] hdc1[1]
      4216960 blocks [2/1] [_U]
      [==>..................]  recovery = 13.1% (553984/4216960) finish=33.1min speed=1840K/sec

When it is done:

shell# cat /proc/mdstat
Personalities : [raid1] [multipath]
md0 : active raid1 hda1[0] hdc1[1]
      4216960 blocks [2/2] [UU]

unused devices: <none>

Step 7: GRUB configuration

We want to have the same GRUB installation on both disks and to always boot on the first device (/dev/hda).
So that if the boot disk fails, then we just have to place the remaining disk at the place of the failed disk and boot.
To achieve this, the boot track of both disk must be the same.

GRUB calls the disks hd0, hd1, etc...
In our case, /dev/hda is hd0, and /dev/hdc is hd1.

GRUB is already set up on /dev/hda and we do not change it.
The setup of the boot track of the /dev/hdc disk is more complicated:

shell# grub
grub> install (hd0,0)/boot/grub/stage1 (hd1) (hd0,0)/boot/grub/stage2 p (hd0,0)/boot/grub/menu.lst

This tells grub to install itself on hd1 (/dev/hdc), but when the booting will occur (if one day the hd0 disk fails and we put hd1 at the place of hd0), stage1 and stage 2 will have to be searched on hd0.

Final Step : Change the partition type of /dev/hda1

Use fdisk, as explained at step 4.
The resulting partition table is:

Disk /dev/hda: 4325 MB, 4325529600 bytes
255 heads, 63 sectors/track, 525 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/hda1               1         525     4217031   fd  Linux raid autodetect

Then we can reboot and we have a complete RAID 1 system.

The superblock looks like:

shell# mdadm -D /dev/md0
/dev/md0:
        Version : 00.90.02
  Creation Time : Fri Dec 23 05:04:24 2005
     Raid Level : raid1
     Array Size : 4216960 (4.02 GiB 4.32 GB)
    Device Size : 4216960 (4.02 GiB 4.32 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Sat Feb 11 03:49:48 2006
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

           UUID : 890a764b:627353ea:ab248d0a:40cc9e20
         Events : 0.12961

    Number   Major   Minor   RaidDevice State
       0      22        1        0      active sync   /dev/hdc1
       1       3        1        1      active sync   /dev/hda1

Troubleshooting

In case you get the error:

shell# mdadm /dev/md0 -a /dev/hda1
mdadm: hot add failed for /dev/hda1: No space left on device

Look at /var/log/kern.log. You might see:

[date] localhost kernel: md0: disk size 4216960 blocks < array size 4297280
[date] localhost kernel: md: export_rdev(hda1)

This is because you did not set a correct size with the mdadm --grow command.

Maintenance

To remove a disk from the RAID array:

shell# mdadm --stop /dev/md0

If a disk is removed and then added back, the software RAID system does not take it automatically into account inside the RAID array.
We must issue (example with /dev/hdc1):

shell# mdadm /dev/md0 -a /dev/hdc1

and the recovery process starts.