FirstServed Tech Blog - FirstServed and the Art of Server Tuning

Since XenServer and XenEnterprise do not support installing the operating system on an MD software RAID device during the installation, you’ll have to undertake a few steps afterwards if you want to mirror your boot disks.

This article is largely based on Harry de Jong’s solution, as posted on the XenSource forums, with a few differences:

• There’s no need to copy mdadm.static from another server to the Xen host;
• The initrd ramdisk’s configuration is a bit simpler;
• There’s no need to swap the physical disks, which means the procedure can be accomplished remotely.

Create identical partitions to your boot disk /dev/sda on your second disk /dev/sdb

If /dev/sda looks like this:

/dev/sdb should look like this:

Do not forget to set the partition type to ‘fd’ (Linux RAID autodetect) instead of the default 83 (Linux).

Create the necessary device nodes

Create your MD RAID 1 arrays (with missing disks)

/dev/sda2 is just an empty, unmounted partition used by XenServer for upgrades.

Copy the Xen Storage Manager data over to RAID

Replace the uuid in the vgextend line above with the uuid of your own Xen SR.  Use ‘sm info’ to display your SR’s uuid.

Remove /dev/sda3 from the SR volume group and add it to the RAID array

Mount /dev/md0 and copy the filesystem to it

Make a new initrd ramdisk containing the MD RAID drivers

Modify /mnt/etc/fstab so the system will mount / from /dev/md0 instead of LABEL=/-main.  Replace "LABEL=/-main" by "/dev/md0".

Create a new boot image and uncompress it:

Since mkinitrd looks at /etc/fstab to determine what device the root volume is on, it has now added the necessary raid drivers to the new boot image.

Uncompress the current ramdisk and add the raid drivers to it

Now add the raid module from the new ramdisk and modify the init file:

Add the following lines before the second line containing "/sbin/udevstart":

Add the following lines before the line containing "echo Creating root device":

Note: if you’ve created other MD raid devices, add a ‘raidautorun’ statement for them as well.

Copy the new ramdisk to the /mnt/boot folder and add modify GRUB’s boot menu

Replace "root=LABEL=/-main" by "root=/dev/md0" in all menu entries.

Unmount /dev/md0

Set up the Master Boot Record on /dev/sdb

Modify GRUB’s boot menu to boot from the second disk

Replace the line in the first XenServer entry that says "root (hd0,0)" with "root (hd1,0)". Also replace "root=LABEL=/-main" by "root=/dev/md0" in all menu entries.  This will allow you to boot from /dev/md0 instead of /dev/sda.  In case something goes wrong, you could always reboot using one of the other entries, which still point to /dev/sda.

Reboot

If all goes well, you should see your system mounting /dev/md0 as the filesystem root.  If not, reboot using one of the other GRUB menu entries and check out the previous steps.

Change /dev/sda’s partition types from Linux to Linux RAID autodetect

The result should be something like this:

Add the missing partitons to the RAID arrays

Then wait for the sync to complete:

Copy the running RAID setup to /etc/mdadm.conf

Be sure to add a line ‘DEVICE partitions’ at the top of /etc/mdadm.conf if you want to define MD devices that span a whole disk instead of just a partition.  Without this line, MD won’t be able to detect your drives as MD components at boot.

That’s it!

We recently took to installing our Linux servers from a USB stick.  This is much more efficient than downloading four of five cd iso’s time and again, since Linux distros like Fedora and CentOS keep spawning new releases constantly.  So we just write the rescue cd iso to a usb stick, and proceed with a network install.  This works like a charm, except that for a few times running, after completing the installation, instead of being met with the grub boot menu, we were dropped to the Grub shell brutally.  This had us stmied for a while, until we realized that for some dumb reason or other, Grub insists on recognizing the USB stick we install from as device hd0.  This means that during installation, Grub writes itself a nice config file containing the line

which points to the second drive, in our case our first hard disk, /dev/sda.  After installation – without the USB stick, Grub sees /dev/sda as the first device, hd0, and proceeds to boot the hd1 device, which now is /dev/sdb instead of /dev/sda.  Since the MBR (master boot record) was only written to /dev/sda, there’s no way Grub can boot from /dev/sdb.

In order to solve all this, from within the Grub shell, do the following:

This will instruct Grub to see both the hard disks as hd0 – since in case the first hard disk fails, /dev/sdb is effectively the first device, and install the MBR and Grub on them.  After this, reboot and do not forget to edit your grub.conf.  Change the line that reads

to

This way, Grub will once again boot from its first device as it was meant to.

Installing Grub and the MBR on the second device of your RAID 1 array should be done in any case, even if you don’t experience USB stick troubles.  If you do not, in case of a failure of your first hard disk, your system will be unable to boot until you install the MBR and Grub on your second hard disk.