FirstServed Tech Blog - FirstServed and the Art of Server Tuning

SNMPD has problems with xen net drivers.
Why?
Xen net emulates a blank MAC address (FF:FF:FF:FF:FF:FF)
Which in it’s turn generates a faulty IPv6 address which is duplicated amongst the interfaces.
SNMPD cannot handle these duplicated addresses an detects an IP conflict.

In our Xen confidurations, we like to configure different VLANs on the Dom0 network interfaces, which are then bridged to the DomU guests.  Not only is this more secure than configuring the VLAN interfaces on the virtual machine, it’s also the only way we know of that will work with Windows guests, since on Windows you need proprietary software for Broadcom or Intel NICs in order to configure VLANs, and the Xen Virtual Network Adapter NIC certainly doesn’t provide any software to this end.

We were pretty annoyed to say the least, when we found out that the brand new XenServer 4.0 – which costs five times as much as it’s predecessor XenServer 3.2 – doesn’t allow one to add VLANs to the host interfaces.  Both the XenCenter console and the xe vlan-create command return ‘This operation is not allowed with your current license’.
Hope was not lost, however, since we figured out the following workaround:

In /etc/sysconfig/network-scripts, comment out the following line in ifcfg-eth0:

Add a new VLAN interface, called ifcfg-eth0.142:

Add the following line to /etc/sysconfig/network:

You can now test the new setup without restarting by using the following commands:

You should now have connectivity on your newly created VLAN interface.

The init scripts bundled with Xen offer little or no support for VLANs.  One way to configure VLANs is to do it at the DomU level, but that’s a trick that’s only feasible on Linux guests, since on Windows VLAN support for ethernet cards is implemented by proprietary software of the NIC vendor, and there’s certainly no such thing as Windows software for the Xen Virtual NIC.

In case the Dom0 doesn’t need active interfaces on the VLAN bridges, you can follow the solution outlined on this page: Bridging domains to tagged VLANs in Xen.

I found a much simpler solution.  The reason Xen’s network-bridge doesn’t play ball with VLAN interfaces is that at a certain point, /sbin/ifdown is called to bring down the net device – a prerequisite to rename it and reassign it to the bridge.  Ifdown however has the undesirable side effect of removing any configured VLAN’s on the device it’s bringing down, effectively deleting the interface entirely.  So when running the network-bridge script, you’ll get error messages saying the device was not found:

I modified xen-network-common.sh so ifdown is no longer called directly.  Instead of it, the ip address of the interface is flushed manually, then the device’s link is brought down, leaving the vlan configured on the interface intact.  The following script is based on Xen 3.0.3′s xen-network-common.sh script.

You can the start up the needed bridges from within your my-network-bridge script – assuming you change (network-script network-bridge) to (network-script my-network-bridge) in xend-config.sxp:

The above example gives you the choice of four different bridges for each domU, some of which are transparently bound to a VLAN.

Originally, it was possible to access the QEMU console from within a VNC viewer window, by pressing Ctrl-Alt-1.  Since this made for a major security breach – users are able to mount any file on Dom0 this way, the feature has been disabled in recent Xen releases.  That makes for a major problem, since mounting, ejecting and changing CD’s in Windows seems only possible by rebooting the Virtual Machine.

After a bit of trial and error, we came to a solution that allows the mounting and ejecting of CD’s from within Windows:

In your Xen guest config file, you must specify an empty CD device – since without it Windows will fail to recognize a CD-rom device:

Mounting a CD-rom

Execute xm block-list to view the configured bloack devices for your Xen guest:

Note the cdrom’s device number and detach it from the guest. Use the –force switch, else detaching will fail.

Now reattach the device with the correct path specified ( phy:/dev/cdrom, file:/path/to/some/iso, … ):

Unmounting a CD-rom

Eject the cdrom from Windows by right-clicking on its icon and selecting ‘Eject’.

Eject the cdrom physically from Dom0 if needed:

Remounting a CD-rom

Now’s the fun part: it appears that if you try to remount exactly the same backend device, e.g. /dev/cdrom, the Windows HVM guest will not be signalled that a new device has been inserted.  A workaround for this is to attach and to detach another device first – any will do, as long as it’s different:

Then attach your new device:

Since XenServer and XenEnterprise do not support installing the operating system on an MD software RAID device during the installation, you’ll have to undertake a few steps afterwards if you want to mirror your boot disks.

This article is largely based on Harry de Jong’s solution, as posted on the XenSource forums, with a few differences:

• There’s no need to copy mdadm.static from another server to the Xen host;
• The initrd ramdisk’s configuration is a bit simpler;
• There’s no need to swap the physical disks, which means the procedure can be accomplished remotely.

Create identical partitions to your boot disk /dev/sda on your second disk /dev/sdb

If /dev/sda looks like this:

/dev/sdb should look like this:

Do not forget to set the partition type to ‘fd’ (Linux RAID autodetect) instead of the default 83 (Linux).

Create the necessary device nodes

Create your MD RAID 1 arrays (with missing disks)

/dev/sda2 is just an empty, unmounted partition used by XenServer for upgrades.

Copy the Xen Storage Manager data over to RAID

Replace the uuid in the vgextend line above with the uuid of your own Xen SR.  Use ‘sm info’ to display your SR’s uuid.

Remove /dev/sda3 from the SR volume group and add it to the RAID array

Mount /dev/md0 and copy the filesystem to it

Make a new initrd ramdisk containing the MD RAID drivers

Modify /mnt/etc/fstab so the system will mount / from /dev/md0 instead of LABEL=/-main.  Replace "LABEL=/-main" by "/dev/md0".

Create a new boot image and uncompress it:

Since mkinitrd looks at /etc/fstab to determine what device the root volume is on, it has now added the necessary raid drivers to the new boot image.

Uncompress the current ramdisk and add the raid drivers to it

Now add the raid module from the new ramdisk and modify the init file:

Add the following lines before the second line containing "/sbin/udevstart":

Add the following lines before the line containing "echo Creating root device":

Note: if you’ve created other MD raid devices, add a ‘raidautorun’ statement for them as well.

Copy the new ramdisk to the /mnt/boot folder and add modify GRUB’s boot menu

Replace "root=LABEL=/-main" by "root=/dev/md0" in all menu entries.

Unmount /dev/md0

Set up the Master Boot Record on /dev/sdb

Modify GRUB’s boot menu to boot from the second disk

Replace the line in the first XenServer entry that says "root (hd0,0)" with "root (hd1,0)". Also replace "root=LABEL=/-main" by "root=/dev/md0" in all menu entries.  This will allow you to boot from /dev/md0 instead of /dev/sda.  In case something goes wrong, you could always reboot using one of the other entries, which still point to /dev/sda.

Reboot

If all goes well, you should see your system mounting /dev/md0 as the filesystem root.  If not, reboot using one of the other GRUB menu entries and check out the previous steps.

Change /dev/sda’s partition types from Linux to Linux RAID autodetect

The result should be something like this:

Add the missing partitons to the RAID arrays

Then wait for the sync to complete:

Copy the running RAID setup to /etc/mdadm.conf

Be sure to add a line ‘DEVICE partitions’ at the top of /etc/mdadm.conf if you want to define MD devices that span a whole disk instead of just a partition.  Without this line, MD won’t be able to detect your drives as MD components at boot.

That’s it!

What happened to our networking?

Here we were, admiring our brand new Dell Poweredge 2950 server that we were going to use as a Xen host.  Fedora Core 6 installed without a hitch, but when we tried booting the Xen kernel, we lost connectivity on the first network card for Dom0.  The second NIC, which had not been tweaked by Xen’s bridging setup, still worked as a charm.  So, regretfully, we set out to learn once again things we’d rather left unlearned…

Dell, Broadcom and the art of technical support

After googling around a bit, we found that the problem was hardly a new one.  It appears that the Broadcom Netextreme II BCM5708 NIC’s that ship with the Dell Poweredge 2950 do not play ball with Xen’s bridging setup, at least not when support for Dell’s IPMI card ( DRAC ) is enabled.  Xen does some internal tweaking with your network interfaces, and ends up assigning eth0′s mac address to a new virtual interface, which is Dom0′s eth0.  The original eth0 gets renamed to peth0, and gets a mac address of ff:ff:ff:ff:ff:fe, since in normal circumstances this interface should not be visible on the LAN.

It appears that the Broadcom BCM5708 NIC’s filter out all traffic that is not directed at the NIC’s physical mac address, at least when the NIC’s management function is enabled.  The funny thing is that this problem has been acknowledged by Dell as far back as the summer of 2006, but they haven’t seen fit to bring out a firmware update ro solve this problem.  Calling Dell Support brought us no further: there’s nothing known about an eventual release of a patch.  How the guys are going to cope with the new Suse Linux Enterprise Server 10 and Red Hat Enterprise Linux 5, both of whom proudly announce the wonders of Xen virtualisation, I don’t know.

Two solutions for the problem

The first, quick and dirty workaround, is to disable the management functions on your Broadcom NIC’s.  Since this means going without IPMI support, and forgoing remote reboot and serial text console functionality , that’s more of a choice between Scylla and Charibdis.  Anyway, this workaround is documented at:

How to disable the management function on Broadcom NetXtreme II BCM5708 NIC

We chose for a second solution, which is slightly more complex, but has the advantage of keeping IPMI connectivity intact.  Since the Broadcom NIC’s require that the physical NIC’s mac addresses remain unchanged, we had to tweak Xen’s bridging scripts to allow this.  Dom0′s network interfaces, which normally are assigned the physical NIC’s mac addresses, get a locally administered mac address out of Xen’s mac address range instead.  Beware: you should take care to assign different addresses to all Xen servers on a same subnet, or you may experience connectivity problems to your Dom0′s.

So here’s what we did:

In you Xen config file, found at /etc/xen/xend-config.sxp in our installation, change the default network script to a custom one:

Then you have to create your custom bridging script.  Place it in /etc/xen/scripts/my-network-bridge – or the directory your xen scripts reside in.  The script should contain the following lines – the second one is not strictly necessary if you only want top bridge one network interface:

Then finally, modify your network-bridge script like this – the changes are outlined in bold:

Finally, comment out the mac addresses in the ifup scripts, else your OS will complain about setting up eth0 and eth1 with different mac addresses than those mentioned in the ifup scripts:

Restart your services as needed, and you’ll be able to connect to your network interfaces without a hitch!

For faster living and greater comfort, you should use LVM 2 logical volumes for the disks you export to your Windows virtual machines.  This will allow you to backup the disks easily using LVM snapshots, and as an added bonus it’ll be really easy to clone further Windows images once you’ve installed Windows once.

The recipe goes as follows:

Taking a snapshot of your virtual disk

Assuming your initial Windows VM resides on /dev/vgdrbd0/lvdrbd0 – which it does in our installations:

This will create a 1 gigabyte LVM snapshot which we’ll use for backup.  Note that the snapshot volume doesn’t have to be as big as the source volume, since it only stores differences between itself and the original.

Backing up your snapshot using ntfsclone

The obvious way to copy your snapshot to a safe location would be to use dd – you can’t go wrong with a bit per bit copy of your volume, but we chose to use ntfsclone, which is quite a bit faster, since it supports both sparse files and its own special image format.  The image file produced by those options will only be the size of the allocated blocks on the partition, not the size of the entire partition.

ntfsclone operates on partition level, not on disk level, and it’s important to keep this in mind.  We don’t want to backup the disk /dev/vgdrbd0/lvsnapshot, but its partitions – one in this case.  Since the NTFS partition contained on our logical volume normally isn’t in use by the Linux OS, we’ll first have to add its partition tables to our known devices using kpartx:

A new entry has now been added by the device mapper: /dev/mapper/lvsnapshot1.  We’ll backup this volume using ntfsclone:

This may take considerable time, depending on the size of the partition you want to backup.  Get yourself your favourite hot beverage and watch the progress meter creep along.  Do not forget to clean up your system afterwards, else your LVM snapshot will continue to register its differences with its source volume until it fills up completely:

Restoring your Windows VM to your original partition

Restoring your image to the same volume is simply reversing the steps above.  Of course, your target logical volume should not be in use by the Xen domU when you’re restoring.

Do NOT copy your image file to /dev/vgdrbd0/lvdrbd0, since that will copy your image to your disk instead of to your partition.  The net result of this is that you’ll overwrite the first crucial 512 bytes of your disk, which contain your MBR (master boot record) and parition table.

Cloning your Windows VM to another disk

Your target volume should be of exactly the same size as the source volume you’re cloning, and should contain the same partition info.  Else, NTFS may fail booting since the partition’s boot sector will not correlate to the partition layout on the new disk.  There are ways and means to work around this problem, using hex editor magic, but that’s outside the scope of this article.

In order to get an exact copy of your original disk, create an LVM volume of the same size as the original (assuming your original volume had a size of 2.5 gigabyte ):

Since no Windows Install was run on this disk, all you have at this moment is an empty disk, which is missing among others its MBR and partition table.

Copy the first 512 bytes of your original volume to a file using dd and copy them to your newly created logical volume.  They contain the disk’s MBR and partition table.  An alternative would be to copy the configuration using fdisk, but hey, why bother since both disks are identical!

After this, proceed as when restoring to your original partition:

Et voilà!