FirstServed Tech Blog - FirstServed and the Art of Server Tuning

A few months ago, we’ve undertaken the project of migrating all our old Pentium IV servers to a virtual platform.  Apart from the extra flexibility which this offers in terms of provisioning, we’re also benefiting from a huge power saving, since a modern Dell platform can easily host a dozen older servers.

However, after having migrated a Windows server, we observed that the attributes of folders on the filesystem had been modified.  In Windows Explorer, everything seemed all right, but when FTP’ing to certain folders, or when viewing them with the ‘dir’ command, the folder were hidden.  It appeared that the ‘system’ attribute had been set throughout the C drive.

We solved the problem by executing

attrib /S /D -S -H -R

After installing hotfix 2 on your Citrix XenServer 5.0 host, you might notice some problems with the package net-snmpd…

service snmpd start
Starting snmpd: /usr/sbin/snmpd: symbol lookup error: /usr/lib/libnetsnmpagent.so.10: undefined symbol: boot_DynaLoader
[FAILED]
 

Citrix knows of this problem, but for those who cannot wait for a fix, here’s a solution:

• Enable the CenOS yum repositories, in /etc/yum.repos.d/CentOS-Base.repo
• Remove the snmp packages and install the new CentOS packages:
  (for the sake of safety, keep a backup)
  • rpm -qa | grep snmp
net-snmp-libs-5.3.1-19.3.xs361
net-snmp-5.3.1-24.2.xs367
  • rpm -e --repackage net-snmp-5.3.1-24.2.xs367 net-snmp-libs-5.3.1-19.3.xs361
warning: /etc/snmp/snmpd.conf saved as /etc/snmp/snmpd.conf.rpmsave
  • yum install net-snmp
  • service snmpd start

Your repackaged rpms are in /var/spool/repackage/ .
These packages can be reinstalled using:

rpm -i --nomd5 package.rpm

(Note: you need 2 hyphens before nomd5: --nomd5)

Please keep in mind that deploying thirth party rpms and changing the domain controller (dom0) is not supported by Citrix!

While XenServer and DRBD do play together nicely in various setups, problems can be encountered when using a setup similar to this:
 

sda3 -> LVM -> DRBD -> LVM -> LVM

This occured in a complex XenServer setup we were building.
When booting the (paravirtualised) guest we saw this message in the systems log:

kernel: drbd6: bio would need to, but cannot, be split: (vcnt=4,idx=0,size=4096,sector=4423615)

Simultaniously the guest would complain about unreadable sectors on the drive.

The issue is actually rather simple, and documented on the internet.
But while some xensource users switch over to file-based VM’s as a work-around, this is not an option in XenServer.

So, how can this really be solved?
By simply adding an extra parameter to the drbd.conf: 

disk {
    max-bio-bvecs 1;
}

This simple switch effectively cures the whole situation.
If you want more information about this switch, please read the man files on drbd.conf

man drbd.conf

The switch and what it does is fairly well documented there.

Hi,

Lately we encountered a network issue with one of our virtualised Windows 2003 servers.
The symptoms:
Long downloads fail, there is no error, the datastream just stops.
Speed drops to zero and stays there.
But not always, it was pretty unpredictable behaviour.

At first we thought this was an IIS issue, so we began to search in that direction.
Changed some parameters, fiddled a bit with the settings…
But no, we were wrong. This was proved by installing Apache on the system and running into the very same problem.
We decided to put a sniffer between our server and another testing machine, only to discover a LOT of bad TCP/IP packets.

A bit demotivated we began a seemingly endless journey on the internet, searching for people who have a problem that resembled ours.
Until we found a post about someone with a Windows 2000 – Xenserver driver issue.
(http://forums.citrix.com/message.jspa?messageID=1337520)
Our attention headed in the direction of the Windows drivers and not to much longer the almighty Google came up with another Citrix post: (http://forums.citrix.com/thread.jspa?threadID=234961&tstart=0).

This issue kind of resembles an old problem we used to have with Xenserver 3.2 and one of your servers… Anyway, that’s not the problem here, but this does kind of prove there is something fishy with these Xen PV drivers. (Citrix people even admitted it in that page!)

So finally, we fixed it by disabling TCP/IP offloading in Windows.

This way not the Xen network card would handle the creation of the TCP checksums, Windows would.
However, this has one downside, it kind of hogs the first CPU.
I managed to get 100% cpu usage on CPU0 only by downloading stuff through IIS, so make sure not to many services are sitting on CPU0 only! (I reconfigured MS SQL to use all the cpu’s but cpu0 to prevent the server from running into problems.)

So, the key to Windows 2003 on XenServer without TCP/IP related headaches is located in the registry at:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\
DisableTaskOffload=0
(DWORD value. 0 means disable offload, 1 means enable. By default this key will not be there, you can just add it.)

For the real adventurous people out there, you can try RSS (Receive-side Scaling), which should make the other cpu’s available for handling NIC packets.

Personally, I did not test it, but keep in mind that if you DO enable it, this TCP checksum calculation can start having an impact on ALL of your cpu’s.

The key:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\
EnableRSS=1

If now only  Citrix can fix this out of the box, we can be happy!

Greets,

Koen

Hi,

After tweaking your storage repositories you might want to change the value for the default SR.
This can be done using the following command:
xe pool-param-set uuid=1ad826b1-9c9f-fd8a-3a50-b8ce3a0841af default-SR=8312500f-9962-59ff-9b0e-797987963750

Don’t forget to change the uuids to the ones that match your configuration!

Koen

Hi,

After being able to mirror the data of my guests using a DRBD setup between 2 servers, I obviously wanted to copy the config of these guest machines to.
(Otherwise the data mirror would be rather useless.)
But since I did already have all the data perfectly in sync, I did not want to do a complete export of the virtual machine.
I found an option which allows you to backup only the configuration (metadata) of a guest machine. This way you are free to setup your own data mirrorring solution.
The command:
[root@snake ~]# xe vm-export vm=12c2ee4d-16fd-5564-0589-9f302e4aa329 filename=/root/ddk –metadata

To import the data, just copy the file (here /root/ddk) over to your other server and use the following command to import your guest:
[root@jeremy ~]# xe vm-import filename=/root/ddk –metadata
You might need to manually specify the SR, in which case the command becomes:
xe vm-import filename=/root/ddk sr-uuid=8312500f-9962-59ff-9b0e-797987963750 –metadata

(Please note that wordpress doesn’t like the 2 hyphens before the word metadata and displays only one, the option is "- -metadata" without the space between those 2 hyphens.)

Don’t you just love these undocumented switches?

Koen

Hi,

Using DRBD on Xenserver is fairly simple.
Just download the Driver Development kit from Citrix for XenServer 4.1.
Import the DDK to your server.
Download your DRBD release (source code) of choice (we used version 8.2.5).
Put it inside the DDK, extract, run the magical command "make rpm" and off you go!

The packages will be ready soon. Just SCP them out of your DDK to any Xenserver you want to install them on.
No additional dependencies are needed to be resolved.

BEWARE: Citrix doesn’t support custom modules (and most likely they won’t ever support DRBD).

Kind Regards,

Koen

Hi,

In this entry I will try to explain how to create one storage repository on a device used by two Xenservers.
Such a setup may be archieved by using DRBD in a primary-primary state between the Xenservers.
The Xenserver version used for this entry is: 4.1.0

First of all, create some room, this can be done by removing your old repository.
www.firstserved.net/blog/2008/05/08/how-to-remove-a-storage-repository/

On your primary server, this command should cover about everything (PBD + SR creation):
xe sr-create name-label=SR-DRBD0 type=lvm shared=false device-config:device=/dev/drbd0

The secondary server however needs some more care:
As you can see with the command "vgdisplay", the command on the primary server has already created a volume group for the data:
[root@jeremy ~]# vgdisplay
  — Volume group —
  VG Name               VG_XenStorage-8312500f-9962-59ff-9b0e-797987963750
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  1
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                0
  Open LV               0
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               258.15 GB
  PE Size               4.00 MB
  Total PE              66087
  Alloc PE / Size       0 / 0
  Free  PE / Size       66087 / 258.15 GB
  VG UUID               0bdET3-DGv5-KSX2-4rag-vgDA-ClmD-Flq9pX

We can’t just delete this because that would break our primary server.

So we introduce the new storage repository on our secondary server instead of creating a new one using the same parameters and uuid as our primary server:
xe sr-introduce name-label=SR-DRBD0 uuid=8312500f-9962-59ff-9b0e-797987963750 shared=false type=lvm

When we now ask a listing of the Storage Repositories, we get this new entry:
uuid ( RO)                : 8312500f-9962-59ff-9b0e-797987963750
          name-label ( RW): SR-DRBD0
    name-description ( RW):
                host ( RO): <not in database>
                type ( RO): lvm
        content-type ( RO): disks
As you can see, some data is still missing.
This data will be filled up after creating the PBD.
(The command xe sr-introduce doesn’t create a PBD for the introduced SR.)
For the creation of a PBD, you will need the hosts uuid, which can be found using:
[root@jeremy ~]# xe host-list
uuid ( RO)                : 38e2b381-697d-4a11-971d-53ed1dcecd72
          name-label ( RW): jeremy.firstserved.net
    name-description ( RO): Default install of XenServer

Now we are ready to create the PBD:
xe pbd-create host-uuid=38e2b381-697d-4a11-971d-53ed1dcecd72 sr-uuid=8312500f-9962-59ff-9b0e-797987963750 device-config:device=/dev/drbd0

Now we can see the new PBD in the listing:
[root@jeremy ~]# xe pbd-list
uuid ( RO)                  : 19f7f1df-7679-bd59-604e-37110e72ad3f
             host-uuid ( RO): 38e2b381-697d-4a11-971d-53ed1dcecd72
               sr-uuid ( RO): 8312500f-9962-59ff-9b0e-797987963750
         device-config (MRO): device: /dev/drbd0
    currently-attached ( RO): false

The last action to do is to attach the PBD to the system:
xe pbd-plug uuid=19f7f1df-7679-bd59-604e-37110e72ad3f

And you should be ready to go!

Koen

Hi,

This entry describes how to remove a Xen storage repository from a XenServer 4.1 Server.
Removal of a Storage repository may be needed after damage to the disk has occured or when organising one or more xen servers in a custom setup.

When trying to remove a SR, you might encounter nasty error messages like:
[root@jeremy ~]# xe pbd-destroy uuid=4c057786-4fd3-eefa-c098-39a2e9eabbf2
You attempted an operation that was not allowed.
reason: PBD is currently attached
[root@jeremy ~]# xe sr-forget uuid=d22f5ac3-4f8d-cb7c-85ed-eb32288d8371
The SR is still connected to a host via a PBD. It cannot be destroyed.
sr: d22f5ac3-4f8d-cb7c-85ed-eb32288d8371 (Local storage)

So here is a small explanation on how to really remove your repository.
First of all, find the uuid of your Storage repositiry using  "xe sr-list"
uuid ( RO)                : e98c3da7-14f7-c2ae-3a15-c50965244cc9
          name-label ( RW): storage
    name-description ( RW):
                host ( RO): jeremy.firstserved.net
                type ( RO): lvm
        content-type ( RO): disk

Now search the corresponding PBD entry using "xe pbd-list":
uuid ( RO)                  : 4c057786-4fd3-eefa-c098-39a2e9eabbf2
             host-uuid ( RO): 38e2b381-697d-4a11-971d-53ed1dcecd72
               sr-uuid ( RO): e98c3da7-14f7-c2ae-3a15-c50965244cc9
         device-config (MRO): device: /dev/drbd0
    currently-attached ( RO): true

As you can see, the SR (Storage Repository) can easily be tracked using the uuid.

Now unplug the PBD from the SR:
xe pbd-unplug uuid=4c057786-4fd3-eefa-c098-39a2e9eabbf2

This allows you to either forget or destroy the SR:
xe sr-destroy uuid=e98c3da7-14f7-c2ae-3a15-c50965244cc9

As you can see, the storage repository has been removed from the system:
[root@jeremy ~]# xe sr-list
uuid ( RO)                : b7060810-f98f-ddfc-2c1a-e829570ed373
          name-label ( RW): Removable storage
    name-description ( RW):
                host ( RO): jeremy.firstserved.net
                type ( RO): udev
        content-type ( RO): disk

uuid ( RO)                : b89f976a-46e3-3e88-febd-685d624cffa3
          name-label ( RW): XenServer Tools
    name-description ( RW): XenServer Tools ISOs
                host ( RO): jeremy.firstserved.net
                type ( RO): iso
        content-type ( RO): iso

uuid ( RO)                : 861138a5-f359-74a7-93f1-c20fbdb1c8a0
          name-label ( RW): DVD drives
    name-description ( RW): Physical DVD drives
                host ( RO): jeremy.firstserved.net
                type ( RO): udev
        content-type ( RO): iso

Greets,

Koen

Limit dom0 to fixed size of memory.
Don’t use up all system memory.
Otherwise dom0 will suffocate the xen net driver.
Which causes loss of network connectivity for some guests.