FirstServed Tech Blog - FirstServed and the Art of Server Tuning

Here is a quick and dirty iptables based NAT solution for linux servers:

iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
iptables -A FORWARD -i eth0 -o eth1 -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A FORWARD -i eth1 -o eth0 --source 192.168.0.1 -j ACCEPT
iptables -A FORWARD -i eth1 -o eth0 -j REJECT

Here we will provide internet access to the host 192.168.0.1 behind eth1 over our own internet connection on eth0.
The ip of eth0 can be dynamic. The host 192.168.0.1 will be using the ip of eth0 on the internet.

We only accept traffic from the ip 192.168.0.1, the rest is rejected.

Oh, and don’t forget to enable ipv4 forwarding in the kernel!
Add:

net.ipv4.ip_forward = 1

to /etc/sysctl.conf and run:

sysctl -p /etc/sysctl.conf

Or just run:

echo 1 > /proc/sys/net/ipv4/ip_forward

Your Milage May Vary…

When working in a clustered environment you might encounter situations where you need to sort merged access_logs from 2 different servers for one website. In this case the timestamps may not be ordered corretly due to time offsets in the individual servers or delays in the logging proces. The ordered data is needed since some statistics processing engines expect correctly sorted data.

The sorting can easily be done by the following command:

sort -t ' ' -k 4.9,4.12n -k 4.5,4.7M -k 4.2,4.3n -k 4.14,4.15n -k 4.17,4.18n -k 4.20,4.21n access_log.1 > access_log.1.sorted

Adjust, change and implement to match your own personal preference!

If the good old and trustworthy CentOs image you are using is suddenly being VERY slow when performing a "yum upgrade" command, try this rule to speed things back up a bit:

yum clean all

Most likely yum is using some old cached information.

We encountered this problem when upgrading from CentOS 5.0 to 5.3.

We thought it would be good to share this simple command to regenerate SSH keys:

rm -fr /etc/ssh/*key*
service sshd restart

Please be careful when executing this command, it will remove files without confirmation!

In normal situations you would never need this…
However it can be usefull when cloning machines.

After installing hotfix 2 on your Citrix XenServer 5.0 host, you might notice some problems with the package net-snmpd…

service snmpd start
Starting snmpd: /usr/sbin/snmpd: symbol lookup error: /usr/lib/libnetsnmpagent.so.10: undefined symbol: boot_DynaLoader
[FAILED]
 

Citrix knows of this problem, but for those who cannot wait for a fix, here’s a solution:

• Enable the CenOS yum repositories, in /etc/yum.repos.d/CentOS-Base.repo
• Remove the snmp packages and install the new CentOS packages:
  (for the sake of safety, keep a backup)
  • rpm -qa | grep snmp
net-snmp-libs-5.3.1-19.3.xs361
net-snmp-5.3.1-24.2.xs367
  • rpm -e --repackage net-snmp-5.3.1-24.2.xs367 net-snmp-libs-5.3.1-19.3.xs361
warning: /etc/snmp/snmpd.conf saved as /etc/snmp/snmpd.conf.rpmsave
  • yum install net-snmp
  • service snmpd start

Your repackaged rpms are in /var/spool/repackage/ .
These packages can be reinstalled using:

rpm -i --nomd5 package.rpm

(Note: you need 2 hyphens before nomd5: --nomd5)

Please keep in mind that deploying thirth party rpms and changing the domain controller (dom0) is not supported by Citrix!

While XenServer and DRBD do play together nicely in various setups, problems can be encountered when using a setup similar to this:
 

sda3 -> LVM -> DRBD -> LVM -> LVM

This occured in a complex XenServer setup we were building.
When booting the (paravirtualised) guest we saw this message in the systems log:

kernel: drbd6: bio would need to, but cannot, be split: (vcnt=4,idx=0,size=4096,sector=4423615)

Simultaniously the guest would complain about unreadable sectors on the drive.

The issue is actually rather simple, and documented on the internet.
But while some xensource users switch over to file-based VM’s as a work-around, this is not an option in XenServer.

So, how can this really be solved?
By simply adding an extra parameter to the drbd.conf: 

disk {
    max-bio-bvecs 1;
}

This simple switch effectively cures the whole situation.
If you want more information about this switch, please read the man files on drbd.conf

man drbd.conf

The switch and what it does is fairly well documented there.

One of our websites is using gettext for displaying dutch and french language. Based on the host header, we define the locale/environment variable. Sometimes it seemed that the locale and the environment variables were getting lost.

After some testing we found the problem, just look at the warning on the PHP website:

"The locale information is maintained per process, not per thread. If you are running PHP on a  multithreaded server api like IIS or Apache on Windows you may experience sudden changes of  locale settings while a script is running although the script itself never called setlocale()  itself. This happens due to other scripts running in different threads of the same process  at the same time changing the processwide locale using setlocale()."

This an easy to implement yet very usefull feature.
For instance, we use it to provide our dedicated servers with a redundant path to the network.

This small walkthrough is based on CentOS, but I’m sure you’ll be able to implement it in other distributions to after having read it.

First of all:
Enable the module in /etc/modprobe.conf and pass the necessary parameters:

alias bond0 bonding
options bond0 mode=balance-alb miimon=100

More information about these parameters and the module can be found here:
http://sourceforge.net/project/showfiles.php?group_id=24692&package_id=146474 (project documentation)
http://surfnet.dl.sourceforge.net/sourceforge/bonding/bonding.txt (direct link)

Now you have actually created your bonding device, the only thing left now is to configure it:

Change directory to the network configuration scripts:

[root@server ~]# cd /etc/sysconfig/network-scripts/

Change the scripts for the underlying interfaces, these should be slaves to the bond:

[root@server network-scripts]# cat ifcfg-eth0
DEVICE=eth0
BOOTPROTO=static
ONBOOT=yes
TYPE=Ethernet
MASTER=bond0
SLAVE=yes
[root@server network-scripts]# cat ifcfg-eth1
DEVICE=eth1
BOOTPROTO=static
ONBOOT=yes
TYPE=Ethernet
MASTER=bond0
SLAVE=yes

Now it is time to configure the bond itself, for this example I’ve chosen a DHCP configuration:

[root@server network-scripts]# cat ifcfg-bond0
DEVICE=bond0
BOOTPROTO=dhcp
ONBOOT=yes
[root@server network-scripts]#

You can now restart the network and your bond will be active:

service network restart

Greets,

Koen

You want to use DRBD for mirroring your disks over the network, and put LVM on top of your DRBD disk for flexible disk management?  Try it like this:

1. Setting up DRDB
2. Setting up LVM

However, LVM is rather greedy when it comes to hogging block level devices at boot time.  So, after restarting, it will happily discover your scsi disk, md device, or whatever, and DRBD will spawn some nasty output like:

Starting DRBD resources: [ d0 d1 ioctl(,SET_DISK_CONFIG,) failed:
Invalid argument
Lower device is already mounted.

cmd /sbin/drbdsetup /dev/drbd1 disk /dev/md3 internal -1 failed!

The solution is to edit your lvm.conf to filter out the block devices you've assigned as DRBD devices.  Try modifying your LCM configuration like this ( found in /etc/lvm/lvm.conf on Fedora and Red Hat distros ):

# By default we accept every block device:
# Comment this out:
# filter = [ "a/.*/" ]

# Filter out your DRBD block devices, accept the rest:
filter = [ "r/md3", "a/.*/" ]

After this, rescan your LVM configuration by executing:

pvscan
vgscan
vgchange -a y

And finally start your DRBD daemon:

/etc/init.d drbd start