Highly Available WEB Servers Cluster Configuration by using rsync and heartbeat Linux based Utilities

Highly Available  WEB Servers Cluster Configuration  by using rsync and  heartbeat Linux based Utilities

Written by: Muhammad Farrukh Siddique (LPIC)

Operating System on both machines: CentOS-5.3 Final  Required RPMs: Heartbeat,rsync,httpd
Service to be mirrored: web service (httpd)
First Machine name (fqdn): node1.ha.int 
Second Machine name (fqdn): node2.ha.int
IP Address of node1: 192.168.3.224
IP Address of node2: 192.168.3.225
Default Gateway of both machines: 192.168.3.1
DNS Server: 192.168.2.11

make sure that you have httpd installed. You can use rsycn to sync any file/directory for any service but in this case we will use httpd. Now first configure Network settings for both machines and check by pinging and resolving hostnames of each other that everything is going fine. We will also generate ssh keys so that each machine can login the other machine without password. Defining a dns server is necessary if internet access is required specially in case of using Yellow Update Manager (yum). Otherwise heartbeat and rsync will work absolutely fine without any dns.

Configuring node1:

[root@node1 ~]# cat /etc/sysconfig/network
NETWORKING=yes
NETWORKING_IPV6=no
HOSTNAME=node1.ha.int
[root@node1 ~]#

[root@node1 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
BOOTPROTO=none
HWADDR=00:0c:29:3a:36:94
ONBOOT=yes
NETMASK=255.255.255.0
IPADDR=192.168.3.224
GATEWAY=192.168.3.1
TYPE=Ethernet
[root@node1 ~]#

[root@node1 ~]# cat /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 localhost.localdomain localhost
::1 localhost6.localdomain6 localhost6

192.168.3.224 node1.ha.int node1
192.168.3.225 node2.ha.int node2
[root@node1 ~]#

[root@node1 ~]# cat /etc/resolv.conf

nameserver 192.168.2.11
[root@node1 ~]#

Configuring node2:
——————

[root@node2 ~]# cat /etc/sysconfig/network
NETWORKING=yes
NETWORKING_IPV6=no
HOSTNAME=node2.ha.int
[root@node2 ~]#

[root@node2 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
BOOTPROTO=static
ONBOOT=yes
HWADDR=00:0c:29:30:5b:e3
NETMASK=255.255.255.0
IPADDR=192.168.3.225
GATEWAY=192.168.3.1
TYPE=Ethernet
[root@node2 ~]#

[root@node2 ~]# cat /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 localhost.localdomain localhost
::1 localhost6.localdomain6 localhost6

192.168.3.224 node1.ha.int node1
192.168.3.225 node2.ha.int node2
[root@node2 ~]#

[root@node2 ~]# cat /etc/resolv.conf

nameserver 192.168.2.11
[root@node2 ~]#

Configuring ssh keys for both machines:

[root@node1 ~]# ssh-keygen -t rsa
[root@node1 ~]# ssh-keygen -t dsa
[root@node1 ~]# cat /root/.ssh/*.pub > /root/.ssh/authorized_keys
Now replace the folder /root/.ssh on node2

[root@node1 ~]# scp -r /root/.ssh/ node2:/root/
this time u have to enter the password

[root@node1 ~]# ssh-keyscan -t rsa node1 node2
[root@node1 ~]# ssh-keyscan -t dsa node1 node2

Now remember to login through ssh from each machine one time to save the key permanently if your /etc/hosts file contains both fqdn and short names against each IP Address then login through ssh separately by fqdn and short names for the first time.

Install rsycn rpm package, by default it is installed in CentOS-5.3 the command is found as /usr/bin/rsync u can also check the exact path in your distribution normally by command

[root@node1 ~]# which rsync
/usr/bin/rsync
[root@node1 ~]#

Now schedule rsycin with cron daemon

[root@node1 ~]# crontab -e

*/1 * * * * /usr/bin/rsync -avz –perms –delete –links -e ssh /var/www/
node2:/var/www/

~
~
~
~
~
~
~
~
“/tmp/crontab.XXXXed1Rxx” 2L, 92C

[root@node1 ~]#

this cron file is saved under /var/spool/cron/root
*/1 shows it will check the synchronizing directories after every one minute and
if found any change then transfer the changed files/directories only.
/usr/bin/rysnc is the command to be run
-avz a for archive v for verbose and z for compression. You can adjust these
settings according to your need
–perms means retain the original permissions
–delete means delete extra files from node2 which are not found on node1
–links means copy the symbolic links as it is.
-e means define rsh command, in this case it is ssh
/var/www is the directory on node1 to be synchronized.
node2:/var/www is the path to be synchronized on node2. You can also make it
specific to some user e.g: user@node2:/var/www/

for further options see man pages
# man rsync

while configuring rsync on node2, just replace “node2” with “node1” , then the
cron job on node sould look like as:

*/1 * * * * /usr/bin/rsync -avz –perms –delete –links -e ssh /var/www/
node1:/var/www/

Now on both machines run the following commands
# chkconfig crond off
#/etc/init.d/crond stop

Installing heartbeat

On both machines

[root@node1 ~]# yum install -y heartbeat-pils heartbeat-stonith
[root@node2 ~]# yum install -y heartbeat-pils heartbeat-stonith
after installing these packages run the command below on both machines
[root@node1 ~]# yum install -y heartbeat
[root@node1 ~]# yum install -y heartbeat

Configuring Heartbeat:

Remember you have to create the required three files in /etc/ha.d directory eithor by using vi editor or someother tool
–> ha.cf
–> haresources
–> authkeys

[root@node1 ~]# cat /etc/ha.d/ha.cf
logfacility local0
keepalive 2
#deadtime 30 # USE THIS!!!
deadtime 10
bcast eth0
#serial /dev/ttyS0
baud 19200
auto_failback off
node node1.ha.int
node node2.ha.int
[root@node1 ~]#

[root@node1 ~]# cat /etc/ha.d/authkeys
auth 3
3 md5 centos
[root@node1 ~]#

[root@node1 ~]# chmod 600 /etc/ha.d/authkeys

[root@node1 ~]# scp /etc/ha.d/ha.cf node2:/etc/ha.d/
[root@node1 ~]# scp /etc/ha.d/authkeys node2:/etc/ha.d/

[root@node1 ~]# cat /etc/ha.d/haresources
node1.ha.int IPaddr::192.168.3.226/24/eth0 httpd crond
[root@node1 ~]#

[root@node2 ~]# cat /etc/ha.d/haresources
node2.ha.int IPaddr::192.168.3.226/24/eth0 httpd crond
[root@node2 ~]#

ha.cf and authkeys files must be same on both machines but haresources file contains its own hostname on each machine, rest of the haresources file is the same.

Remember to stop httpd service on both machines and also run the following commands on both machines to stop httpd and to start heartbeat at boot up.
#chkconfig httpd off
#chkconfig –level 35 heartbeat on

Now start heartbeat
[root@node1 ~]# /etc/init.d/heartbeat start
Starting High-Availability services:
2009/07/06_16:56:56 INFO: Resource is stopped
[ OK ]
[root@node1 ~]#

[root@node2 ~]# /etc/init.d/heartbeat start
Starting High-Availability services:
2009/07/06_17:23:53 INFO: Resource is stopped
[ OK ]
[root@node2 ~]#