Highly Available Linux Cluster for (LAMP) Apache, MySQL, PHP using DRBD and HeartBeat on CentOS / RedHat / Fedora

         

Disclaimer:
The following steps are performed on Virtual machines and tested successfully.
It is advised to test first on raw / virtual machines and then in production environment.

We will setup for Apache Webserver and MySQL on High Availability Cluster (2 Nodes).

Test Scenario:

We have two virtual machines running on Oracle VirtualBox:

OS: CentOS 5.10
RAM: 512MB
HDD: 2 virtual hard disks (8GB + 3GB)

1. cos-5-1 192.168.1.8 Primary server

2. cos-5-2 192.168.1.9 Secondry server

Setup for IP to name resolve

## we don’t have DNS we need this step ##

Basic Setup Configuration.

[root@cos-5-1 ~]# vim /etc/hosts

192.168.1.8 cos-5-1
 192.168.1.9 cos-5-2

save & exit

[root@cos-5-1 ~]# ping cos-5-1
 PING cos-5-1 (192.168.1.8) 56(84) bytes of data.
 64 bytes from cos-5-1 (192.168.1.8): icmp_seq=1 ttl=64 time=4.15 ms
 64 bytes from cos-5-1 (192.168.1.8): icmp_seq=2 ttl=64 time=0.126 ms
 64 bytes from cos-5-1 (192.168.1.8): icmp_seq=3 ttl=64 time=1.88 ms

[1]+ Stopped ping cos-5-1
 [root@cos-5-1 ~]# ping cos-5-2
 PING cos-5-2 (192.168.1.9) 56(84) bytes of data.
 64 bytes from cos-5-2 (192.168.1.9): icmp_seq=1 ttl=64 time=1.32 ms
 64 bytes from cos-5-2 (192.168.1.9): icmp_seq=2 ttl=64 time=0.523 ms
 64 bytes from cos-5-2 (192.168.1.9): icmp_seq=3 ttl=64 time=1.79 ms

[2]+ Stopped ping cos-5-2
 [root@cos-5-1 ~]#

[root@cos-5-1 ~]# scp /etc/hosts 192.168.1.9:/etc/hosts

On Node1 servers:

stop unwanted services on both servers

[root@cos-5-1 ~]# /etc/init/sendmail stop

[root@cos-5-1 ~]# chkconfig --level 235 sendmail off

[root@cos-5-1 ~]# iptables -F

[root@cos-5-1 ~]# service iptables save

[root@cos-5-2 ~]# /etc/init/sendmail stop

[root@cos-5-2 ~]# chkconfig --level 235 sendmail off

[root@cos-5-2 ~]# iptables -F

[root@cos-5-2 ~]# service iptables save

=========================================
NTP Configuration
=========================================

[root@cos-5-1 ~]# rpm -qa | grep ntp
 ntp-4.2.2p1-17.el5.centos

[root@cos-5-1 ~]# vi /etc/ntp.conf

Change as following.

### Commented By Ahmad Adnan on May 31 2014 ###
 #server 0.centos.pool.ntp.org
 #server 1.centos.pool.ntp.org
 #server 2.centos.pool.ntp.org

server 127.127.1.0 # local clock

#fudge 127.127.1.0 stratum 10

# driftfile /var/lib/ntp/drift

keys /etc/ntp/keys

save quit.

[root@cos-5-1 ~]# /etc/init.d/ntpd start
 [root@cos-5-1 ~]# chkconfig -–level 235 ntpd on

[root@cos-5-2 ~]# rpm -qa | grep ntp
 ntp-4.2.2p1-17.el5.centos

[root@cos-5-2 ~]# vi /etc/ntp.conf

server 192.168.1.8 ### add this line on second server ###

#server 0.centos.pool.ntp.org
 #server 1.centos.pool.ntp.org
 #server 2.centos.pool.ntp.org

#server 127.127.1.0 # local clock ##### #####
 #fudge 127.127.1.0 stratum 10

driftfile /var/lib/ntp/drift

keys /etc/ntp/keys

save & exit

[root@cos-5-2 ~]# /etc/init.d/ntpd start
 [root@cos-5-2 ~]# chkconfig --level 235 ntpd on

[root@cos-5-2 ~]# ntpdate -u 192.168.1.8

[root@cos-5-2 ~]# watch ntpq -p -n

[root@cos-5-1 ~]# watch ntpq -p -n

=========================================
PARTITION SETUP On Both Servers.
=========================================
Partition setup on both server identical same with fdisk

We have 3GB disks on both servers.

Partition Setup for Cluster Servers

We need to create LVM partition

[root@cos-5-1 ~]# fdisk -l
 fdisk
 [root@cos-5-1 ~]# fdisk /dev/sdb

[root@cos-5-1 ~]# fdisk /dev/sd
 sda sda1 sda2 sdb sdb1

[root@cos-5-2 ~]# fdisk /dev/sdb

### To know more about fdisk ###

Command (m for help): m
 Command action
 a toggle a bootable flag
 b edit bsd disklabel
 c toggle the dos compatibility flag
 d delete a partition
 l list known partition types
 m print this menu
 n add a new partition
 o create a new empty DOS partition table
 p print the partition table
 q quit without saving changes
 s create a new empty Sun disklabel
 t change a partition’s system id
 u change display/entry units
 v verify the partition table
 w write table to disk and exit
 x extra functionality (experts only)

Command (m for help): p

Disk /dev/sdb: 4294 MB, 4294967296 bytes
 255 heads, 63 sectors/track, 522 cylinders
 Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
 /dev/sdb1 1 522 4192933+ 8e Linux LVM

Command (m for help): d
 Selected partition 1

Command (m for help): n
 Command action
 e extended
 p primary partition (1-4)
 p
 Partition number (1-4): 1
 First cylinder (1-391, default 1):
 Using default value 1
 Last cylinder or +size or +sizeM or +sizeK (1-391, default 391): 391

Command (m for help): p

Disk /dev/sdb: 3221 MB, 3221225472 bytes
 255 heads, 63 sectors/track, 391 cylinders
 Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
 /dev/sdb1 1 366 2939863+ 83 Linux

Command (m for help): t
 Selected partition 1
 Hex code (type L to list codes): 8e
 Changed system type of partition 1 to 8e (Linux LVM)

Command (m for help): p

Disk /dev/sdb: 4294 MB, 4294967296 bytes
 255 heads, 63 sectors/track, 522 cylinders
 Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
 /dev/sdb1 1 487 3911796 8e Linux LVM

Command (m for help):

Command (m for help): w

[root@cos-5-1 ~]# partprobe

=======================================================

Create Physical Volume for LVM this is second step for LVM partition.

[root@cos-5-1 ~]# pvcreat /dev/sdb1

Create Volume Group with this command

[root@cos-5-1 ~]# vgcreate vgdrbd /dev/sdb1

Create Logical volume partition

[root@cos-5-1 ~]# lvcreate -n lvdrbd /dev/mapper/vgdrbd -L +293M

Rounding up size to full physical extent 296.00 MB
 Logical volume "lvdrbd" created

Note: Create LVM on Both servers identical same ……………….

=======================================================
add these three values in sysctl.conf

[root@cos-5-1 ~]#vi /etc/sysctl.conf

net.ipv4.conf.eth0.arp_ignore = 1

net.ipv4.conf.all.arp_announce = 2

net.ipv4.conf.eth0.arp_announce = 2

save & quit

[root@cos-5-1 ~]# sysctl -p

net.ipv4.ip_forward = 0
 net.ipv4.conf.default.rp_filter = 1
 net.ipv4.conf.default.accept_source_route = 0
 kernel.sysrq = 0
 kernel.core_uses_pid = 1
 net.ipv4.tcp_syncookies = 1
 kernel.msgmnb = 65536
 kernel.msgmax = 65536
 kernel.shmmax = 4294967295
 kernel.shmall = 268435456
 net.ipv4.conf.eth0.arp_ignore = 1
 net.ipv4.conf.all.arp_announce = 2
 net.ipv4.conf.eth0.arp_announce = 2
 [root@cos-5-1 ~]#

=========================================
DRBD Setup
(Distributed Replicated Block Device)
=========================================

Install drbd82 & kmod-drbd82 rpms using yum command.

[root@cos-5-1 ~]#yum install -y drbd82 kmod-drbd82

open /etc/drbd.conf

[root@cos-5-1 ~]#vim /etc/drbd.conf

global {
 usage-count yes;
 }

common {
 syncer { rate 10M; }
 }

resource r0 {
 protocol C;
 handlers {
 pri-on-incon-degr "echo o > /proc/sysrq-trigger ; halt -f”;
 pri-lost-after-sb "echo o > /proc/sysrq-trigger ; halt -f”;
 local-io-error "echo o > /proc/sysrq-trigger ; halt -f”;
 outdate-peer "/usr/lib/heartbeat/drbd-peer-outdater -t 5";
 }

startup {
 }

disk {
 on-io-error detach;
 }

net {
 after-sb-0pri disconnect;
 after-sb-1pri disconnect;
 after-sb-2pri disconnect;
 rr-conflict disconnect;
 }

syncer {
 rate 10M;
 al-extents 257;
 }

on cos-5-1 {
 device /dev/drbd0;
 disk /dev/vgdrbd/lvdrbd;
 address 192.168.1.8:7788;
 meta-disk internal;
 }

on node2 {
 device /dev/drbd0;
 disk /dev/vgdrbd/lvdrbd;
 address 192.168.1.9:7788;
 meta-disk internal;
 }

}

save it……..

[root@cos-5-1 ~]#
 [root@cos-5-1 ~]# scp /etc/drbd.conf cos-5-2:/etc/drbd.conf

change in cos-5-2 as following

on cos-5-2 {
 device /dev/drbd0;
 disk /dev/vgdrbd/lvdrbd;
 address 192.168.1.9:7788;
 meta-disk internal;
 }

on node2 {
 device /dev/drbd0;
 disk /dev/vgdrbd/lvdrbd;
 address 192.168.1.8:7788;
 meta-disk internal;
 }

We need to run module on both servers to run drbd

Load DRBD module both nodes:

[root@cos-5-1 ~]# modprobe drbd

[root@cos-5-1 ~]# echo "modprobe drbd” >> /etc/rc.local

[root@cos-5-2 ~]# modprobe drbd

[root@cos-5-2 ~]# echo "modprobe drbd” >> /etc/rc.local

=========================================
##### run this on both servers ######
=========================================

[root@cos-5-1 ~]#drbdadm create-md r0

[root@cos-5-2 ~]#drbdadm create-md r0

--== Thank you for participating in the global usage survey ==--
 The server's response is:

you are the 25278th user to install this version
 ...
 ....
 .....
 ......
 Writing meta data...
 initialising activity log
 NOT initialized bitmap
 New drbd meta data block sucessfully created.
 success

[root@cos-5-1 ~]#drbdadm attach r0

[root@cos-5-2 ~]#drbdadm attach r0

[root@cos-5-1 ~]#drbdadm syncer r0

[root@cos-5-2 ~]#drbdadm syncer r0

[root@cos-5-1 ~]#drbdadm connect r0

[root@cos-5-2 ~]#drbdadm connect r0

=========================================
On Primary Node only
=========================================

[root@cos-5-1 ~]#drbdadm -- --overwrite-data-of-peer primary r0

=========================================
 On both Nodes:
 =========================================
 [root@cos-5-1 ~]#drbdadm up all

[root@cos-5-2 ~]#drbdadm up all

On Primary Node only

[root@cos-5-1 ~]#drbdadm -- primary all #### ON Node one Only ####

[root@cos-5-1 ~]#watch cat /proc/drbd
 Every 2.0s: cat /proc/drbd Sat May 31 01:30:23 2014

version: 8.2.6 (api:88/proto:86-88)
 GIT-hash: 3e69822d3bb4920a8c1bfdf7d647169eba7d2eb4 build by buildsvn@c5-i386-bui
 ld, 2008-10-03 11:42:32
 0: cs:Connected st:Primary/Secondary ds:UpToDate/UpToDate C r---
 ns:1727 nr:25067 dw:26794 dr:3688 al:31 bm:36 lo:0 pe:0 ua:0 ap:0 oos:0
 =========================================
 only on cos-5-1 i.e. Primary Node
 =========================================

[root@cos-5-1 ~]#mkfs.ext3 /dev/drbd0

[root@cos-5-1 ~]#mkdir /data/

[root@cos-5-1 ~]#mount /dev/drbd0 /data/

[root@cos-5-1 ~]#df -hk

Filesystem 1K-blocks Used Available Use% Mounted on
 /dev/mapper/VolGroup00-LogVol00
 5967432 2625468 3033948 47% /
 /dev/sda1 101086 12074 83793 13% /boot
 tmpfs 257720 0 257720 0% /dev/shm
 /dev/drbd0 4031516 107600 3719128 3% /data

[root@cos-5-1 ~]#

On cos-5-2 ####### Secondry Node #######

[root@cos-5-2 ~]#mkdir /data

=========================================
Heartbeat Setup:
=========================================

Install heartbeat package using yum

Note: Internet connection is required or configure yum repository on your local machine with extras.

Server cos-5-1

[root@cos-5-1 ~]#yum install -y heartbeat heartbeat-pils heartbeat-stonith heartbeat-devel

[root@cos-5-1 ~]#vim /etc/ha.d/ha.cf ## Create this file and copy this text ##

logfacility local0
 keepalive 2
 deadtime 10
 bcast eth0
 #serial /dev/ttyS0
 baud 19200

### Active Active state ###
 auto_failback on
 node cos-5-1
 node cos-5-2

save & quit.

[root@cos-5-1 ~]#vi /etc/ha.d/haresources

cos-5-1 IPaddr::192.168.1.190/24/eth0 drbddisk::r0 Filesystem::/dev/drbd0::/data::ext3 httpd mysqld

Server cos-5-2
 [root@cos-5-2 ~]#yum install -y heartbeat heartbeat-pils heartbeat-stonith heartbeat-devel

[root@cos-5-2 ~]#vim /etc/ha.d/ha.cf ## Create this file and copy this text ##

logfacility local0
 keepalive 2
 deadtime 10
 bcast eth0
 #serial /dev/ttyS0
 baud 19200

### Active Active state ###
 auto_failback on
 node cos-5-1
 node cos-5-2

save & quit.

[root@cos-5-2 ~]#vi /etc/ha.d/haresources

cos-5-2 IPaddr::192.168.1.190/24/eth0 drbddisk::r0 Filesystem::/dev/drbd0::/data::ext3 httpd mysqld

On Both Servers:

[root@cos-5-1 ~]#vi /etc/ha.d/authkeys

auth 3
 3 md5 redhat

######### Use Long name as password #########

both NODE:

[root@cos-5-1 ~]#chmod 600 /etc/ha.d/authkeys

[root@cos-5-1 ~]#scp /etc/ha.d/authkeys cos-5-2:/etc/ha.d/authkeys

[root@cos-5-1 ~]#chkconfig --level 235 heartbeat on

Note: if you have problem mounting /dev/drbd0 on /data then run these commands to check the status if you found the drbddisk stopped then start it.

[root@cos-5-1 ~]#/etc/ha.d/resource.d/drbddisk r0 status
 [root@cos-5-1 ~]#/etc/ha.d/resource.d/drbddisk r0 stop
 [root@cos-5-1 ~]#/etc/ha.d/resource.d/drbddisk r0 start

[root@cos-5-1 data]# service drbd status
 drbd driver loaded OK; device status:
 version: 8.2.6 (api:88/proto:86-88)
 GIT-hash: 3e69822d3bb4920a8c1bfdf7d647169eba7d2eb4 build by buildsvn@c5-i386-build, 2008-10-03 11:42:32
 m:res cs st ds p mounted fstype
 0:r0 Connected Primary/Secondary UpToDate/UpToDate C /data ext3

we can see that servers are in Primary/Secondary state and working well with /data directory mounted.

To takeover the machine cos-5-1 to node2 forcefully.

[root@cos-5-1 ~]#/usr/lib/heartbeat/hb_takeover

=========================================
Apache and MySQL Setup:
=========================================

[root@cos-5-1 ~]#yum install -y httpd mysql-server

[root@cos-5-1 ~]#vi /etc/httpd/conf/httpd.conf

search using / and edit as following:

DocumentRoot "/data/www/html"

save & exit

[root@cos-5-1 ~]# cd /data

[root@cos-5-1 data]# mkdir -p www/html

[root@cos-5-1 data]# chown -R apache:apache www

[root@cos-5-1 data]# vi /etc/my.cnf

change the path as following:

#datadir=/var/lib/mysql

datadir=/data/mysql

#socket=/var/lib/mysql/mysql.sock

socket=/data/mysql/mysql.sock

[root@cos-5-1 data]# mkdir mysql

[root@cos-5-1 data]# chown -R mysql:mysql mysql

Note: This is required on only primary server i.e cos-5-1

[root@cos-5-2 ~]#yum install -y httpd mysql-server

[root@cos-5-1 ~]#scp /etc/httpd/conf/httpd.conf cos-5-2:/etc/httpd/conf/httpd.conf
 [root@cos-5-1 ~]#scp /etc/my.cnf cos-5-2:/etc/my.cnf

On both servers

[root@cos-5-1 ~]#/etc/init/heartbeat start

[root@cos-5-1 ~]#ifconfig
 eth0 Link encap:Ethernet HWaddr
 inet addr:192.168.1.8 Bcast:192.168.1.255 Mask:255.255.255.0
 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
 RX packets:45718 errors:0 dropped:0 overruns:0 frame:0
 TX packets:136777 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:1000
 RX bytes:4187282 (3.9 MiB) TX bytes:200501202 (191.2 MiB)

eth0:0 Link encap:Ethernet HWaddr 08:00:27:46:B1:8E
 inet addr:192.168.1.190 Bcast:192.168.1.255 Mask:255.255.255.0
 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

[root@cos-5-2 ~]#/etc/init.d/heartbeat start

[root@cos-5-2 ~]#ifconfig

Note: We must use Virtual IP/Service IP which we define in heartbeat i.e. 192.168.1.190 as default IP for accessing Webserver and MySQL.

ALHAMDULILLAH It’s Done.

Please note:
=========================================
When Primary Node i.e. cos-5-1 is Up
(No Virtual Interface shown on cos-5-2)
=========================================

[root@cos-5-2 ha.d]# ifconfig
 eth0 Link encap:Ethernet HWaddr 08:00:27:6A:72:75
 inet addr:192.168.1.9 Bcast:192.168.1.255 Mask:255.255.255.0
 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
 RX packets:13694 errors:0 dropped:0 overruns:0 frame:0
 TX packets:11256 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:1000
 RX bytes:3739203 (3.5 MiB) TX bytes:1669898 (1.5 MiB)

=========================================
 When Primary Node i.e. cos-5-1 is Down
 Virtual Interface shows up
 =========================================
 [root@cos-5-2 ha.d]# ifconfig
 eth0 Link encap:Ethernet HWaddr 08:00:27:6A:72:75
 inet addr:192.168.1.9 Bcast:192.168.1.255 Mask:255.255.255.0
 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
 RX packets:13779 errors:0 dropped:0 overruns:0 frame:0
 TX packets:11365 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:1000
 RX bytes:3768827 (3.5 MiB) TX bytes:1682793 (1.6 MiB)

eth0:0 Link encap:Ethernet HWaddr 08:00:27:6A:72:75
 inet addr:192.168.1.190 Bcast:192.168.1.255 Mask:255.255.255.0
 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

=========================================
When Primary Node i.e. cos-5-1 is UP
(No DRBD drive is mounted)
=========================================

[root@cos-5-2 ~]# df -h
 Filesystem Size Used Avail Use% Mounted on
 /dev/mapper/VolGroup00-LogVol00
 17G 3.1G 13G 20% /
 /dev/sda1 99M 25M 69M 27% /boot
 tmpfs 252M 0 252M 0% /dev/shm

=========================================
 When Primary Node i.e. cos-5-1 is Down
 DRBD drive is mounted on /data
 =========================================
 [root@cos-5-2 ha.d]# df -h
 Filesystem Size Used Avail Use% Mounted on
 /dev/mapper/VolGroup00-LogVol00
 17G 3.1G 13G 20% /
 /dev/sda1 99M 25M 69M 27% /boot
 tmpfs 252M 0 252M 0% /dev/shm
 /dev/drbd0 287M 19M 254M 7% /data