Highly Available Linux Cluster for (LAMP) Apache, MySQL, PHP using DRBD and HeartBeat on CentOS / RedHat / Fedora
Disclaimer:
The following steps are performed on Virtual machines and tested successfully.
It is advised to test first on raw / virtual machines and then in production environment.
We will setup for Apache Webserver and MySQL on High Availability Cluster (2 Nodes).
Test Scenario:
We have two virtual machines running on Oracle VirtualBox:
OS: CentOS 5.10
RAM: 512MB
HDD: 2 virtual hard disks (8GB + 3GB)
1. cos-5-1 192.168.1.8 Primary server
2. cos-5-2 192.168.1.9 Secondry server
Setup for IP to name resolve
## we don’t have DNS we need this step ##
Basic Setup Configuration.
[root@cos-5-1 ~]# vim /etc/hosts
192.168.1.8 cos-5-1 192.168.1.9 cos-5-2
save & exit
[root@cos-5-1 ~]# ping cos-5-1 PING cos-5-1 (192.168.1.8) 56(84) bytes of data. 64 bytes from cos-5-1 (192.168.1.8): icmp_seq=1 ttl=64 time=4.15 ms 64 bytes from cos-5-1 (192.168.1.8): icmp_seq=2 ttl=64 time=0.126 ms 64 bytes from cos-5-1 (192.168.1.8): icmp_seq=3 ttl=64 time=1.88 ms
[1]+ Stopped ping cos-5-1 [root@cos-5-1 ~]# ping cos-5-2 PING cos-5-2 (192.168.1.9) 56(84) bytes of data. 64 bytes from cos-5-2 (192.168.1.9): icmp_seq=1 ttl=64 time=1.32 ms 64 bytes from cos-5-2 (192.168.1.9): icmp_seq=2 ttl=64 time=0.523 ms 64 bytes from cos-5-2 (192.168.1.9): icmp_seq=3 ttl=64 time=1.79 ms
[2]+ Stopped ping cos-5-2 [root@cos-5-1 ~]#
[root@cos-5-1 ~]# scp /etc/hosts 192.168.1.9:/etc/hosts
On Node1 servers:
stop unwanted services on both servers
[root@cos-5-1 ~]# /etc/init/sendmail stop
[root@cos-5-1 ~]# chkconfig --level 235 sendmail off
[root@cos-5-1 ~]# iptables -F
[root@cos-5-1 ~]# service iptables save
[root@cos-5-2 ~]# /etc/init/sendmail stop
[root@cos-5-2 ~]# chkconfig --level 235 sendmail off
[root@cos-5-2 ~]# iptables -F
[root@cos-5-2 ~]# service iptables save
=========================================
NTP Configuration
=========================================
[root@cos-5-1 ~]# rpm -qa | grep ntp ntp-4.2.2p1-17.el5.centos
[root@cos-5-1 ~]# vi /etc/ntp.conf
Change as following.
### Commented By Ahmad Adnan on May 31 2014 ### #server 0.centos.pool.ntp.org #server 1.centos.pool.ntp.org #server 2.centos.pool.ntp.org
server 127.127.1.0 # local clock
#fudge 127.127.1.0 stratum 10
# driftfile /var/lib/ntp/drift
keys /etc/ntp/keys
save quit.
[root@cos-5-1 ~]# /etc/init.d/ntpd start [root@cos-5-1 ~]# chkconfig -–level 235 ntpd on
[root@cos-5-2 ~]# rpm -qa | grep ntp ntp-4.2.2p1-17.el5.centos
[root@cos-5-2 ~]# vi /etc/ntp.conf
server 192.168.1.8 ### add this line on second server ###
#server 0.centos.pool.ntp.org #server 1.centos.pool.ntp.org #server 2.centos.pool.ntp.org
#server 127.127.1.0 # local clock ##### ##### #fudge 127.127.1.0 stratum 10
driftfile /var/lib/ntp/drift
keys /etc/ntp/keys
save & exit
[root@cos-5-2 ~]# /etc/init.d/ntpd start [root@cos-5-2 ~]# chkconfig --level 235 ntpd on
[root@cos-5-2 ~]# ntpdate -u 192.168.1.8
[root@cos-5-2 ~]# watch ntpq -p -n
[root@cos-5-1 ~]# watch ntpq -p -n
=========================================
PARTITION SETUP On Both Servers.
=========================================
Partition setup on both server identical same with fdisk
We have 3GB disks on both servers.
Partition Setup for Cluster Servers
We need to create LVM partition
[root@cos-5-1 ~]# fdisk -l fdisk [root@cos-5-1 ~]# fdisk /dev/sdb
[root@cos-5-1 ~]# fdisk /dev/sd sda sda1 sda2 sdb sdb1
[root@cos-5-2 ~]# fdisk /dev/sdb
### To know more about fdisk ###
Command (m for help): m Command action a toggle a bootable flag b edit bsd disklabel c toggle the dos compatibility flag d delete a partition l list known partition types m print this menu n add a new partition o create a new empty DOS partition table p print the partition table q quit without saving changes s create a new empty Sun disklabel t change a partition’s system id u change display/entry units v verify the partition table w write table to disk and exit x extra functionality (experts only)
Command (m for help): p
Disk /dev/sdb: 4294 MB, 4294967296 bytes 255 heads, 63 sectors/track, 522 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System /dev/sdb1 1 522 4192933+ 8e Linux LVM
Command (m for help): d Selected partition 1
Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 1 First cylinder (1-391, default 1): Using default value 1 Last cylinder or +size or +sizeM or +sizeK (1-391, default 391): 391
Command (m for help): p
Disk /dev/sdb: 3221 MB, 3221225472 bytes 255 heads, 63 sectors/track, 391 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System /dev/sdb1 1 366 2939863+ 83 Linux
Command (m for help): t Selected partition 1 Hex code (type L to list codes): 8e Changed system type of partition 1 to 8e (Linux LVM)
Command (m for help): p
Disk /dev/sdb: 4294 MB, 4294967296 bytes 255 heads, 63 sectors/track, 522 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System /dev/sdb1 1 487 3911796 8e Linux LVM
Command (m for help):
Command (m for help): w
[root@cos-5-1 ~]# partprobe
=======================================================
Create Physical Volume for LVM this is second step for LVM partition.
[root@cos-5-1 ~]# pvcreat /dev/sdb1
Create Volume Group with this command
[root@cos-5-1 ~]# vgcreate vgdrbd /dev/sdb1
Create Logical volume partition
[root@cos-5-1 ~]# lvcreate -n lvdrbd /dev/mapper/vgdrbd -L +293M
Rounding up size to full physical extent 296.00 MB Logical volume "lvdrbd" created
Note: Create LVM on Both servers identical same ……………….
=======================================================
add these three values in sysctl.conf
[root@cos-5-1 ~]#vi /etc/sysctl.conf
net.ipv4.conf.eth0.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.eth0.arp_announce = 2
save & quit
[root@cos-5-1 ~]# sysctl -p
net.ipv4.ip_forward = 0 net.ipv4.conf.default.rp_filter = 1 net.ipv4.conf.default.accept_source_route = 0 kernel.sysrq = 0 kernel.core_uses_pid = 1 net.ipv4.tcp_syncookies = 1 kernel.msgmnb = 65536 kernel.msgmax = 65536 kernel.shmmax = 4294967295 kernel.shmall = 268435456 net.ipv4.conf.eth0.arp_ignore = 1 net.ipv4.conf.all.arp_announce = 2 net.ipv4.conf.eth0.arp_announce = 2 [root@cos-5-1 ~]#
=========================================
DRBD Setup
(Distributed Replicated Block Device)
=========================================
Install drbd82 & kmod-drbd82 rpms using yum command.
[root@cos-5-1 ~]#yum install -y drbd82 kmod-drbd82
open /etc/drbd.conf
[root@cos-5-1 ~]#vim /etc/drbd.conf
global {
usage-count yes;
}
common {
syncer { rate 10M; }
}
resource r0 {
protocol C;
handlers {
pri-on-incon-degr "echo o > /proc/sysrq-trigger ; halt -f”;
pri-lost-after-sb "echo o > /proc/sysrq-trigger ; halt -f”;
local-io-error "echo o > /proc/sysrq-trigger ; halt -f”;
outdate-peer "/usr/lib/heartbeat/drbd-peer-outdater -t 5";
}
startup {
}
disk {
on-io-error detach;
}
net {
after-sb-0pri disconnect;
after-sb-1pri disconnect;
after-sb-2pri disconnect;
rr-conflict disconnect;
}
syncer {
rate 10M;
al-extents 257;
}
on cos-5-1 {
device /dev/drbd0;
disk /dev/vgdrbd/lvdrbd;
address 192.168.1.8:7788;
meta-disk internal;
}
on node2 {
device /dev/drbd0;
disk /dev/vgdrbd/lvdrbd;
address 192.168.1.9:7788;
meta-disk internal;
}
}
save it……..
[root@cos-5-1 ~]# [root@cos-5-1 ~]# scp /etc/drbd.conf cos-5-2:/etc/drbd.conf
change in cos-5-2 as following
on cos-5-2 {
device /dev/drbd0;
disk /dev/vgdrbd/lvdrbd;
address 192.168.1.9:7788;
meta-disk internal;
}
on node2 {
device /dev/drbd0;
disk /dev/vgdrbd/lvdrbd;
address 192.168.1.8:7788;
meta-disk internal;
}
We need to run module on both servers to run drbd
Load DRBD module both nodes:
[root@cos-5-1 ~]# modprobe drbd
[root@cos-5-1 ~]# echo "modprobe drbd” >> /etc/rc.local
[root@cos-5-2 ~]# modprobe drbd
[root@cos-5-2 ~]# echo "modprobe drbd” >> /etc/rc.local
=========================================
##### run this on both servers ######
=========================================
[root@cos-5-1 ~]#drbdadm create-md r0
[root@cos-5-2 ~]#drbdadm create-md r0
--== Thank you for participating in the global usage survey ==-- The server's response is:
you are the 25278th user to install this version ... .... ..... ...... Writing meta data... initialising activity log NOT initialized bitmap New drbd meta data block sucessfully created. success
[root@cos-5-1 ~]#drbdadm attach r0
[root@cos-5-2 ~]#drbdadm attach r0
[root@cos-5-1 ~]#drbdadm syncer r0
[root@cos-5-2 ~]#drbdadm syncer r0
[root@cos-5-1 ~]#drbdadm connect r0
[root@cos-5-2 ~]#drbdadm connect r0
=========================================
On Primary Node only
=========================================
[root@cos-5-1 ~]#drbdadm -- --overwrite-data-of-peer primary r0
========================================= On both Nodes: ========================================= [root@cos-5-1 ~]#drbdadm up all
[root@cos-5-2 ~]#drbdadm up all
On Primary Node only
[root@cos-5-1 ~]#drbdadm -- primary all #### ON Node one Only ####
[root@cos-5-1 ~]#watch cat /proc/drbd Every 2.0s: cat /proc/drbd Sat May 31 01:30:23 2014
version: 8.2.6 (api:88/proto:86-88) GIT-hash: 3e69822d3bb4920a8c1bfdf7d647169eba7d2eb4 build by buildsvn@c5-i386-bui ld, 2008-10-03 11:42:32 0: cs:Connected st:Primary/Secondary ds:UpToDate/UpToDate C r--- ns:1727 nr:25067 dw:26794 dr:3688 al:31 bm:36 lo:0 pe:0 ua:0 ap:0 oos:0 ========================================= only on cos-5-1 i.e. Primary Node =========================================
[root@cos-5-1 ~]#mkfs.ext3 /dev/drbd0
[root@cos-5-1 ~]#mkdir /data/
[root@cos-5-1 ~]#mount /dev/drbd0 /data/
[root@cos-5-1 ~]#df -hk
Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/VolGroup00-LogVol00 5967432 2625468 3033948 47% / /dev/sda1 101086 12074 83793 13% /boot tmpfs 257720 0 257720 0% /dev/shm /dev/drbd0 4031516 107600 3719128 3% /data
[root@cos-5-1 ~]#
On cos-5-2 ####### Secondry Node #######
[root@cos-5-2 ~]#mkdir /data
=========================================
Heartbeat Setup:
=========================================
Install heartbeat package using yum
Note: Internet connection is required or configure yum repository on your local machine with extras.
Server cos-5-1
[root@cos-5-1 ~]#yum install -y heartbeat heartbeat-pils heartbeat-stonith heartbeat-devel
[root@cos-5-1 ~]#vim /etc/ha.d/ha.cf ## Create this file and copy this text ##
logfacility local0 keepalive 2 deadtime 10 bcast eth0 #serial /dev/ttyS0 baud 19200
### Active Active state ### auto_failback on node cos-5-1 node cos-5-2
save & quit.
[root@cos-5-1 ~]#vi /etc/ha.d/haresources
cos-5-1 IPaddr::192.168.1.190/24/eth0 drbddisk::r0 Filesystem::/dev/drbd0::/data::ext3 httpd mysqld
Server cos-5-2 [root@cos-5-2 ~]#yum install -y heartbeat heartbeat-pils heartbeat-stonith heartbeat-devel
[root@cos-5-2 ~]#vim /etc/ha.d/ha.cf ## Create this file and copy this text ##
logfacility local0 keepalive 2 deadtime 10 bcast eth0 #serial /dev/ttyS0 baud 19200
### Active Active state ### auto_failback on node cos-5-1 node cos-5-2
save & quit.
[root@cos-5-2 ~]#vi /etc/ha.d/haresources
cos-5-2 IPaddr::192.168.1.190/24/eth0 drbddisk::r0 Filesystem::/dev/drbd0::/data::ext3 httpd mysqld
On Both Servers:
[root@cos-5-1 ~]#vi /etc/ha.d/authkeys
auth 3 3 md5 redhat
######### Use Long name as password #########
both NODE:
[root@cos-5-1 ~]#chmod 600 /etc/ha.d/authkeys
[root@cos-5-1 ~]#scp /etc/ha.d/authkeys cos-5-2:/etc/ha.d/authkeys
[root@cos-5-1 ~]#chkconfig --level 235 heartbeat on
Note: if you have problem mounting /dev/drbd0 on /data then run these commands to check the status if you found the drbddisk stopped then start it.
[root@cos-5-1 ~]#/etc/ha.d/resource.d/drbddisk r0 status [root@cos-5-1 ~]#/etc/ha.d/resource.d/drbddisk r0 stop [root@cos-5-1 ~]#/etc/ha.d/resource.d/drbddisk r0 start
[root@cos-5-1 data]# service drbd status drbd driver loaded OK; device status: version: 8.2.6 (api:88/proto:86-88) GIT-hash: 3e69822d3bb4920a8c1bfdf7d647169eba7d2eb4 build by buildsvn@c5-i386-build, 2008-10-03 11:42:32 m:res cs st ds p mounted fstype 0:r0 Connected Primary/Secondary UpToDate/UpToDate C /data ext3
we can see that servers are in Primary/Secondary state and working well with /data directory mounted.
To takeover the machine cos-5-1 to node2 forcefully.
[root@cos-5-1 ~]#/usr/lib/heartbeat/hb_takeover
=========================================
Apache and MySQL Setup:
=========================================
[root@cos-5-1 ~]#yum install -y httpd mysql-server
[root@cos-5-1 ~]#vi /etc/httpd/conf/httpd.conf
search using / and edit as following:
DocumentRoot "/data/www/html"
save & exit
[root@cos-5-1 ~]# cd /data
[root@cos-5-1 data]# mkdir -p www/html
[root@cos-5-1 data]# chown -R apache:apache www
[root@cos-5-1 data]# vi /etc/my.cnf
change the path as following:
#datadir=/var/lib/mysql
datadir=/data/mysql
#socket=/var/lib/mysql/mysql.sock
socket=/data/mysql/mysql.sock
[root@cos-5-1 data]# mkdir mysql
[root@cos-5-1 data]# chown -R mysql:mysql mysql
Note: This is required on only primary server i.e cos-5-1
[root@cos-5-2 ~]#yum install -y httpd mysql-server
[root@cos-5-1 ~]#scp /etc/httpd/conf/httpd.conf cos-5-2:/etc/httpd/conf/httpd.conf [root@cos-5-1 ~]#scp /etc/my.cnf cos-5-2:/etc/my.cnf
On both servers
[root@cos-5-1 ~]#/etc/init/heartbeat start
[root@cos-5-1 ~]#ifconfig eth0 Link encap:Ethernet HWaddr inet addr:192.168.1.8 Bcast:192.168.1.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:45718 errors:0 dropped:0 overruns:0 frame:0 TX packets:136777 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:4187282 (3.9 MiB) TX bytes:200501202 (191.2 MiB)
eth0:0 Link encap:Ethernet HWaddr 08:00:27:46:B1:8E inet addr:192.168.1.190 Bcast:192.168.1.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
[root@cos-5-2 ~]#/etc/init.d/heartbeat start
[root@cos-5-2 ~]#ifconfig
Note: We must use Virtual IP/Service IP which we define in heartbeat i.e. 192.168.1.190 as default IP for accessing Webserver and MySQL.
ALHAMDULILLAH It’s Done.
Please note:
=========================================
When Primary Node i.e. cos-5-1 is Up
(No Virtual Interface shown on cos-5-2)
=========================================
[root@cos-5-2 ha.d]# ifconfig eth0 Link encap:Ethernet HWaddr 08:00:27:6A:72:75 inet addr:192.168.1.9 Bcast:192.168.1.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:13694 errors:0 dropped:0 overruns:0 frame:0 TX packets:11256 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:3739203 (3.5 MiB) TX bytes:1669898 (1.5 MiB)
========================================= When Primary Node i.e. cos-5-1 is Down Virtual Interface shows up ========================================= [root@cos-5-2 ha.d]# ifconfig eth0 Link encap:Ethernet HWaddr 08:00:27:6A:72:75 inet addr:192.168.1.9 Bcast:192.168.1.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:13779 errors:0 dropped:0 overruns:0 frame:0 TX packets:11365 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:3768827 (3.5 MiB) TX bytes:1682793 (1.6 MiB)
eth0:0 Link encap:Ethernet HWaddr 08:00:27:6A:72:75 inet addr:192.168.1.190 Bcast:192.168.1.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
=========================================
When Primary Node i.e. cos-5-1 is UP
(No DRBD drive is mounted)
=========================================
[root@cos-5-2 ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/VolGroup00-LogVol00 17G 3.1G 13G 20% / /dev/sda1 99M 25M 69M 27% /boot tmpfs 252M 0 252M 0% /dev/shm
========================================= When Primary Node i.e. cos-5-1 is Down DRBD drive is mounted on /data ========================================= [root@cos-5-2 ha.d]# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/VolGroup00-LogVol00 17G 3.1G 13G 20% / /dev/sda1 99M 25M 69M 27% /boot tmpfs 252M 0 252M 0% /dev/shm /dev/drbd0 287M 19M 254M 7% /data