High Availability Linux Cluster for SQUID Proxy using DRBD and HeartBeat on CentOS /RHEL / Fedora by Babar Zahoor
High Availability Linux Cluster Setup using DRBD and Heart Beat on CentOS 5.x 6.x /RHEL 5.x 6.X/ Fedora
#### This How To belongs to My video on High Availability Squid Cache using DRBD and HeartBeat ####
OS CentOS 5.3 on both machines.redhat-logo1
We will setup for Transparent squid on High Availability Cluster.
Packages are available on CentOS extras repository.
Our Scenario
We have two servers
baber 192.168.1.50 Primary server
farrukh 192.168.1.60 Secondry server
Setup for ip to name resolve ## we don’t have DNS we need this step ##
Basic Setup Configuration.
[[email protected] ~]# vim /etc/hosts 192.168.1.50 baber 192.168.1.60 farrukh save & exit [[email protected] ~]# ping baber PING baber (192.168.1.50) 56(84) bytes of data. 64 bytes from baber (192.168.1.50): icmp_seq=1 ttl=64 time=4.15 ms 64 bytes from baber (192.168.1.50): icmp_seq=2 ttl=64 time=0.126 ms 64 bytes from baber (192.168.1.50): icmp_seq=3 ttl=64 time=1.88 ms [1]+ Stopped ping baber [[email protected] ~]# ping farrukh PING farrukh (192.168.1.60) 56(84) bytes of data. 64 bytes from farrukh (192.168.1.60): icmp_seq=1 ttl=64 time=1.32 ms 64 bytes from farrukh (192.168.1.60): icmp_seq=2 ttl=64 time=0.523 ms 64 bytes from farrukh (192.168.1.60): icmp_seq=3 ttl=64 time=1.79 ms [2]+ Stopped ping farrukh
[[email protected] ~]#
[[email protected] ~]# scp /etc/hosts 192.168.1.60:/etc/hosts
On Node1 servers:
Please before going to next step, stop unwanted services on both servers
[[email protected] ~]# /etc/init/sendmail stop [[email protected] ~]# chkconfig --level 235 sendmail off [[email protected] ~]# iptables -F [[email protected] ~]#service iptables save [[email protected] ~]# /etc/init/sendmail stop [[email protected] ~]# chkconfig --level 235 sendmail off [[email protected] ~]# iptables -F [[email protected] ~]#service iptables save [[email protected] ~]# rpm -qa | grep ntp ntp-4.2.2p1-9.el5.centos.1 [[email protected] ~]#
Then we need to open ntp server configuration file.
[[email protected] ~]#vi /etc/ntp.conf # Permit time synchronization with our time source, but do not # permit the source to query or modify the service on this system. restrict default kod nomodify notrap nopeer noquery # Permit all access over the loopback interface. This could # be tightened as well, but to do so would effect some of # the administrative functions. restrict 127.0.0.1 # Hosts on local network are less restricted. #restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap # Use public servers from the pool.ntp.org project. # Please consider joining the pool (http://www.pool.ntp.org/join.html). ### Edited By Babar Zahoor Jun 16 2009 ### #server 0.centos.pool.ntp.org #server 1.centos.pool.ntp.org #server 2.centos.pool.ntp.org #broadcast 192.168.1.255 key 42 # broadcast server #broadcastclient # broadcast client #broadcast 224.0.1.1 key 42 # multicast server #multicastclient 224.0.1.1 # multicast client #manycastserver 239.255.254.254 # manycast server #manycastclient 239.255.254.254 key 42 # manycast client # Undisciplined Local Clock. This is a fake driver intended for backup # and when no outside source of synchronized time is available. ########## for server use this and on clients comment this and use server serverIP ################## server 127.127.1.0 # local clock #fudge 127.127.1.0 stratum 10 # Drift file. Put this in a directory which the daemon can write to. # No symbolic links allowed, either, since the daemon updates the file # by creating a temporary in the same directory and then rename()'ing # it to the file. # driftfile /var/lib/ntp/drift # Key file containing the keys and key identifiers used when operating # with symmetric key cryptography. # Specify the key identifiers which are trusted. # trustedkey 4 8 42 # Specify the key identifier to use with the ntpdc utility. # requestkey 8 # Specify the key identifier to use with the ntpq utility. #controlkey 8 keys /etc/ntp/keys save quit. [[email protected] ~]# [[email protected] ~]# /etc/init.d/ntpd start [[email protected] ~]# chkconfig --level 235 ntpd on [[email protected] ~]# vim ntp.conf # Permit time synchronization with our time source, but do not # permit the source to query or modify the service on this system. restrict default kod nomodify notrap nopeer noquery # Permit all access over the loopback interface. This could # be tightened as well, but to do so would effect some of # the administrative functions. #restrict 127.0.0.1 #estrict -6 ::1 # Hosts on local network are less restricted. #restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap # Use public servers from the pool.ntp.org project. # Please consider joining the pool (http://www.pool.ntp.org/join.html). server 192.168.1.50 ### add this line on second server ### #server 0.centos.pool.ntp.org #server 1.centos.pool.ntp.org #server 2.centos.pool.ntp.org #broadcast 192.168.1.255 key 42 # broadcast server #broadcastclient # broadcast client #broadcast 224.0.1.1 key 42 # multicast server #multicastclient 224.0.1.1 # multicast client #manycastserver 239.255.254.254 # manycast server #manycastclient 239.255.254.254 key 42 # manycast client # Undisciplined Local Clock. This is a fake driver intended for backup # and when no outside source of synchronized time is available. #server 127.127.1.0 # local clock ##### ##### #fudge 127.127.1.0 stratum 10 # Drift file. Put this in a directory which the daemon can write to. # No symbolic links allowed, either, since the daemon updates the file # by creating a temporary in the same directory and then rename()'ing # it to the file. driftfile /var/lib/ntp/drift # Key file containing the keys and key identifiers used when operating # with symmetric key cryptography. keys /etc/ntp/keys # Specify the key identifiers which are trusted. #trustedkey 4 8 42 # Specify the key identifier to use with the ntpdc utility. #requestkey 8 # Specify the key identifier to use with the ntpq utility. #controlkey 8 save & exit [[email protected] ~]# /etc/init.d/ntpd start [[email protected] ~]# chkconfig --level 235 ntpd on [[email protected] ~]# ntpdate -u 192.168.1.50 [[email protected] ~]# watch ntpq -p -n [[email protected] ~]# watch ntpq -p -n
PARTITION SETUP On Both Servers.
Partition setup on both server identical same with fdisk here we have 3GB disks on both servers, here we will setup partition for HA Cluster Servers. We need to create LVM partitions on both machines, we will explain one server named as farrukh.
[[email protected] ~]# fdisk -l [[email protected] ~]# fdisk /dev/sdb [[email protected] ~]# fdisk /dev/sd sda sda1 sda2 sdb sdb1
[[email protected] ~]# fdisk /dev/sdb Command (m for help): m Command action a toggle a bootable flag b edit bsd disklabel c toggle the dos compatibility flag d delete a partition l list known partition types m print this menu n add a new partition o create a new empty DOS partition table p print the partition table q quit without saving changes s create a new empty Sun disklabel t change a partition's system id u change display/entry units v verify the partition table w write table to disk and exit x extra functionality (experts only) Command (m for help): p Disk /dev/sdb: 4294 MB, 4294967296 bytes 255 heads, 63 sectors/track, 522 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sdb1 1 522 4192933+ 8e Linux LVM Command (m for help): d Selected partition 1 Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 1 First cylinder (1-522, default 1): Using default value 1 Last cylinder or +size or +sizeM or +sizeK (1-522, default 522): +4000M Command (m for help): p Disk /dev/sdb: 4294 MB, 4294967296 bytes 255 heads, 63 sectors/track, 522 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sdb1 1 487 3911796 83 Linux Command (m for help): t Selected partition 1 Hex code (type L to list codes): 8e Changed system type of partition 1 to 8e (Linux LVM) Command (m for help): p Disk /dev/sdb: 4294 MB, 4294967296 bytes 255 heads, 63 sectors/track, 522 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sdb1 1 487 3911796 8e Linux LVM Command (m for help): Command (m for help): w [[email protected] ~]# partprobe
New Create Physical Volume for LVM this is second step for LVM partition
[[email protected] ~]# pvcreat /dev/sdb1 Create Volume Group with this command [[email protected] ~]# vgcreate vgdrbd /dev/sdb1 Create Logical volume partition [[email protected] ~]# lvcreate -n lvdrbd /dev/mapper/vgdrbd -L +4000M
Note: Create LVM on Both servers identical same ……………….
Note:Please also add these three values in sysctl.conf
[[email protected] ~]#vi /etc/sysctl.conf net.ipv4.conf.eth0.arp_ignore = 1 net.ipv4.conf.all.arp_announce = 2 net.ipv4.conf.eth0.arp_announce = 2 save & quit [[email protected] ~]# sysctl -p net.ipv4.ip_forward = 0 net.ipv4.conf.default.rp_filter = 1 net.ipv4.conf.eth0.arp_ignore = 1 net.ipv4.conf.all.arp_announce = 2 net.ipv4.conf.eth0.arp_announce = 2 net.ipv4.conf.default.accept_source_route = 0 kernel.sysrq = 0 kernel.core_uses_pid = 1 net.ipv4.tcp_syncookies = 1 kernel.msgmnb = 65536 kernel.msgmax = 65536 kernel.shmmax = 4294967295 kernel.shmall = 268435456 [[email protected] ~]#
DRBD Setup
Now install drbd82 & kmod-drbd82 or latest available package rpms using yum command on both servers
[[email protected] ~]#yum install -y drbd82 kmod-drbd82
Now open /etc/drbd.conf using any text editor, I am using here VIM for this purpose
[[email protected] ~]#vim /etc/drbd.conf global { usage-count yes; } common { syncer { rate 10M; } } resource r0 { protocol C; handlers { pri-on-incon-degr "echo o > /proc/sysrq-trigger ; halt -f"; pri-lost-after-sb "echo o > /proc/sysrq-trigger ; halt -f"; local-io-error "echo o > /proc/sysrq-trigger ; halt -f"; outdate-peer "/usr/lib/heartbeat/drbd-peer-outdater -t 5"; } startup { } disk { on-io-error detach; } net { after-sb-0pri disconnect; after-sb-1pri disconnect; after-sb-2pri disconnect; rr-conflict disconnect; } syncer { rate 10M; al-extents 257; } ####add the below information according to your HA server setup#### on baber { device /dev/drbd0; disk /dev/VGdrbd/lvdrbd; address 192.168.1.50:7788; meta-disk internal; } on node2 { device /dev/drbd0; disk /dev/VGdrbd/lvdrbd; address 192.168.1.60:7788; meta-disk internal; } } save it........ [[email protected] ~]# [[email protected] ~]# scp /etc/drbd.conf farrukh:/etc/drbd.conf Now we need to run modules on both servers to run drbd
Load DRBD module both nodes:
[[email protected] ~]# modprobe drbd [[email protected] ~]# echo "modprobe drbd" >> /etc/rc.local [[email protected] ~]# modprobe drbd [[email protected] ~]# echo "modprobe drbd" >> /etc/rc.local ##### Please run these command on both servers ######
[[email protected] ~]#drbdadm create-md r0 [[email protected] ~]#drbdadm create-md r0 [[email protected] ~]#drbdadm attach r0 [[email protected] ~]#drbdadm attach r0 [[email protected] ~]#drbdadm syncer r0 [[email protected] ~]#drbdadm syncer r0 [[email protected] ~]#drbdadm connect r0 [[email protected] ~]#drbdadm connect r0 Please run below command on Primary Node only
[[email protected] ~]#drbdadm -- --overwrite-data-of-peer primary r0
Now please run below commands on both Nodes:
[[email protected] ~]#drbdadm up all [[email protected] ~]#drbdadm up all Please run below command on Primary Node only
[[email protected] ~]#drbdadm -- primary all [[email protected] ~]#watch cat /proc/drbd
Please run this command on Primary Node only
[[email protected] ~]#mkfs.ext3 /dev/drbd0
[[email protected] ~]#mkdir /data/
[[email protected] ~]#mount /dev/drbd0 /data/ [[email protected] ~]# [[email protected] ~]# df -hk Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/VolGroup00-LogVol00 5967432 2625468 3033948 47% / /dev/sda1 101086 12074 83793 13% /boot tmpfs 257720 0 257720 0% /dev/shm /dev/drbd0 4031516 107600 3719128 3% /data [[email protected] ~]# Please run this command on Secondary Node only and no need to mount the partition here
[[email protected] ~]#mkdir /data
Heartbeat Setup:
Install heartbeat package using yum and for installing these packages Internet connection is required or you can configure yum repository on your local machine with extras where the servers / both nodes will be able to found these packages
[[email protected] ~]#yum install -y heartbeat heartbeat-pils heartbeat-stonith heartbeat-devel
Now we will configure HA configuration file, if you can’t find this file please create new file and copy below test into that file
[[email protected] ~]#vim /etc/ha.d/ha.cf ## Create this file and copy this text ## logfacility local0 keepalive 2 #deadtime 30 # USE THIS!!! deadtime 10 # we use two heartbeat links, eth2 and serial 0 bcast eth0 ####### We can use eth1 instead of eth0 it's better option ######## #serial /dev/ttyS0 baud 19200 auto_failback on ################## Active Active state ################# node baber node farrukh save & quit. Again below configuration on same primary server "Baber"
[[email protected] ~]#vi /etc/ha.d/haresources baber IPaddr::192.168.1.190/24/eth0 drbddisk::r0 Filesystem::/dev/drbd0::/data::ext3 squid Now please configure HARESOURCES file on secondary Server farrukh:
[[email protected] ~]#vi /etc/ha.d/haresources
farrukh IPaddr::192.168.1.190/24/eth0 drbddisk::r0 Filesystem::/dev/drbd0::/data::ext3 squid Below configuration for authorization to access the remote resources on Both Servers:
[[email protected] ~]#vi /etc/ha.d/authkeys auth 3 3 md5 redhat ######### Use Long name as password ######### Again on both Nodes
[[email protected] ~]#chmod 600 /etc/ha.d/authkeys [[email protected] ~]#scp /etc/ha.d/authkeys farrukh:/etc/ha.d/authkeys [[email protected] ~]#chkconfig --level 235 heartbeat on
Note: if you have problem mounting /dev/drbd0 on /data then run these commands to check the status if you found the drbddisk stopped then start it.
[[email protected] ~]#/etc/ha.d/resource.d/drbddisk r0 status [[email protected] ~]#/etc/ha.d/resource.d/drbddisk r0 start [[email protected] ~]#/etc/ha.d/resource.d/drbddisk r0 restart [[email protected] data]# service drbd status drbd driver loaded OK; device status: version: 8.0.13 (api:86/proto:86) GIT-hash: ee3ad77563d2e87171a3da17cc002ddfd1677dbe build by [email protected], 2008-10-02 13:31:44 m:res cs st ds p mounted fstype 0:r0 Connected Primary/Secondary UpToDate/UpToDate C /data ext3
We can see that servers are in Primary/Secondary state and working well with /data directory mounted and to takeover the machine node1 to node2 forcefully
[[email protected] ~]#/usr/lib/heartbeat/hb_takeover Now configuration of service on both server and we are configuring Squid Transparent
[[email protected] ~]#vim /etc/sysctl.conf # Controls IP packet forwarding net.ipv4.ip_forward = 1 #### If it is 0 make it 1 for packet forwarding #### save it Then
[[email protected] ~]#scp /etc/sysctl.conf farrukh:/etc/sysctl.conf [[email protected] ~]#sysctl -p [[email protected] ~]# sysctl -p [[email protected] ~]#yum install -y squid [[email protected] ~]#vim /etc/squid/squid.conf search these options using / and edit as required http_port 3128 transparent acl our_networks src 192.168.1.0/24 192.168.2.0/24 http_access allow our_networks cache_dir ufs /data/squid 1000 32 256 ##### cache directories must be at /data/squid ##### visible_hostname squid.ha-cluster.com save & exit [[email protected] ~]# cd /data [[email protected] ~]# mkdir squid [[email protected] ~]# chown squid:squid squid Note: This setup is required on only on primary server i.e baber
[[email protected] ~]#scp /etc/squid/squid.conf farrukh:/etc/squid/squid.conf [[email protected] ~]#iptables -F [[email protected] ~]#iptables -t nat -A PREROUTING -p tcp -i eth0 --dport 80 -j REDIRECT --to-port 3128 [[email protected] ~]#iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE [[email protected] ~]#service iptables save [[email protected] ~]#iptables -F [[email protected] ~]#iptables -t nat -A PREROUTING -p tcp -i eth0 --dport 80 -j REDIRECT --to-port 3128 [[email protected] ~]#iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE [[email protected] ~]#service iptables save On both servers
[[email protected] ~]#/etc/init/heartbeat start [[email protected] ~]#ifconfig [[email protected] ~]#tail -f /var/log/squid/access.log [[email protected] ~]#/etc/init/heartbeat start [[email protected] ~]#ifconfig Note: We must use VIP/Service IP which we define in heartbeat i.e. 192.168.1.190 as default gateway IP for accessing the internet transparently.
ALHAMDULILLAH We have Done it………….