软件工程
当前位置 :首页信息技术软件工程

centos下lvs+keepalived实现负载均衡全纪录

来源: 作者: 时间:2012-11-24 点击:

试验环境

172.1.1.200 VIP
172.1.1.168 Master-LVS-Director
172.1.1.169 Backup-LVS-Director
172.1.1.54 RealServer1
172.1.1.55 RealServer2

注意:VIP(172.1.1.200)不能被占用!

一、配置LVS

1、安装ipvsadm
[root@172-1-1-168 ~]# yum install -y ipvsadm


2、安装keepalived

确认当前运行的内核
[root@172-1-1-168 ~]# ls -1 /usr/src/kernels
2.6.18-308.20.1.el5-i686


如果没有请安装linux内核请执行下面命令安装
[root@172-1-1-168 ~]# yum install -y kernel-devel


紧接着安装keepalived
[root@172-1-1-168 ~]# wget http://www.keepalived.org/software/keepalived-1.2.7.tar.gz
[root@172-1-1-168 ~]# tar -zxvf keepalived-1.2.7.tar.gz
[root@172-1-1-168 ~]# cd keepalived-1.2.7
[root@172-1-1-168 keepalived-1.2.7]# ./configure --sysconfdir=/etc/ --sbindir=/usr/sbin/ --with-kernel-dir=/usr/src/kernels/2.6.18-308.20.1.el5-i686
Keepalived configuration
------------------------
Keepalived version : 1.2.7
Compiler : gcc
Compiler flags : -g -O2 -DETHERTYPE_IPV6=0x86dd
Extra Lib : -lpopt -lssl -lcrypto
Use IPVS Framework : Yes
IPVS sync daemon support : Yes
IPVS use libnl : No
Use VRRP Framework : Yes
Use VRRP VMAC : No
SNMP support : No
Use Debug flags : No
[root@172-1-1-168 keepalived-1.2.7]# make && make install
[root@172-1-1-168 keepalived-1.2.7]# /usr/sbin/keepalived --version
Keepalived v1.2.7 (11/20,2012)


这样keepalived就安装成功了

3、配置keepalived
[root@172-1-1-168 ~]# vim /etc/keepalived/keepalived.conf

! Configuration File for keepalived

# 全局定义
global_defs {
notification_email {
admin@rschome.com
}
notification_email_from admin@rschome.com

#smtp主机地址
smtp_server 127.0.0.1
smtp_connect_timeout 30

#运行Keepalived服务器的一个标识
#发邮件时显示在邮件标题中的信息
router_id LVS_MASTER
}

#VIP
vrrp_instance VI_1 {

#指定实例的初始状态(角色)。在两台都启动时系统会马上选择priority值高的作为MASTER
state MASTER

#VT_1 实例绑定的网卡
interface eth0

#VRID 标记(0-255)
virtual_router_id 51

#优先级,BACKUP的值一定要低于MASTER
priority 100

#检查间隔
advert_int 1

#设置认证
authentication {
#认证类型
auth_type PASS
#认证密码
auth_pass 123456
}

#VIP 这个IP在发生MASTER 到 BACKUP切换时会随之add或del,所以每台服务器上可以不绑定
#虚拟地址,而都放入virtual_ipaddress块中(可以多个),keepalived会自动使用ip地址进
#行绑定(不需要依赖ifcfg-eth0),利用ip add show eth0可以看到加入的VIP
virtual_ipaddress {
172.1.1.200
}
}

#定义virtual_server (HTTP | 80)
virtual_server 172.1.1.200 80 {
delay_loop 6 # service polling的delay时间
lb_algo wlc # 调度算法
lb_kind DR # LVS工作方式
persistence_timeout 50 # 会话保持时间
protocol TCP # 协议类型(TCP|UDP)

#定义rs1,每一个rs都需要下面的一个配置段
real_server 172.1.1.54 80 {
weight 1 # 权值 默认1,0为失效
# inhibit_on_failure # 在服务器健康检查失败后不从IPVS中删除而将其权值标记为0

# TCP方式的健康检查
TCP_CHECK {
connect_timeout 10 # 连接超时时间
nb_get_retry 3 # 重试次数
delay_before_retry 3 # 重试间隔
connect_port 80 # 健康检查端口
}
}

# 定义rs2
real_server 172.1.1.55 80 {
weight 1
TCP_CHECK {
connect_timeout 10
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
}

在Backup-LVS-Director(172.1.1.169)上执行上面相同的操作!
不过需要删除state MASTER,并修改priority的值小于MASTER的,router_id也要调整以和MASTER区别!

二、配置RealServer

为了方便起见,在RealServer上编写一个启动脚本,如下:
[root@172-1-1-54 ~]# vim rs_start.sh

#!/bin/bash
#Description : RealServer Start!
#Write by : rschome.com
#Last Modefiy : 2012/11/20

VIP=172.1.1.200
LVS_TYPE=DR

startrs()
{
echo "start LVS of REALServer"

if [ "$LVS_TYPE" == "DR" ];then
/sbin/ifconfig lo:0 $VIP broadcast $VIP netmask 255.255.255.255 up
/sbin/route add -host $VIP dev lo:0
else
/sbin/ifconfig tunl0 $VIP netmask 255.255.255.255 broadcast $VIP up
/sbin/route add -host $VIP dev tunl0
fi
#echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
#echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
#echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
#echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
}

stoprs()
{
if [ "$LVS_TYPE" == "DR" ];then
/sbin/ifconfig lo:0 down
echo "close LVS Directorserver"
else
/sbin/ifconfig tunl0 down
echo "close LVS Tunnel server"
fi
#echo "0" >/proc/sys/net/ipv4/conf/lo/arp_ignore
#echo "0" >/proc/sys/net/ipv4/conf/lo/arp_announce
#echo "0" >/proc/sys/net/ipv4/conf/all/arp_ignore
#echo "0" >/proc/sys/net/ipv4/conf/all/arp_announce
}

# ============ Main ===========

case $1 in
"start")
startrs;;
"stop")
stoprs;;
"*")
echo "Usage $0 {start|stop}"
exit 1
esac


注意:上面的有关arp的设置注释掉了,好像我加了这个arp的设置就无法访问虚拟IP。
该脚本默认启动LVS/DR模式,通过修改脚本变量可以实现LVS/Tunnel模式的切换。

运行脚本进行rs1设置后执行相关的服务
[root@172-1-1-54 ~]# sh rs_start.sh start
[root@172-1-1-54 ~]# service httpd restart

rs2也执行与上面相同的操作。
启动之后查看ifconfig信息:
[root@172-1-1-54 ~]# ifconfig lo:0
lo:0 Link encap:Local Loopback
inet addr:172.1.1.200 Mask:255.255.255.255
UP LOOPBACK RUNNING MTU:16436 Metric:1

如果看到上面有虚拟IP的信息inet addr:172.1.1.200说明RealServer启动成功!

三、启动LVS

Master/Backup LVS启动keepalived服务
[root@172-1-1-168 ~]# service keepalived start

Backup-LVS 同样执行如上命令启动keepalived

查看Master-LVS上eth0接口在启动keepalived前后变化

启动keepalived之前
[root@172-1-1-168 ~]# ip add show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
link/ether 00:0c:29:46:cb:e4 brd ff:ff:ff:ff:ff:ff
inet 172.1.1.168/24 brd 172.1.1.255 scope global eth0
inet6 fe80::20c:29ff:fe46:cbe4/64 scope link
valid_lft forever preferred_lft forever


启动keepalived之后
[root@172-1-1-168 ~]# ip add show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
link/ether 00:0c:29:46:cb:e4 brd ff:ff:ff:ff:ff:ff
inet 172.1.1.168/24 brd 172.1.1.255 scope global eth0
inet 172.1.1.200/32 scope global eth0
inet6 fe80::20c:29ff:fe46:cbe4/64 scope link
valid_lft forever preferred_lft forever


查看LVS运行情况
[root@172-1-1-168 ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 172.1.1.200:80 wlc persistent 50
-> 172.1.1.55:80 Route 1 0 0
-> 172.1.1.54:80 Route 1 0 0

这样LVS就全部启动成功了!

四、负载均衡测试


这里假定两个RealServer节点配置www服务的网页文件根目录均为/var/www/html目录,然后分别执行如下操作:
在RealServer1(172.1.1.54)执行:
[root@172-1-1-54 ~]# echo "This is CentOS2." >/var/www/html/index.html

在RealServer2(172.1.1.55)执行:
[root@172-1-1-55 ~]# echo "This is CentOS3." >/var/www/html/index.html

接着打开浏览器,访问 http://172.1.1.200 这个地址,然后不断刷新此页面,如果能分别看到“This is CentOS2.”和“This is CentOS3.”就表明LVS已经在进行负载均衡了。
注意:如果在Master-LVS-Director和Backup-LVS-Director上有httpd运行请停止,否则keepalived重启后会默认访问该httpd。


如果虚拟IP不能访问,可以考虑进行以下网络及防火墙处理:
在Master-LVS-Director和Backup-LVS-Director上分别执行
防火墙处理:
[root@172-1-1-168 ~]# iptables -t nat -A PREROUTING -p tcp -d 172.1.1.200 --dport 80 -j REDIRECT
[root@172-1-1-168 ~]# iptables -I INPUT -i eth0 -p tcp --dport 80 -j ACCEPT
[root@172-1-1-168 ~]# iptables -I OUTPUT -p tcp --sport 80 -j ACCEPT
[root@172-1-1-168 ~]# /etc/rc.d/init.d/iptables save
[root@172-1-1-168 ~]# service iptables restart

网络处理:
[root@172-1-1-168 ~]# ifconfig eth0:0 172.1.1.200 netmask 255.255.255.0 broadcast 172.1.1.200
[root@172-1-1-168 ~]# route add -host 172.1.1.200 dev eth0:0
[root@172-1-1-168 ~]# echo "1" >/proc/sys/net/ipv4/ip_forward


五、故障测试


1、RS故障

切换到其中的一台rs上,如172.1.1.55,则停止172.1.1.54的httpd服务
[root@172-1-1-54 ~]# service httpd stop


这时查看Master/Backup LVS上的的日志输出
[root@172-1-1-168 ~]# tail -f /var/log/messages

Nov 20 22:15:36 172-1-1-168 Keepalived_healthcheckers[4396]: TCP connection to [172.1.1.54]:80 failed !!!
Nov 20 22:15:36 172-1-1-168 Keepalived_healthcheckers[4396]: Removing service [172.1.1.54]:80 from VS [172.1.1.200]:80
Nov 20 22:15:36 172-1-1-168 Keepalived_healthcheckers[4396]: Remote SMTP server [127.0.0.1]:25 connected.
Nov 20 22:15:38 172-1-1-168 Keepalived_healthcheckers[4396]: SMTP alert successfully sent.

[root@172-1-1-169 ~]# tail -f /var/log/messages

Nov 20 22:15:36 172-1-1-168 Keepalived_healthcheckers[4396]: TCP connection to [172.1.1.54]:80 failed !!!
Nov 20 22:15:36 172-1-1-168 Keepalived_healthcheckers[4396]: Removing service [172.1.1.54]:80 from VS [172.1.1.200]:80
Nov 20 22:15:36 172-1-1-168 Keepalived_healthcheckers[4396]: Remote SMTP server [127.0.0.1]:25 connected.
Nov 20 22:15:38 172-1-1-168 Keepalived_healthcheckers[4396]: SMTP alert successfully sent.


通过日志可以看出Master与Backup几乎同时感知了RS1服务器已经故障,并且从IPVS中移除故障rs(或者将其权值标记为0也就是不可用)。并且向指定的邮箱发送邮件,Master和Backup都会发送邮件,其邮件标题会根据router_id的值区分出Master和Backup
我在测试过程中发现两台RS切换的很慢,不知道是邮件设置的原因还是没有备份机的原因,或者是公司网络的原因都不确定!

2、Master LVS-Router故障

停止Master-LVS的keepalived服务,人为造成故障
[root@172-1-1-168 ~]# service keepalived stop


这时查看Backup-LVS的log信息
[root@172-1-1-169 ~]# tail -f /var/log/messages

Sep 3 11:23:28 localhost Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE
Sep 3 11:23:29 localhost Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE
Sep 3 11:23:29 localhost Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs.
Sep 3 11:23:29 localhost Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 172.1.1.200
Sep 3 11:23:29 localhost Keepalived_vrrp: Netlink reflector reports IP 172.1.1.200 added
Sep 3 11:23:29 localhost Keepalived_healthcheckers: Netlink reflector reports IP 172.1.1.200 added
Sep 3 11:23:34 localhost Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 172.1.1.200


由日志可以看出,Backup-LVS监测到Master-LVS故障后立即将自己的身份切换为Master然后将VIP设置到自己的eth0端口上并发送ARP广播。

现在我手动将Master-LVS的keepalived的服务起来,然后再查看Backup-LVS的log信息
[root@172-1-1-168 ~]# service keepalived start

[root@172-1-1-169 ~]# tail -f /var/log/messages

Sep 3 11:30:44 localhost Keepalived_vrrp: VRRP_Instance(VI_1) Received higher prio advert
Sep 3 11:30:44 localhost Keepalived_vrrp: VRRP_Instance(VI_1) Entering BACKUP STATE
Sep 3 11:30:44 localhost Keepalived_vrrp: VRRP_Instance(VI_1) removing protocol VIPs.
Sep 3 11:30:44 localhost Keepalived_vrrp: Netlink reflector reports IP 172.1.1.200 removed
Sep 3 11:30:44 localhost Keepalived_healthcheckers: Netlink reflector reports IP 172.1.1.200 removed
Sep 3 11:30:44 localhost Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 172.1.1.200


由Backup-LVS的日志可以看到,其检测到比自己优先级高的实例后将自己的身份切换成了Backup,然后将VIP从eth0端口移除,并发送ARP广播给Master-LVS。

六、参考文章


1、http://www.keepalived.org/documentation.html
2、http://www.linuxso.com/fuzai/9218.html
3、http://bbs.linuxtone.org/thread-1077-1-1.html

 

上一篇:
下一篇: 已经是最后一篇
最新评论共有 位网友发表了评论
发表评论
用户名: 密码: 游客发言不需要密码