系统环境

标题 版本 备注
系统版本 ubuntu 22.04.3 LTS (Jammy Jellyfish) -
内核版本 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux -
Keepalived v2.2.8 -
机器一 192.168.3.101 -
机器二 192.168.3.102 -
VIP 192.168.3.222 VIP 地址
VIP 192.168.3.111 VIP 地址

先决条件

在 Master 和 Backup 机器上安装好 keepalived 服务（注意：无需启动服务）

关于 keepalived 的安装请参考：在 ubuntu 22.04 中，通过源码编译安装 keepalived 服务

由于该篇内容为双活双主模式（即互为主备模式），所以接下来的配置就不区分主备

双活双主配置

机器一：192.168.3.101

1.编辑 keepalived 主配置文件 /usr/local/keepalived/etc/keepalived/keepalived.conf，配置内容为：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
$ sudo sh -c 'cat << EOF > /usr/local/keepalived/etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
   notification_email {
        root@localhost
   }
   notification_email_from veazhi@localhost
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id veazhi101
   vrrp_mcast_group4 224.0.0.111
}

vrrp_instance VI_1 {
    state MASTER
    interface enp1s0
    virtual_router_id 101
    priority 199
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.3.222
    }

    notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master"
    notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup"
    notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"
}

vrrp_instance VI_2 {
    state BACKUP
    interface enp1s0
    virtual_router_id 102
    priority 98
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 2222
    }
    virtual_ipaddress {
        192.168.3.111
    }

    notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master"
    notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup"
    notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"
}
EOF'

2.启动 keepalived 服务之前，我们先看下本机的网卡配置内容：

1
2
3
4
5
6
7
8
9
10
11
12
13
$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:99:26:6d brd ff:ff:ff:ff:ff:ff
    inet 192.168.3.101/24 brd 192.168.3.255 scope global enp1s0
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe99:266d/64 scope link 
       valid_lft forever preferred_lft forever

3.打开另一个终端，动态监控下系统日志输出，然后启动 keepalived 服务：

1
$ sudo systemctl start keepalived.service

同时，另一个终端的日志输出：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
$ tail -f /var/log/syslog
Apr  2 23:10:04 vm001-ubuntu2204 systemd[1]: Starting LVS and VRRP High Availability Monitor...
Apr  2 23:10:04 vm001-ubuntu2204 Keepalived[2116]: Starting Keepalived v2.2.8 (04/04,2023), git commit v2.2.7-154-g292b299e+
Apr  2 23:10:04 vm001-ubuntu2204 Keepalived[2116]: WARNING - keepalived was built for newer Linux 5.15.143, running on Linux 5.15.0-101-generic #111-Ubuntu SMP Tue Mar 5 20:16:58 UTC 2024
Apr  2 23:10:04 vm001-ubuntu2204 Keepalived[2116]: Command line: '/usr/local/keepalived/sbin/keepalived' '-D'
Apr  2 23:10:04 vm001-ubuntu2204 Keepalived[2116]: Opening file '/usr/local/keepalived/etc/keepalived/keepalived.conf'.
Apr  2 23:10:04 vm001-ubuntu2204 Keepalived[2116]: Configuration file /usr/local/keepalived/etc/keepalived/keepalived.conf
Apr  2 23:10:04 vm001-ubuntu2204 Keepalived[2117]: NOTICE: setting config option max_auto_priority should result in better keepalived performance
Apr  2 23:10:04 vm001-ubuntu2204 Keepalived[2117]: Starting VRRP child process, pid=2118
Apr  2 23:10:04 vm001-ubuntu2204 systemd[1]: Started LVS and VRRP High Availability Monitor.
Apr  2 23:10:04 vm001-ubuntu2204 Keepalived_vrrp[2118]: Registering Kernel netlink reflector
Apr  2 23:10:04 vm001-ubuntu2204 Keepalived_vrrp[2118]: Registering Kernel netlink command channel
Apr  2 23:10:04 vm001-ubuntu2204 Keepalived_vrrp[2118]: Script user 'keepalived_script' does not exist
Apr  2 23:10:04 vm001-ubuntu2204 Keepalived_vrrp[2118]: WARNING - script '/usr/local/keepalived/etc/keepalived/notify.sh' is not executable for uid:gid 0:0 - disabling.
Apr  2 23:10:04 vm001-ubuntu2204 Keepalived_vrrp[2118]: message repeated 5 times: [ WARNING - script '/usr/local/keepalived/etc/keepalived/notify.sh' is not executable for uid:gid 0:0 - disabling.]
Apr  2 23:10:04 vm001-ubuntu2204 Keepalived_vrrp[2118]: SECURITY VIOLATION - scripts are being executed but script_security not enabled.
Apr  2 23:10:04 vm001-ubuntu2204 Keepalived_vrrp[2118]: Assigned address 192.168.3.101 for interface enp1s0
Apr  2 23:10:04 vm001-ubuntu2204 Keepalived_vrrp[2118]: Assigned address fe80::5054:ff:fe99:266d for interface enp1s0
Apr  2 23:10:04 vm001-ubuntu2204 Keepalived_vrrp[2118]: Registering gratuitous ARP shared channel
Apr  2 23:10:04 vm001-ubuntu2204 Keepalived_vrrp[2118]: (VI_1) removing VIPs.
Apr  2 23:10:04 vm001-ubuntu2204 Keepalived_vrrp[2118]: (VI_2) removing VIPs.
Apr  2 23:10:04 vm001-ubuntu2204 Keepalived[2117]: Startup complete
Apr  2 23:10:04 vm001-ubuntu2204 Keepalived_vrrp[2118]: (VI_1) Entering BACKUP STATE (init)
Apr  2 23:10:04 vm001-ubuntu2204 Keepalived_vrrp[2118]: (VI_2) Entering BACKUP STATE (init)
Apr  2 23:10:04 vm001-ubuntu2204 Keepalived_vrrp[2118]: VRRP sockpool: [ifindex(  2), family(IPv4), proto(112), fd(12,13) multicast, address(224.0.0.111)]
Apr  2 23:10:07 vm001-ubuntu2204 Keepalived_vrrp[2118]: (VI_1) Receive advertisement timeout
Apr  2 23:10:07 vm001-ubuntu2204 Keepalived_vrrp[2118]: (VI_1) Entering MASTER STATE
Apr  2 23:10:07 vm001-ubuntu2204 Keepalived_vrrp[2118]: (VI_1) setting VIPs.
Apr  2 23:10:07 vm001-ubuntu2204 Keepalived_vrrp[2118]: (VI_1) Sending/queueing gratuitous ARPs on enp1s0 for 192.168.3.222
Apr  2 23:10:07 vm001-ubuntu2204 Keepalived_vrrp[2118]: Sending gratuitous ARP on enp1s0 for 192.168.3.222
Apr  2 23:10:07 vm001-ubuntu2204 Keepalived_vrrp[2118]: message repeated 4 times: [ Sending gratuitous ARP on enp1s0 for 192.168.3.222]
Apr  2 23:10:08 vm001-ubuntu2204 Keepalived_vrrp[2118]: (VI_2) Receive advertisement timeout
Apr  2 23:10:08 vm001-ubuntu2204 Keepalived_vrrp[2118]: (VI_2) Entering MASTER STATE
Apr  2 23:10:08 vm001-ubuntu2204 Keepalived_vrrp[2118]: (VI_2) setting VIPs.
Apr  2 23:10:08 vm001-ubuntu2204 Keepalived_vrrp[2118]: (VI_2) Sending/queueing gratuitous ARPs on enp1s0 for 192.168.3.111
Apr  2 23:10:08 vm001-ubuntu2204 Keepalived_vrrp[2118]: Sending gratuitous ARP on enp1s0 for 192.168.3.111
Apr  2 23:10:08 vm001-ubuntu2204 Keepalived_vrrp[2118]: message repeated 4 times: [ Sending gratuitous ARP on enp1s0 for 192.168.3.111]
Apr  2 23:10:12 vm001-ubuntu2204 Keepalived_vrrp[2118]: (VI_1) Sending/queueing gratuitous ARPs on enp1s0 for 192.168.3.222
Apr  2 23:10:12 vm001-ubuntu2204 Keepalived_vrrp[2118]: Sending gratuitous ARP on enp1s0 for 192.168.3.222
Apr  2 23:10:12 vm001-ubuntu2204 Keepalived_vrrp[2118]: message repeated 4 times: [ Sending gratuitous ARP on enp1s0 for 192.168.3.222]
Apr  2 23:10:13 vm001-ubuntu2204 Keepalived_vrrp[2118]: (VI_2) Sending/queueing gratuitous ARPs on enp1s0 for 192.168.3.111
Apr  2 23:10:13 vm001-ubuntu2204 Keepalived_vrrp[2118]: Sending gratuitous ARP on enp1s0 for 192.168.3.111

日志分析说明：

4.再次查看本机网卡内容：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:99:26:6d brd ff:ff:ff:ff:ff:ff
    inet 192.168.3.101/24 brd 192.168.3.255 scope global enp1s0
       valid_lft forever preferred_lft forever
    inet 192.168.3.222/32 scope global enp1s0
       valid_lft forever preferred_lft forever
    inet 192.168.3.111/32 scope global enp1s0
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe99:266d/64 scope link 
       valid_lft forever preferred_lft forever

机器二：192.168.3.102

1.编辑 keepalived 主配置文件 /usr/local/keepalived/etc/keepalived/keepalived.conf，配置内容为：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
$ sudo sh -c 'cat << EOF > /usr/local/keepalived/etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
   notification_email {
      root@localhost
   }
   notification_email_from veazhi@localhost
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id veazhi102
   vrrp_mcast_group4 224.0.0.111
}

vrrp_instance VI_1 {
    state BACKUP
    interface enp1s0
    virtual_router_id 101
    priority 198
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.3.222
    }

    notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master"
    notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup"
    notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"
}

vrrp_instance VI_2 {
    state MASTER
    interface enp1s0
    virtual_router_id 102
    priority 99
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 2222
    }
    virtual_ipaddress {
        192.168.3.111
    }

    notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master"
    notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup"
    notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"
}
EOF'

2.启动 keepalived 服务之前，我们先看下本机的网卡配置内容：

1
2
3
4
5
6
7
8
9
10
11
12
13
$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:2b:a8:e1 brd ff:ff:ff:ff:ff:ff
    inet 192.168.3.102/24 brd 192.168.3.255 scope global enp1s0
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe2b:a8e1/64 scope link 
       valid_lft forever preferred_lft forever

3.打开另一个终端，动态监控下系统日志输出，然后启动 keepalived 服务：

1
$ sudo systemctl start keepalived.service

同时，另一个终端的日志输出：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
$ tail -f /var/log/syslog
Apr  2 23:10:38 vm002-ubuntu systemd[1]: Starting LVS and VRRP High Availability Monitor...
Apr  2 23:10:38 vm002-ubuntu Keepalived[2082]: Starting Keepalived v2.2.8 (04/04,2023), git commit v2.2.7-154-g292b299e+
Apr  2 23:10:38 vm002-ubuntu Keepalived[2082]: WARNING - keepalived was built for newer Linux 5.15.143, running on Linux 5.15.0-101-generic #111-Ubuntu SMP Tue Mar 5 20:16:58 UTC 2024
Apr  2 23:10:38 vm002-ubuntu systemd[1]: Started LVS and VRRP High Availability Monitor.
Apr  2 23:10:38 vm002-ubuntu Keepalived[2082]: Command line: '/usr/local/keepalived/sbin/keepalived' '-D'
Apr  2 23:10:38 vm002-ubuntu Keepalived[2082]: Opening file '/usr/local/keepalived/etc/keepalived/keepalived.conf'.
Apr  2 23:10:38 vm002-ubuntu Keepalived[2082]: Configuration file /usr/local/keepalived/etc/keepalived/keepalived.conf
Apr  2 23:10:38 vm002-ubuntu Keepalived[2083]: NOTICE: setting config option max_auto_priority should result in better keepalived performance
Apr  2 23:10:38 vm002-ubuntu Keepalived[2083]: Starting VRRP child process, pid=2084
Apr  2 23:10:38 vm002-ubuntu Keepalived_vrrp[2084]: Registering Kernel netlink reflector
Apr  2 23:10:38 vm002-ubuntu Keepalived_vrrp[2084]: Registering Kernel netlink command channel
Apr  2 23:10:38 vm002-ubuntu Keepalived_vrrp[2084]: Script user 'keepalived_script' does not exist
Apr  2 23:10:38 vm002-ubuntu Keepalived_vrrp[2084]: WARNING - script '/usr/local/keepalived/etc/keepalived/notify.sh' is not executable for uid:gid 0:0 - disabling.
Apr  2 23:10:38 vm002-ubuntu Keepalived_vrrp[2084]: message repeated 5 times: [ WARNING - script '/usr/local/keepalived/etc/keepalived/notify.sh' is not executable for uid:gid 0:0 - disabling.]
Apr  2 23:10:38 vm002-ubuntu Keepalived_vrrp[2084]: SECURITY VIOLATION - scripts are being executed but script_security not enabled.
Apr  2 23:10:38 vm002-ubuntu Keepalived_vrrp[2084]: Assigned address 192.168.3.102 for interface enp1s0
Apr  2 23:10:38 vm002-ubuntu Keepalived_vrrp[2084]: Assigned address fe80::5054:ff:fe2b:a8e1 for interface enp1s0
Apr  2 23:10:38 vm002-ubuntu Keepalived_vrrp[2084]: Registering gratuitous ARP shared channel
Apr  2 23:10:38 vm002-ubuntu Keepalived_vrrp[2084]: (VI_1) removing VIPs.
Apr  2 23:10:38 vm002-ubuntu Keepalived_vrrp[2084]: (VI_2) removing VIPs.
Apr  2 23:10:38 vm002-ubuntu Keepalived[2083]: Startup complete
Apr  2 23:10:38 vm002-ubuntu Keepalived_vrrp[2084]: (VI_1) Entering BACKUP STATE (init)
Apr  2 23:10:38 vm002-ubuntu Keepalived_vrrp[2084]: (VI_2) Entering BACKUP STATE (init)
Apr  2 23:10:38 vm002-ubuntu Keepalived_vrrp[2084]: VRRP sockpool: [ifindex(  2), family(IPv4), proto(112), fd(12,13) multicast, address(224.0.0.111)]
Apr  2 23:10:39 vm002-ubuntu Keepalived_vrrp[2084]: (VI_2) received lower priority (98) advert from 192.168.3.101 - discarding
Apr  2 23:10:41 vm002-ubuntu Keepalived_vrrp[2084]: message repeated 2 times: [ (VI_2) received lower priority (98) advert from 192.168.3.101 - discarding]
Apr  2 23:10:41 vm002-ubuntu Keepalived_vrrp[2084]: (VI_2) Receive advertisement timeout
Apr  2 23:10:41 vm002-ubuntu Keepalived_vrrp[2084]: (VI_2) Entering MASTER STATE
Apr  2 23:10:41 vm002-ubuntu Keepalived_vrrp[2084]: (VI_2) setting VIPs.
Apr  2 23:10:41 vm002-ubuntu Keepalived_vrrp[2084]: (VI_2) Sending/queueing gratuitous ARPs on enp1s0 for 192.168.3.111
Apr  2 23:10:41 vm002-ubuntu Keepalived_vrrp[2084]: Sending gratuitous ARP on enp1s0 for 192.168.3.111
Apr  2 23:10:41 vm002-ubuntu Keepalived_vrrp[2084]: message repeated 4 times: [ Sending gratuitous ARP on enp1s0 for 192.168.3.111]
Apr  2 23:10:46 vm002-ubuntu Keepalived_vrrp[2084]: (VI_2) Sending/queueing gratuitous ARPs on enp1s0 for 192.168.3.111
Apr  2 23:10:46 vm002-ubuntu Keepalived_vrrp[2084]: Sending gratuitous ARP on enp1s0 for 192.168.3.111
Apr  2 23:10:46 vm002-ubuntu Keepalived_vrrp[2084]: message repeated 4 times: [ Sending gratuitous ARP on enp1s0 for 192.168.3.111]

与此同时， 机器一：192.168.3.101 上的日志输出：

1
2
3
Apr  2 23:10:13 vm001-ubuntu2204 Keepalived_vrrp[2118]: message repeated 4 times: [ Sending gratuitous ARP on enp1s0 for 192.168.3.111]
Apr  2 23:10:41 vm001-ubuntu2204 Keepalived_vrrp[2118]: (VI_2) Master received advert from 192.168.3.102 with higher priority 99, ours 98
Apr  2 23:10:41 vm001-ubuntu2204 Keepalived_vrrp[2118]: (VI_2) Entering BACKUP STATE

日志分析说明：

这段日志记录了 Keepalived 的 VRRP 实例在 vm001-ubuntu2204 上的一些活动。下面是日志的分析：

  1. 重复发送 Gratuitous ARP：

    • Apr 2 23:10:13 vm001-ubuntu2204 Keepalived_vrrp[2118]: message repeated 4 times: [ Sending gratuitous ARP on enp1s0 for 192.168.3.111]
    • Keepalived 实例在网络接口 enp1s0 上重复发送 Gratuitous ARP 消息，以更新网络中其他设备的 ARP 缓存，确保它们知道虚拟 IP 地址 192.168.3.111 目前由该实例管理。

  2. Master 收到来自其他节点的广播：

    • Apr 2 23:10:41 vm001-ubuntu2204 Keepalived_vrrp[2118]: (VI_2) Master received advert from 192.168.3.102 with higher priority 99, ours 98
    • VRRP 实例 VI_2 当前是 Master。它收到了来自 IP 地址 192.168.3.102 的另一个 VRRP 节点发送的广播。该广播宣称了更高的优先级（99），而当前节点的优先级是 98。

  3. 切换到备份状态：

    • Apr 2 23:10:41 vm001-ubuntu2204 Keepalived_vrrp[2118]: (VI_2) Entering BACKUP STATE
    • 由于收到了更高优先级的广播，VRRP 实例 VI_2 决定将自己从 Master 状态切换到备份状态。这意味着另一个节点（具有更高优先级的节点）将成为新的 Master。

综上所述，这些日志反映了 Keepalived 在 VRRP 实例间进行状态转换的过程。通过这种方式，Keepalived 确保了在网络中的 VRRP 节点之间高可用性和故障转移能力，以确保网络服务的连续性。

4.再次查看本机网卡内容：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
$ ip a
$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:2b:a8:e1 brd ff:ff:ff:ff:ff:ff
    inet 192.168.3.102/24 brd 192.168.3.255 scope global enp1s0
       valid_lft forever preferred_lft forever
    inet 192.168.3.111/32 scope global enp1s0
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe2b:a8e1/64 scope link 
       valid_lft forever preferred_lft forever

与此同时， 机器一：192.168.3.101 上的网卡内容：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:99:26:6d brd ff:ff:ff:ff:ff:ff
    inet 192.168.3.101/24 brd 192.168.3.255 scope global enp1s0
       valid_lft forever preferred_lft forever
    inet 192.168.3.222/32 scope global enp1s0
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe99:266d/64 scope link 
       valid_lft forever preferred_lft forever

故障模拟

模拟机器一故障

机器一：192.168.3.101

1.停止 keepalived 服务:

1
$ sudo systemctl stop keepalived.service

2.同时，另一个终端的日志输出：

1
2
3
4
5
6
7
8
9
Apr  2 23:30:21 vm001-ubuntu2204 systemd[1]: Stopping LVS and VRRP High Availability Monitor...
Apr  2 23:30:21 vm001-ubuntu2204 Keepalived[2117]: Stopping
Apr  2 23:30:21 vm001-ubuntu2204 Keepalived_vrrp[2118]: (VI_1) sent 0 priority
Apr  2 23:30:21 vm001-ubuntu2204 Keepalived_vrrp[2118]: (VI_1) removing VIPs.
Apr  2 23:30:22 vm001-ubuntu2204 Keepalived_vrrp[2118]: Stopped - used 0.063499 user time, 0.245590 system time
Apr  2 23:30:22 vm001-ubuntu2204 Keepalived[2117]: CPU usage (self/children) user: 0.001304/0.063701 system: 0.000000/0.246370
Apr  2 23:30:22 vm001-ubuntu2204 Keepalived[2117]: Stopped Keepalived v2.2.8 (04/04,2023), git commit v2.2.7-154-g292b299e+
Apr  2 23:30:22 vm001-ubuntu2204 systemd[1]: keepalived.service: Deactivated successfully.
Apr  2 23:30:22 vm001-ubuntu2204 systemd[1]: Stopped LVS and VRRP High Availability Monitor.

日志分析说明：

这段日志记录了在 vm001-ubuntu2204 上停止 Keepalived 服务的过程。Keepalived 是一个用于实现服务器高可用的软件，主要通过虚拟路由冗余协议（VRRP）来实现。下面是日志的详细分析：

  1. 停止服务:

    • Apr 2 23:30:21 vm001-ubuntu2204 systemd[1]: Stopping LVS and VRRP High Availability Monitor...
      • 系统正在停止 Linux Virtual Server (LVS) 和 VRRP 高可用性监视器，即 Keepalived 服务。

  2. Keepalived 开始停止:

    • Apr 2 23:30:21 vm001-ubuntu2204 Keepalived[2117]: Stopping
      • Keepalived 主进程开始停止。

  3. 发送优先级为 0 的广播:

    • Apr 2 23:30:21 vm001-ubuntu2204 Keepalived_vrrp[2118]: (VI_1) sent 0 priority
      • Keepalived 的 VRRP 子进程向其他 VRRP 对等体发送了一个优先级为 0 的广播，表示它正在放弃 Master 角色。

  4. 移除 VIPs:

    • Apr 2 23:30:21 vm001-ubuntu2204 Keepalived_vrrp[2118]: (VI_1) removing VIPs.
      • VRRP 子进程开始移除配置在它上的虚拟IP地址（VIPs）。

  5. 服务停止统计:

    • Apr 2 23:30:22 vm001-ubuntu2204 Keepalived_vrrp[2118]: Stopped - used 0.063499 user time, 0.245590 system time
      • Keepalived VRRP 子进程已经停止，展示了在用户态和内核态消耗的时间。

  6. CPU 使用统计:

    • Apr 2 23:30:22 vm001-ubuntu2204 Keepalived[2117]: CPU usage (self/children) user: 0.001304/0.063701 system: 0.000000/0.246370
      • 展示了 Keepalived 主进程及其子进程在用户态和内核态的 CPU 使用情况。

  7. Keepalived 停止完成:

    • Apr 2 23:30:22 vm001-ubuntu2204 Keepalived[2117]: Stopped Keepalived v2.2.8 (04/04,2023), git commit v2.2.7-154-g292b299e+
      • Keepalived 完全停止，显示了 Keepalived 的版本和 Git 提交信息。

  8. 服务停止确认:

    • Apr 2 23:30:22 vm001-ubuntu2204 systemd[1]: keepalived.service: Deactivated successfully.
    • Apr 2 23:30:22 vm001-ubuntu2204 systemd[1]: Stopped LVS and VRRP High Availability Monitor.
      • Systemd 系统管理器确认 Keepalived 服务已成功停止。

这一系列日志显示了 Keepalived 服务在 vm001-ubuntu2204 上从开始停止到完全停止的整个过程。

再次查看本机 IP ：

1
2
3
4
5
6
7
8
9
10
11
12
13
$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:99:26:6d brd ff:ff:ff:ff:ff:ff
    inet 192.168.3.101/24 brd 192.168.3.255 scope global enp1s0
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe99:266d/64 scope link 
       valid_lft forever preferred_lft forever

机器二：192.168.3.102

于此同时，查看该机终端的日志输出：

1
2
3
4
5
6
7
8
9
Apr  2 23:30:21 vm002-ubuntu Keepalived_vrrp[2084]: (VI_1) Backup received priority 0 advertisement
Apr  2 23:30:21 vm002-ubuntu Keepalived_vrrp[2084]: (VI_1) Receive advertisement timeout
Apr  2 23:30:21 vm002-ubuntu Keepalived_vrrp[2084]: (VI_1) Entering MASTER STATE
Apr  2 23:30:21 vm002-ubuntu Keepalived_vrrp[2084]: (VI_1) setting VIPs.
Apr  2 23:30:21 vm002-ubuntu Keepalived_vrrp[2084]: (VI_1) Sending/queueing gratuitous ARPs on enp1s0 for 192.168.3.222
Apr  2 23:30:21 vm002-ubuntu Keepalived_vrrp[2084]: Sending gratuitous ARP on enp1s0 for 192.168.3.222
Apr  2 23:30:21 vm002-ubuntu Keepalived_vrrp[2084]: message repeated 4 times: [ Sending gratuitous ARP on enp1s0 for 192.168.3.222]
Apr  2 23:30:26 vm002-ubuntu Keepalived_vrrp[2084]: (VI_1) Sending/queueing gratuitous ARPs on enp1s0 for 192.168.3.222
Apr  2 23:30:26 vm002-ubuntu Keepalived_vrrp[2084]: Sending gratuitous ARP on enp1s0 for 192.168.3.222

日志分析说明：

这些日志来自 Keepalived 的 VRRP (Virtual Router Redundancy Protocol) 实例运行在一个名为 vm002-ubuntu 的机器上。Keepalived 用于高可用性解决方案，确保服务可靠性通过在多台服务器间共享一个虚拟IP（VIP）。这里的情况是针对 VRRP 实例 VI_1 的行为和状态变化。

  1. Backup 收到优先级为 0 的广播:

    • Apr 2 23:30:21 vm002-ubuntu Keepalived_vrrp[2084]: (VI_1) Backup received priority 0 advertisement
    • 这表示 VI_1 实例在备用（Backup）模式下收到了一个优先级为 0 的广播。通常，优先级为 0 的广播用于通知其他VRRP成员，发送广播的主机即将停止参与VRRP选举，这通常是一个主动退位的信号。

  2. 接收广播超时:

    • Apr 2 23:30:21 vm002-ubuntu Keepalived_vrrp[2084]: (VI_1) Receive advertisement timeout
    • 这表示 VI_1 实例在等待来自其他VRRP成员的广播时超时了。这种情况下，如果该实例是备用（Backup）服务器，它将认为主服务器（Master）不可达或出现问题，准备接管角色。

  3. 进入 MASTER 状态:

    • Apr 2 23:30:21 vm002-ubuntu Keepalived_vrrp[2084]: (VI_1) Entering MASTER STATE
    • 基于前述情况，VI_1 实例判断主服务器不可用或其优先级较低，因此决定提升自身为 Master 状态，开始承担提供服务的职责。

  4. 设置 VIPs:

    • Apr 2 23:30:21 vm002-ubuntu Keepalived_vrrp[2084]: (VI_1) setting VIPs.
    • 作为新的 Master，实例配置自己的网络接口上的虚拟IP地址（VIP），在这个案例中是 192.168.3.222，确保流向该地址的流量能被当前主机接收和处理。

  5. 发送/队列化 gratuitous ARPs:

    • Apr 2 23:30:21 vm002-ubuntu Keepalived_vrrp[2084]: Sending gratuitous ARP on enp1s0 for 192.168.3.222
    • 实例发送 Gratuitous ARP 广播，这是为了更新网络中其他设备的 ARP 缓存，确保当它们尝试访问虚拟IP地址时，流量能够被正确地路由到新的主服务器上。重复发送几次是为了确保更新的广播能被网络上尽可能多的设备接收到。

总结，这些日志说明了一个 VRRP 实例从备用状态过渡到主状态的过程，这通常发生在当前主服务器不可用或者存在网络分区时。此过程保证了服务的高可用性，即使在某些服务器发生故障时也能保持服务不中断。

再次查看本机 IP：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:2b:a8:e1 brd ff:ff:ff:ff:ff:ff
    inet 192.168.3.102/24 brd 192.168.3.255 scope global enp1s0
       valid_lft forever preferred_lft forever
    inet 192.168.3.111/32 scope global enp1s0
       valid_lft forever preferred_lft forever
    inet 192.168.3.222/32 scope global enp1s0
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe2b:a8e1/64 scope link 
       valid_lft forever preferred_lft forever

恢复机器一故障

机器一：192.168.3.101

启动 keepalived 服务:

1
$ sudo systemctl start keepalived.service

同时，另一个终端的日志输出：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
Apr  2 23:35:42 vm001-ubuntu2204 systemd[1]: Starting LVS and VRRP High Availability Monitor...
Apr  2 23:35:42 vm001-ubuntu2204 Keepalived[2255]: Starting Keepalived v2.2.8 (04/04,2023), git commit v2.2.7-154-g292b299e+
Apr  2 23:35:42 vm001-ubuntu2204 systemd[1]: Started LVS and VRRP High Availability Monitor.
Apr  2 23:35:42 vm001-ubuntu2204 Keepalived[2255]: WARNING - keepalived was built for newer Linux 5.15.143, running on Linux 5.15.0-101-generic #111-Ubuntu SMP Tue Mar 5 20:16:58 UTC 2024
Apr  2 23:35:42 vm001-ubuntu2204 Keepalived[2255]: Command line: '/usr/local/keepalived/sbin/keepalived' '-D'
Apr  2 23:35:42 vm001-ubuntu2204 Keepalived[2255]: Opening file '/usr/local/keepalived/etc/keepalived/keepalived.conf'.
Apr  2 23:35:42 vm001-ubuntu2204 Keepalived[2255]: Configuration file /usr/local/keepalived/etc/keepalived/keepalived.conf
Apr  2 23:35:42 vm001-ubuntu2204 Keepalived[2256]: NOTICE: setting config option max_auto_priority should result in better keepalived performance
Apr  2 23:35:42 vm001-ubuntu2204 Keepalived[2256]: Starting VRRP child process, pid=2257
Apr  2 23:35:42 vm001-ubuntu2204 Keepalived_vrrp[2257]: Registering Kernel netlink reflector
Apr  2 23:35:42 vm001-ubuntu2204 Keepalived_vrrp[2257]: Registering Kernel netlink command channel
Apr  2 23:35:42 vm001-ubuntu2204 Keepalived_vrrp[2257]: Script user 'keepalived_script' does not exist
Apr  2 23:35:42 vm001-ubuntu2204 Keepalived_vrrp[2257]: WARNING - script '/usr/local/keepalived/etc/keepalived/notify.sh' is not executable for uid:gid 0:0 - disabling.
Apr  2 23:35:42 vm001-ubuntu2204 Keepalived_vrrp[2257]: message repeated 5 times: [ WARNING - script '/usr/local/keepalived/etc/keepalived/notify.sh' is not executable for uid:gid 0:0 - disabling.]
Apr  2 23:35:42 vm001-ubuntu2204 Keepalived_vrrp[2257]: SECURITY VIOLATION - scripts are being executed but script_security not enabled.
Apr  2 23:35:42 vm001-ubuntu2204 Keepalived_vrrp[2257]: Assigned address 192.168.3.101 for interface enp1s0
Apr  2 23:35:42 vm001-ubuntu2204 Keepalived_vrrp[2257]: Assigned address fe80::5054:ff:fe99:266d for interface enp1s0
Apr  2 23:35:42 vm001-ubuntu2204 Keepalived_vrrp[2257]: Registering gratuitous ARP shared channel
Apr  2 23:35:42 vm001-ubuntu2204 Keepalived_vrrp[2257]: (VI_1) removing VIPs.
Apr  2 23:35:42 vm001-ubuntu2204 Keepalived_vrrp[2257]: (VI_2) removing VIPs.
Apr  2 23:35:42 vm001-ubuntu2204 Keepalived[2256]: Startup complete
Apr  2 23:35:42 vm001-ubuntu2204 Keepalived_vrrp[2257]: (VI_1) Entering BACKUP STATE (init)
Apr  2 23:35:42 vm001-ubuntu2204 Keepalived_vrrp[2257]: (VI_2) Entering BACKUP STATE (init)
Apr  2 23:35:42 vm001-ubuntu2204 Keepalived_vrrp[2257]: VRRP sockpool: [ifindex(  2), family(IPv4), proto(112), fd(12,13) multicast, address(224.0.0.111)]
Apr  2 23:35:42 vm001-ubuntu2204 Keepalived_vrrp[2257]: (VI_1) received lower priority (198) advert from 192.168.3.102 - discarding
Apr  2 23:35:44 vm001-ubuntu2204 Keepalived_vrrp[2257]: message repeated 2 times: [ (VI_1) received lower priority (198) advert from 192.168.3.102 - discarding]
Apr  2 23:35:45 vm001-ubuntu2204 Keepalived_vrrp[2257]: (VI_1) Receive advertisement timeout
Apr  2 23:35:45 vm001-ubuntu2204 Keepalived_vrrp[2257]: (VI_1) Entering MASTER STATE
Apr  2 23:35:45 vm001-ubuntu2204 Keepalived_vrrp[2257]: (VI_1) setting VIPs.
Apr  2 23:35:45 vm001-ubuntu2204 Keepalived_vrrp[2257]: (VI_1) Sending/queueing gratuitous ARPs on enp1s0 for 192.168.3.222
Apr  2 23:35:45 vm001-ubuntu2204 Keepalived_vrrp[2257]: Sending gratuitous ARP on enp1s0 for 192.168.3.222
Apr  2 23:35:45 vm001-ubuntu2204 Keepalived_vrrp[2257]: message repeated 4 times: [ Sending gratuitous ARP on enp1s0 for 192.168.3.222]
Apr  2 23:35:50 vm001-ubuntu2204 Keepalived_vrrp[2257]: (VI_1) Sending/queueing gratuitous ARPs on enp1s0 for 192.168.3.222
Apr  2 23:35:50 vm001-ubuntu2204 Keepalived_vrrp[2257]: Sending gratuitous ARP on enp1s0 for 192.168.3.222

日志分析说明：

当机器一上的 keepalived 恢复后，会初始化 2 个IP 状态：首先是移除 2 个 VIP，然后初始化 VIP1 和 VIP2 的 BACKUP 状态。由于 VRRP 实例 VI_1 广播超时（即在预定时间内没有从其他VRRP成员那里收到预期的心跳信号，这通常意味着没有其他实例或者是它们无法通信），所以它进入了 MASTER 状态（Entering MASTER STATE）。这表示它现在认为自己是负责管理指定的VIP（虚拟IP地址）的主节点。这个状态转换是VRRP协议保证虚拟IP地址可用性的一部分，当主节点失效时，备份节点会接管，成为新的主节点。进入MASTER状态后，VI_1实例开始设置其负责的VIP（192.168.3.222），意味着它会在其网络接口上配置这个IP地址，从而开始接收发送到该地址的流量。

这段日志继续描述了Keepalived及其VRRP实例VI_1的行为，重点是实例在接收广播超时后的状态转换。下面是对这些行为的分析：

  1. 接收广播超时：

    • VI_1实例因为接收广播超时（Receive advertisement timeout），即在预定时间内没有从其他VRRP成员那里收到预期的心跳信号，这通常意味着没有其他实例或者是它们无法通信。

  2. 进入MASTER状态：

    • 由于超时，VI_1实例进入了MASTER状态（Entering MASTER STATE）。这表示它现在认为自己是负责管理指定的VIP（虚拟IP地址）的主节点。这个状态转换是VRRP协议保证虚拟IP地址可用性的一部分，当主节点失效时，备份节点会接管，成为新的主节点。

  3. 设置VIPs：

    • 进入MASTER状态后，VI_1实例开始设置其负责的VIP（192.168.3.222），意味着它会在其网络接口上配置这个IP地址，从而开始接收发送到该地址的流量。

  4. 发送免费ARP：

    • Sending/queueing gratuitous ARPs on enp1s0 for 192.168.3.222指的是Keepalived在网络上广播免费ARP消息，以通知网络中的其他设备，虚拟IP地址192.168.3.222现在由enp1s0接口上的这台机器负责。这有助于快速更新网络中其他设备的ARP缓存，确保向该虚拟IP地址发送的数据能够被正确路由到新的主节点。

这个过程是VRRP协议设计的核心，旨在确保高可用性和网络中虚拟IP地址的无缝故障转移。通过这种方式，如果当前主节点因为任何原因宕机，备份节点可以迅速接管，最小化对服务的影响。

再次查看本机 IP ：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:99:26:6d brd ff:ff:ff:ff:ff:ff
    inet 192.168.3.101/24 brd 192.168.3.255 scope global enp1s0
       valid_lft forever preferred_lft forever
    inet 192.168.3.222/32 scope global enp1s0
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe99:266d/64 scope link 
       valid_lft forever preferred_lft forever

机器二：192.168.3.102

于此同时，查看该机终端的日志输出：

1
2
3
4
Apr  2 23:30:26 vm002-ubuntu Keepalived_vrrp[2084]: message repeated 4 times: [ Sending gratuitous ARP on enp1s0 for 192.168.3.222]
Apr  2 23:35:45 vm002-ubuntu Keepalived_vrrp[2084]: (VI_1) Master received advert from 192.168.3.101 with higher priority 199, ours 198
Apr  2 23:35:45 vm002-ubuntu Keepalived_vrrp[2084]: (VI_1) Entering BACKUP STATE
Apr  2 23:35:45 vm002-ubuntu Keepalived_vrrp[2084]: (VI_1) removing VIPs.

日志分析说明：

具体来看，vm002-ubuntu的日志揭示了一个重要的VRRP事件：

  1. 接收到具有更高优先级的广播：

    • vm002-ubuntu上的Keepalived_vrrp实例VI_1接收到了一个来自IP地址192.168.3.101的广播，该广播宣称有更高的优先级（199），而vm002-ubuntu上的VI_1实例的优先级为198。VRRP协议规定，如果一个备份节点接收到一个来自另一个节点的广播，且该广播宣称的优先级高于自身，它必须将自己降级为备份状态。

  2. 进入备份状态：

    • 根据接收到的具有更高优先级的广播，VI_1实例转换到备份状态（Entering BACKUP STATE）。这表明它现在不再主导处理发送到VIP（在这个案例中是192.168.3.222）的流量。

  3. 移除VIPs：

    • 进入备份状态后，VI_1实例开始移除其之前负责的VIP（removing VIPs.）。在实践中，这意味着该节点不再在其网络接口上配置有这个VIP，从而不会处理发送到该VIP的数据包。

此过程是VRRP协议确保网络服务高可用性的核心机制。通过这种方式，如果主节点因为任何原因宕机，备份节点可以迅速接管，成为新的主节点。同样，如果原主节点恢复并宣布具有更高的优先级，它可以重新接管VIP，保证服务的持续可用性。

这种动态的优先级调整和主/备转换确保了网络中服务的高可用性和故障恢复能力，是Keepalived和VRRP协议设计的重要特征。

再次查看本机 IP：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:2b:a8:e1 brd ff:ff:ff:ff:ff:ff
    inet 192.168.3.102/24 brd 192.168.3.255 scope global enp1s0
       valid_lft forever preferred_lft forever
    inet 192.168.3.111/32 scope global enp1s0
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe2b:a8e1/64 scope link 
       valid_lft forever preferred_lft forever