前言:
Redis 的几种常见使用方式包括:
- Redis 单副本
- Redis 多副本(主从)
- Redis Sentinel(哨兵)
- Redis Cluster
- Redis 自研
Redis Cluster是一种服务器 Sharding 技术,redis 3.0版本开始正式提供。Sentinel基本已经实现了高可用,但是每台机器都存储相同内容,很浪费内存,所以Redis Cluster实现了分布式存储。每台机器节点上存储不同的内容。
Redis cluster具有以下优点:
- 组件all-in-box,部署简单,节约机器资源
- 性能比proxy模式好
- 自动故障转移、Slot迁移中数据可用
- 官方原生集群方案,更新与支持有保障
因此,本文主要探讨的就是官方版本的Redis集群,
部署规划:
计划使用三台服务器,具体信息如下:
Redis集群规划
ID | IP | 部署的组件 | 部署方式 | Redis版本 |
1 | 192.168.217.16 | redis单机实例两个 | 源码编译 | Redis-5.0.4 |
2 | 192.168.217.17 | redis单机实例两个 | 源码编译 | Redis-5.0.4 |
3 | 192.168.217.18 | redis单机实例两个 | 源码编译 | Redis-5.0.4 |
部署步骤:
一,
在官网下载Redis-5.0.4,官网地址:Index of /releases/
下载地址:Index of /releases/
将安装包上传到服务器上,解压备用。
二,
进入解压后的目录,可以看到有redis.conf
[root@master redis-5.0.4]# ll total 256 -rw-rw-r-- 1 root root 99445 Mar 19 2019 00-RELEASENOTES -rw-rw-r-- 1 root root 53 Mar 19 2019 BUGS -rw-rw-r-- 1 root root 1894 Mar 19 2019 CONTRIBUTING -rw-rw-r-- 1 root root 1487 Mar 19 2019 COPYING drwxrwxr-x 6 root root 192 Sep 8 23:39 deps -rw-rw-r-- 1 root root 11 Mar 19 2019 INSTALL -rw-rw-r-- 1 root root 151 Mar 19 2019 Makefile -rw-rw-r-- 1 root root 4223 Mar 19 2019 MANIFESTO -rw-rw-r-- 1 root root 20555 Mar 19 2019 README.md -rw-rw-r-- 1 root root 62147 Sep 9 01:17 redis.conf -rwxrwxr-x 1 root root 275 Mar 19 2019 runtest -rwxrwxr-x 1 root root 280 Mar 19 2019 runtest-cluster -rwxrwxr-x 1 root root 281 Mar 19 2019 runtest-sentinel -rw-rw-r-- 1 root root 9710 Mar 19 2019 sentinel.conf drwxrwxr-x 3 root root 8192 Sep 8 23:51 src drwxrwxr-x 10 root root 167 Mar 19 2019 tests drwxrwxr-x 8 root root 4096 Mar 19 2019 utils
三,
编译前的依赖安装
配置一个本地仓库,本地仓库的配置就不说了,安装依赖命令:
yum install autoconf gcc gcc-c++ -y
四,
编译和编译安装:
make
[root@slave1 redis-5.0.4]# make cd src && make all make[1]: Entering directory `/root/redis-5.0.4/src' CC Makefile.dep make[1]: Leaving directory `/root/redis-5.0.4/src' make[1]: Entering directory `/root/redis-5.0.4/src' rm -rf redis-server redis-sentinel redis-cli redis-benchmark redis-check-rdb redis-check-aof *.o *.gcda *.gcno *.gcov redis.info lcov-html Makefile.dep dict-benchmark (cd ../deps && make distclean) make[2]: Entering directory `/root/redis-5.0.4/deps' (cd hiredis && make clean) > /dev/null || true (cd linenoise && make clean) > /dev/null || true (cd lua && make clean) > /dev/null || true (cd jemalloc && [ -f Makefile ] && make distclean) > /dev/null || true (rm -f .make-*) 。。。。。。。。。。。。。。。。。。。。。。。。。。 。。。。。。。。。。。。。。。。。。。。。。。。
make install
[root@slave1 redis-5.0.4]# make install PREFIX=/usr/local/redis-cluster/redis-node1 cd src && make install make[1]: Entering directory `/root/redis-5.0.4/src' CC Makefile.dep make[1]: Leaving directory `/root/redis-5.0.4/src' make[1]: Entering directory `/root/redis-5.0.4/src' Hint: It's a good idea to run 'make test' ;) INSTALL install INSTALL install INSTALL install INSTALL install INSTALL install make[1]: Leaving directory `/root/redis-5.0.4/src'
此时,在/usr/local/redis-cluster/redis-node1下就有了第一个实例文件,继续make install ,命令如下:
[root@slave1 redis-5.0.4]# make install PREFIX=/usr/local/redis-cluster/redis-node1
此目录下现有文件如下:
[root@slave1 bin]# pwd /usr/local/redis-cluster/redis-node1/bin [root@slave1 bin]# ll total 32696 -rwxr-xr-x 1 root root 4365912 Sep 9 10:24 redis-benchmark -rwxr-xr-x 1 root root 8101856 Sep 9 10:24 redis-check-aof -rwxr-xr-x 1 root root 8101856 Sep 9 10:24 redis-check-rdb -rwxr-xr-x 1 root root 4806256 Sep 9 10:24 redis-cli lrwxrwxrwx 1 root root 12 Sep 9 10:24 redis-sentinel -> redis-server -rwxr-xr-x 1 root root 8101856 Sep 9 10:24 redis-server
三台服务器都这么操作,也就是最后三个服务器总共有/usr/local/redis-cluster/redis-node1 /usr/local/redis-cluster/redis-node2 六个这样的目录。
五,
配置文件的修改
由于节点比较多,总共是六个,因此,配置文件采用模板方式,先修改192.168.217.16服务器内的安装目录内的redis.conf,修改的地方有这么几个:
--->daemonize no 修改为daemonize yes
--->bind 127.0.0.1 修改为 bind 192.168.217.16 127.0.0.1(在17服务器上就修改为17,18服务器上修改为18,必须要有127.0.0.1)
--->protected-mode yes 修改为protected-mode yes
--->port 6379 修改为port 16379(修改成这样是为了安全,毕竟默认端口不安全嘛)
--->appendonly no 修改为 appendonly yes #aof模式的开启
--->cluster-enabled yes 这一行注释去掉,开启集群模式
--->cluster-config-file redis-6379.conf 这一行去掉注释,修改为cluster-config-file 192.168.217.16-node1.conf
--->logfile "" 修改为logfile /usr/local/redis-cluster/redis-node1/bin/node1.log
这样修改完毕后,我们在16服务器上就有了一个模板文件,然后将此文件拷贝到 /usr/local/redis-cluster/redis-node1/bin目录下:
cp 192.168.217.16-node1.conf /usr/local/redis-cluster/redis-node1/bin/192.168.217.16-node1.conf
复制到node2目录下,这个时候名字修改一下:
cp 192.168.217.16-node1.conf /usr/local/redis-cluster/redis-node2/bin/192.168.217.16-node2.conf
编辑一哈node2的配置文件(两个地方需要修改):
cluster-config-file 192.168.217.16-node1.conf这一段 修改为cluster-config-file 192.168.217.16-node2.conf port 16379 字段修改为port 16380
其它节点照着做就可以了,配置文件内的IP地址和cluster-config-file 字段以及logs的文件形式与实际的配置文件名字相符就可以了,例如,17服务器node1下的配置文件名称为192.168.217.17-node1.conf,文件内的cluster-config-file 字段是192.168.217.17-node1.conf,logfiel字段是logfile /usr/local/redis-cluster/redis-node1/bin/node1.log ,port 是16379,如果是node2,那么,文件内port就是16380,字段是192.168.217.17-node2.conf,logs字段是logfile /usr/local/redis-cluster/redis-node2/bin/node2.log
六,
启动脚本
以16服务器上为例:
[root@master redis-5.0.4]# pwd /root/redis-5.0.4 [root@master redis-5.0.4]# cp -a utils/redis_init_script /etc/init.d/redis-node1 [root@master redis-5.0.4]# cp -a utils/redis_init_script /etc/init.d/redis-node2
编辑启动脚本redis-node1和redis-node2;
port和可执行文件路径配置文件路径一一对应
[root@master init.d]# cat /etc/init.d/redis-node1 #!/bin/sh # # Simple Redis init.d script conceived to work on Linux systems # as it does use of the /proc filesystem. ### BEGIN INIT INFO # Provides: redis_6379 # Default-Start: 2 3 4 5 # Default-Stop: 0 1 6 # Short-Description: Redis data structure server # Description: Redis data structure server. See https://redis.io ### END INIT INFO REDISPORT=16379 EXEC=/usr/local/redis-cluster/redis-node1/bin/redis-server CLIEXEC=/usr/local/redis-cluster/redis-node1/bin/redis-cli PIDFILE=/var/run/redis_${REDISPORT}.pid CONF="/usr/local/redis-cluster/redis-node1/bin/192.168.217.16-node1.conf" case "$1" in start) if [ -f $PIDFILE ] then echo "$PIDFILE exists, process is already running or crashed" else echo "Starting Redis server..." $EXEC $CONF fi ;; stop) if [ ! -f $PIDFILE ] then echo "$PIDFILE does not exist, process is not running" else PID=$(cat $PIDFILE) echo "Stopping ..." $CLIEXEC -p $REDISPORT shutdown while [ -x /proc/${PID} ] do echo "Waiting for Redis to shutdown ..." sleep 1 done echo "Redis stopped" fi ;; *) echo "Please use start or stop as first argument" ;; esac [root@master init.d]# cat /etc/init.d/redis-node2 #!/bin/sh # # Simple Redis init.d script conceived to work on Linux systems # as it does use of the /proc filesystem. ### BEGIN INIT INFO # Provides: redis_6379 # Default-Start: 2 3 4 5 # Default-Stop: 0 1 6 # Short-Description: Redis data structure server # Description: Redis data structure server. See https://redis.io ### END INIT INFO REDISPORT=16380 EXEC=/usr/local/redis-cluster/redis-node2/bin/redis-server CLIEXEC=/usr/local/redis-cluster/redis-node2/bin/redis-cli PIDFILE=/var/run/redis_${REDISPORT}.pid CONF="/usr/local/redis-cluster/redis-node2/bin/192.168.217.16-node2.conf" case "$1" in start) if [ -f $PIDFILE ] then echo "$PIDFILE exists, process is already running or crashed" else echo "Starting Redis server..." $EXEC $CONF fi ;; stop) if [ ! -f $PIDFILE ] then echo "$PIDFILE does not exist, process is not running" else PID=$(cat $PIDFILE) echo "Stopping ..." $CLIEXEC -p $REDISPORT shutdown while [ -x /proc/${PID} ] do echo "Waiting for Redis to shutdown ..." sleep 1 done echo "Redis stopped" fi ;; *) echo "Please use start or stop as first argument" ;; esac
七,
启动各个实例(以16节点的node1为例,其它照做)
chkconfig --add redis-node1.conf&&chkconfig redis-node1 on service redis-node1 start(启动节点1) service redis-node1 stop(关闭节点1)
查看进程,看到这样就算基本完成了:
[root@master init.d]# ps -ef |grep redis root 6029 1 0 11:13 ? 00:00:04 /usr/local/redis-cluster/redis-node1/bin/redis-server 192.168.217.16:16379 [cluster] root 6082 1 0 11:13 ? 00:00:04 /usr/local/redis-cluster/redis-node2/bin/redis-server 192.168.217.16:16380 [cluster] root 19864 16681 0 11:47 pts/0 00:00:00 grep --color=auto redis [root@master init.d]# netstat -antup |grep 163 tcp 0 0 192.168.217.16:16379 0.0.0.0:* LISTEN 6029/redis-server 1 tcp 0 0 192.168.217.16:16380 0.0.0.0:* LISTEN 6082/redis-server 1
八,
最后一哆嗦,集群建立,建立命令为;
[root@master bin]# ./redis-cli --cluster create 192.168.217.16:16379 192.168.217.16:16380 192.168.217.17:16379 192.168.217.17:16380 192.168.217.18:16379 192.168.217.18:16380 --cluster-replicas 1
输入如下;
[root@master bin]# ./redis-cli --cluster create 192.168.217.16:16379 192.168.217.16:16380 192.168.217.17:16379 192.168.217.17:16380 192.168.217.18:16379 192.168.217.18:16380 --cluster-replicas 1 >>> Performing hash slots allocation on 6 nodes... Master[0] -> Slots 0 - 5460 Master[1] -> Slots 5461 - 10922 Master[2] -> Slots 10923 - 16383 Adding replica 192.168.217.17:16380 to 192.168.217.16:16379 Adding replica 192.168.217.18:16380 to 192.168.217.17:16379 Adding replica 192.168.217.16:16380 to 192.168.217.18:16379 M: b6e8d09653f87d16ee5971ef332c0a0cbd448d5c 192.168.217.16:16379 slots:[0-5460] (5461 slots) master S: 11b1cefbb59905142f2e429b58b1ddaca1b87969 192.168.217.16:16380 replicates 0754abc970dc7fe790e45fce27512b5c0dbe3678 M: 4c7d99015b46ef2837070f3b7f4c4f3b1a15fd80 192.168.217.17:16379 slots:[5461-10922] (5462 slots) master S: 61321ce9323ab88af6cf7da5384438c594f91e9f 192.168.217.17:16380 replicates b6e8d09653f87d16ee5971ef332c0a0cbd448d5c M: 0754abc970dc7fe790e45fce27512b5c0dbe3678 192.168.217.18:16379 slots:[10923-16383] (5461 slots) master S: 2ba9a13f828cae6490982ef50837589690c0f25e 192.168.217.18:16380 replicates 4c7d99015b46ef2837070f3b7f4c4f3b1a15fd80 Can I set the above configuration? (type 'yes' to accept): yes >>> Nodes configuration updated >>> Assign a different config epoch to each node >>> Sending CLUSTER MEET messages to join the cluster Waiting for the cluster to join ............ >>> Performing Cluster Check (using node 192.168.217.16:16379) M: b6e8d09653f87d16ee5971ef332c0a0cbd448d5c 192.168.217.16:16379 slots:[0-5460] (5461 slots) master 1 additional replica(s) M: 0754abc970dc7fe790e45fce27512b5c0dbe3678 192.168.217.18:16379 slots:[10923-16383] (5461 slots) master 1 additional replica(s) M: 4c7d99015b46ef2837070f3b7f4c4f3b1a15fd80 192.168.217.17:16379 slots:[5461-10922] (5462 slots) master 1 additional replica(s) S: 2ba9a13f828cae6490982ef50837589690c0f25e 192.168.217.18:16380 slots: (0 slots) slave replicates 4c7d99015b46ef2837070f3b7f4c4f3b1a15fd80 S: 11b1cefbb59905142f2e429b58b1ddaca1b87969 192.168.217.16:16380 slots: (0 slots) slave replicates 0754abc970dc7fe790e45fce27512b5c0dbe3678 S: 61321ce9323ab88af6cf7da5384438c594f91e9f 192.168.217.17:16380 slots: (0 slots) slave replicates b6e8d09653f87d16ee5971ef332c0a0cbd448d5c [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered.
九,
单元测试
(1)
查看集群信息
[root@master bin]# ./redis-cli -h 192.168.217.16 -p 16379 -c 192.168.217.16:16379> cluster info cluster_state:ok cluster_slots_assigned:16384 cluster_slots_ok:16384 cluster_slots_pfail:0 cluster_slots_fail:0 cluster_known_nodes:6 cluster_size:3 cluster_current_epoch:6 cluster_my_epoch:1 cluster_stats_messages_ping_sent:216 cluster_stats_messages_pong_sent:200 cluster_stats_messages_sent:416 cluster_stats_messages_ping_received:195 cluster_stats_messages_pong_received:216 cluster_stats_messages_meet_received:5 cluster_stats_messages_received:416
(2)
集群查看数据:
[root@master bin]# ./redis-cli -h 192.168.217.18 -p 16380 -c 192.168.217.18:16380> set AAA jingqu -> Redirected to slot [3205] located at 192.168.217.16:16379 OK 192.168.217.16:16379> get AAA "jingqu" 192.168.217.16:16379> [root@master bin]# ./redis-cli -h 192.168.217.16 -p 16379 -c 192.168.217.16:16379> get AAA "jingqu" 192.168.217.16:16379>
(3)
故障模拟
下线192.168.217.16:16379,进程里看到确实少了一个节点
192.168.217.16:16379> shutdown not connected> not connected> [root@master bin]# ps -ef |grep redis root 23087 1 0 11:54 ? 00:00:01 /usr/local/redis-cluster/redis-node2/bin/redis-server 192.168.217.16:16380 [cluster] root 25714 16681 0 12:01 pts/0 00:00:00 grep --color=auto redis
192.168.217.18:16380 ,在下线一个
[root@master bin]# ./redis-cli -h 192.168.217.18 -p 16380 -c 192.168.217.18:16380> shutdown not connected>
登录192.168.217.18:1637,仍然可以查询到数据,可以看到数据是由192.168.217.17:16380提供:
[root@master bin]# ./redis-cli -h 192.168.217.18 -p 16379 -c 192.168.217.18:16379> get AAA -> Redirected to slot [3205] located at 192.168.217.17:16380 "jingqu"
至此,Redis的集群搭建完毕了。