一、概述
Redis3.0版本之后支持Cluster,目前redis cluster支持的特性有:
節(jié)點(diǎn)自動(dòng)發(fā)現(xiàn),slave->master 選舉,集群容錯(cuò),Hot resharding:在線(xiàn)分片
進(jìn)群管理:cluster xxx,基于配置(nodes-port.conf)的集群管理,ASK 轉(zhuǎn)向/MOVED 轉(zhuǎn)向機(jī)制.
二、redis cluster安裝
1、下載和解包
cd /usr/local/
wget http://download.redis.io/releases/redis-4.0.1.tar.gz
tar -xvzf redis-4.0.1.tar.gz
2、 編譯安裝
cd redis-4.0.1
make && make install
3、創(chuàng)建redis節(jié)點(diǎn)
選擇2臺(tái)服務(wù)器,分別為:192.168.59.131,192.168..59.130.每分服務(wù)器有3個(gè)節(jié)點(diǎn)。
先在192.168.59.131創(chuàng)建3個(gè)節(jié)點(diǎn):
cd /usr/local/
//創(chuàng)建集群目錄
mkdir redis_cluster
//分別代表三個(gè)節(jié)點(diǎn) 其對(duì)應(yīng)端口 7000 7001 7002
mkdir 7000 7001 7002
//以創(chuàng)建7000節(jié)點(diǎn)為例,拷貝到7000目錄
cp /usr/local/redis-4.0.1/redis.conf ./redis_cluster/7000/
//拷貝到7001目錄
cp /usr/local/redis-4.0.1/redis.conf ./redis_cluster/7001/
//拷貝到7002目錄
cp /usr/local/redis-4.0.1/redis.conf ./redis_cluster/7002/
分別對(duì)7001,7002、7003文件夾中的3個(gè)文件修改對(duì)應(yīng)的配置
daemonize yes //redis后臺(tái)運(yùn)行
pidfile /var/run/redis_7000.pid //pidfile文件對(duì)應(yīng)7000,7002,7003
bind 192.168.59.131 //綁定ip地址
port 7000 //端口7000,7002,7003
cluster-enabled yes //開(kāi)啟集群 把注釋#去掉
cluster-config-file nodes_7000.conf //集群的配置 配置文件首次啟動(dòng)自動(dòng)生成 7000,7001,7002
cluster-node-timeout 5000 //請(qǐng)求超時(shí) 設(shè)置5秒夠了
appendonly yes //aof日志開(kāi)啟 有需要就開(kāi)啟,它會(huì)每次寫(xiě)操作都記錄一條日志
在192.168.59.130創(chuàng)建3個(gè)節(jié)點(diǎn):對(duì)應(yīng)的端口改為7003,7004,7005.配置對(duì)應(yīng)的改一下就可以了
4、兩臺(tái)機(jī)啟動(dòng)各節(jié)點(diǎn)
第一臺(tái)機(jī)器上執(zhí)行
redis-server redis_cluster/7000/redis.conf
redis-server redis_cluster/7001/redis.conf
redis-server redis_cluster/7002/redis.conf
另外一臺(tái)機(jī)器上執(zhí)行
redis-server redis_cluster/7003/redis.conf
redis-server redis_cluster/7004/redis.conf
redis-server redis_cluster/7005/redis.conf
5、查看服務(wù)
ps -ef | grep redis #查看是否啟動(dòng)成功
netstat -tnlp | grep redis #可以看到redis監(jiān)聽(tīng)端口
三、創(chuàng)建集群
前面已經(jīng)準(zhǔn)備好了搭建集群的redis節(jié)點(diǎn),接下來(lái)我們要把這些節(jié)點(diǎn)都串連起來(lái)搭建集群。
官方提供了一個(gè)工具:redis-trib.rb(/usr/local/redis-4.0.1/src/redis-trib.rb) 看后綴就知道它是用ruby寫(xiě)的一個(gè)程序,所以我們還得安裝ruby.
yum -y install ruby ruby-devel rubygems rpm-build
再用 gem 這個(gè)命令來(lái)安裝 redis接口 gem是ruby的一個(gè)工具包.
gem install redis
接下來(lái)運(yùn)行一下redis-trib.rb
[root@localhost local]# /usr/local/redis-4.0.1/src/redis-trib.rb
Usage: redis-trib <command> <options> <arguments ...>
create host1:port1 ... hostN:portN
--replicas <arg>
check host:port
info host:port
fix host:port
--timeout <arg>
reshard host:port
--from <arg>
--to <arg>
--slots <arg>
--yes
--timeout <arg>
--pipeline <arg>
rebalance host:port
--weight <arg>
--auto-weights
--use-empty-masters
--timeout <arg>
--simulate
--pipeline <arg>
--threshold <arg>
add-node new_host:new_port existing_host:existing_port
--slave
--master-id <arg>
del-node host:port node_id
set-timeout host:port milliseconds
call host:port command arg arg .. arg
import host:port
--from <arg>
--copy
--replace
help (show this help)
For check, fix, reshard, del-node, set-timeout you can specify the host and port of any working node in the cluster.
確認(rèn)所有的節(jié)點(diǎn)都啟動(dòng),接下來(lái)使用參數(shù)create 創(chuàng)建 (在192.168.59.131中來(lái)創(chuàng)建)
/usr/local/redis-4.0.1/src/redis-trib.rb create --replicas 1 192.168.59.131:7000 192.168.59.131:7001 192.168.59.131:7003 192.168.59.130:7003 192.168.59.130:7004 192.168.59.130:7005
--replicas 1 表示 自動(dòng)為每一個(gè)master節(jié)點(diǎn)分配一個(gè)slave節(jié)點(diǎn) 上面有6個(gè)節(jié)點(diǎn),程序會(huì)按照一定規(guī)則生成 3個(gè)master(主)3個(gè)slave(從)
防火墻一定要開(kāi)放監(jiān)聽(tīng)的端口,否則會(huì)創(chuàng)建失敗。
運(yùn)行中,提示Can I set the above configuration? (type 'yes' to accept): yes //輸入yes
[root@localhost local]# /usr/local/redis-4.0.1/src/redis-trib.rb create --replicas 1 192.168.59.131:7000 192.168.59.131:7001 192.168.59.131:7003 192.168.59.130:7003 192.168.59.130:7004 192.168.59.130:7005
>>> Creating cluster
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
192.168.59.131:7000
192.168.59.130:7003
192.168.59.131:7001
Adding replica 192.168.59.130:7004 to 192.168.59.131:7000
Adding replica 192.168.59.131:7002 to 192.168.59.130:7003
Adding replica 192.168.59.130:7005 to 192.168.59.131:7001
M: 048937fd4ac542c135e7527f681be12348e8c8d0 192.168.59.131:7000
slots:0-5460 (5461 slots) master
M: 2039bc32bf55d21d74296cdc91a316022a7afc4b 192.168.59.131:7001
slots:10923-16383 (5461 slots) master
S: 76514c28d0a40f9751ccaf117d1f8f1d0878475a 192.168.59.131:7002
replicates a58a304964ff10d10f4323341f7084c9f60f5124
M: a58a304964ff10d10f4323341f7084c9f60f5124 192.168.59.130:7003
slots:5461-10922 (5462 slots) master
S: 7512d76b416d09619d3b639a94945a676d7b8826 192.168.59.130:7004
replicates 048937fd4ac542c135e7527f681be12348e8c8d0
S: 7824b9721cb21de60c8d132d668fd3ac5c4eae49 192.168.59.130:7005
replicates 2039bc32bf55d21d74296cdc91a316022a7afc4b
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join..
>>> Performing Cluster Check (using node 192.168.59.131:7000)
M: 048937fd4ac542c135e7527f681be12348e8c8d0 192.168.59.131:7000
slots:0-5460 (5461 slots) master
1 additional replica(s)
S: 76514c28d0a40f9751ccaf117d1f8f1d0878475a 192.168.59.131:7002
slots: (0 slots) slave
replicates a58a304964ff10d10f4323341f7084c9f60f5124
M: a58a304964ff10d10f4323341f7084c9f60f5124 192.168.59.130:7003
slots:5461-10922 (5462 slots) master
1 additional replica(s)
S: 7512d76b416d09619d3b639a94945a676d7b8826 192.168.59.130:7004
slots: (0 slots) slave
replicates 048937fd4ac542c135e7527f681be12348e8c8d0
M: 2039bc32bf55d21d74296cdc91a316022a7afc4b 192.168.59.131:7001
slots:10923-16383 (5461 slots) master
1 additional replica(s)
S: 7824b9721cb21de60c8d132d668fd3ac5c4eae49 192.168.59.130:7005
slots: (0 slots) slave
replicates 2039bc32bf55d21d74296cdc91a316022a7afc4b
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
到這里集群已經(jīng)初步搭建好了。
四、測(cè)試
- get 和 set數(shù)據(jù)
[root@localhost local]# redis-cli -h 192.168.59.131 -c -p 7002
192.168.59.131:7002> keys *
(empty list or set)
[root@localhost local]# redis-cli -h 192.168.59.130 -c -p 7005
192.168.59.130:7005> set hello world
-> Redirected to slot [866] located at 192.168.59.131:7000
OK
192.168.59.131:7000>
[root@localhost local]# redis-cli -h 192.168.59.131 -c -p 7002
192.168.59.131:7002> get hello
-> Redirected to slot [866] located at 192.168.59.131:7000
"world"
192.168.59.131:7000>
- 模擬故障切換
A. 先把192.168.59.131 其中一個(gè)Master服務(wù)Down掉,(192.168.59.131有2個(gè)Master, 1個(gè)Slave) ,
測(cè)試一下,依然沒(méi)有問(wèn)題,集群依然能繼續(xù)工作。
原因: redis集群 通過(guò)選舉方式進(jìn)行容錯(cuò),保證一個(gè)服務(wù)掛了還能跑,這個(gè)選舉是全部集群超過(guò)半數(shù)以上的Master發(fā)現(xiàn)其他Master掛了后,會(huì)將其他對(duì)應(yīng)的Slave節(jié)點(diǎn)升級(jí)成Master.
B. 要是掛的是192.168.59.131怎么辦? 結(jié)果是,(error) CLUSTERDOWN The cluster is down 沒(méi)辦法,超過(guò)半數(shù)掛了那整個(gè)集群就無(wú)法工作了。
五 、常用命令
- 查看集群情況
redis-trib.rb check ip:port #檢查集群狀態(tài)
redis-cli -c -h ip -p port #使用-c進(jìn)入集群命令模式
redis-trib.rb rebalance ip:port --auto-weights #重新分配權(quán)重
[root@localhost script]# /usr/local/redis-4.0.1/src/redis-trib.rb info 192.168.59.131:7000
192.168.59.130:7003 (a58a3049...) -> 0 keys | 5462 slots | 1 slaves.
192.168.59.130:7005 (7824b972...) -> 0 keys | 5461 slots | 1 slaves.
192.168.59.130:7004 (7512d76b...) -> 0 keys | 5461 slots | 1 slaves.
[OK] 0 keys in 3 masters.
0.00 keys per slot on average.
增加、刪除集群節(jié)點(diǎn)
redis-trib.rb add-node ip:port(新增節(jié)點(diǎn)) ip:port(現(xiàn)有效節(jié)點(diǎn))
redis-trib.rb del-node ip:port id(目標(biāo)節(jié)點(diǎn)的id) #刪除master節(jié)點(diǎn)之前首先要使用reshard移除master的全部slot重新劃分slot
redis-trib.rb reshard ip:port
[root@localhost script]# /usr/local/redis-4.0.1/src/redis-trib.rb reshard 192.168.59.131:7001
>>> Performing Cluster Check (using node 192.168.59.131:7001)
S: 2039bc32bf55d21d74296cdc91a316022a7afc4b 192.168.59.131:7001
slots: (0 slots) slave
replicates 7824b9721cb21de60c8d132d668fd3ac5c4eae49
M: 7824b9721cb21de60c8d132d668fd3ac5c4eae49 192.168.59.130:7005
slots:10923-16383 (5461 slots) master
1 additional replica(s)
S: 048937fd4ac542c135e7527f681be12348e8c8d0 192.168.59.131:7000
slots: (0 slots) slave
replicates 7512d76b416d09619d3b639a94945a676d7b8826
M: 7512d76b416d09619d3b639a94945a676d7b8826 192.168.59.130:7004
slots:0-5460 (5461 slots) master
1 additional replica(s)
S: 76514c28d0a40f9751ccaf117d1f8f1d0878475a 192.168.59.131:7002
slots: (0 slots) slave
replicates a58a304964ff10d10f4323341f7084c9f60f5124
M: a58a304964ff10d10f4323341f7084c9f60f5124 192.168.59.130:7003
slots:5461-10922 (5462 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
How many slots do you want to move (from 1 to 16384)? 5461
What is the receiving node ID? 7824b9721cb21de60c8d132d668fd3ac5c4eae49
Please enter all the source node IDs.
Type 'all' to use all the nodes as source nodes for the hash slots.
Type 'done' once you entered all the source nodes IDs.
Source node #1:7512d76b416d09619d3b639a94945a676d7b8826
Source node #2:done
Ready to move 5461 slots.
Source nodes:
M: 7512d76b416d09619d3b639a94945a676d7b8826 192.168.59.130:7004
slots:0-5460 (5461 slots) master
1 additional replica(s)
Destination node:
M: 7824b9721cb21de60c8d132d668fd3ac5c4eae49 192.168.59.130:7005
slots:10923-16383 (5461 slots) master
1 additional replica(s)
Resharding plan:
Moving slot 0 from 7512d76b416d09619d3b639a94945a676d7b8826
........................
.........................
Moving slot 5460 from 7512d76b416d09619d3b639a94945a676d7b8826
Do you want to proceed with the proposed reshard plan (yes/no)? yes
Moving slot 0 from 192.168.59.130:7004 to 192.168.59.130:7005:
........................
.........................
Moving slot 5460 from 192.168.59.130:7004 to 192.168.59.130:7005:
[root@localhost script]#
- 將master轉(zhuǎn)換為salve
cluster replicate master-id #轉(zhuǎn)換前6380端必須沒(méi)有slots
[root@localhost script]# redis-cli -h 192.168.59.130 -c -p 7004
192.168.59.130:7004> cluster replicate 7824b9721cb21de60c8d132d668fd3ac5c4eae49
OK
192.168.59.130:7004> quit
六、深入了解
-
redis-cluster架構(gòu)圖
(1)所有的redis節(jié)點(diǎn)彼此互聯(lián)(PING-PONG機(jī)制),內(nèi)部使用二進(jìn)制協(xié)議優(yōu)化傳輸速度和帶寬.
(2)節(jié)點(diǎn)的fail是通過(guò)集群中超過(guò)半數(shù)的節(jié)點(diǎn)檢測(cè)失效時(shí)才生效.
(3)客戶(hù)端與redis節(jié)點(diǎn)直連,不需要中間proxy層.客戶(hù)端不需要連接集群所有節(jié)點(diǎn),連接集群中任何一個(gè)可用節(jié)點(diǎn)即可
(4)redis-cluster把所有的物理節(jié)點(diǎn)映射到[0-16383]slot上,cluster 負(fù)責(zé)維護(hù)node<->slot<->value-
redis-cluster選舉容錯(cuò)
(1)領(lǐng)著選舉過(guò)程是集群中所有master參與,如果半數(shù)以上master節(jié)點(diǎn)與master節(jié)點(diǎn)通信超過(guò)(cluster-node-timeout),認(rèn)為當(dāng)前master節(jié)點(diǎn)掛掉.
(2):什么時(shí)候整個(gè)集群不可用(cluster_state:fail),當(dāng)集群不可用時(shí),所有對(duì)集群的操作做都不可用,收到((error) CLUSTERDOWN The cluster is down)錯(cuò)誤
a:如果集群任意master掛掉,且當(dāng)前master沒(méi)有slave.集群進(jìn)入fail狀態(tài),也可以理解成進(jìn)群的slot映射[0-16383]不完成時(shí)進(jìn)入fail狀態(tài).
b:如果進(jìn)群超過(guò)半數(shù)以上master掛掉,無(wú)論是否有slave集群進(jìn)入fail狀態(tài).
-

