需要角色進(jìn)程:nameNode(2個(gè),一個(gè)active,一個(gè)standby);dataNode(3個(gè)),zookeeper(3個(gè)),zkfc(健康檢查和自動(dòng)故障轉(zhuǎn)移需要的進(jìn)程,2個(gè),和兩個(gè)nameNode在一起),JNN(3個(gè),日志節(jié)點(diǎn),用來同步兩個(gè)nameNode)。
1.前置準(zhǔn)備:java環(huán)境,hadoop下載,環(huán)境變量配置,nameNode節(jié)點(diǎn)向其他節(jié)點(diǎn)的免密鑰登錄(兩個(gè)nameNode節(jié)點(diǎn)需互相免密鑰登錄,從namenode節(jié)點(diǎn)最好也向其余節(jié)點(diǎn)免密鑰登錄,這樣也可以在從節(jié)點(diǎn)執(zhí)行start-dfs.sh等命令),詳見hadoop1.x偽分布式環(huán)境搭建。
2.配置hdfs-site.xml(之前配置的secondaryNamenode就不需要了):
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
? <name>dfs.nameservices</name>
? <value>mycluster</value>
</property>
<property>
? <name>dfs.ha.namenodes.mycluster</name>
? <value>n1,n2</value>
</property>
<property>
? <name>dfs.namenode.rpc-address.mycluster.n1</name>
? <value>wang-108:8020</value>
</property>
<property>
? <name>dfs.namenode.rpc-address.mycluster.n2</name>
? <value>wang-109:8020</value>
</property>
<property>
? <name>dfs.namenode.http-address.mycluster.n1</name>
? <value>wang-108:50070</value>
</property>
<property>
? <name>dfs.namenode.http-address.mycluster.n2</name>
? <value>wang-109:50070</value>
</property>
<property>
? <name>dfs.namenode.shared.edits.dir</name>
? <value>qjournal://wang-108:8485;wang-109:8485;wang-208:8485/mycluster</value>
</property>
<property>
? <name>dfs.journalnode.edits.dir</name>
? <value>/var/sxt/hadoop/ha/jn</value>
</property>
<property>
? <name>dfs.client.failover.proxy.provider.mycluster</name>
? <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
<property>
? <name>dfs.ha.fencing.methods</name>
? <value>sshfence</value>
</property>
<property>
? <name>dfs.ha.fencing.ssh.private-key-files</name>
? <value>/root/.ssh/id_dsa</value>
</property>
<property>
? <name>dfs.ha.automatic-failover.enabled</name>
? <value>true</value>
</property>
3.配置core-site.xml:
<property>
<name>fs.defaultFS</name>
<value>hdfs://mycluster</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/var/sxt/hadoop/ha</value>
</property>
<property>
? <name>ha.zookeeper.quorum</name>
? <value>wang-109:2181,wang-208:2181,wang-209:2181</value>
</property>
4.配置zookeeper(至少需要在3個(gè)節(jié)點(diǎn)安裝),下載,解壓,進(jìn)入conf,使用zoo-sample.cfg模板,mv zoo-sample.cfg zoo.cfg,設(shè)置data目錄dataDir=/var/sxt/zk,然后在末尾添加server.1=node01:2888:3888和server.2=node02:2888:3888和server.3=node03:2888:3888, 然后在dataDir目錄下添加myid文件,內(nèi)容為當(dāng)前節(jié)點(diǎn)對(duì)于的id號(hào)(node01為1),然后將zookeeper分發(fā)至其余兩個(gè)節(jié)點(diǎn)(scp -r zookeeper-3.4.10 node02:`pwd`/),在三個(gè)節(jié)點(diǎn)配置zookeeper的環(huán)境變量,最后啟動(dòng)zookeeper(zkServer.sh start),可使用zkServer.sh status查看狀態(tài),一起啟動(dòng)的話,id號(hào)最大的為leader,其它為follwer
5.在三個(gè)節(jié)點(diǎn)啟動(dòng)journalnode,hadoop-daemon.sh start journalnode
6.在主節(jié)點(diǎn)格式化hdfs(hdfs namenode -format)
7.在主節(jié)點(diǎn)啟動(dòng)namenode(hadoop-daemon.sh start namenode),然后在從節(jié)點(diǎn)執(zhí)行hdfs namenode -bootstrapStandby(同步主節(jié)點(diǎn)剛剛格式化的信息)
8.在主節(jié)點(diǎn)初始化zk( hdfs zkfc -formatZK ),此時(shí)在有zookeeper的節(jié)點(diǎn)上執(zhí)行zkCli.sh連入查看節(jié)點(diǎn),在zk中會(huì)多一個(gè)節(jié)點(diǎn)/hadoop-ha/mycluster,就是在hdfs-site.xml配置的namenode集群的名稱
9.在主節(jié)點(diǎn)啟動(dòng)hdfs,start-dfs.sh
測(cè)試:在瀏覽器查看node01:50070,會(huì)發(fā)現(xiàn)主和從節(jié)點(diǎn)皆可訪問,一個(gè)active,一個(gè)standBy(從節(jié)點(diǎn)不能訪問文件目錄),此時(shí)可嘗試停掉主節(jié)點(diǎn)的namenode(hadoop-daemon.sh stop namenode),然后發(fā)現(xiàn)從自動(dòng)變?yōu)榱酥鳎ㄔ趜ookeeper中看也會(huì)發(fā)現(xiàn)相關(guān)信息變更為了從節(jié)點(diǎn))
關(guān)閉集群:在主節(jié)點(diǎn)執(zhí)行 stop-dfs.sh;再次啟動(dòng):start-dfs.sh