背景

• 由于項(xiàng)目中需要用到實(shí)時(shí)處理框架，基于現(xiàn)有架構(gòu)選擇spark streaming消費(fèi)kafka使用scala語(yǔ)言進(jìn)行開(kāi)發(fā)。出于服務(wù)器資源緊張以及登錄測(cè)試服務(wù)器需要?jiǎng)討B(tài)驗(yàn)證，故在本地搭建環(huán)境進(jìn)行測(cè)試。

環(huán)境準(zhǔn)備

1. 配置免密登錄并修改/etc/hosts
   • 127.0.0.1 localhost
   • 127.0.0.1 hadoop000
   • sudo scutil --set HostName hadoop000 -- 修改終端登錄顯示
2. 下載安裝JDK1.8:
   • https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
   • 配置環(huán)境變量: vim ~/.bash_profile

export MAVEN_HOME="/Users/xieyin/soft/apache-maven-3.6.1/bin"
export PATH="$MAVEN_HOME/bin:$PATH"
export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_211.jdk/Contents/Home
export SCALA_HOME=/Users/xieyin/Soft/scala-2.11.8
export PATH=$PATH:$SCALA_HOME/bin:$JAVA_HOME/bin:/usr/local/Cellar/unrar/5.7.5/bin
export PATH="/usr/local/Cellar/redis/5.0.5/bin:$PATH"
export PATH="/usr/local/opt/gnu-sed/libexec/gnubin:$PATH"
export KE_HOME=/Users/xieyin/Soft/kafka-eagle/kafka-eagle-web-1.3.9
export PATH=$KE_HOME/bin:$PATH
export HADOOP_HOME=/Users/xieyin/Soft/hadoop-2.6.0-cdh5.15.1
export PATH=$HADOOP_HOME/bin:$PATH
export HBASE_HOME=/Users/xieyin/Soft/hbase-1.2.0-cdh5.15.1
export PATH=$HBASE_HOME/bin:$PATH

3. Mac默認(rèn)安裝最新版本kafka2.3.1、zookeeper3.4.14

# 安裝
brew install zookeeper
brew install kafka
# 啟動(dòng)
brew services start zookeeper
brew services start kafka

4. 下載安裝hadoop
   • http://archive.cloudera.com/cdh5/cdh/5/hadoop-2.6.0-cdh5.15.1.tar.gz
   • 修改配置文件

# site-core.xml
  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://hadoop000:9000</value>
  </property>
  <property>
    <name>hadoop.tmp.dir</name>
    <value>/Users/xieyin/tmp/hadoop2.6</value>
  </property>

# hdfs-site.xml
<property>
  <name>dfs.replication</name>
    <value>1</value>
    <description>設(shè)置副本數(shù)</description>
</property>

<property>
   <name>dfs.namenode.name.dir</name>
   <value>/Users/xieyin/tmp/hadoop2.6/dfs/name</value>
   <description>設(shè)置存放NameNode的文件路徑</description>
</property>

<property>
   <name>dfs.datanode.data.dir</name>
   <value>/Users/xieyin/tmp/hadoop2.6/dfs/data</value>
  <description>設(shè)置存放DataNode的文件路徑</description>
</property>

<property>
    <name>dfs.permissions</name>
    <value>false</value>
    <description>權(quán)限還是不要的好</description>
</property>


# copy一份cp mapred-site.xml.template mapred-site.xml 
# mapred-site.xml
<property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
</property>

# yarn-site.xml
<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>
<property>
    <name>yarn.nodemanager.vmem-check-enabled</name>
    <value>false</value>
    <description>由于我的測(cè)試機(jī)內(nèi)存少，所以就關(guān)閉虛擬內(nèi)存檢測(cè)</description>
</property>

5. 下載安裝HBASE
   • http://archive.cloudera.com/cdh5/cdh/5/hbase-1.2.0-cdh5.15.1.tar.gz
   • 修改配置文件

# hbase-env.sh
export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_211.jdk/Contents/Home
export HBASE_MANAGES_ZK=false
# hbase-site.xml
<property>
    <name>hbase.rootdir</name>
    <value>hdfs://hadoop000:9000/hbase</value>
</property>
<property>
    <name>hbase.cluster.distributed</name>
    <value>true</value>
</property>
<property>
    <name>hbase.zookeeper.quorum</name>
    <value>hadoop000:2181</value>
</property>

啟動(dòng)hadoop

1. 格式化namenode
   • $HADOOP_HOME/bin/hadoop namenode -format
2. 啟動(dòng)HDFS
   • $HADOOP_HOME/sbin/start-dfs.sh
3. 打開(kāi)web UI
   http://hadoop000:50070/
   
   image.png

啟動(dòng)HBASE

1. start-hbase.sh
   • 日志: /Users/xieyin/Soft/hbase-1.2.0-cdh5.15.1/logs/hbase-xieyin-master-hadoop000.log
2. web UI
   • http://hadoop000:60010/
     
     image.png