1、ELK日志收集
Elasticsearch : 數(shù)據(jù)庫，存數(shù)據(jù) java
Logstash:收集日志，過濾數(shù)據(jù) java
Kibana:分析，過濾，展示 java
Filebeat: 收集日志，傳輸?shù)紼S GO
- 安裝kibana（收集Nginx日志）
#優(yōu)化時間同步
yum install ntpdate -y
# 上傳kibana包#依賴java環(huán)境。（如沒有需安裝java-1.8.0-openjdk.x86_64）
rpm -ivh kibana-6.6.0-x86_64.rpm
# 查看配置文件
[root@db01 /data/soft]# rpm -qc kibana
[root@db01 ~]# grep '^[a-z]' /etc/kibana/kibana.yml
server.port: 5601
server.host: "10.0.0.51" # kibana服務器ip
server.name: "db01"
elasticsearch.hosts: ["http://10.0.0.51:9200"] # 填寫es地址
kibana.index: ".kibana"
# 重啟kibana
systemctl restart kibana
#測試環(huán)境部署安裝（Nginx、db壓測軟件）
yum install nginx httpd-tools -y
systemctl start nginx
ab -n 100 -c 100 http://10.0.0.51/
# 觀察日志
tail -f /var/log/nginx/access.log
#安裝filebeat，上傳安裝包
rpm -ivh filebeat-6.6.0-x86_64.rpm
cp /etc/filebeat/filebeat.yml /tmp/ 備份配置文件
[root@db01 /data/soft]# vim /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
enabled: true # 開啟
paths:
- /var/log/nginx/access.log #收集Nginx日志
output.elasticsearch:
hosts: ["10.0.0.51:9200"] # 填寫ES服務器IP
# 重啟filebeat
systemctl restart filebeat
2.kibana-Web頁面配置
image.png
image.png
image.png
image.png
-
單條件查詢
image.png -
多條件查詢
image.png -
多條件沖突時
image.png -
排除過濾
image.png
3.filebeat原理
把filebeat部署到需要采集數(shù)據(jù)的服務器中，用于數(shù)據(jù)收集，并將收集到的數(shù)據(jù)發(fā)個elasticsearch服務器存儲。
filebeat只負責傳輸最新的數(shù)據(jù)，不會關(guān)心es服務器是否存儲
filebeat每次收集日志都會記錄一個坐標點。當filebeat服務器停止期間有新的數(shù)據(jù)產(chǎn)生，會在filebeat啟動時有新數(shù)據(jù)產(chǎn)生時，會將上次記錄的發(fā)送點為起點，將最新的數(shù)據(jù)發(fā)送到es服務器中。以前已經(jīng)發(fā)送過的數(shù)據(jù)，并不會二次發(fā)送?。?！
- 修改Nginx日志為json格式
修改nginx 配置文件（nginx多臺配置一樣）
image.png
[root@db01 ~]# vim /etc/nginx/nginx.conf
log_format json '{ "time_local": "$time_local", '
'"remote_addr": "$remote_addr", '
'"referer": "$http_referer", '
'"request": "$request", '
'"status": $status, '
'"bytes": $body_bytes_sent, '
'"agent": "$http_user_agent", '
'"x_forwarded": "$http_x_forwarded_for", '
'"up_addr": "$upstream_addr",'
'"up_host": "$upstream_http_host",'
'"upstream_time": "$upstream_response_time",'
'"request_time": "$request_time"'
' }';
# 語法檢查，并重啟Nginx服務
nginx -t
systemctl restart nginx
-
修改filebeat配置（優(yōu)化日志收集）nginx多臺配置一樣。
image.png
[root@db01 ~]# vim /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/nginx/access.log
json.keys_under_root: true
json.overwrite_keys: true
output.elasticsearch:
hosts: ["10.0.0.51:9200"]
index: "nginx-%{[beat.version]}-%{+yyyy.MM}"
setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.enabled: false
setup.template.overwrite: true
# 重啟filebeat
systemctl restart filebeat
更改json格式后。日志會與之前的沖突，最好備份以前的日志
elasticsearch-Web頁面和kibana-Web頁面都需重新創(chuàng)建于配置?。?！
image.png
4.分別收集Nginx的正確和錯誤日志
image.png
[root@db01 ~]# vim /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/nginx/access.log
json.keys_under_root: true
json.overwrite_keys: true
tags: ["access"]
- type: log
enabled: true
paths:
- /var/log/nginx/error.log
tags: ["error"]
setup.kibana:
host: "10.0.0.51:5601"
output.elasticsearch:
hosts: ["10.0.0.51:9200"]
# index: "nginx-%{[beat.version]}-%{+yyyy.MM}"
indices:
- index: "nginx-access-%{[beat.version]}-%{+yyyy.MM}"
when.contains:
tags: "access"
- index: "nginx-error-%{[beat.version]}-%{+yyyy.MM}"
when.contains:
tags: "error"
setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.enabled: false
setup.template.overwrite: true
# 重啟filebeat服務
systemctl restart filebeat
在kibana和es頁面刪除舊的索引，并創(chuàng)建新的索引
image.png
image.png
image.png
5.收集Tomcat日志
#安裝Tmocat,#帶webapps的包為軟件測試包
yum install tomcat tomcat-webapps tomcat-admin-webapps tomcat-docs-webapp tomcat-javadoc -y
[root@db01 ~]# systemctl start tomcat
[root@db01 ~]# netstat -lntup|grep 8080
tcp6 0 0 :::8080 :::* LISTEN 3956/java
image.png
-
修改Tmocat配置文件。改寫為json格式輸出日志
image.png
vim /etc/tomcat/server.xml
# 刪除原本139行，粘貼如下內(nèi)容
pattern="{"clientip":"%h","ClientUser":"%l","authenticated":"%u","AccessTime":"%t&
quot;,"method":"%r","status":"%s","SendBytes":"%b","Query?string":"%q","partner":&
quot;%{Referer}i","AgentVersion":"%{User-Agent}i"}"/>
# 重啟Tomcat，備份或清空原來日志
systemctl restart tomcat
- filebeat配置文件中添加Tomcat的解析
[root@db01 ~]# cat /etc/filebeat/filebeat.yml
filebeat.inputs:
##################Nginx###################
- type: log
enabled: true
paths:
- /var/log/nginx/access.log
json.keys_under_root: true
json.overwrite_keys: true
tags: ["access"]
- type: log
enabled: true
paths:
- /var/log/nginx/error.log
tags: ["error"]
##################Tomcat###################
- type: log
enabled: true
paths:
- /var/log/tomcat/localhost_access_log.*.txt # tomcat 日志文件相對特殊，以日期格式命名。所以此時*表可收集變動日志
json.keys_under_root: true
json.overwrite_keys: true
tags: ["tomcat"]
##################output###################
setup.kibana:
host: "10.0.0.51:5601"
output.elasticsearch:
hosts: ["10.0.0.51:9200"]
# index: "nginx-%{[beat.version]}-%{+yyyy.MM}"
indices:
- index: "nginx-access-%{[beat.version]}-%{+yyyy.MM}"
when.contains:
tags: "access"
- index: "nginx-error-%{[beat.version]}-%{+yyyy.MM}"
when.contains:
tags: "error"
- index: "tomcat-access-%{[beat.version]}-%{+yyyy.MM}" # tomcat按照月來收集日志創(chuàng)建索引
when.contains:
tags: "tomcat"
setup.template.name: "nginx" # 此處沒加Tomcat的使用，此時只使用一次，前面Nginx已經(jīng)使用過。所有Tomcat可以不用加，也可以按名字識別
setup.template.pattern: "nginx-*"
setup.template.enabled: false
setup.template.overwrite: true
# 重啟filebeat服務
systemctl restart filebeat
在Tomcat隨便點擊產(chǎn)生一些日志后，在kibana頁面添加索引，觀察es和Kinbana的Web頁面是否正常
image.png
image.png
6.java日志收集
官方地址https://www.elastic.co/guide/en/beats/filebeat/6.6/multiline-examples.html
因為java日志的輸出信息非常多，需要將多行拼成一個事件，所以需要多行匹配模式，因為elasticsearch本身就是java開發(fā)的，所以我們可以直接收集ES的日志
image.png
- 代碼
[root@db01 ~]# cat /etc/filebeat/filebeat.yml
filebeat.inputs:
##################Nginx###################
- type: log
enabled: true
paths:
- /var/log/nginx/access.log
json.keys_under_root: true
json.overwrite_keys: true
tags: ["access"]
- type: log
enabled: true
paths:
- /var/log/nginx/error.log
tags: ["error"]
##################Tomcat###################
- type: log
enabled: true
paths:
- /var/log/tomcat/localhost_access_log.*.txt
json.keys_under_root: true
json.overwrite_keys: true
tags: ["tomcat"]
##################es###################
- type: log
enabled: true
paths:
- /var/log/elasticsearch/elasticsearch.log
tags: ["es"]
multiline.pattern: '^\['
multiline.negate: true
multiline.match: after
##################output###################
setup.kibana:
host: "10.0.0.51:5601"
output.elasticsearch:
hosts: ["10.0.0.51:9200"]
# index: "nginx-%{[beat.version]}-%{+yyyy.MM}"
indices:
- index: "nginx-access-%{[beat.version]}-%{+yyyy.MM}"
when.contains:
tags: "access"
- index: "nginx-error-%{[beat.version]}-%{+yyyy.MM}"
when.contains:
tags: "error"
- index: "tomcat-access-%{[beat.version]}-%{+yyyy.MM}"
when.contains:
tags: "tomcat"
- index: "es-java-%{[beat.version]}-%{+yyyy.MM}"
when.contains:
tags: "es"
setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.enabled: false
setup.template.overwrite: true
#重啟filebeat
systemctl restart filebeat
嘗試在es的配置文件自造錯誤。然后恢復觀察日志。
在kibana創(chuàng)建索引觀察Es報錯日志變化
image.png
image.png
7.收集docker日志
- 安裝docker（部署Nginx）
rm -fr /etc/yum.repos.d/local.repo
curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
wget -O /etc/yum.repos.d/docker-ce.repo https://mirrors.ustc.edu.cn/docker-ce/linux/centos/docker-ce.repo
sed -i 's#download.docker.com#mirrors.tuna.tsinghua.edu.cn/docker-ce#g' /etc/yum.repos.d/docker-ce.repo
yum install docker-ce -y
systemctl start docker
vim /etc/docker/daemon.json
{
"registry-mirrors": ["https://registry.docker-cn.com"]
}
systemctl restart docker
#運行nginx鏡像
docker pull nginx
docker run --name nginx -p 80:80 -d nginx
docker ps
docker start afa04f1bbeb9
docker exec -it afa04f1bbeb9 /bin/bash
docker logs -f nginx
------------------------------------------------------------------第二版鏡像
docker commit nginx nginx:v2
docker images
docker run --name nginx -p 8080:80 -d nginx:v2
# 刷點nginx日志查看是否有日志產(chǎn)生
#docker容器的日志目錄（編碼目錄對應相應的容器id）
[root@db02 /var/lib/docker/containers]# pwd
/var/lib/docker/containers
- 配置filebeat收集單個docker日志（缺陷：日志正確錯誤混亂）
官方文檔：https://www.elastic.co/guide/en/beats/filebeat/6.7/filebeat-input-docker.html
# 首先查看docker容器的id(inspect 跟容器的名稱)
docker inspect nginx-test|grep -w "Id"
# 配置文件
vim /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: docker
containers.ids:
- '2338d5038f7a2eac96d84d6cf424fb1829bd754ec5e0df944bdd29ba6d61a54e' # /var/lib/docker/containers/目錄下對應的容器id
tags: ["docker-nginx"] # 日志內(nèi)打標簽
output.elasticsearch:
hosts: ["10.0.0.51:9200"]
index: "docker-nginx-%{[beat.version]}-%{+yyyy.MM.dd}"
setup.template.name: "docker"
setup.template.pattern: "docker-*"
setup.template.enabled: false
setup.template.overwrite: true
- 收集多個容器日志（分類不同服務容器，區(qū)分正確錯誤日志）
假如我們有多個docker鏡像或者重新提交了新鏡像，那么直接指定ID的就不是太方便了。如果直接配置filebeat存到es里臺機器所有的容器日志都會混在一起沒有辦法區(qū)分多容器日志收集處理。
其實收集的日志本質(zhì)來說還是文件，而這個日志是以容器json,log命名存放在默認目錄下的json格式文件。但是每個容器的ID都不一樣，為了區(qū)分不同服務器運行的不同容器，可以使用docker-compose通過給容器添加labels標簽來作為區(qū)分
然后filebeat把容器日志當作普通的json格式來解析并傳輸?shù)絜s
- 安裝docker-compose（容器編排工具）
#1.安裝docker-compose
yum install -y python2-pip
#2.這里使用pip安裝，默認源為國外，可以使用國內(nèi)加速，相關(guān)網(wǎng)站#https://mirrors.tuna.tsinghua.edu.cn/help/pypi/
#pip加速操作命令
pip install -i https://pypi.tuna.tsinghua.edu.cn/simple pip -U
pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple
#3.繼續(xù)安裝docker-compose#報錯可嘗試執(zhí)行yum install python-devel
pip install docker-compose
#4.檢查
docker-compose version
image.png
- 創(chuàng)建docker-compose.yml
[root@db02 ~]# cat docker-compose.yml
version: '3' # 固定寫法
services: # 相當于一個服務，下面容器都屬于這個服務
nginx:
image: nginx:v2 # 容器名稱
# 設置labels
labels:
service: nginx #設置標簽
# logging設置增加labels.service
logging:
options:
labels: "service" # 將標簽記錄到日志中去
ports:
- "8080:80"
db:
image: nginx:latest
# 設置labels
labels:
service: db # 容器標簽
# logging設置增加labels.service
logging:
options:
labels: "service"
ports:
- "80:80"
#啟動docker-compose
docker-compose up #此時會創(chuàng)建2個容器，并且前臺啟動，會罕住。新開一個窗口docker ps查看
image.png
- 配置filebeat
[root@db02 ~]# vim /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/lib/docker/containers/*/*-json.log
json.keys_under_root: true
json.overwrite_keys: true
output.elasticsearch:
hosts: ["10.0.0.51:9200"]
indices:
- index: "docker-nginx-access-%{[beat.version]}-%{+yyyy.MM}"
when.contains:
attrs.service: "nginx"
stream: "stdout"
- index: "docker-nginx-error-%{[beat.version]}-%{+yyyy.MM}"
when.contains:
attrs.service: "nginx"
stream: "stderr"
- index: "docker-db-access-%{[beat.version]}-%{+yyyy.MM}"
when.contains:
attrs.service: "db"
stream: "stdout"
- index: "docker-db-error-%{[beat.version]}-%{+yyyy.MM}"
when.contains:
attrs.service: "db"
stream: "stderr"
setup.template.name: "docker"
setup.template.pattern: "docker-*"
setup.template.enabled: false
setup.template.overwrite: true
image.png
image.png
image.png
8.filebeat modules配置（自帶的日志收集模塊，有缺陷）
使用nginx模版配置需要安裝2個插件，默認從官方下載速度太慢，可以提前下載然后離線安裝
https://www.elastic.co/guide/en/elasticsearch/plugins/6.6/ingest-geoip.html
https://www.elastic.co/guide/en/elasticsearch/plugins/6.6/plugin-management-custom-url.html
# 在線安裝
[root@elk-175 ~]# /usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-user-agent
[root@elk-175 ~]# /usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-geoip
[root@elk-175 ~]# wget https://artifacts.elastic.co/downloads/elasticsearch-plugins/ingest-user-agent/ingest-user-agent-6.6.0.zip
[root@elk-175 ~]# wget https://artifacts.elastic.co/downloads/elasticsearch-plugins/ingest-geoip/ingest-geoip-6.6.0.zip
#離線下載好安裝包，上傳服務器進行安裝
[root@db02 ~]# /usr/share/elasticsearch/bin/elasticsearch-plugin install file:///root/ingest-geoip-6.6.0.zip
[root@db02 ~]# /usr/share/elasticsearch/bin/elasticsearch-plugin install file:///root/ingest-user-agent-6.6.0.zip
- 操作步驟
#1.filebeat 配置文件添加模塊路徑
filebeat.config.modules: # 配置filebeat modules模塊
path: ${path.config}/modules.d/*.yml # modules 的配置文件在哪里
reload.enabled: true # 自動載入
reload.period: 10s
#2.模塊會更改json格式，所有吧nginx的日志改為普通格式
vim /etc/nginx/nginx.conf
main
>/var/log/nginx/access.log
>/var/log/nginx/error.log
nginx -t
systemctl restart nginx
#3.命令行輸入激活模塊
filebeat modules enable nginx
#4. 重啟Es
systemctl restart elasticsearch
#5.重啟filebaet
systemctl restart filebeat
- fiebleat 完整配置
[root@db01 /etc/filebeat]# vim filebeat.yml
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml #模塊路徑
reload.enabled: true
reload.period: 10s
setup.kibana:
host: "10.0.0.51:5601"
output.elasticsearch:
hosts: ["10.0.0.51:9200"]
indices:
- index: "nginx-access-%{[beat.version]}-%{+yyyy.MM}"
when.contains:
fileset.name: "access"
- index: "nginx-error-%{[beat.version]}-%{+yyyy.MM}"
when.contains:
fileset.name: "error"
setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.enabled: false
setup.template.overwrite: true
單機模式發(fā)布了遠程es服務器。本地可以發(fā)。集群模發(fā)生es正常
9.使用Redis作為緩存
filebeat 缺陷：只支持傳輸?shù)絾闻_Redis服務去，不支持Redis集群和哨兵
#1.安裝Redis并啟動
#2.修改nginx配置文件日志輸出格式為json格式
#3.修改filebeat配置文件output到Redis中
[root@db01 /etc/logstash/conf.d]# vim /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/nginx/access.log
json.keys_under_root: true
json.overwrite_keys: true
tags: ["access"]
- type: log
enabled: true
paths:
- /var/log/nginx/error.log
tags: ["error"]
setup.kibana:
host: "10.0.0.51:5601"
output.redis:
hosts: ["10.0.0.51"] # Redis服務器IP
key: "filebeat" # 將日志存入Redis的filebeat鍵值中
db: 0
timeout: 5
--------------------------------------------------------------------------------------------------------------------------------------------
#4.配置logstash配置文件
[root@db01 ~]# cat /etc/logstash/conf.d/redis.conf
input {
redis {
host => "10.0.0.51" #Redis 服務器地址
port => "6379"
db => "0"
key => "filebeat" # 取Redis中filebeat鍵值的內(nèi)容
data_type => "list" # 類型
}
}
filter {
mutate {
convert => ["upstream_time", "float"] # 兩行Nginx與PHP解析的時間，轉(zhuǎn)換為浮點型，方便大小排序
convert => ["request_time", "float"]
}
}
output {
if "access" in [tags] { # key值中含有access的傳輸?shù)紼s服務器nginx_access索引
elasticsearch {
hosts => "http://10.0.0.51:9200"
manage_template => false
index => "nginx_access-%{+yyyy.MM.dd}"
}
}
if "error" in [tags] { # # key值中含有error的傳輸?shù)紼s服務器nginx_error索引
elasticsearch {
hosts => "http://10.0.0.51:9200"
manage_template => false
index => "nginx_error-%{+yyyy.MM.dd}"
}
}
}
#5.啟動logstash（前臺啟動，啟動速度慢）
[root@db01 /etc/elasticsearch]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/redis.conf
10.kibana數(shù)據(jù)圖制作
上傳測試數(shù)據(jù)，然后開始kibana-web頁面設置
image.png
image.png
image.png
image.png
image.png
image.png
image.png
image.png
image.png
image.png
image.png
11.kibana監(jiān)控ES集群
image.png
image.png
image.png
12.使用kafka作為緩存
三臺服務器配置hosts，并可以互相ping通
[root@db01 ~]# vim /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
10.0.0.51 db01
10.0.0.52 db02
10.0.0.53 db03
- 下載安裝并驗證zookeeper
zookeeper下載地址http://zookeeper.apache.org/releases.html
kafka下載地址http://kafka.apache.org/downloads.html
zookeeper集群特性：整個集群中只要有超過集群數(shù)量一半的zookeeper工作是正常的，那么整個集群對外就是可用的，例如有2臺服務器做一個zaookeeper，只要有任何一臺故障或宕機，那么這個zookeeper集群就是不可用的了。因為剩下的一臺沒有超過集群的一半的數(shù)量,但是假如有三臺zookeeper組成一個集群,那么損壞一臺還剩兩臺,大于3臺的一半,所以損壞一臺還是可以正常運行的,但是再損壞一臺就只剩下一臺,集群就不可用了.
如果是4臺組成,損壞一臺正常,損壞兩臺還剩兩臺,不滿足集群總數(shù)的一半,所以3臺的集群和4臺的集群算壞兩臺的結(jié)果都是集群不可用.所以這也是為什么集群一般是奇數(shù)的原因.
#上傳zookeeper-3.4.11.tar.gz和kafka_2.11-1.0.0.tgz
[root@db01 /data/soft]# tar zxf zookeeper-3.4.11.tar.gz -C /opt/
[root@db01 /data/soft]# ln -s /opt/zookeeper-3.4.11/ /opt/zookeeper
[root@db01 /data/soft]# ll /opt/
[root@db01 /data/soft]# mkdir -p /data/zookeeper
[root@db01 /data/soft]# cp /opt/zookeeper/conf/zoo_sample.cfg /opt/zookeeper/conf/zoo.cfg #將簡易配置文件修改為正式配置文件
# 編輯保留的配置文件
[root@db01 /data/soft]# vim /opt/zookeeper/conf/zoo.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/data/zookeeper
clientPort=2181
server.1=10.0.0.51:2888:3888
server.2=10.0.0.52:2888:3888
server.3=10.0.0.53:2888:3888
# 用rsync發(fā)送zookeeper給另外兩臺服務器
[root@db01 /opt]# rsync -avz zookeeper* db02:/opt/
[root@db01 /opt]# rsync -avz zookeeper* db03:/opt/
#用rsync 發(fā)送數(shù)據(jù)目錄
[root@db01 /opt]# rsync -avz /data/zookeeper db02:/data/
[root@db01 /opt]# rsync -avz /data/zookeeper db03:/data/
# 修改三臺服務器的myid
[root@db01 /opt]# echo "1" > /data/zookeeper/myid # 填寫的數(shù)字必須對應zoo,cfg配置中的server.$
[root@db02 /opt]# echo "2" > /data/zookeeper/myid
[root@db03 /opt]# echo "3" > /data/zookeeper/myid
# 各節(jié)點啟動zookeeper
[root@db01 /opt]# /opt/zookeeper/bin/zkServer.sh start
[root@db01 /opt]# /opt/zookeeper/bin/zkServer.sh status
#Mode: leader為主、Mode: follower為從（目前集群為高可用架構(gòu)，可以壞一臺）
- zookeeper 簡單操作命令測試
連接到任意節(jié)點生成數(shù)據(jù)：
我們在節(jié)點1生成數(shù)據(jù)，然后在其他節(jié)點驗證數(shù)據(jù)
#其中一個點節(jié)點創(chuàng)建一條hello的數(shù)據(jù)
[root@db01 /opt]# /opt/zookeeper/bin/zkCli.sh -server db01:2181
[zk: db01:2181(CONNECTED) 0] create /test "hello"
#集群其他節(jié)點可查看到
[root@db02 /opt]# /opt/zookeeper/bin/zkCli.sh -server db02:2181
[zk: db02:2181(CONNECTED) 0] get /test
hello
cZxid = 0x100000002
ctime = Tue Apr 07 20:07:30 CST 2020
mZxid = 0x100000002
mtime = Tue Apr 07 20:07:30 CST 2020
pZxid = 0x100000002
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 5
numChildren = 0
- 安裝部署kafka
# 上傳kafka安裝包
[root@db01 /data/soft]# tar xf kafka_2.11-1.0.0.tgz -C /opt/
[root@db01 /data/soft]# ln -s /opt/kafka_2.11-1.0.0/ /opt/kafka
[root@db01 /data/soft]# mkdir /opt/kafka/logs
[root@db01 /data/soft]# vim /opt/kafka/config/server.properties
broker.id=1 # 填寫的數(shù)字必須對應zoo,cfg配置中的server.$
listeners=PLAINTEXT://10.0.0.51:9092 # 本機ip地址
log.dirs=/opt/kafka/logs
log.retention.hours=24
zookeeper.connect=10.0.0.51:2181,10.0.0.52:2181,10.0.0.53:2181 #zookeeper集群ip
#將配置文件和kafka一同發(fā)送到集群中的其他機器
[root@db01 /data/soft]# rsync -avz /opt/kafka* db02:/opt/
[root@db01 /data/soft]# rsync -avz /opt/kafka* db03:/opt/
#其他兩臺機器修改kafka配置文件
[root@db02 /opt]# vim kafka/config/server.properties
broker.id=2 # 填寫的數(shù)字必須對應zoo,cfg配置中的server.$
listeners=PLAINTEXT://10.0.0.52:9092 # 本機IP
log.dirs=/opt/kafka/logs
log.retention.hours=24
zookeeper.connect=10.0.0.51:2181,10.0.0.52:2181,10.0.0.53:2181
----------------------------------------------------------------------------------------------------------------------------------------------
[root@db03 /opt]# vim kafka/config/server.properties
broker.id=3 # 填寫的數(shù)字必須對應zoo,cfg配置中的server.$
listeners=PLAINTEXT://10.0.0.53:9092 # 本機IP
log.dirs=/opt/kafka/logs
log.retention.hours=24
zookeeper.connect=10.0.0.51:2181,10.0.0.52:2181,10.0.0.53:2181
- 各節(jié)點啟動kafka
# 節(jié)點1，可以先前臺啟動，方便查看日志報錯（3臺測試）
[root@db01 /data/soft]# /opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
# 最后一行出現(xiàn)KafkaServer id 和started 字樣，就表明啟動成功了， 然后就可以放到后臺啟動了（集群3臺啟動）
[root@db01 /data/soft]# /opt/kafka/bin/kafka-server-start.sh -daemon /opt/kafka/config/server.properties #（3臺都啟動）
[root@db01 /data/soft]# tail -f /opt/kafka/logs/server.log
# 創(chuàng)建測試
[root@db01 ~]# /opt/kafka/bin/kafka-topics.sh --create --zookeeper 10.0.0.51:2181,10.0.0.52:2181,10.0.0.53:2181 --partitions 3 --replication-factor 3 --topic kafkatest
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
Created topic "kafkatest".
#測試獲取TOPPID，任意一臺機測試
[root@db01 ~]# /opt/kafka/bin/kafka-topics.sh --describe --zookeeper 10.0.0.51:2181,10.0.0.52:2181,10.0.0.53:2181 --topic kafkatest
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
Topic:kafkatest PartitionCount:3 ReplicationFactor:3 Configs:
Topic: kafkatest Partition: 0 Leader: 2 Replicas: 2,1,3 Isr: 2,1,3
Topic: kafkatest Partition: 1 Leader: 3 Replicas: 3,2,1 Isr: 3,2,1
Topic: kafkatest Partition: 2 Leader: 1 Replicas: 1,3,2 Isr: 1,3,2
# 數(shù)據(jù)測試創(chuàng)建一個名為messagetest的topic
[root@db01 ~]# /opt/kafka/bin/kafka-topics.sh --create --zookeeper 10.0.0.51:2181,10.0.0.52:2181,10.0.0.53:2181 --partitions 3 --replication-factor 3 --topic messagetest
#登錄發(fā)送消息
#發(fā)送消息:注意,端口是 kafka的9092,而不是zookeeper的2181
#producer.sh發(fā)送消失這、consumer.sh接收消息者
[root@db01 ~]# /opt/kafka/bin/kafka-console-producer.sh --broker-list 10.0.0.51:9092,10.0.0.52:9092,10.0.0.53:9092 --topic messagetest
#其他節(jié)點登錄接收消息者腳本
[root@db02 ~]# /opt/kafka/bin/kafka-console-consumer.sh --zookeeper 10.0.0.51:2181,10.0.0.52:2181,10.0.0.53:2181 --topic messagetest --from-beginning
[root@db03 ~]# /opt/kafka/bin/kafka-console-consumer.sh --zookeeper 10.0.0.51:2181,10.0.0.52:2181,10.0.0.53:2181 --topic messagetest --from-beginning
- 配置filebeat配置文件（發(fā)送到kafka）
[root@db01 ~]# vim /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/nginx/access.log
json.keys_under_root: true
json.overwrite_keys: true
tags: ["access"]
- type: log
enabled: true
paths:
- /var/log/nginx/error.log
tags: ["error"]
setup.kibana:
host: "10.0.0.51:5601"
output.kafka:
hosts: ["10.0.0.51:9092","10.0.0.52:9092","10.0.0.53:9092"]
topic: elklog # 創(chuàng)建一個話題框
- logstatsh配置文件
[root@db01 ~]# vim /etc/logstash/conf.d/kafka.conf
input{
kafka{
bootstrap_servers=>"10.0.0.51:9092" # kafka集群中任意一臺
topics=>["elklog"] # 對應filebeat 的話題框
group_id=>"logstash"
codec => "json"
}
}
filter {
mutate {
convert => ["upstream_time", "float"] convert => ["request_time", "float"]
}
}
output {
if "access" in [tags] {
elasticsearch {
hosts => "http://10.0.0.51:9200"
manage_template => false
index => "nginx_access-%{+yyyy.MM}"
}
}
if "error" in [tags] {
elasticsearch {
hosts => "http://10.0.0.51:9200"
manage_template => false
index => "nginx_error-%{+yyyy.MM}"
}
}
}
--------------------------------------------------------------------------------------------------------
#啟動logstash
[root@db01 /etc/elasticsearch]# systemctl restart filebeat
[root@db01 /etc/elasticsearch]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/kafka.conf
13.使用nginx+keepalived代理多臺redis
image.png
redis集群方案有哨兵和集群，但可惜的是filebeat和logstash都不支持這兩種方案。但是我們可采用keepalived的方式解決
1.使用Nginx+keepalived反向代理負載均衡到后面的多臺Redis
2.考慮Redis故障切換數(shù)據(jù)一致性的問題，所有最好我們使用2臺Redis，并且只工作一臺，另外一臺作為backup，只有第一臺壞掉后，第二臺才會工作。
3.filebeat的oputut的Redis地址為Keepalived的虛擬IP
4.logstash可以啟動多個節(jié)點來加速讀取Redis的數(shù)據(jù)
5.后端可以采用多臺es集群來做支撐
- 安裝keepalived
#兩臺服務器安裝keepalived
[root@db01 ~]# yum install keepalived
[root@db01 ~]# vim /etc/keepalived/keepalived.conf
router_id lb01
}
vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 50
priority 150 # 權(quán)重
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
10.0.0.3
}
------------------------------------------------------------------------------------------------------------------------------------------
[root@db02 ~]# vim /etc/keepalived/keepalived.conf
vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 50
priority 100 #權(quán)重改小
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
10.0.0.3
}
}
- 安裝redis
mkdir /data/soft -p
cd /data/soft
# 上傳安裝包
tar xf redis-5.0.7.tar.gz -C /opt/
ln -s /opt/redis-5.0.7 /opt/redis
cd /opt/redis
make
make install
mkdir -p /opt/redis_6379/{conf,pid,logs}
mkdir -p /data/redis_6379
vim /opt/redis_6379/conf/redis_6379.conf
daemonize yes
bind 127.0.0.1 10.0.0.52
port 6379
pidfile /opt/redis_6379/pid/redis_6379.pid
logfile /opt/redis_6379/logs/redis_6379.log
save 900 1
save 300 10
save 60 10000
dbfilename redis.rdb
dir /data/redis_6379/
#啟動
redis-server /opt/redis_6379/conf/redis_6379.conf
- 配置Nginx反向代理(4層代理)
# 在配置文件底部加入stram模塊
[root@db01 /data]# vim /etc/nginx/nginx.conf
stream {
upstream redis {
server 10.0.0.52:6379 max_fails=2 fail_timeout=10s;
server 10.0.0.53:6379 max_fails=2 fail_timeout=10s backup; #backup的參數(shù)是在IP52Redis存活時不啟動。宕機時才啟動頂替52繼續(xù)工作（備用服務器）
}
server {
listen 6379;
proxy_connect_timeout 1s;
proxy_timeout 3s;
proxy_pass redis;
}
}
# 語法檢查后重啟Nginx服務
[root@db01 /data]# nginx -t
[root@db01 /data]# systemctl restart nginx
[root@db01 /data]# ss -lntup|grep nginx
tcp LISTEN 0 128 *:6379 *:* users:(("nginx",pid=2261,fd=7),("ngin",pid=2260,fd=7))
tcp LISTEN 0 128 *:80 *:* users:(("nginx",pid=2261,fd=6),("ngin",pid=2260,fd=6))
# 遠程登錄測試是否反向代理成功。（0.3是keepalived的VIP網(wǎng)卡）
# Nginx配置文件stream不加backup時，正常是負載均衡
[root@db01 /data]# redis-cli -h 10.0.0.3
- 配置filebeta配置文件
[root@db01 ~]# vim /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/nginx/access.log
json.keys_under_root: true
json.overwrite_keys: true
tags: ["access"]
- type: log
enabled: true
paths:
- /var/log/nginx/error.log
tags: ["error"]
setup.kibana:
host: "10.0.0.51:5601"
output.redis:
hosts: ["10.0.0.3"] # 填寫keepalived的IP地址
key: "filebeat"
#重啟filebeat
systemctl resatrt filebeat
- 配置logstash配置文件
[root@db01 ~]# vim /etc/logstash/conf.d/redis.conf
input {
redis {
host => "10.0.0.3" # 填寫keepalived的IP地址
port => "6379"
db => "0"
key => "filebeat"
data_type => "list"
}
}
filter {
mutate {
convert => ["upstream_time", "float"]
convert => ["request_time", "float"]
}
}
output {
if "access" in [tags] {
elasticsearch {
hosts => "http://10.0.0.51:9200"
manage_template => false
index => "nginx_access-%{+yyyy.MM.dd}"
}
}
if "error" in [tags] {
elasticsearch {
hosts => "http://10.0.0.51:9200"
manage_template => false
index => "nginx_error-%{+yyyy.MM.dd}"
}
}
}
#啟動logstash
[root@db01 /data]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/redis.conf
#ab壓測產(chǎn)生數(shù)據(jù)
[root@db02 /opt/redis]# ab -n 20000 -c 20 http://10.0.0.51/tt
image.png
es每存儲一條數(shù)據(jù)，Redis中就會少一條時間。成功替ES服務器減少壓力。、
image.png
image.png
