filebeat+redis+logstash+els集群+kibana完成搜索過(guò)程

1.實(shí)驗(yàn)環(huán)境：

??使用8臺(tái)CentOS主機(jī)，實(shí)現(xiàn)filebeat+redis+logstash+els集群(3臺(tái))+kibana來(lái)完成搜索日志相關(guān)內(nèi)容，目標(biāo)：filebeat來(lái)完成收集本機(jī)http數(shù)據(jù)，收集完成后發(fā)送給redis，redis主要是來(lái)避免數(shù)據(jù)量過(guò)大，logstash處理不過(guò)來(lái)，logstash是用來(lái)格式化數(shù)據(jù)，將收集來(lái)的數(shù)據(jù)格式化成指定格式，els集群是將格式化完成的數(shù)據(jù)，進(jìn)行文檔分析，，構(gòu)建索引，提供查詢等操作，kibana提供圖形化界面查詢的組件
邏輯拓?fù)鋱D


邏輯拓?fù)鋱D.png

2.實(shí)驗(yàn)步驟

本實(shí)驗(yàn)所用的四個(gè)軟件包全部都是5.6版本
下載相關(guān)網(wǎng)站：https://www.elastic.co/cn/products
配置前注意事項(xiàng)：1.關(guān)閉防火墻。2.關(guān)閉SELinux。3.同步時(shí)間
步驟1.實(shí)現(xiàn)收集httpd服務(wù)的日志文件，并將數(shù)據(jù)發(fā)送給redis服務(wù)
http+filebeat服務(wù)器相關(guān)配置

[root@filebeat ~]# yum install -y httpd
[root@filebeat ~]# echo test > /var/www/html/index.html
[root@filebeat ~]# systemctl start httpd
[root@filebeat ~]# rpm -ivh filebeat-5.6.10-x86_64.rpm
相關(guān)配置文件
/etc/filebeat/filebeat.full.yml  #模板配置文件
/etc/filebeat/filebeat.yml 主配置文件
配置redis需要從模板文件中將模板復(fù)制到主配置文件中
output.redis:
  enabled: true   #開(kāi)啟
  hosts: ["172.18.100.2:6379"] #redis服務(wù)器
  port: 6379 
  key: filebeat  #key的名字
  password:  centos #密碼若沒(méi)有設(shè)置則不用填
  db: 0 #寫(xiě)入哪個(gè)數(shù)據(jù)庫(kù)
  datatype: list  #數(shù)據(jù)類型
  worker: 1  #開(kāi)幾個(gè)進(jìn)行寫(xiě)數(shù)據(jù)
  loadbalance: true  #是否支持將多個(gè)redis中寫(xiě)入
[root@filebeat ~]# systemctl start filebeat

redis相關(guān)配置

[root@redis ~]# yum install -y redis
[root@redis ~]# vim /etc/redis.conf
bind 0.0.0.0
port 6379
requirepass centos
[root@nginx1 ~]# systemctl start redis
增加訪問(wèn)日志，在redis中查詢
[root@nginx1 ~]# redis-cli -a centos
127.0.0.1:6379> KEYS *
1) "filebeat"   #即可驗(yàn)證成功

步驟2配置logstash從redis中拿數(shù)據(jù)，并且格式化，然后存入elasticsearch，并且顯示
logstash相關(guān)配置，配置該服務(wù)之前需要安裝JVM相關(guān)組件

[root@nginx2 ~]# rpm -ivh logstash-5.6.10.rpm
[root@nginx2 ~]# cd /etc/logstash/conf.d/
[root@nginx2 conf.d]# vim redis-logstash-els.conf  #創(chuàng)建文件，只要以.conf結(jié)尾即可
input {
        redis {
                batch_count => 1
                data_type => "list"
                key => "filebeat"
                host => "172.18.100.2"
                port => 6379
                password => "centos"
                threads => 5
        }
}
filter {
        grok {
                match => {
                        "message" => "%{HTTPD_COMBINEDLOG}"
                }
                        remove_field => "message"
        }
        date {
                match => ["timestamp","dd/MM/YYYY:H:m:s Z"]
                remove_field => "timestamp"
        }

}
output {
        stdout {
                codec => rubydebug
        }
}
在終端顯示格式化好的內(nèi)容
[root@nginx2 conf.d]# /usr/share/logstash/bin/logstash -f redis-logstash-els.conf
{
        "request" => "/",
          "agent" => "\"curl/7.29.0\"",
         "offset" => 93516,
           "auth" => "-",
          "ident" => "-",
     "input_type" => "log",
           "verb" => "GET",
         "source" => "/var/log/httpd/access_log",
           "type" => "log",
           "tags" => [
        [0] "_dateparsefailure"
    ],
       "referrer" => "\"-\"",
     "@timestamp" => 2018-06-20T15:21:20.094Z,
       "response" => "200",
          "bytes" => "5",
       "clientip" => "127.0.0.1",
           "beat" => {
            "name" => "filebeat.test.com",
        "hostname" => "filebeat.test.com",
         "version" => "5.6.10"
    },
       "@version" => "1",
    "httpversion" => "1.1",
      "timestamp" => "20/Jun/2018:11:21:19 -0400"
}
將output修改成傳遞給els集群
output {
        elasticsearch {
                hosts => ["http://172.18.100.4:9200/","http://172.18.100.5:9200/","http://172.18.100.6:9200/"]
                index => "logstash-%{+YYYY.MM.dd}"
                document_type => "apache_logs"
        }
}


檢查沒(méi)有錯(cuò)誤即可
[root@nginx2 conf.d]# /usr/share/logstash/bin/logstash -f redis-logstash-els.conf  -t
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
Configuration OK

步驟3配置els集群服務(wù)，需要先安裝JVM服務(wù)
節(jié)點(diǎn)1：

[root@tomcat1 ~]# rpm -ivh elasticsearch-5.6.10.rpm
[root@els1 ~]# vim /etc/elasticsearch/elasticsearch.yml
cluster.name: myels
node.name: els.test.com
network.host: 172.18.100.4
http.port: 9200
discovery.zen.ping.unicast.hosts: ["172.18.100.4", "172.18.100.5","172.18.100.6"]
discovery.zen.minimum_master_nodes: 2

節(jié)點(diǎn)2：

[root@tomcat1 ~]# rpm -ivh elasticsearch-5.6.10.rpm
[root@els1 ~]# vim /etc/elasticsearch/elasticsearch.yml
cluster.name: myels
node.name: els.test.com
network.host: 172.18.100.5
http.port: 9200
discovery.zen.ping.unicast.hosts: ["172.18.100.4", "172.18.100.5","172.18.100.6"]
discovery.zen.minimum_master_nodes: 2

節(jié)點(diǎn)3：

[root@tomcat1 ~]# rpm -ivh elasticsearch-5.6.10.rpm
[root@els1 ~]# vim /etc/elasticsearch/elasticsearch.yml
cluster.name: myels
node.name: els.test.com
network.host: 172.18.100.6
http.port: 9200
discovery.zen.ping.unicast.hosts: ["172.18.100.4", "172.18.100.5","172.18.100.6"]
discovery.zen.minimum_master_nodes: 2

在els任意一個(gè)節(jié)點(diǎn)上查看數(shù)據(jù)

[root@els1 ~]# curl -XGET 'http://172.18.100.4:9200/logstash-2018.06.21?pretty=true'  顯示傳過(guò)來(lái)的數(shù)據(jù)
    "settings" : {
      "index" : {
        "refresh_interval" : "5s",
        "number_of_shards" : "5",
        "provided_name" : "logstash-2018.06.21",
        "creation_date" : "1529545212157",
        "number_of_replicas" : "1",
        "uuid" : "3n74gNpCQUyCLq58vAwL6A",
        "version" : {
          "created" : "5061099"
        }
      }
    }
  }
}

步驟4：配置Nginx反向代理，若其中有一個(gè)故障，還可以

[root@mysql1 ~]# yum install -y nginx
[root@mysql1 ~]# vim /etc/nginx/conf.d/test.conf
upstream ser {
        server 172.18.100.4:9200;
        server 172.18.100.5:9200;
        server 172.18.100.6:9200;
}
server {
        listen 80;
        server_name www.test.com;
        root /app/;
        index index.html;
        location / {
                proxy_pass http://ser;
        }
}

步驟5：配置kibana實(shí)現(xiàn)圖形化查看

server.host: "0.0.0.0"
server.basePath: ""
server.name: "172.18.100.8"
elasticsearch.url: "http://172.18.100.7:80"  #反向代理服務(wù)器
elasticsearch.preserveHost: true
kibana.index: ".kibana"
image.png

image.png
?著作權(quán)歸作者所有,轉(zhuǎn)載或內(nèi)容合作請(qǐng)聯(lián)系作者
【社區(qū)內(nèi)容提示】社區(qū)部分內(nèi)容疑似由AI輔助生成，瀏覽時(shí)請(qǐng)結(jié)合常識(shí)與多方信息審慎甄別。
平臺(tái)聲明：文章內(nèi)容（如有圖片或視頻亦包括在內(nèi)）由作者上傳并發(fā)布，文章內(nèi)容僅代表作者本人觀點(diǎn)，簡(jiǎn)書(shū)系信息發(fā)布平臺(tái)，僅提供信息存儲(chǔ)服務(wù)。

相關(guān)閱讀更多精彩內(nèi)容

友情鏈接更多精彩內(nèi)容