prometheus配置詳解

本文按照官方文檔的相關(guān)內(nèi)容整理整理的配置語法以及實(shí)現(xiàn)功能

1.prometheus 配置文件主體

# 此片段指定的是prometheus的全局配置， 比如采集間隔，抓取超時(shí)時(shí)間等.
global:
  # 抓取間隔
  [ scrape_interval: <duration> | default = 1m ]

  # 抓取超時(shí)時(shí)間
  [ scrape_timeout: <duration> | default = 10s ]

  # 評估規(guī)則間隔
  [ evaluation_interval: <duration> | default = 1m ]

  # 外部一些標(biāo)簽設(shè)置
  external_labels:
    [ <labelname>: <labelvalue> ... ]

  # File to which PromQL queries are logged.
  # Reloading the configuration will reopen the file.
  [ query_log_file: <string> ]

# 此片段指定報(bào)警規(guī)則文件， prometheus根據(jù)這些規(guī)則信息，會推送報(bào)警信息到alertmanager中。
rule_files:
  [ - <filepath_glob> ... ]

# 此片段指定抓取配置，prometheus的數(shù)據(jù)采集通過此片段配置。
scrape_configs:
  [ - <scrape_config> ... ]

# 此片段指定報(bào)警配置， 這里主要是指定prometheus將報(bào)警規(guī)則推送到指定的alertmanager實(shí)例地址。
alerting:
  alert_relabel_configs:
    [ - <relabel_config> ... ]
  alertmanagers:
    [ - <alertmanager_config> ... ]

# 指定后端的存儲的寫入api地址。
remote_write:
  [ - <remote_write> ... ]

# 指定后端的存儲的讀取api地址。
remote_read:
  [ - <remote_read> ... ]

2.scrape_configs配置詳解

一個scrape_config 片段指定一組目標(biāo)和參數(shù)， 目標(biāo)就是實(shí)例，指定采集的端點(diǎn)， 參數(shù)描述如何采集這些實(shí)例， 配置文件格式如下

# The job name assigned to scraped metrics by default.
job_name: <job_name>

# 抓取間隔,默認(rèn)繼承g(shù)lobal值。
[ scrape_interval: <duration> | default = <global_config.scrape_interval> ]

# 抓取超時(shí)時(shí)間,默認(rèn)繼承g(shù)lobal值。
[ scrape_timeout: <duration> | default = <global_config.scrape_timeout> ]

# 抓取路徑， 默認(rèn)是/metrics
[ metrics_path: <path> | default = /metrics ]

# honor_labels controls how Prometheus handles conflicts between labels that are
# already present in scraped data and labels that Prometheus would attach
# server-side ("job" and "instance" labels, manually configured target
# labels, and labels generated by service discovery implementations).
#
# If honor_labels is set to "true", label conflicts are resolved by keeping label
# values from the scraped data and ignoring the conflicting server-side labels.
#
# If honor_labels is set to "false", label conflicts are resolved by renaming
# conflicting labels in the scraped data to "exported_<original-label>" (for
# example "exported_instance", "exported_job") and then attaching server-side
# labels.
#
# Setting honor_labels to "true" is useful for use cases such as federation and
# scraping the Pushgateway, where all labels specified in the target should be
# preserved.
#
# Note that any globally configured "external_labels" are unaffected by this
# setting. In communication with external systems, they are always applied only
# when a time series does not have a given label yet and are ignored otherwise.
[ honor_labels: <boolean> | default = false ]

# honor_timestamps controls whether Prometheus respects the timestamps present
# in scraped data.
#
# If honor_timestamps is set to "true", the timestamps of the metrics exposed
# by the target will be used.
#
# If honor_timestamps is set to "false", the timestamps of the metrics exposed
# by the target will be ignored.
[ honor_timestamps: <boolean> | default = true ]

# 指定采集使用的協(xié)議，http或者h(yuǎn)ttps。
[ scheme: <scheme> | default = http ]

# 指定url參數(shù)。
params:
  [ <string>: [<string>, ...] ]

# 指定認(rèn)證信息。
basic_auth:
  [ username: <string> ]
  [ password: <secret> ]
  [ password_file: <string> ]

# 指定token的數(shù)值， 用戶get metrics認(rèn)證使用
[ bearer_token: <secret> ]

# 指定獲取token的文件， 用戶get metrics認(rèn)證使用
[ bearer_token_file: /path/to/bearer/token/file ]

# 指定獲取metrics時(shí)需要的tls證書
tls_config:
  [ <tls_config> ]

# Optional proxy URL.
[ proxy_url: <string> ]

# List of Azure service discovery configurations.
azure_sd_configs:
  [ - <azure_sd_config> ... ]

# List of Consul service discovery configurations.
consul_sd_configs:
  [ - <consul_sd_config> ... ]

# List of DNS service discovery configurations.
dns_sd_configs:
  [ - <dns_sd_config> ... ]

# List of EC2 service discovery configurations.
ec2_sd_configs:
  [ - <ec2_sd_config> ... ]

# List of OpenStack service discovery configurations.
openstack_sd_configs:
  [ - <openstack_sd_config> ... ]

# List of file service discovery configurations.
file_sd_configs:
  [ - <file_sd_config> ... ]

# List of GCE service discovery configurations.
gce_sd_configs:
  [ - <gce_sd_config> ... ]

# List of Kubernetes service discovery configurations.
kubernetes_sd_configs:
  [ - <kubernetes_sd_config> ... ]

# List of Marathon service discovery configurations.
marathon_sd_configs:
  [ - <marathon_sd_config> ... ]

# List of AirBnB's Nerve service discovery configurations.
nerve_sd_configs:
  [ - <nerve_sd_config> ... ]

# List of Zookeeper Serverset service discovery configurations.
serverset_sd_configs:
  [ - <serverset_sd_config> ... ]

# List of Triton service discovery configurations.
triton_sd_configs:
  [ - <triton_sd_config> ... ]

# 靜態(tài)指定服務(wù)job
static_configs:
  [ - <static_config> ... ]

# 控制采集哪些數(shù)據(jù)標(biāo)簽，可以刪除不必要的標(biāo)簽
relabel_configs:
  [ - <relabel_config> ... ]

# 添加、編輯或修改指標(biāo)的標(biāo)簽值或標(biāo)簽格式。
metric_relabel_configs:
  [ - <relabel_config> ... ]

# Per-scrape limit on number of scraped samples that will be accepted.
# If more than this number of samples are present after metric relabelling
# the entire scrape will be treated as failed. 0 means no limit.
[ sample_limit: <int> | default = 0 ]

因?yàn)椴渴鹪趉ubernetes環(huán)境中所以我只在意基于kubernetes_sd_configs的服務(wù)發(fā)現(xiàn)和static_configs靜態(tài)文件的發(fā)現(xiàn)

2.1 relabel_configs

relable_configss是功能強(qiáng)大的工具，就是Relabel可以在Prometheus采集數(shù)據(jù)之前，通過Target實(shí)例的Metadata信息，動態(tài)重新寫入Label的值。除此之外，我們還能根據(jù)Target實(shí)例的Metadata信息選擇是否采集或者忽略該Target實(shí)例。

relabel_configs

配置格式如下：

# The source labels select values from existing labels. Their content is concatenated
# using the configured separator and matched against the configured regular expression
# for the replace, keep, and drop actions.
[ source_labels: '[' <labelname> [, ...] ']' ]

# 默認(rèn)分隔符
[ separator: <string> | default = ; ]

# Label to which the resulting value is written in a replace action.
# It is mandatory for replace actions. Regex capture groups are available.
[ target_label: <labelname> ]

# Regular expression against which the extracted value is matched.
[ regex: <regex> | default = (.*) ]

# Modulus to take of the hash of the source label values.
[ modulus: <uint64> ]

# Replacement value against which a regex replace is performed if the
# regular expression matches. Regex capture groups are available.
[ replacement: <string> | default = $1 ]

# Action to perform based on regex matching.
[ action: <relabel_action> | default = replace ]

其中action主要包括:

replace：默認(rèn)，通過regex匹配source_label的值，使用replacement來引用表達(dá)式匹配的分組
keep：刪除regex與連接不匹配的目標(biāo) source_labels
drop：刪除regex與連接匹配的目標(biāo) source_labels
labeldrop：刪除regex匹配的標(biāo)簽
labelkeep：刪除regex不匹配的標(biāo)簽
hashmod：設(shè)置target_label為modulus連接的哈希值source_labels
labelmap：匹配regex所有標(biāo)簽名稱。然后復(fù)制匹配標(biāo)簽的值進(jìn)行分組，replacement分組引用（{1},{2},…）替代

prometheus中的數(shù)值都是key:value格式， 其中replace、keep、drop都是對value的操作， labelmap、labeldrop、labelkeep都是對key的操作

replace用法

replace是action的默認(rèn)值， 通過regex匹配source_label的值，使用replacement來引用表達(dá)式匹配的分組

  - action: replace
    regex: ([^:]+)(?::\d+)?;(\d+)
    replacement: $1:$2
    source_labels:
    - __address__
    - __meta_kubernetes_service_annotation_prometheus_io_port
    target_label: __address__

上面的列子中address的值為$1:$2， 其中 $1 是正則表達(dá)式([^:]+)(?::\d+)?從address中獲取， $2是正則表達(dá)式(\d+)從(\d+)中獲取， 最后的address的數(shù)值為192.168.1.1:9100

keep用法

relabel_configs:
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe]
    action: keep
    regex: true

上面的例子只要匹配__meta_kubernetes_service_annotation_prometheus_io_probe=true數(shù)據(jù)就保留， 反正source_labels中的值沒有匹配regex中的值就丟棄

drop用法

drop 的使用和keep剛好相反， 還是使用keep的例子:

relabel_configs:
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe]
    action: keep
    regex: true

上面的例子只要__meta_kubernetes_service_annotation_prometheus_io_probe這個標(biāo)簽的值為true就丟棄， 反之如果__meta_kubernetes_service_annotation_prometheus_io_probe!=true的數(shù)據(jù)就保留

labelmap用法

labelmap的用法和上面說到replace、keep、drop不同， labelmap匹配的是標(biāo)簽名稱， 而replace、keep、drop匹配的是value

relabel_configs:
  - action: labelmap
    regex: __meta_kubernetes_service_label_(.+)

上面例子中只要匹配到正則表達(dá)式__meta_kubernetes_service_label_(.+)的標(biāo)簽， 就將標(biāo)簽重寫為(.+)中的內(nèi)容， 效果如下：

原標(biāo)簽： __meta_kubernetes_service_label_test=111 
重寫后： test=111

hashmod用法

待續(xù)

2.1.6 labeldrop用法

使用labeldrop則可以對Target標(biāo)簽進(jìn)行過濾，刪除符合過濾條件的標(biāo)簽，例如：

relabel_configs:
  - action: labeldrop
    regex: __meta_kubernetes_service_label_(.+)

該配置會使用regex匹配當(dāng)前target中的所有標(biāo)簽， 刪除符合規(guī)則的標(biāo)簽， 反之保留不符合規(guī)則的

labelkeep用法

使用labelkeep則可以對Target標(biāo)簽進(jìn)行過濾，僅保留符合過濾條件的標(biāo)簽，例如：

relabel_configs:
  - action: labelkeep
    regex: __meta_kubernetes_service_label_(.+)

該配置會使用regex匹配當(dāng)前target中的所有標(biāo)簽， 保留符合規(guī)則的標(biāo)簽， 反之不符合的移除

2.2 metric_relabel_configs

上面我們說到relabel_config是獲取metrics之前對標(biāo)簽的重寫， 對應(yīng)的metric_relabel_configs是對獲取metrics之后對標(biāo)簽的操作， metric_relabel_configs能夠確定我們保存哪些指標(biāo)，刪除哪些指標(biāo)，以及這些指標(biāo)將是什么樣子。

metric_relabel_configs的配置和relabel_config的配置基本相同， 如果需要配置相關(guān)參數(shù)請參考2.scrape_configs

2.2 static_configs

主要用途為指定exporter獲取metrics數(shù)據(jù)的目標(biāo)， 可以指定prometheus、 mysql、 nginx等目標(biāo)

    scrape_configs:
    - job_name: prometheus
      static_configs:
      - targets:
        - localhost:9090

此規(guī)則主要是用于抓取prometheus自己數(shù)據(jù)的配置， targets列表中的為prometheus 獲取metrics的地址和端口， 因?yàn)闆]有指定metrics_path所以使用默認(rèn)的/metrics中獲取數(shù)據(jù)，

簡單理解就是， prometheus訪問 http://localhost:9090/metrics 獲取監(jiān)控?cái)?shù)據(jù)

還可以配置指定exporter中的目的地址， 如獲取node_exporter的數(shù)據(jù)

scrape_configs: 
- job_name: node 
  static_configs: 
  - targets: 
    - 10.40.58.153:9100 
    - 10.40.61.116:9100 
    - 10.40.58.154:9100

簡單理解為分別訪問 http://10.40.58.153:9100/metrics http://10.40.58.154:9100/metrics http://10.40.61.116:9100/metrics 獲取metrics數(shù)據(jù)

2.3 kubernetes_sd_configs

kubernetes的服務(wù)發(fā)現(xiàn)可以刮取以下幾種數(shù)據(jù)

  • node
  • service
  • pod
  • endpoints
  • ingress

通過指定kubernetes_sd_config的模式為endpoints，Prometheus會自動從Kubernetes中發(fā)現(xiàn)到所有的endpoints節(jié)點(diǎn)并作為當(dāng)前Job監(jiān)控的Target實(shí)例。如下所示，

kubernetes_sd_configs:   
- role: endpoints

配置實(shí)例一

該配置是使用kubernetes的發(fā)現(xiàn)機(jī)制發(fā)現(xiàn)kube-apiservers

scrape_configs:
- bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  job_name: kubernetes-apiservers 
  kubernetes_sd_configs:   
  - role: endpoints
  relabel_configs:
  - action: keep
    regex: default;kubernetes;https
    source_labels:
    - __meta_kubernetes_namespace
    - __meta_kubernetes_service_name
    - __meta_kubernetes_endpoint_port_name
  scheme: https
  tls_config:
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    insecure_skip_verify: true

上面的刮取配置定義了如下信息：

  • job名稱為kubernetes-apiservers(job-name: kubernetes-apiservers)
  • 獲取kubernetes中endpoints的相關(guān)信息(role: endpoints)
  • 使用https的方式獲取信息(scheme: https)
  • target的需要滿足default名稱空間下service名字為kubernetes，并且端口為https
    • __meta_kubernetes_namespace=～default
    • __meta_kubernetes_service_name=～kubernetes
    • __meta_kubernetes_endpoint_port_name=～=https

配置實(shí)例二

該配置是自動發(fā)現(xiàn)kubernetes中的endpoints

- job_name: 'kubernetes-service-endpoints'

kubernetes_sd_configs:
  - role: endpoints

relabel_configs:
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
    action: keep
    regex: true
    
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
    action: replace
    target_label: __scheme__
    regex: (https?)
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
    action: replace
    target_label: __metrics_path__
    regex: (.+)
  - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
    action: replace
    target_label: __address__
    regex: ([^:]+)(?::\d+)?;(\d+)
    replacement: $1:$2
    
  - action: labelmap
    regex: __meta_kubernetes_service_label_(.+)
    
  - source_labels: [__meta_kubernetes_namespace]
    action: replace
    target_label: kubernetes_namespace
  - source_labels: [__meta_kubernetes_service_name]
    action: replace
    target_label: kubernetes_name
  - source_labels: [__meta_kubernetes_pod_node_name]
    action: replace
    target_label: kubernetes_node

可以看到relable_configs中的規(guī)則很多， 具體的內(nèi)容如下

  • job名稱為kubernetes-service-endpoints(job-name: kubernetes-service-endpoints)
  • 獲取kubernetes中endpoints的相關(guān)信息(role: endpoints)
  • 使用http的方式獲取信息(沒有配置使用默認(rèn)配置http)
  • relabel配置部分：

    • annotations中必須存在prometheus.io/scrape: "true"配置才會被promethues發(fā)現(xiàn)

    • __scheme__的值為__meta_kubernetes_service_annotation_prometheus_io_scheme的value， 需要滿足正則表達(dá)式(https?)

    • __metrics_path__的值為__meta_kubernetes_service_annotation_prometheus_io_path的value， 滿足正則表達(dá)式(.+)

    • __address__的value替換為IP:port的方式

    • kubernetes_namespace的value replace為__meta_kubernetes_namespace的value

    • kubernetes_name的value replace為__meta_kubernetes_service_name的value

    • kubernetes_node的value replace為__meta_kubernetes_pod_node_name的value

獲取的metrics的信息如下：

up{app="prometheus",app_kubernetes_io_managed_by="Helm",chart="prometheus-11.3.0",component="node-exporter",heritage="Helm",instance="10.40.61.116:9100",job="kubernetes-service-endpoints",kubernetes_name="prometheus-node-exporter",kubernetes_namespace="devops",kubernetes_node="py-modelo2o08cn-p005.pek3.example.com",release="prometheus"}
最后編輯于 ：
?著作權(quán)歸作者所有,轉(zhuǎn)載或內(nèi)容合作請聯(lián)系作者
【社區(qū)內(nèi)容提示】社區(qū)部分內(nèi)容疑似由AI輔助生成，瀏覽時(shí)請結(jié)合常識與多方信息審慎甄別。
平臺聲明：文章內(nèi)容（如有圖片或視頻亦包括在內(nèi)）由作者上傳并發(fā)布，文章內(nèi)容僅代表作者本人觀點(diǎn)，簡書系信息發(fā)布平臺，僅提供信息存儲服務(wù)。

相關(guān)閱讀更多精彩內(nèi)容

友情鏈接更多精彩內(nèi)容