背景
由于內(nèi)部服務(wù)器安裝GitLab，但是發(fā)型GitLab運(yùn)行一段時(shí)間后就無(wú)法訪問(wèn)，重啟后又可以
終極分析原因
由于之前升級(jí)python的原因，導(dǎo)致安裝GitLab的時(shí)候安裝prometheus失敗
解決方案
查看GitLab狀態(tài)
gitlab-ctl status
發(fā)現(xiàn)prometheus啟動(dòng)失敗
run: gitaly: (pid 1508) 19980339s; run: log: (pid 1500) 19980339s
run: gitlab-monitor: (pid 1505) 19980339s; run: log: (pid 1496) 19980339s
run: gitlab-workhorse: (pid 1513) 19980339s; run: log: (pid 1510) 19980339s
run: logrotate: (pid 15567) 1913s; run: log: (pid 1502) 19980339s
run: nginx: (pid 1509) 19980339s; run: log: (pid 1498) 19980339s
run: node-exporter: (pid 1504) 19980339s; run: log: (pid 1494) 19980339s
run: postgres-exporter: (pid 1506) 19980339s; run: log: (pid 1497) 19980339s
run: postgresql: (pid 1532) 19980339s; run: log: (pid 1514) 19980339s
down: prometheus: 6s, normally up; run: log: (pid 17174) 1076s
run: sidekiq: (pid 1823) 19980328s; run: log: (pid 1499) 19980339s
run: unicorn: (pid 1512) 19980339s; run: log: (pid 1501) 19980339s
查看日志
tail -f -n 100 /var/log/gitlab/prometheus/current
發(fā)現(xiàn)LevelDB存在問(wèn)題，提示需要修復(fù)
2018-09-26_13:23:42.79129 time="2018-09-26T21:23:42+08:00" level=info msg="Listening on localhost:11002" source="web.go:341"
2018-09-26_13:23:42.79331 time="2018-09-26T21:23:42+08:00" level=error msg="Could not open the fingerprint-to-metric index for archived series. Please try a 3rd party tool to repair LevelDB in directory "/var/opt/gitlab/prometheus/data/archived_fingerprint_to_metric". If unsuccessful or undesired, delete the whole directory and restart Prometheus for crash recovery. You will lose all archived time series." source="persistence.go:213"
2018-09-26_13:23:42.79333 time="2018-09-26T21:23:42+08:00" level=error msg="Error opening memory series storage: leveldb: manifest corrupted (field 'comparer'): missing [file=MANIFEST-000785]" source="main.go:192"
網(wǎng)上查找相關(guān)資料，說(shuō)運(yùn)行以下命令修復(fù)
sudo -u gitlab-prometheus python -c "import leveldb; leveldb.RepairDB('/var/opt/gitlab/prometheus/data/archived_fingerprint_to_metric')"
但是運(yùn)行提示錯(cuò)誤：
Traceback (most recent call last):
File "<string>", line 1, in <module>
ImportError: No module named leveldb
需要先安裝leveldb
pip install leveldb
然后再運(yùn)行命令... 這時(shí)發(fā)現(xiàn)pip也出現(xiàn)錯(cuò)誤
如提示pip相關(guān)錯(cuò)誤，那可能是因?yàn)橹皃ython升級(jí)問(wèn)題，導(dǎo)致錯(cuò)誤
這個(gè)時(shí)候你需要重新安裝setuptools和pip，參考升級(jí)地址：
https://blog.csdn.net/uisoul/article/details/90216021
升級(jí)完成后，重新執(zhí)行修復(fù)命令，就可以了
pip install leveldb
sudo -u gitlab-prometheus python -c "import leveldb; leveldb.RepairDB('/var/opt/gitlab/prometheus/data/archived_fingerprint_to_metric')"
