環(huán)境準(zhǔn)備

httpbin 服務(wù)

httpbin 是一個使用 Python + Flask 編寫的 HTTP 請求響應(yīng)服務(wù)。

該服務(wù)主要用于測試 HTTP 庫，你可以向他發(fā)送請求，然后他會按照指定的規(guī)則將你的請求返回。

部署 httpbin 服務(wù)：

kubectl apply -f samples/httpbin/httpbin.yaml

fortio 服務(wù)

Fortio是一個微服務(wù)（http, grpc)負載測試庫，命令行工具，高級回顯服務(wù)器和web UI在go (golang)。Fortio允許指定一組每秒查詢負載，并記錄延遲直方圖和其他有用的統(tǒng)計數(shù)據(jù)。

部署 fortio 服務(wù)：

kubectl apply -f samples/httpbin/sample-client/fortio-deploy.yaml

httpbin 服務(wù)作為接收請求的服務(wù)端， fortio 服務(wù)作為發(fā)送請求的客戶端。

熔斷配置

創(chuàng)建目標(biāo)規(guī)則，訪問httpbin 服務(wù)時應(yīng)用熔斷配置：

kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: httpbin
spec:
  host: httpbin
  trafficPolicy:
    connectionPool:
      http:
        http1MaxPendingRequests: 1
        maxRequestsPerConnection: 1
EOF

在 fortio 服務(wù)中向 httpbin 服務(wù)的發(fā)出并發(fā)請求：

export FORTIO_POD=$(kubectl get pod -l app=fortio -n istio-demo -o jsonpath={.items..metadata.name})
kubectl exec -it "$FORTIO_POD" -c fortio -- /usr/bin/fortio load -c 5 -qps 0 -n 5 -loglevel Warning http://httpbin:8000/get

返回結(jié)果如下：

I0725 11:43:55.895950 4039039 request.go:621] Throttling request took 1.130727121s, request: GET:https://172.24.28.71:6443/apis/ceph.rook.io/v1?timeout=32s
03:43:58 I logger.go:127> Log level is now 3 Warning (was 2 Info)
Fortio 1.17.1 running at 0 queries per second, 104->104 procs, for 5 calls: http://httpbin:8000/get
Starting at max qps with 5 thread(s) [gomax 104] for exactly 5 calls (1 per thread + 0)
03:43:58 W http_client.go:806> [3] Non ok http code 503 (HTTP/1.1 503)
03:43:58 W http_client.go:806> [1] Non ok http code 503 (HTTP/1.1 503)
03:43:58 W http_client.go:806> [2] Non ok http code 503 (HTTP/1.1 503)
Ended after 5.398808ms : 5 calls. qps=926.13
Aggregated Function Time : count 5 avg 0.0035955544 +/- 0.001035 min 0.002619591 max 0.005337051 sum 0.017977772
# range, mid point, percentile, count
>= 0.00261959 <= 0.003 , 0.0028098 , 60.00, 3
> 0.004 <= 0.005 , 0.0045 , 80.00, 1
> 0.005 <= 0.00533705 , 0.00516853 , 100.00, 1
# target 50% 0.0029049
# target 75% 0.00475
# target 90% 0.00516853
# target 99% 0.0053202
# target 99.9% 0.00533537
Sockets used: 5 (for perfect keepalive, would be 5)
Jitter: false
Code 200 : 2 (40.0 %)
Code 503 : 3 (60.0 %)
Response Header Sizes : count 5 avg 92 +/- 112.7 min 0 max 230 sum 460
Response Body/Total Sizes : count 5 avg 419.8 +/- 177.3 min 275 max 637 sum 2099
All done 5 calls (plus 0 warmup) 3.596 ms avg, 926.1 qps

可以看到，在發(fā)送的5個請求中，有3個請求返回 503 。

查看Envoy日志

執(zhí)行下面的命令，查看fortio 服務(wù)的Envoy日志：

kubectl logs -l app=fortio -c istio-proxy

可以看到fortio服務(wù)對httpbin服務(wù)的調(diào)用的日志：

{
     "authority": "httpbin:8000",
     "bytes_received": 0,
     "bytes_sent": 81,
     "connection_termination_details": null,
     "downstream_local_address": "172.24.146.239:8000",
     "downstream_remote_address": "172.24.158.166:55030",
     "duration": 1,
     "method": "GET",
     "path": "/get",
     "protocol": "HTTP/1.1",
     "request_id": "3aa81a10-cddf-9b3b-b704-c1099b05c6fb",
     "requested_server_name": null,
     "response_code": 503,
     "response_code_details": "via_upstream",
     "response_flags": "-",
     "route_name": "allow_any",
     "start_time": "2022-07-22T09:19:04.465Z",
     "upstream_cluster": "PassthroughCluster",
     "upstream_host": "172.24.146.239:8000",
     "upstream_local_address": "172.24.158.166:55042",
     "upstream_service_time": "1",
     "upstream_transport_failure_reason": null,
     "user_agent": "fortio.org/fortio-1.17.1",
     "x_forwarded_for": null
}

其中，response_flags為-，沒有任何異常。

執(zhí)行下面的命令，查看httpbin 服務(wù)的Envoy日志：

kubectl logs -l app=httpbin -c istio-proxy

可以看到httpbin服務(wù)被fortio服務(wù)調(diào)用的Envoy日志：

{
     "authority": "httpbin:8000",
     "bytes_received": 0,
     "bytes_sent": 81,
     "connection_termination_details": null,
     "downstream_local_address": "172.24.158.96:80",
     "downstream_remote_address": "172.24.158.166:55042",
     "duration": 0,
     "method": "GET",
     "path": "/get",
     "protocol": "HTTP/1.1",
     "request_id": "3aa81a10-cddf-9b3b-b704-c1099b05c6fb",
     "requested_server_name": null,
     "response_code": 503,
     "response_code_details": "upstream_reset_before_response_started{overflow}",
     "response_flags": "UO",
     "route_name": "default",
     "start_time": "2022-07-22T09:19:04.466Z",
     "upstream_cluster": "inbound|80||",
     "upstream_host": null,
     "upstream_local_address": null,
     "upstream_service_time": null,
     "upstream_transport_failure_reason": null,
     "user_agent": "fortio.org/fortio-1.17.1",
     "x_forwarded_for": null
}

其中，response_flags為UO，表示因為熔斷產(chǎn)生的上游（upstream）溢出。

深入分析

通過Envoy日志，我們可以做出一些分析和判斷：

當(dāng)httpbin服務(wù)的請求正常的時候，調(diào)用過程如下圖：

當(dāng)httpbin服務(wù)的熔斷被觸發(fā)的時候，調(diào)用過程如下圖：

當(dāng)httpbin服務(wù)熔斷被觸發(fā)時，返回503狀態(tài)碼的是httpbin服務(wù)的Envoy。

清理

kubectl delete destinationrule httpbin
kubectl delete -f samples/httpbin/httpbin.yaml
kubectl delete -f samples/httpbin/sample-client/fortio-deploy.yaml

最后，感謝你這么帥，還給我點贊。

封面