why

保證服務(wù)在高壓的時候不會出現(xiàn)異常

what

多線程 重復(fù)地對服務(wù)的接口進(jìn)行調(diào)用，去看服務(wù)的表現(xiàn)：

• tps （transaction peer second）服務(wù)每秒可以處理的請求個數(shù)
• tpr （time peer request）每條請求的處理時間 平均值 99.9%的情況 99%的情況
• errorrate 錯誤率

how

開始壓之前

首先，我們要有一個并發(fā)測試的工具

• 定時執(zhí)行
• 壓力逐漸增加
  都是通過修改config.xml來實(shí)現(xiàn)

<?xml version="1.0" encoding="UTF-8"?>
<tasks>
<task>
<suitename>ApiPayTest</suitename>
<methodname>queryUserByOutId</methodname>
<israndom>true</israndom>
<isrampup>true</isrampup>
<interval>100</interval>
<poolcount>5</poolcount>
<threadcount>10</threadcount>
<sampletype>bouncetime</sampletype>
<client_timeout>3000</client_timeout>
<sleeptime>100</sleeptime>
<value>false</value>
<startTime>2014/10/15/10/35</startTime>
<stopTime>2014/10/15/10/50</stopTime>
</task>
</tasks>
</pre>

然后，我們寫測試用例

• 繼承Worktest，重寫working方法
• 準(zhǔn)備工作寫到構(gòu)造函數(shù)中
• 返回多余時間

<pre><code>package com.xiaomi.cashpay.works;
import java.util.UUID;
public class Notifytest extends Worktest {
private static IdManager.Iface idManagerService;
public Notifytest() {
init();
}
private void init() {
notifyService = ThriftClientFactory.createClient(Notify.Iface.class,
1000);
this.idManagerService = ThriftClientFactory.createClient(
IdManager.Iface.class, 1000);
receiver = "http://10.101.30.185/cashpay/notify1.php";
}
@Override
public String working() throws TException {
long s1 = System.currentTimeMillis();
try {
notifyid = this.logic_Merchant_UniqueId();
} catch (TException e) {
e.printStackTrace();
System.out.println("idmanager TException" + e.getMessage());
}
long s2 = System.currentTimeMillis();
long extratime = (s2 - s1);
SendResponse res = notifyService.send(req);
return res.getResEnum().toString() + " " + notifyid+ "~" + extratime ;
}
public static void main(String[] argv) {
Notifytest test = new Notifytest();
try {
System.out.println(test.working());
} catch (TException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
</code></pre>

其實(shí)，我們還需要一個壓測環(huán)境

• zookeeper
• resin
• db

終于開始壓了

看服務(wù)的性能

• top
• vmstat
• collectl

看數(shù)據(jù)庫的性能

• perfcounter

壓完之后

用腳本static.py分析我們的log

xiaoguo.jpg

<pre><code>import os
fp=open('Pthread-info.log','r+')
fpnew=open('b.txt','a+')
count=0
linenumber = 0
first=True
processtime =[]
for line in fp:
linenumber = linenumber+1
s = line.split()

    if True == first:
            minutetemp=s[1][0:16]
            first=False
            count = count+1
            processtime.append(int(float(s[4])))
    elif minutetemp==s[1][0:16]:
            processtime.append(int(float(s[4])))
            count = count+1
    else:
            processtime.sort()
            pos_eighty =int(len(processtime)*0.8)
            pos_ninty = int(len(processtime)*0.9)
            pos_doublenine = int(len(processtime)*0.99)
            sum(processtime)
            fpnew.write(minutetemp+" "+str(int(count/60.0))+" "+str(sum(processtime)/len(processtime))+" "+str(processtime[pos_eighty])+" "+str(processtime[pos_ninty])+" "+str(processtime[pos_doublenine])+'\n')
            minutetemp=s[1][0:16]
            count = 1
            processtime=[]
            processtime.append(int(float(s[4])))

processtime.sort()
pos_eighty =int(len(processtime)0.8)
pos_ninty = int(len(processtime)0.9)
pos_doublenine = int(len(processtime)*0.99)
fpnew.write(minutetemp+" "+str(int(count/60.0))+" "+str(sum(processtime)/len(processtime))+" "+str(processtime[pos_eighty])+" "+str(processtime[pos_ninty])+" "+str(processtime[pos_doublenine])+'\n')
print(linenumber)
</code></pre>

看服務(wù)本身的log

• awk
  分割
  默認(rèn)為空格
  -F ''
  修改為需要的分割符
  自定義變量
  awk '{begin{count=0}}'

• grep
  -E
  或

• sort -r

• unique

log.jpg

通用的幾個腳本：
分析服務(wù)處理請求的平均時間，最大值，最小值
<pre><code>ls cashpay-core.log.2014121522* cashpay-core.log.2014121523* cashpay-core.log.2014121600* cashpay-core.log.2014121601* cashpay-core.log.2014121602* | xargs zgrep "INFO" | grep queryAccountByUserId | awk 'BEGIN {max=-1;print ""; print "count,total time, average, max, min"}; {gsub(":","",$3); gsub("\.","",$3); gsub("]","",$3); if (!s[$5]) {s[$5]=$3; e[$5]=$3} else {e[$5]=$3; t=(e[$5]-s[$5]); if (t < 40000 && t >0) { count++; sum+=t; if (max==-1) {max = t; min =t; } if (t>max) {max =t;} if ( t<min) min=t } }}; END {if (count >0) {print count"\t"sum"\t"sum/count"\t"max"\t"min}}'
</code></pre>
找到服務(wù)的具體處理時間和這次調(diào)用對應(yīng)的logid
<pre><code>zgrep "100%] - createCharge" cashpay-core.log.2014121720 | awk -F'[' '{print $2}' | awk -F 'ms,' '{print$1 $2}'|awk '{print$1" " $4}'|awk -F':' '{print $1" "$2}'|awk '{print$1" " $3}'|awk -F',' '{if(NF>1)print $0}'|sort</code>
</pre>
分析Pthread-error.log中的connection-refused錯誤
<pre><code>
grep -B 12 "HttpHostConnectException" Pthread-error.log.2014121615|awk '{if (NR%14==1){ printf$0} else if(NR%14==13){print" "$0}}'|awk '{print$2" "$3" " $10}'|awk -F ']' '{print$1$2}'|awk -F'.' '{print $1}'|awk -F ':' '{print $1":"$2}'|awk '{print$2}'|awk 'begin{count=0;minute="start"} {if($1!=minute){minute=$1; print $0" " count;count=0}else{ count++;}}'
</code></pre>