python工具庫(kù)介紹-requests:人性化的HTTP

Requests是Python基于Apache2 Licensed許可證的人性化HTTP庫(kù)。

Python標(biāo)準(zhǔn)庫(kù)中urllib2提供了不少HTTP 功能,但API不系統(tǒng)。它有點(diǎn)過(guò)時(shí),完成最簡(jiǎn)單的任務(wù)也需要大量工作。

下面我們用實(shí)例演示訪問(wèn)github。

>>> import requests
>>> r = requests.get('https://api.github.com/user', auth=('ouyangchongwu@test.com', 'password'))
>>> r.status_code
200
>>> r.headers['content-type']
'application/json; charset=utf-8'
>>> r.encoding
'utf-8'
>>> r.text
u'{"login":"oychw",...}'
>>> r.json()
{u'disk_usage': 176, u'private_gists': 0, ...}

Requests為Python處理了所有HTTP/1.1操作, 與Web服務(wù)的無(wú)縫集成。不需要為URL手動(dòng)添加查詢字符串或POST數(shù)據(jù)進(jìn)行表單處理?;?a target="_blank" rel="nofollow">urllib3, 能自動(dòng)處理Keep-alive和HTTP連接池。

特點(diǎn):

  • 國(guó)際化域名和 URLs

  • Keep-Alive & 連接池

  • 持久的 Cookie 會(huì)話

  • 類(lèi)瀏覽器的SSL認(rèn)證

  • 基本/摘要式的身份認(rèn)證

  • 優(yōu)雅的鍵/值 Cookie

  • 自動(dòng)解壓

  • Unicode響應(yīng)體

  • 多段文件上傳

  • 連接超時(shí)

  • 支持 .netrc

  • 適用于 Python 2.6—3.4

  • 線程安全

用戶手冊(cè)

簡(jiǎn)介

Requests關(guān)注PEP 20的部分:

  • Beautiful is better than ugly.(美麗優(yōu)于丑陋)

  • Explicit is better than implicit.(明確優(yōu)于含糊)

  • Simple is better than complex.(簡(jiǎn)單優(yōu)于復(fù)雜)

  • Complex is better than complicated.(復(fù)雜優(yōu)于繁瑣)

  • Readability counts.(可讀性)

安裝

  • 安裝: 推薦:pip install requests,其次:easy_install requests

  • 最新代碼:

快速入門(mén)

  • 發(fā)送請(qǐng)求:

下面獲取Github的公共時(shí)間線,并在httpbin演示其他HTTP操作:

>>> import requests
>>> r = requests.get('https://github.com/timeline.json')
>>> r = requests.post("http://httpbin.org/post")
>>> r = requests.put("http://httpbin.org/put")
>>> r = requests.delete("http://httpbin.org/delete")
>>> r = requests.head("http://httpbin.org/get")
>>> r = requests.options("http://httpbin.org/get")
  • 在URL中傳遞參數(shù)

URL的查詢字符串(query string)例如, httpbin.org/get?key=val,在Requests可以用字典的形式構(gòu)建。比如傳遞key1=value1和key2=value2到 httpbin.org/get:

>>> import requests
>>> payload = {'key1': 'value1', 'key2': 'value2'}
>>> r = requests.get("http://httpbin.org/get", params=payload)
>>> print(r.url)
http://httpbin.org/get?key2=value2&key1=value1
>>> payload = {'key1': 'value1', 'key2[]': ['value2', 'value3']}
>>> r = requests.get("http://httpbin.org/get", params=payload)
>>> print(r.url)
http://httpbin.org/get?key1=value1&key2%5B%5D=value2&key2%5B%5D=value3

注意字典里值為None的鍵會(huì)忽略。上面第2個(gè)例子訪問(wèn)的是http://httpbin.org/get?key1=value1&key2[]=value2&key2[]=value3 。注意key后面需要添加中括號(hào)對(duì)。

  • 響應(yīng)
>>> import requests
>>> r = requests.get('https://api.github.com/events')
>>> r.text
>>> u'[{"id":"2636319727","type":"PullRequestReviewCommentEvent","actor":{"id":1148601,"login":"i ...}]
>>> r.encoding
'utf-8'
>>> r.encoding = 'ISO-8859-1'

Requests會(huì)自動(dòng)解碼服務(wù)器的返回。大多數(shù)unicode字符集都能無(wú)縫解碼。請(qǐng)求發(fā)出時(shí)Requests會(huì)基于響應(yīng)的HTTP頭部推測(cè)響應(yīng)的編碼。同時(shí)還可以設(shè)置和查詢編碼。改變編碼后,訪問(wèn) r.text 將會(huì)使用 r.encoding 。

  • 二進(jìn)制響應(yīng)

r.content可以以字節(jié)的方式顯示響應(yīng)。

>>> r.content
b'[{"repository":{"open_issues":0,"url":"https://github.com/...

傳輸格式gzip和deflate會(huì)自動(dòng)轉(zhuǎn)碼。處理圖片實(shí)例:

>>> from PIL import Image
>>> from StringIO import StringIO
>>> i = Image.open(StringIO(r.content))
  • Json響應(yīng)

Requests內(nèi)置了JSON解碼器:

>>> import requests
>>> r = requests.get('https://github.com/timeline.json')
>>> r.json()
{u'documentation_url': u'https://developer.github.com/v3/activity/events/#list-public-events', u'message': u"Hello there, wayfaring stranger. If you're reading this then you probably didn't see our blog post a couple of years back announcing that this API would go away: http://git.io/17AROg Fear not, you should be able to get what you need from the shiny new Events API instead."}

JSON解碼失敗時(shí)r.json 就會(huì)拋出異常。例如, 401 (Unauthorized) , ValueError: No JSON object could be decoded等。

  • 原始響應(yīng)

極端的情況下需要查看服務(wù)器的原始套接字響應(yīng),請(qǐng)求時(shí)設(shè)置 stream=True:

>>> import requests
>>> r = requests.get('https://api.github.com/events', stream=True)
>>> r.raw
<urllib3.response.HTTPResponse object at 0x7f807dd6f4d0>
>>> r.raw.read(10)
'\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x03'

通常需要存為文件:

with open(filename, 'wb') as fd:
    for chunk in r.iter_content(chunk_size):
        fd.write(chunk)

Response.iter_content 能減少直接使用Response.raw的大量處理,下載流時(shí)尤其推薦。

  • 自定義頭
>>> import requests
>>> import json
>>> url = 'https://api.github.com/some/endpoint'
>>> payload = {'some': 'data'}
>>> headers = {'content-type': 'application/json'}
>>> r = requests.post(url, data=json.dumps(payload), headers=headers)
  • 更加復(fù)雜的POST請(qǐng)求

表單直接以字典形式發(fā)送:

>>> import requests
>>> payload = {'key1': 'value1', 'key2': 'value2'}
>>> r = requests.post("http://httpbin.org/post", data=payload)
>>> print(r.text)
{
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {
    "key1": "value1", 
    "key2": "value2"
  }, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate, compress", 
    "Content-Length": "23", 
    "Content-Type": "application/x-www-form-urlencoded", 
    "Host": "httpbin.org", 
    "User-Agent": "python-requests/2.2.1 CPython/2.7.6 Linux/3.13.0-53-generic"
  }, 
  "json": null, 
  "origin": "119.122.150.177", 
  "url": "http://httpbin.org/post"
}

string則會(huì)被直接發(fā)布出去。Github API v3中接受編碼為JSON的POST/PATCH數(shù)據(jù)

>>> import requests
>>> import json
>>> url = 'https://api.github.com/some/endpoint'
>>> payload = {'some': 'data'}
>>> r = requests.post(url, data=json.dumps(payload))
  • POST復(fù)雜編碼的文件
>>> import requests
>>> url = 'http://httpbin.org/post'
>>> files = {'file': open('/home/andrew/test.xls', 'rb')}
>>>  r = requests.post(url, files=files)
>>> url = 'http://httpbin.org/post'
>>> r.text
u'{\n  "args": {}, \n  "data": "", \n  "files": {\n ...  "url": "http://httpbin.org/post"\n}\n'

可以顯式地設(shè)置文件名,文件類(lèi)型和請(qǐng)求頭:

>>> import requests
>>> files = {'file': ('report.xls', open('/home/andrew/test.xls', 'rb'), 'application/vnd.ms-excel', {'Expires': '0'})}
>>> url = 'http://httpbin.org/post'
>>> r = requests.post(url, files=files)
>>> r.text
u'{\n  "args": {}, \n  "data": ""..."url": "http://httpbin.org/post"\n}\n'

還可以直接用文字代替文件:

>>> import requests
>>> url = 'http://httpbin.org/post'
>>> files = {'file': ('report.csv', 'some,data,to,send\nanother,row,to,send\n')}
>>> r = requests.post(url, files=files)
>>> r.text
u'{\n  "args": {}, \n  "data": "", \n  "files": {\n  ...  "json": null, \n  "origin": "14.153.22.104", \n  "url": "http://httpbin.org/post"\n}\n'

multipart/form-data不支持特別大的文件,建議使用requests-toolbelt,參考:toolbelt

  • 響應(yīng)狀態(tài)碼
>>> import requests
>>> r = requests.get('http://httpbin.org/get')
>>> r.status_code
200
>>> r.status_code == requests.codes.ok
True
>>> bad_r = requests.get('http://httpbin.org/status/404')
>>> bad_r.status_code
404
>>> bad_r.raise_for_status()
>>> r.raise_for_status()

上面的requests.codes.ok是內(nèi)置的狀態(tài)碼查詢對(duì)象??梢允褂?Response.raise_for_status()跑出失敗請(qǐng)求(4XX客戶端錯(cuò)誤或5XX服務(wù)器異常),我們可以通過(guò) Response.raise_for_status() 來(lái)拋出異常。r的返回為200,所以返回None,不產(chǎn)生異常。

  • 響應(yīng)頭
>>> r.headers
{'content-length': '275', 'server': 'nginx', 'connection': 'keep-alive', 'access-control-allow-credentials': 'true', 'date': 'Tue, 10 Mar 2015 08:21:36 GMT', 'access-control-allow-origin': '*', 'content-type': 'application/json'}

>>> r.headers['Content-Type']
'application/json'

>>> r.headers.get('content-type')
 'application/json'

根據(jù) RFC 2616,HTTP頭部不區(qū)分大小寫(xiě)。根據(jù)RFC 7230,接收方會(huì)對(duì)服務(wù)端對(duì)同一key的不同value進(jìn)行組合。

  • Cookies

可以訪問(wèn)響應(yīng)中包含的Cookie:

>>> import requests
>>> url = 'http://automationtesting.sinaapp.com/login'
>>> r = requests.get(url)
>>> r.cookies.keys()
['saeut', 'trac_form_token', 'trac_session']
>>> r.cookies['saeut']
 'CkMPGlT+tfQiXS9uGYviAg=='

使用cookies參數(shù)可以發(fā)送你的cookies到服務(wù)器:

>>> import requests
>>> url = 'http://httpbin.org/cookies'
>>> cookies = dict(cookies_are='working')
>>> r = requests.get(url, cookies=cookies)
>>> r.text
u'{\n  "cookies": {\n    "cookies_are": "working"\n  }\n}\n'
  • 重定向與請(qǐng)求歷史

默認(rèn)對(duì)HEAD以外其他所有動(dòng)作進(jìn)行位置重定向。Response.history可以看到重定向的記錄。

>>> import requests
>>> r = requests.get('http://github.com')
>>> r.url
u'https://github.com/'
>>> r.status_code
200
>>> r.history
[<Response [301]>]

GET, OPTIONS, POST, PUT, PATCH 或者 DELETE可以通過(guò)allow_redirects參數(shù)禁用重定向,這個(gè)設(shè)置對(duì)HEAD也生效:

>>> import requests
>>> r = requests.get('http://github.com', allow_redirects=False)
>>> r.status_code
301
>>> r.history
[]
>>> r = requests.head('http://github.com', allow_redirects=True)
>>> r.url
u'https://github.com/'
>>> r.history
[<Response [301]>]
  • 超時(shí)

超時(shí)告訴requests在經(jīng)過(guò)timeout參數(shù)的秒之后停止等待響應(yīng):

>>> import requests
>>> requests.get('http://github.com', timeout=0.1)    
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/dist-packages/requests/api.py", line 55, in get
    return request('get', url, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 455, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 558, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 387, in send
    raise Timeout(e)
requests.exceptions.Timeout: (<urllib3.connectionpool.HTTPConnectionPool object at 0x7f807dd6f050>, 'Connection to github.com timed out. (connect timeout=0.1)')
>>>

注: 超時(shí)不是對(duì)整個(gè)響應(yīng)下載的時(shí)間限制, 而且指定時(shí)間沒(méi)有收到服務(wù)器返回就拋出異常。

  • 錯(cuò)誤與異常

ConnectionError:網(wǎng)絡(luò)問(wèn)題(如DNS失敗、拒絕連接等)。

HTTPError: 比較罕見(jiàn)的無(wú)效HTTP響應(yīng)時(shí)。

Timeout:請(qǐng)求超時(shí)。

TooManyRedirects:超過(guò)了設(shè)定的最大重定向次數(shù)。

requests.exceptions.RequestException是所有具體異常的基類(lèi)。

高級(jí)用法

  • Session對(duì)象

Session對(duì)象能夠跨請(qǐng)求保持參數(shù),Session實(shí)例發(fā)出的所有請(qǐng)求共享cookies。

Session對(duì)象具有主Requests API的所有方法。

>>> s = requests.Session()
>>> s.get('http://httpbin.org/cookies/set/sessioncookie/123456789')
<Response [200]>
>>> r = s.get("http://httpbin.org/cookies")
>>> print(r.text)
{
  "cookies": {
    "sessioncookie": "123456789"
  }
}

Session也可為request方法提供缺省數(shù)據(jù),添加屬性即可:

>>> import requests
>>> s = requests.Session()
>>> s.auth = ('user', 'pass')
>>> s.headers.update({'x-test': 'true'})
>>> s.get('http://httpbin.org/headers', headers={'x-test2': 'true'})
<Response [200]>

傳遞給request方法的字典都會(huì)與已有session層的值合并。方法層的參數(shù)會(huì)覆蓋會(huì)話的參數(shù)。在方法層參數(shù)中將鍵值設(shè)置為None,會(huì)被自動(dòng)忽略key。參考: session api

  • 請(qǐng)求(Request)和響應(yīng)(Response)對(duì)象

requests.get()等請(qǐng)求主要做兩件的事情。一為構(gòu)建Request 對(duì)象。二為收到服務(wù)器響應(yīng)時(shí)產(chǎn)生Response 對(duì)象。Response對(duì)象包含服務(wù)器返回和原來(lái)的 Request 對(duì)象。

>>> import requests
>>> r = requests.get('http://en.wikipedia.org/wiki/Monty_Python')
>>> r.headers
{'content-length': '67559', ...}
>>> r.request.headers
{'Connection': 'keep-alive', 'Accept-Encoding': ...}
  • 預(yù)請(qǐng)求

當(dāng)從API或會(huì)話調(diào)用接收Response對(duì)象時(shí),request屬性實(shí)際上是PreparedRequest。如果你需要修改body或header,可以如下方式進(jìn)行處理:

from requests import Request, Session

s = Session()
req = Request('GET', url,
    data=data,
    headers=header
)
prepped = req.prepare()

# do something with prepped.body
# do something with prepped.headers

resp = s.send(prepped,
    stream=stream,
    verify=verify,
    proxies=proxies,
    cert=cert,
    timeout=timeout
)

print(resp.status_code)

這里沒(méi)有對(duì)Request對(duì)象進(jìn)行特殊處理,而是修改PreparedRequest對(duì)象。然后用requests.*Session.*.發(fā)送。

上述代碼沒(méi)有Request Session。Session層狀態(tài),如cookie不會(huì)使用。用Session.prepare_request()替換Request.prepare()即可增加狀態(tài)支持:

from requests import Request, Session

s = Session()
req = Request('GET',  url,
    data=data
    headers=headers
)

prepped = s.prepare_request(req)

# do something with prepped.body
# do something with prepped.headers

resp = s.send(prepped,
    stream=stream,
    verify=verify,
    proxies=proxies,
    cert=cert,
    timeout=timeout
)

print(resp.status_code)
  • SSL證書(shū)驗(yàn)證

使用 verify 參數(shù)可以像web瀏覽器一樣為HTTPS請(qǐng)求驗(yàn)證SSL證書(shū):

參考資料

最后編輯于
?著作權(quán)歸作者所有,轉(zhuǎn)載或內(nèi)容合作請(qǐng)聯(lián)系作者
【社區(qū)內(nèi)容提示】社區(qū)部分內(nèi)容疑似由AI輔助生成,瀏覽時(shí)請(qǐng)結(jié)合常識(shí)與多方信息審慎甄別。
平臺(tái)聲明:文章內(nèi)容(如有圖片或視頻亦包括在內(nèi))由作者上傳并發(fā)布,文章內(nèi)容僅代表作者本人觀點(diǎn),簡(jiǎn)書(shū)系信息發(fā)布平臺(tái),僅提供信息存儲(chǔ)服務(wù)。

相關(guān)閱讀更多精彩內(nèi)容

友情鏈接更多精彩內(nèi)容