https://www.runoob.com/w3cnote/python-pip-install-usage.html
pip相關(guān)網(wǎng)站

1.準(zhǔn)備工作

1.1 用到技術(shù)

python3
requests: http 爬取 json 數(shù)據(jù)

1.2 安裝

pip install requests

1.3 導(dǎo)入

import requests

2. URL分析

2.1 復(fù)制 json 接口請(qǐng)求 URL

進(jìn)入掘金個(gè)人主頁(yè)，打開開發(fā)者工具，點(diǎn)擊“專欄” tab ，在開發(fā)者工具"Network->XHR->Name->get_entry_by_self->Headers->Request URL" 復(fù)制 url。

將上面復(fù)制的 url 賦值給下方代碼中 juejin_youliang_api_full_url 變量。

import requests

juejin_youliang_api_full_url = 'https://timeline-merger-ms.juejin.im/v1/get_entry_by_self?src=web&uid=59bf12c05188256c6d77d0db&device_id=1568894045269&token=eyJhY2Nlc3NfdG9rZW4iOiJ2cVFna25YNU9oM2VFSVM4IiwicmVmcmVzaF90b2tlbiI6ImlwYXJUazFzUGJyNlZVQUgiLCJ0b2tlbl90eXBlIjoibWFjIiwiZXhwaXJlX2luIjoyNTkyMDAwfQ%3D%3D&targetUid=59bf12c05188256c6d77d0db&type=post&limit=20&order=createdAt'

def decode_url(url):
    adr, query = url.split("?")
    params = { kv.split("=")[0]:kv.split("=")[1] for kv in query.split("&")}
    return adr, params

decode_url(juejin_youliang_api_full_url)
print (decode_url(juejin_youliang_api_full_url))

輸出：

('https://timeline-merger-ms.juejin.im/v1/get_entry_by_self',
 {'src': 'web', 
'uid': '59bf12c05188256c6d77d0db',
 'device_id': '1568894045269', 
'token': 'eyJhY2Nlc3NfdG9rZW4iOiJ2cVFna25YNU9oM2VFSVM4IiwicmVmcmVzaF90b2tlbiI6ImlwYXJUazFzUGJyNlZVQUgiLCJ0b2tlbl90eXBlIjoibWFjIiwiZXhwaXJlX2luIjoyNTkyMDAwfQ%3D%3D', 
'targetUid': '59bf12c05188256c6d77d0db',
 'type': 'post', 
'limit': '20',
 'order': 'createdAt'
})

3. 抓取數(shù)據(jù)

助手函數(shù):

def encode_url(url, params):
    query = "&".join(["{}={}".format(k, v) for k, v in params.items()])
    return "{}?{}".format(url, query)

def get_juejin_url(uid, device_id, token):
    url = "https://timeline-merger-ms.juejin.im/v1/get_entry_by_self"
    params = {'src': 'web',
              'uid': uid,
              'device_id': device_id,
              'token': token,
              'targetUid': uid,
              'type': 'post',
              'limit': 20,
              'order': 'createdAt'}
    return encode_url(url, params)

requests 抓取數(shù)據(jù)：

uid='59bf12c05188256c6d77d0db'
device_id='1568894045269'
token='eyJhY2Nlc3NfdG9rZW4iOiJ2cVFna25YNU9oM2VFSVM4IiwicmVmcmVzaF90b2tlbiI6ImlwYXJUazFzUGJyNlZVQUgiLCJ0b2tlbl90eXBlIjoibWFjIiwiZXhwaXJlX2luIjoyNTkyMDAwfQ%3D%3D'

url = get_juejin_url(uid, device_id, token)

headers = {
  'Origin': 'https://juejin.im',
  'Referer': 'https://juejin.im/user/5bd2b8b25188252a784d19d7/posts',
  'Sec-Fetch-Mode': 'cors',
  'User-Agent': 'Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Mobile Safari/537.36'
}

res = requests.get(url, headers=headers)

if res.status_code == 200:
    json_data = res.json()
    print('獲取數(shù)據(jù)陳成功')
    print(json_data)
else:
    print('數(shù)據(jù)獲取失敗，請(qǐng)檢查token是否失效')

4. 分析數(shù)據(jù)

4.1 分析 json 數(shù)據(jù) 找到文章列表字段

輸出：

json_data = res.json()
    
    for k, v in json_data.items():
        print(k, ':', v)

t --host localhost --port 57248 /Users/qiuchendichen/Desktop/python腳本/demo.py 
s : 1
m : ok
d : {'total': 2, 'entrylist': [{'collectionCount': 0, 'userRankIndex': 0, 'buildTime': 1568894959.9655, 'commentsCount': 0 .... .... ....