爬蟲實(shí)戰(zhàn)第三天

任務(wù)

爬取小豬短租北京地區(qū)（http://bj.xiaozhu.com/） 租房信息（前三頁）。

成果

將爬取的信息寫入到了MongoDB中，并且查詢了價格大于等于500/晚的租房信息。

源碼

from bs4 import BeautifulSoup
from pymongo import MongoClient
import requests

pages = ['http://bj.xiaozhu.com/search-duanzufang-p{}-0/'.format(str(i)) for i in range(1, 4)]
info = []
client = MongoClient('localhost', 27017)
xiao_zhu = client['xiao_zhu']
xiao_zhu_sheet = xiao_zhu['xiao_zhu_sheet']


def get_info(url):
    wb_data = requests.get(url)
    soup = BeautifulSoup(wb_data.text, 'lxml')
    data = {
        'title': soup.select('div.pho_info > h4 > em')[0].get_text(),
        'address': soup.select('div.pho_info > p > span')[0].get_text().strip(' ').strip('\n'),
        'price': int(soup.select('#pricePart > div.day_l > span')[0].get_text()),
        # 圖片鏈接在chrome中不是直接打開而是下載，在IE中可以直接打開
        'house_image': soup.select('#curBigImage')[0]['src'],
        'master_name': soup.select('#floatRightBox > div.js_box.clearfix > div.w_240 > h6 > a')[0]['title'],
        'master_sex': soup.select('#floatRightBox > div.js_box.clearfix > div.w_240 > h6 > span')[0]['class'][0].split('_')[1],
        'master_image': soup.select('#floatRightBox > div.js_box.clearfix > div.member_pic > a > img')[0]['src']
    }
    xiao_zhu_sheet.insert_one(data)


def get_url(start_url):
    wb_data = requests.get(start_url)
    soup = BeautifulSoup(wb_data.text, 'lxml')
    urls = soup.select('#page_list > ul > li > a')
    return urls

for page in pages:
    urls = get_url(page)
    for url in urls:
        try:
            get_info(url['href'])
        except Exception as e:
            pass

'''
'price'關(guān)鍵字的屬性類型必須為數(shù)值型，這樣才能比較大小
$lt/$lte/$gt/$gte/$ne （l == less g == greater e == equal n == not）
使用print(type(xiao_zhu_sheet.find({'price': {'$gte': 500}})[0]))發(fā)現(xiàn)每個item實(shí)際上是一個dict
'''
for item in xiao_zhu_sheet.find({'price': {'$gte': 500}}):
    print(item)

小結(jié)

• Pymongo操作MongoDB首先建立client連接（感覺有點(diǎn)類似于MySQL中的conn??）,然后通過連接用python進(jìn)行操作MongDB，建立具體的db和collection。
• Pymongo具體語法參考： http://api.mongodb.com/python/current/