背景
公司視頻應用,需要針對用戶的操作數(shù)據(jù)分析,得出應用內(nèi)所有的視頻的權重分析,方便以后更好的進行推薦視頻操作.目前已有的log格式主要有:
action vid uid
view video video_001 NaN
play video video_001 user_00x
用戶分為注冊用戶與游客兩種,操作主要有view,like,comment,upload,download幾種.為了方便處理,第一階段只考慮針對游客訪問的視頻進行處理.這樣形成一個只有action與uid的字典形式.處理數(shù)據(jù)主要使用python的pandas庫.
清洗數(shù)據(jù)
日志初期action包括view,like,comment,upload,uid包括用戶id以及代表游客的NaN空值.第一步需要把這些action具體量化,為了簡單量化初期:view = 1 , like = 2 , comment = 3 , upload = 3 , download = 3.日志文件以csv格式存儲.導入數(shù)據(jù)后進行處理:
import pandas as pd
# pandas讀取csv
data = pd.read_csv('data.csv')
# action為playvideo的設置為1,即打分為1
data.loc[data['action']=='playvideo','action']=1
# 把NaN(即游客)值重置為0
data=data.fillna(value = '0')
# 過濾出uid值為0的數(shù)據(jù)
tourist_data = data.loc[data['uid']=='0']
del tourist_data['uid']
# tourist_data為action與uic的矩陣
DataFrame如下
action vid
0 1 2c9f91345c3ed855015c5ee9cb904681
1 1 2c9f91345c3ed855015c52649f962d4f
2 -1 2c9f91345c2007f8015c2deee14c18cb
3 1 2c9f91345c3ed855015c5ee9cb904681
4 3 2c9f91345c2007f8015c2deee14c18cb
針對這個數(shù)據(jù),需要進行清洗.找出重復的vid,并相加對應的action.得出的矩陣就是需要的vid的score排列.
首先DataFrame生成vid的list
vid_list=tourist_data['vid'].values.tolist()
問題分解為尋找list中重復的數(shù)值,并把數(shù)值對應的index記錄下來.需要用到兩個庫enumerate與defaultdict
enumerate可以把list生成帶有index的dict
defaultdict可以對list形成的dict進行統(tǒng)計處理
# 生成帶有index的列表
In [189]: [(v,i) for i,v in enumerate(vid_list)]
Out[189]:
[('2c9f91345c3ed855015c5ee9cb904681', 0),
('2c9f91345c3ed855015c52649f962d4f', 1),
('2c9f91345c2007f8015c2deee14c18cb', 2),
('2c9f91345c3ed855015c5ee9cb904681', 3),
('2c9f91345c2007f8015c2deee14c18cb', 4),
('2c9f91345c3ed855015c52649f962d4f', 5),
('2c9f91345c2007f8015c2deee14c18cb', 6),
('2c9f91345bf13cac015bfce28ef31002', 7),
('2c9f91345c3ed855015c5ee9cb904681', 8),
('2c9f91345c3ed855015c52649f962d4f', 9)]
# 利用defaultdict生成對重復vid處理后的dict
In [190]: vid_dict = defaultdict(list)
In [191]: for key, value in [(v, i) for i, v in enumerate(vid_list)]:
...: vid_dict[key].append(value)
...:
In [192]: vid_dict
Out[192]:
defaultdict(list,
{'2c9f91345bf13cac015bfce28ef31002': [7],
'2c9f91345c2007f8015c2deee14c18cb': [2, 4, 6],
'2c9f91345c3ed855015c52649f962d4f': [1, 5, 9],
'2c9f91345c3ed855015c5ee9cb904681': [0, 3, 8]})
vid對應的list即為對應的index位置,利用index位置就可以為score_list進行處理累加
# 生成score list
score_list = tourist_data['action'].values.tolist()
In [222]: vid_dict = defaultdict(list)
In [223]: for key, value in [(v, i) for i, v in enumerate(vid_list)]:
...: vid_dict[key].append(value)
In [224]: vid_dict
Out[224]:
defaultdict(list,
{'2c9f91345bf13cac015bfce28ef31002': [7],
'2c9f91345c2007f8015c2deee14c18cb': [2, 4, 6],
'2c9f91345c3ed855015c52649f962d4f': [1, 5, 9],
'2c9f91345c3ed855015c5ee9cb904681': [0, 3, 8]})
In [227]: rank_list=[]
...: for i in vid_dict:
...: score = 0
...: for index in vid_dict[i]:
...: score += int(score_list[index])
...: rank_list.append(score)
In [231]: vid_list = []
In [232]: for i in vid_dict:
...: vid_list.append(i)
In [233]: vid_list
Out[233]:
['2c9f91345bf13cac015bfce28ef31002',
'2c9f91345c2007f8015c2deee14c18cb',
'2c9f91345c3ed855015c5ee9cb904681',
'2c9f91345c3ed855015c52649f962d4f']
In [234]: rank_list
Out[234]: [1, 4, 5, 3]
In [237]: vid_score = pd.DataFrame({'score':rank_list,'vid':vid_list})
生成score與vid的矩陣
針對生成的數(shù)據(jù)進行分析
使用altair對數(shù)據(jù)進行畫圖分析
# 導入csv文件
import pandas as pd
from altair import Chart, load_dataset
%matplotlib inline
vids_score = pd.read_csv('./res/vid_score.csv')
bins = [0, 10, 20, 30, 40, 50,60,70,80,90, 100,150,200,300,400,500,1000,1500]
scores = pd.cut(vids_score['score'], bins)
def get_stats(group):
return {'count': group.count()}
grouped = vids_score['score'].groupby(scores)
bin_counts = grouped.apply(get_stats).unstack()
bin_counts
bin_counts.index = ['0~10', '10~20', '20~30', '30~40', '40~50', '50~60', '60~70',
'70~80', '80~90', '90~100','100-150','150-200','200-300','300-400','400-500','500-1000','1000-1500']
bin_counts.index.name = 'score'
plt=bin_counts.plot(kind='bar', alpha=0.5, rot=1,width = 0.8,align='center',figsize=(15,4))
