詳解Python數(shù)據(jù)分析--Pandas知識點

更新時間：2019年03月23日 14:14:45 作者：Star_Zhao

這篇文章主要介紹了Python數(shù)據(jù)分析--Pandas知識點，文中通過示例代碼介紹的非常詳細(xì)，對大家的學(xué)習(xí)或者工作具有一定的參考學(xué)習(xí)價值，需要的朋友們下面隨著小編來一起學(xué)習(xí)學(xué)習(xí)吧

本文主要是總結(jié)學(xué)習(xí)pandas過程中用到的函數(shù)和方法, 在此記錄, 防止遺忘

1. 重復(fù)值的處理

利用drop_duplicates()函數(shù)刪除數(shù)據(jù)表中重復(fù)多余的記錄, 比如刪除重復(fù)多余的ID.

 import pandas as pd
 df = pd.DataFrame({"ID": ["A1000","A1001","A1002", "A1002"],
         "departmentId": [60001,60001, 60001, 60001]})
 df.drop_duplicates()

2. 缺失值的處理

缺失值是數(shù)據(jù)中因缺少信息而造成的數(shù)據(jù)聚類, 分組, 截斷等

2.1 缺失值產(chǎn)生的原因

主要原因可以分為兩種: 人為原因和機械原因.

1) 人為原因: 由于人的主觀失誤造成數(shù)據(jù)的缺失, 比如數(shù)據(jù)錄入人員的疏漏;

2) 機械原因: 由于機械故障導(dǎo)致的數(shù)據(jù)收集或者數(shù)據(jù)保存失敗從而造成數(shù)據(jù)的缺失.

2.2 缺失值的處理方式

缺失值的處理方式通常有三種: 補齊缺失值, 刪除缺失值, 刪除缺失值, 保留缺失值.

1) 補齊缺失值: 使用計算出來的值去填充缺失值, 例如樣本平均值.

使用fillna()函數(shù)對缺失值進(jìn)行填充, 使用mean()函數(shù)計算樣本平均值.

import pandas as pd
import numpy as np
df = pd.DataFrame({'ID':['A10001', 'A10002', 'A10003', 'A10004'], 
          "Salary":[11560, np.NaN, 12988,12080]})
#用Salary字段的樣本均值填充缺失值
df["Salary"] = df["Salary"].fillna(df["Salary"].mean())
df

2) 刪除缺失值: 當(dāng)數(shù)據(jù)量大時且缺失值占比較小可選用刪除缺失值的記錄.

示例: 刪除entrytime中缺失的值,采用dropna函數(shù)對缺失值進(jìn)行刪除:

 import pandas as pd
df = pd.DataFrame({"ID": ["A1000","A1001","A1002"],
         "entrytime": ["2015-05-06",pd.NaT,"2016-07-01" ]})
df.dropna()

3) 保留缺失值.

3. 刪除前后空格

使用strip()函數(shù)刪除前后空格.

 import pandas as pd
 df = pd.DataFrame({"ID": ["A1000","A1001","A1002"],
         "Surname": [" Zhao ","Qian"," Sun " ]})
 df["Surname"] = df["Surname"].str.strip()
 df

4. 查看數(shù)據(jù)類型

查看所有列的數(shù)據(jù)類型使用dtypes, 查看單列使用dtype, 具體用法如下:

import pandas as pd
df = pd.DataFrame({"ID": [100000,100101,100201],"Surname": [" Zhao ","Qian"," Sun " ]})
#查看所有列的數(shù)據(jù)類型
df.dtypes
#查看單列的數(shù)據(jù)類型
df["ID"].dtype

5. 修改數(shù)據(jù)類型

使用astype()函數(shù)對數(shù)據(jù)類型進(jìn)行修改, 用法如下

import pandas as pd
df = pd.DataFrame({"ID": [100000,100101,100201],"Surname": [" Zhao ","Qian"," Sun " ]})
#將ID列的類型轉(zhuǎn)化為字符串的格式
df["ID"].astype(str)

6. 字段的抽取

使用slice(start, end)函數(shù)可完成字段的抽取, 注意start是從0開始且不包含end. 比如抽取前兩位slice(0, 2).

import pandas as pd
df = pd.DataFrame({"ID": [100000,100101,100201],"Surname": [" Zhao ","Qian"," Sun " ]})
#需要將ID列的類型轉(zhuǎn)換為字符串, 否則無法使用slice()函數(shù)
 df["ID"]= df["ID"].astype(str)
#抽取ID前兩位
df["ID"].str.slice(0,2)

7. 字段的拆分

使用split()函數(shù)進(jìn)行字段的拆分, split(pat=None, n = -1, expand=True)函數(shù)包含三個參數(shù):

第一個參數(shù)則是分隔的字符串, 默認(rèn)是以空格分隔

第二個參數(shù)則是分隔符使用的次數(shù), 默認(rèn)分隔所有

第三個參數(shù)若是True, 則在不同的列展開, 否則以序列的形式顯示.

import pandas as pd
df = pd.DataFrame({"ID": [100000,100101,100201],"Surname_Age": ["Zhao_23","Qian_33","Sun_28" ]})
#對Surname_Age字段進(jìn)行拆分
df_new = df["Surname_Age"].str.split("_", expand =True)
df_new

8. 字段的命名

有兩種方式一種是使用rename()函數(shù), 另一種是直接設(shè)置columns參數(shù)

import pandas as pd
df = pd.DataFrame({"ID": [100000,100101,100201],"Surname_Age": ["Zhao_23","Qian_33","Sun_28" ]})
#第一種方法使用rename()函數(shù)
# df_new = df["Surname_Age"].str.split("_", expand =True).rename(columns={0: "Surname", 1: "Age"})
# df_new
#第二種方法直接設(shè)置columns參數(shù)
df_new = df["Surname_Age"].str.split("_", expand =True)
df_new.columns = ["Surname","Age"]
df_new

兩種方式同樣的結(jié)果:

9. 字段的合并

使用merge()函數(shù)對字段進(jìn)行合并操作.

 import pandas as pd
 df = pd.DataFrame({"ID": [100000,100101,100201],"Surname_Age": ["Zhao_23","Qian_33","Sun_28" ]})
 df_new = df["Surname_Age"].str.split("_", expand =True)
 df_new.columns = ["Surname","Age"]
 #使用merge函數(shù)對兩表的字段進(jìn)行合并操作.
pd.merge(df, df_new, left_index =True, right_index=True)

10. 字段的刪除

利用drop()函數(shù)對字段進(jìn)行刪除.

import pandas as pd
df = pd.DataFrame({"ID": [100000,100101,100201],"Surname_Age": ["Zhao_23","Qian_33","Sun_28" ]})
df_new = df["Surname_Age"].str.split("_", expand =True)
df_new.columns = ["Surname","Age"]
df_mer= pd.merge(df, df_new, left_index =True, right_index=True)
#drop()刪除字段,第一個參數(shù)指要刪除的字段,axis=1表示字段所在列,inplace為True表示在當(dāng)前表執(zhí)行刪除.
df_mer.drop("Surname_Age", axis = 1, inplace =True)
df_mer

刪除Surname_Age字段成功:

11. 記錄的抽取

1) 關(guān)系運算: df[df.字段名關(guān)系運算符數(shù)值], 比如抽取年齡大于30歲的記錄.

import pandas as pd
df = pd.DataFrame({"ID": [100000,100101,100201],"Surname_Age": ["Zhao_23","Qian_33","Sun_28" ]})
df_new = df["Surname_Age"].str.split("_", expand =True)
df_new.columns = ["Surname","Age"]
df_mer= pd.merge(df, df_new, left_index =True, right_index=True)
df_mer.drop("Surname_Age", axis = 1, inplace =True)
#將Age字段數(shù)據(jù)類型轉(zhuǎn)化為整型
df_mer["Age"] = df_mer["Age"].astype(int)
#抽取Age中大于30的記錄
df_mer[df_mer.Age > 30]

2) 范圍運算: df[df.字段名.between(s1, s2)], 注意既包含s1又包含s2, 比如抽取年齡大于等于23小于等于28的記錄.

df_mer[df_mer.Age.between(23,28)]

3) 邏輯運算: 與(&) 或(|) 非(not)

比如上面的范圍運算df_mer[df_mer.Age.between(23,28)]就等同于df_mer[(df_mer.Age >= 23) & (df_mer.Age <= 28)]

df_mer[(df_mer.Age >= 23 ) & (df_mer.Age <= 28)]

4) 字符匹配: df[df.字段名.str.contains("字符", case = True, na =False)] contains()函數(shù)中case=True表示區(qū)分大小寫, 默認(rèn)為True; na = False表示不匹配缺失值.

import pandas as pd
import numpy as np
df = pd.DataFrame({"ID": [100000,100101,100201],"Surname_Age": ["Zhao_23","Qian_33","Sun_28"],"SpouseAge":[np.NaN,"32",np.NaN]})
#匹配SpouseAge中包含2的記錄
df[df.SpouseAge.str.contains("2",na = False)]

當(dāng)na改為True時, 結(jié)果為:

5) 缺失值匹配:df[pd.isnull(df.字段名)]表示匹配該字段中有缺失值的記錄.

import pandas as pd
import numpy as np
df = pd.DataFrame({"ID": [100000,100101,100201],"Surname_Age": ["Zhao_23","Qian_33","Sun_28"],"SpouseAge":[np.NaN,"32",np.NaN]})
 #匹配SpouseAge中有缺失值的記錄
df[pd.isnull(df.SpouseAge)]

12.記錄的合并

使用concat()函數(shù)可以將兩個或者多個數(shù)據(jù)表的記錄合并一起, 用法: pandas.concat([df1, df2, df3.....])

import pandas as pd
df1 = pd.DataFrame({"ID": ["A10006","A10001"],"Salary": [12000, 20000]})
df2 = pd.DataFrame({"ID": ["A10008"], "Salary": [10000]})
#使用concat()函數(shù)將df1與df2的記錄進(jìn)行合并
pd.concat([df1, df2])