统计推断基础¶
In [1]:
import pandas as pd
import sklearn.datasets as datasets
df=pd.read_csv('http://archive.ics.uci.edu/ml/machine-learning-databases/servo/servo.data',
header=None,names=['motor','screw','pgain','vgain','target'])
df.head()
Out[1]:
描述性统计分析¶
In [2]:
df.describe(include='all')
Out[2]:
Box Plots¶
In [3]:
df.plot(kind='box') # Box Plots
Out[3]:
置信度区间估计¶
In [4]:
import statsmodels.api as sm
# 或者使用DescrStatsW
d1 = sm.stats.DescrStatsW(df.loc[:,'target'])
d1.tconfint_mean(0.05) # alpha=0.05
Out[4]:
In [5]:
%matplotlib inline
import seaborn as sns
from scipy import stats
sns.distplot(df.loc[:,'target'], kde=True, fit=stats.norm) # Histograph
Out[5]:
Q-Q¶
In [6]:
import statsmodels.api as sm
from matplotlib import pyplot as plt
fig = sm.qqplot(df.loc[:,'target'], fit=True, line='45')
对分布的检验¶
- stats.shapiro
Shapiro–Wilk test,是一种基于相关性的检验思路，接近1，说明接近正态分布。H0：样本服从正态分布 - stats.jarque_bera 是一种基于峰度和偏度的检验思路，
In [7]:
import pandas as pd
import scipy
rv=scipy.stats.norm(loc=0,scale=1)
df_rv=pd.DataFrame(rv.rvs(size=4000),columns=['data'])
In [8]:
print('Jarque-Bera test:', stats.jarque_bera(df_rv.data)) # 返回统计量和p-value
print('Shapiro-Wilk test:', stats.shapiro(df_rv.data)) # 返回统计量和p-value
stats.kstest(rvs=df_rv.data,cdf='norm') # 返回统计量和p-value
Out[8]:
单样本T检验¶
In [9]:
't-statistic=%6.4f, p-value=%6.4f, df=%s' %d1.ttest_mean(1.15)
Out[9]:
p-value>0.05,说明不能拒绝原假设，认为均值确实是1.15
两样本T检验¶
先做个描述性统计
In [10]:
df.groupby('motor')['target'].describe()
Out[10]:
挑选前两类做T检验
- 第一步:方差齐次检验
In [11]:
(_,df1),(_,df2),(_,df3),(_,_),(_,_)=df.groupby('motor')['target']
leveneTestRes = stats.levene(df1, df2, center='median')
print('w-value=%6.4f, p-value=%6.4f' %leveneTestRes)
p-value < 0.05，说明拒绝原假设，认为方差不齐
p-value > 0.05，说明无法拒绝原假设，认为方差齐性
- 第二步:T-test
In [12]:
# stats.stats.ttest_ind(df1, df2, equal_var=True)#scipy.stats
'statistic=%6.4f, pvalue=%6.4f, df=%6.4f'%sm.stats.ttest_ind(df1, df2, usevar='unequal')#usevar='pooled'or'unequal'
Out[12]:
pvalue < 0.05 说明拒绝原假设，认为均值不相等 pvalue > 0.05 说明无法拒绝原假设，认为均值相等 http://www.statsmodels.org/stable/generated/statsmodels.stats.weightstats.CompareMeans.ttest_ind.html
方差分析¶
- 单因素方差分析
In [13]:
sepal_length_list = []
for i in df['motor'].unique():
sepal_length_list.append(df[df['motor'] == i]['target'])
stats.f_oneway(*sepal_length_list)
Out[13]:
In [14]:
# 利用回归模型中的方差分析
from statsmodels.formula.api import ols
sm.stats.anova_lm(ols('target ~ C(motor)',data=df).fit())
Out[14]:
- 多因素方差分析
In [15]:
sm.stats.anova_lm(ols('target ~ C(motor) + C(screw)',data=df).fit())
Out[15]:
In [16]:
ana = ols('target ~ C(motor) + C(screw) +C(motor)*C(screw)', data= df).fit()
sm.stats.anova_lm(ana)
Out[16]:
相关分析¶
散点图
In [17]:
import seaborn as sns
sns.jointplot(x='vgain',y='target', data=df,kind='scatter')
Out[17]:
相关性分析:“spearman”,“pearson” 和 "kendall"
- pearson用于线性相关关系
- spearman秩相关系数，不一定线性，但是单调
- kendall序数
In [18]:
df.loc[:,['pgain', 'target']].corr(method='pearson')
# df.loc[:,['pgain', 'target']].corr(method='spearman')
# df.loc[:,['pgain', 'target']].corr(method='kendall')
Out[18]:
卡方检验¶
In [19]:
cross_table=df.pivot_table(index='motor',columns='screw',values='target',aggfunc='count')
cross_table
Out[19]:
In [20]:
print('chisq = %6.4f\n p-value = %6.4f\n dof = %i\n expected_freq = %s'%stats.chi2_contingency(cross_table))