[python skill]利用python实现假设性检验方法

+关注继续查看

hello，大噶好，最近新学习了利用python实现假设性检验的一些方法，下面结合方法的数学原理做简单的总结~

--------------------------------------------------------------------------------------------------------------------

Permutation test on frog data

1、合并n1，n2，无放回地抽取生成与原数据长度相等的两组数据(n1,n2)

def permutation_sample(data1, data2):
"""Generate a permutation sample from two data sets."""

# Concatenate the data sets: data
data = np.concatenate((data1, data2))

# Permute the concatenated array: permuted_data
permuted_data = np.random.permutation(data)

# Split the permuted array into two: perm_sample_1, perm_sample_2
perm_sample_1 = permuted_data[:len(data1)]
perm_sample_2 = permuted_data[len(data1):]

return perm_sample_1, perm_sample_2

2、重复1过程，进行多次实验（如10000次），将每次实验得出数据（n1,n2）求平均，再将平均值做差：

def draw_perm_reps(data_1, data_2, func, size=1):
"""Generate multiple permutation replicates."""

# Initialize array of replicates: perm_replicates
perm_replicates = np.empty(size)

for i in range(size):
# Generate permutation sample
perm_sample_1, perm_sample_2 = permutation_sample(data_1,data_2)

# Compute the test statistic
perm_replicates[i] = func(perm_sample_1,perm_sample_2)

return perm_replicates
def diff_of_means(data_1, data_2):
"""Difference in means of two arrays."""

# The difference of means of data_1, data_2: diff
diff = np.mean(data_1)-np.mean(data_2)

return diff



3、这样我们得到多个（10000）置换排列求得的结果，这些结果能代表模拟抽样总体情况。

Kleinteich and Gorb (Sci. Rep.4, 5225, 2014) performed an interesting experiment with South American horned frogs. They held a plate connected to a force transducer, along with a bait fly, in front of them. They then measured the impact force and adhesive force of the frog's tongue when it struck the target.

K和G博士以前做过一系列实验，研究青蛙舌头的黏力与哪些因素有关。

# Compute difference of mean impact force from experiment: empirical_diff_means
empirical_diff_means = diff_of_means(force_a,force_b)#求得原始数据的平均值的差值

# Draw 10,000 permutation replicates: perm_replicates
perm_replicates = draw_perm_reps(force_a, force_b,
diff_of_means, size=10000)#重复一万次实验之后统计差值分布情况

# Compute p-value: p
p = np.sum(perm_replicates >= empirical_diff_means) / len(perm_replicates)#计算统计差值中比原始数据差值还大的可能

# Print the result
print('p-value =', p)


output：

p-value = 0.0063

-----------------------------------------------------------------------------------------------------------------------------------

Bootstrap hypothesis tests

-----------------------------------------------------------------------------------------------------------------------------------

A one-sample bootstrap hypothesis test

Another juvenile frog was studied, Frog C, and you want to see if Frog B and Frog C have similar impact forces. Unfortunately, you do not have Frog C's impact forces available, but you know they have a mean of 0.55 N. Because you don't have the original data, you cannot do a permutation test, and you cannot assess the hypothesis that the forces from Frog B and Frog C come from the same distribution. You will therefore test another, less restrictive hypothesis: The mean strike force of Frog B is equal to that of Frog C.

To set up the bootstrap hypothesis test, you will take the mean as our test statistic. Remember, your goal is to calculate the probability of getting a mean impact force less than or equal to what was observed for Frog B if the hypothesis that the true mean of Frog B's impact forces is equal to that of Frog C is true. You first translate all of the data of Frog B such that the mean is 0.55 N. This involves adding the mean force of Frog C and subtracting the mean force of Frog B from each measurement of Frog B. This leaves other properties of Frog B's distribution, such as the variance, unchanged.

# Make an array of translated impact forces: translated_force_b
translated_force_b = force_b-np.mean(force_b)+0.55#改变FROG_B的黏力（为什么要改FROG_B的数据呢？？？认为FROG_B的原始数据采集错误吗？）

# Take bootstrap replicates of Frog B's translated impact forces: bs_replicates
bs_replicates = draw_bs_reps(translated_force_b, np.mean, 10000)#bootstrap reps

# Compute fraction of replicates that are less than the observed Frog B force: p
p = np.sum(bs_replicates <= np.mean(force_b)) / 10000#求p

# Print the p-value
print('p = ', p)


output:

p =  0.0046

-----------------------------------------------------------------------------------------------------------------------------------

A bootstrap test for identical distributions

# Compute difference of mean impact force from experiment: empirical_diff_means
empirical_diff_means = diff_of_means(force_a,force_b)

# Concatenate forces: forces_concat
forces_concat = np.concatenate((force_a,force_b))

# Initialize bootstrap replicates: bs_replicates
bs_replicates = np.empty(10000)

for i in range(10000):
# Generate bootstrap sample
bs_sample = np.random.choice(forces_concat, size=len(forces_concat))

# Compute replicate
bs_replicates[i] = diff_of_means(bs_sample[:len(force_a)],
bs_sample[len(force_a):])

# Compute and print p-value: p
p = np.sum(bs_replicates>=empirical_diff_means) / len(bs_replicates)
print('p-value =', p)


output：

p-value = 0.0055

代码过程解析：

1、首先我们认为FROG_A和FROG_B的分布是相同的，那么我们就可以将其合并（forces_concat = np.concatenate((force_a,force_b))）；

2、利用bootstrap replicate，我们可以有放回地抽取force_a，force_b长度的两组数据，求平均值（得到一种可能的客观世界均值），并对平均值做差。

3、重复2步骤10000次，我们就得到了可能的客观世界中两个里在假设情况下均值之差的分布情况。

4、检验比原始数据（empirical_diff_means = diff_of_means(force_a,force_b)）更极端的情况发生的概率是多少，求出p值。

Testing the hypothesis that two samples have the same distribution may be done with a bootstrap test, but a permutation test is preferred because it is more accurate (exact, in fact).

But therein lies the limit of a permutation test; it is not very versatile. We now want to test the hypothesis that Frog A and Frog B have the same mean impact force, but not necessarily the same distribution. This, too, is impossible with a permutation test.

-----------------------------------------------------------------------------------------------------------------------------------

A two-sample bootstrap hypothesis test for difference of means.

# Compute mean of all forces: mean_force
mean_force = np.mean(forces_concat)

# Generate shifted arrays
force_a_shifted = force_a - np.mean(force_a) + mean_force
force_b_shifted = force_b - np.mean(force_b) + mean_force

# Compute 10,000 bootstrap replicates from shifted arrays
bs_replicates_a = draw_bs_reps(force_a_shifted, np.mean, 10000)
bs_replicates_b = draw_bs_reps(force_b_shifted, np.mean, 10000)

# Get replicates of difference of means: bs_replicates
bs_replicates = bs_replicates_a - bs_replicates_b

# Compute and print p-value: p
p = np.sum(bs_replicates>=empirical_diff_means) / len(bs_replicates)
print('p-value =', p)


output：

p-value = 0.0043

that's all thank you~

Python列表的方法
Python列表的方法。 插入的方法，删除的方法，反转列表，排序列表这些方法全都是改的对象
0 0
Python数据分析（四）——plot方法
Python数据分析（四）——plot方法
0 0
【Python基础：面向对象之魔法方法】
【Python基础：面向对象之魔法方法】
0 0
Python中断多重循环的几种方法，你都知道吗？
Python中断多重循环的几种方法，你都知道吗？
0 0
Python 常用字符串方法

0 0
Python学习之路-列表的方法

0 0

0 0
【Python零基础入门篇 · 26】：魔法方法（__doc__、__str__、__module__、__class__的使用）
【Python零基础入门篇 · 26】：魔法方法（__doc__、__str__、__module__、__class__的使用）
0 0
【Python零基础入门篇 · 25】：init和new方法、单例模式
【Python零基础入门篇 · 25】：init和new方法、单例模式
0 0
+关注

Python第五讲——关于爬虫如何做js逆向的思路

Python系列直播第一讲——Python中的一切皆对象

Python 脚本速查手册