[雪峰磁针石博客]数据分析工具pandas快速入门教程5-处理缺失数据

简介:

第5章 缺失数据

介绍

很少没有任何缺失值的数据集。 有许多缺失数据的表示。 在数据库中是NULL值,一些编程语言使用NA。缺失值可以是空字符串:''或者甚至是数值88或99等。Pandas显示缺失值为NaN。

本章将涵盖:

  • 什么是缺失值
  • 如何创建缺失值
  • 如何重新编码并使用缺失值进行计算

什么是缺失值

可以从numpy中获得NaN值,在Python中看到缺失值使用几种方式显示:NaN,NAN或nan,他们都是相等的。

NaN不等于0或空字符串''。


In [1]: from numpy import NaN, NAN, nan

In [2]: print(NaN == True, NaN == False, NaN == 0, NaN == '', sep='|')
False|False|False|False

In [3]: print(NaN == NaN, NaN == nan, NaN == NAN, nan == NAN, sep='|')
False|False|False|False

In [4]: import pandas as pd

In [5]: print(pd.isnull(NaN), pd.isnull(nan), pd.isnull(NAN), sep='|')
True|True|True

In [6]: print(pd.notnull(NaN), pd.notnull(99), pd.notnull("https://china-testing.github.io"), sep='|')
False|True|True
AI 代码解读

缺失值的来源

来自加载数据或数据处理

  • 加载数据

当我们加载数据时,pandas会自动找到该缺少数据的单元格,并填充NaN值。在read_csv函数中,参数na_values, keep_default_na, na_filter用于处理缺失值。比如:na_values=[99]。na_filter设置为False,在读大文件时会提升性能。

5-1.py


import pandas as pd

visited_file = 'data/survey_visited.csv'
print(pd.read_csv(visited_file))
print(pd.read_csv(visited_file, keep_default_na=False))
print(pd.read_csv(visited_file, na_values=[''], keep_default_na=False))
AI 代码解读

执行结果


$ python3 5-1.py 
   ident   site       dated
0    619   DR-1  1927-02-08
1    622   DR-1  1927-02-10
2    734   DR-3  1939-01-07
3    735   DR-3  1930-01-12
4    751   DR-3  1930-02-26
5    752   DR-3         NaN
6    837  MSK-4  1932-01-14
7    844   DR-1  1932-03-22
   ident   site       dated
0    619   DR-1  1927-02-08
1    622   DR-1  1927-02-10
2    734   DR-3  1939-01-07
3    735   DR-3  1930-01-12
4    751   DR-3  1930-02-26
5    752   DR-3            
6    837  MSK-4  1932-01-14
7    844   DR-1  1932-03-22
   ident   site       dated
0    619   DR-1  1927-02-08
1    622   DR-1  1927-02-10
2    734   DR-3  1939-01-07
3    735   DR-3  1930-01-12
4    751   DR-3  1930-02-26
5    752   DR-3         NaN
6    837  MSK-4  1932-01-14
7    844   DR-1  1932-03-22
AI 代码解读
  • 合并数据

import pandas as pd

visited = pd.read_csv('data/survey_visited.csv')
survey = pd.read_csv('data/survey_survey.csv')
print(visited)
print(survey)
vs = visited.merge(survey, left_on='ident', right_on='taken')
print(vs)
AI 代码解读

执行结果


$ python3 5-2.py 
   ident   site       dated
0    619   DR-1  1927-02-08
1    622   DR-1  1927-02-10
2    734   DR-3  1939-01-07
3    735   DR-3  1930-01-12
4    751   DR-3  1930-02-26
5    752   DR-3         NaN
6    837  MSK-4  1932-01-14
7    844   DR-1  1932-03-22
    taken person quant  reading
0     619   dyer   rad     9.82
1     619   dyer   sal     0.13
2     622   dyer   rad     7.80
3     622   dyer   sal     0.09
4     734     pb   rad     8.41
5     734   lake   sal     0.05
6     734     pb  temp   -21.50
7     735     pb   rad     7.22
8     735    NaN   sal     0.06
9     735    NaN  temp   -26.00
10    751     pb   rad     4.35
11    751     pb  temp   -18.50
12    751   lake   sal     0.10
13    752   lake   rad     2.19
14    752   lake   sal     0.09
15    752   lake  temp   -16.00
16    752    roe   sal    41.60
17    837   lake   rad     1.46
18    837   lake   sal     0.21
19    837    roe   sal    22.50
20    844    roe   rad    11.25
    ident   site       dated  taken person quant  reading
0     619   DR-1  1927-02-08    619   dyer   rad     9.82
1     619   DR-1  1927-02-08    619   dyer   sal     0.13
2     622   DR-1  1927-02-10    622   dyer   rad     7.80
3     622   DR-1  1927-02-10    622   dyer   sal     0.09
4     734   DR-3  1939-01-07    734     pb   rad     8.41
5     734   DR-3  1939-01-07    734   lake   sal     0.05
6     734   DR-3  1939-01-07    734     pb  temp   -21.50
7     735   DR-3  1930-01-12    735     pb   rad     7.22
8     735   DR-3  1930-01-12    735    NaN   sal     0.06
9     735   DR-3  1930-01-12    735    NaN  temp   -26.00
10    751   DR-3  1930-02-26    751     pb   rad     4.35
11    751   DR-3  1930-02-26    751     pb  temp   -18.50
12    751   DR-3  1930-02-26    751   lake   sal     0.10
13    752   DR-3         NaN    752   lake   rad     2.19
14    752   DR-3         NaN    752   lake   sal     0.09
15    752   DR-3         NaN    752   lake  temp   -16.00
16    752   DR-3         NaN    752    roe   sal    41.60
17    837  MSK-4  1932-01-14    837   lake   rad     1.46
18    837  MSK-4  1932-01-14    837   lake   sal     0.21
19    837  MSK-4  1932-01-14    837    roe   sal    22.50
20    844   DR-1  1932-03-22    844    roe   rad    11.25
AI 代码解读
  • 用户输入

import pandas as pd
from numpy import NaN, NAN, nan

num_legs = pd.Series({'goat': 4, 'amoeba': nan})
print(num_legs)
scientists = pd.DataFrame({'Name': ['Rosaline Franklin', 'William Gosset'],
                           'Occupation': ['Chemist', 'Statistician'],
                           'Born': ['1920-07-25', '1876-06-13'],
                           'Died': ['1958-04-16', '1937-10-16'],
                           'missing': [NaN, nan]})
print(scientists)
scientists['missing'] = nan
print(scientists)
AI 代码解读

执行结果


$ python3 5-3.py 
amoeba    NaN
goat      4.0
dtype: float64
         Born        Died               Name    Occupation  missing
0  1920-07-25  1958-04-16  Rosaline Franklin       Chemist      NaN
1  1876-06-13  1937-10-16     William Gosset  Statistician      NaN
         Born        Died               Name    Occupation  missing
0  1920-07-25  1958-04-16  Rosaline Franklin       Chemist      NaN
1  1876-06-13  1937-10-16     William Gosset  Statistician      NaN
AI 代码解读
  • 重新索引

5-4.py


import pandas as pd
from numpy import NaN, NAN, nan

gapminder = pd.read_csv('data/gapminder.tsv', sep='\t')
life_exp = gapminder.groupby(['year'])['lifeExp'].mean()
print(life_exp)
print(life_exp.reindex(range(2000, 2010)))
AI 代码解读

执行结果


year
1952    49.057620
1957    51.507401
1962    53.609249
1967    55.678290
1972    57.647386
1977    59.570157
1982    61.533197
1987    63.212613
1992    64.160338
1997    65.014676
2002    65.694923
2007    67.007423
Name: lifeExp, dtype: float64
year
2000          NaN
2001          NaN
2002    65.694923
2003          NaN
2004          NaN
2005          NaN
2006          NaN
2007    67.007423
2008          NaN
2009          NaN
Name: lifeExp, dtype: float64
AI 代码解读

处理缺失数据

  • 统计缺失数据

5-5.py


import pandas as pd
from numpy import NaN, NAN, nan
import numpy as np

ebola = pd.read_csv('data/country_timeseries.csv')
print(ebola.head())
print(ebola.count())
num_rows = ebola.shape[0]
print("num_rows")
print(num_rows)
num_missing = num_rows - ebola.count()
print("num_missing:")
print(num_missing)
print(np.count_nonzero(ebola.isnull()))
print(np.count_nonzero(ebola['Cases_Guinea'].isnull()))
print(ebola.Cases_Guinea.value_counts(dropna=False).head())
AI 代码解读

执行结果


 Date  Day  Cases_Guinea  Cases_Liberia  Cases_SierraLeone  \
0    1/5/2015  289        2776.0            NaN            10030.0   
1    1/4/2015  288        2775.0            NaN             9780.0   
2    1/3/2015  287        2769.0         8166.0             9722.0   
3    1/2/2015  286           NaN         8157.0                NaN   
4  12/31/2014  284        2730.0         8115.0             9633.0   

   Cases_Nigeria  Cases_Senegal  Cases_UnitedStates  Cases_Spain  Cases_Mali  \
0            NaN            NaN                 NaN          NaN         NaN   
1            NaN            NaN                 NaN          NaN         NaN   
2            NaN            NaN                 NaN          NaN         NaN   
3            NaN            NaN                 NaN          NaN         NaN   
4            NaN            NaN                 NaN          NaN         NaN   

   Deaths_Guinea  Deaths_Liberia  Deaths_SierraLeone  Deaths_Nigeria  \
0         1786.0             NaN              2977.0             NaN   
1         1781.0             NaN              2943.0             NaN   
2         1767.0          3496.0              2915.0             NaN   
3            NaN          3496.0                 NaN             NaN   
4         1739.0          3471.0              2827.0             NaN   

   Deaths_Senegal  Deaths_UnitedStates  Deaths_Spain  Deaths_Mali  
0             NaN                  NaN           NaN          NaN  
1             NaN                  NaN           NaN          NaN  
2             NaN                  NaN           NaN          NaN  
3             NaN                  NaN           NaN          NaN  
4             NaN                  NaN           NaN          NaN  
Date                   122
Day                    122
Cases_Guinea            93
Cases_Liberia           83
Cases_SierraLeone       87
Cases_Nigeria           38
Cases_Senegal           25
Cases_UnitedStates      18
Cases_Spain             16
Cases_Mali              12
Deaths_Guinea           92
Deaths_Liberia          81
Deaths_SierraLeone      87
Deaths_Nigeria          38
Deaths_Senegal          22
Deaths_UnitedStates     18
Deaths_Spain            16
Deaths_Mali             12
dtype: int64
num_rows
122
num_missing:
Date                     0
Day                      0
Cases_Guinea            29
Cases_Liberia           39
Cases_SierraLeone       35
Cases_Nigeria           84
Cases_Senegal           97
Cases_UnitedStates     104
Cases_Spain            106
Cases_Mali             110
Deaths_Guinea           30
Deaths_Liberia          41
Deaths_SierraLeone      35
Deaths_Nigeria          84
Deaths_Senegal         100
Deaths_UnitedStates    104
Deaths_Spain           106
Deaths_Mali            110
dtype: int64
1214
29
NaN       29
 86.0      3
 495.0     2
 112.0     2
 390.0     2
Name: Cases_Guinea, dtype: int64
AI 代码解读
  • 处理缺失数据

5-6.py


import pandas as pd
from numpy import NaN, NAN, nan
import numpy as np

ebola = pd.read_csv('data/country_timeseries.csv')
print(ebola.iloc[0:10, 0:5])
print(ebola.fillna(0).iloc[0:10, 0:5])
# 前向填充
print(ebola.fillna(method='ffill').iloc[0:10, 0:5])
# 后向填充
print(ebola.fillna(method='bfill').iloc[0:10, 0:5])

print(ebola.interpolate().iloc[0:10, 0:5])

print(ebola.shape)
ebola_dropna = ebola.dropna()
print(ebola_dropna.shape)
print(ebola_dropna)

ebola['Cases_multiple'] = ebola['Cases_Guinea'] + ebola['Cases_Liberia'] + \
ebola['Cases_SierraLeone']

ebola_subset = ebola.loc[:, ['Cases_Guinea', 'Cases_Liberia',
                             'Cases_SierraLeone', 'Cases_multiple']]
print(ebola_subset.head(n=10))
print(ebola.Cases_Guinea.sum(skipna = True))
print(ebola.Cases_Guinea.sum(skipna = False))
AI 代码解读

执行结果


         Date  Day  Cases_Guinea  Cases_Liberia  Cases_SierraLeone
0    1/5/2015  289        2776.0            NaN            10030.0
1    1/4/2015  288        2775.0            NaN             9780.0
2    1/3/2015  287        2769.0         8166.0             9722.0
3    1/2/2015  286           NaN         8157.0                NaN
4  12/31/2014  284        2730.0         8115.0             9633.0
5  12/28/2014  281        2706.0         8018.0             9446.0
6  12/27/2014  280        2695.0            NaN             9409.0
7  12/24/2014  277        2630.0         7977.0             9203.0
8  12/21/2014  273        2597.0            NaN             9004.0
9  12/20/2014  272        2571.0         7862.0             8939.0
         Date  Day  Cases_Guinea  Cases_Liberia  Cases_SierraLeone
0    1/5/2015  289        2776.0            0.0            10030.0
1    1/4/2015  288        2775.0            0.0             9780.0
2    1/3/2015  287        2769.0         8166.0             9722.0
3    1/2/2015  286           0.0         8157.0                0.0
4  12/31/2014  284        2730.0         8115.0             9633.0
5  12/28/2014  281        2706.0         8018.0             9446.0
6  12/27/2014  280        2695.0            0.0             9409.0
7  12/24/2014  277        2630.0         7977.0             9203.0
8  12/21/2014  273        2597.0            0.0             9004.0
9  12/20/2014  272        2571.0         7862.0             8939.0
         Date  Day  Cases_Guinea  Cases_Liberia  Cases_SierraLeone
0    1/5/2015  289        2776.0            NaN            10030.0
1    1/4/2015  288        2775.0            NaN             9780.0
2    1/3/2015  287        2769.0         8166.0             9722.0
3    1/2/2015  286        2769.0         8157.0             9722.0
4  12/31/2014  284        2730.0         8115.0             9633.0
5  12/28/2014  281        2706.0         8018.0             9446.0
6  12/27/2014  280        2695.0         8018.0             9409.0
7  12/24/2014  277        2630.0         7977.0             9203.0
8  12/21/2014  273        2597.0         7977.0             9004.0
9  12/20/2014  272        2571.0         7862.0             8939.0
         Date  Day  Cases_Guinea  Cases_Liberia  Cases_SierraLeone
0    1/5/2015  289        2776.0         8166.0            10030.0
1    1/4/2015  288        2775.0         8166.0             9780.0
2    1/3/2015  287        2769.0         8166.0             9722.0
3    1/2/2015  286        2730.0         8157.0             9633.0
4  12/31/2014  284        2730.0         8115.0             9633.0
5  12/28/2014  281        2706.0         8018.0             9446.0
6  12/27/2014  280        2695.0         7977.0             9409.0
7  12/24/2014  277        2630.0         7977.0             9203.0
8  12/21/2014  273        2597.0         7862.0             9004.0
9  12/20/2014  272        2571.0         7862.0             8939.0
         Date  Day  Cases_Guinea  Cases_Liberia  Cases_SierraLeone
0    1/5/2015  289        2776.0            NaN            10030.0
1    1/4/2015  288        2775.0            NaN             9780.0
2    1/3/2015  287        2769.0         8166.0             9722.0
3    1/2/2015  286        2749.5         8157.0             9677.5
4  12/31/2014  284        2730.0         8115.0             9633.0
5  12/28/2014  281        2706.0         8018.0             9446.0
6  12/27/2014  280        2695.0         7997.5             9409.0
7  12/24/2014  277        2630.0         7977.0             9203.0
8  12/21/2014  273        2597.0         7919.5             9004.0
9  12/20/2014  272        2571.0         7862.0             8939.0
(122, 18)
(1, 18)
          Date  Day  Cases_Guinea  Cases_Liberia  Cases_SierraLeone  \
19  11/18/2014  241        2047.0         7082.0             6190.0   

    Cases_Nigeria  Cases_Senegal  Cases_UnitedStates  Cases_Spain  Cases_Mali  \
19           20.0            1.0                 4.0          1.0         6.0   

    Deaths_Guinea  Deaths_Liberia  Deaths_SierraLeone  Deaths_Nigeria  \
19         1214.0          2963.0              1267.0             8.0   

    Deaths_Senegal  Deaths_UnitedStates  Deaths_Spain  Deaths_Mali  
19             0.0                  1.0           0.0          6.0  
   Cases_Guinea  Cases_Liberia  Cases_SierraLeone  Cases_multiple
0        2776.0            NaN            10030.0             NaN
1        2775.0            NaN             9780.0             NaN
2        2769.0         8166.0             9722.0         20657.0
3           NaN         8157.0                NaN             NaN
4        2730.0         8115.0             9633.0         20478.0
5        2706.0         8018.0             9446.0         20170.0
6        2695.0            NaN             9409.0             NaN
7        2630.0         7977.0             9203.0         19810.0
8        2597.0            NaN             9004.0             NaN
9        2571.0         7862.0             8939.0         19372.0
84729.0
nan
AI 代码解读

参考资料

目录
打赏
0
0
0
0
971
分享
相关文章
Pandas数据应用:天气数据分析
本文介绍如何使用 Pandas 进行天气数据分析。Pandas 是一个强大的 Python 数据处理库,适合处理表格型数据。文章涵盖加载天气数据、处理缺失值、转换数据类型、时间序列分析(如滚动平均和重采样)等内容,并解决常见报错如 SettingWithCopyWarning、KeyError 和 TypeError。通过这些方法,帮助用户更好地进行气候趋势预测和决策。
162 71
从数据小白到大数据达人:一步步成为数据分析专家
从数据小白到大数据达人:一步步成为数据分析专家
261 92
Probly:开源 AI Excel表格工具,交互式生成数据分析结果与可视化图表
Probly 是一款结合电子表格功能与 Python 数据分析能力的 AI 工具,支持在浏览器中运行 Python 代码,提供交互式电子表格、数据可视化和智能分析建议,适合需要强大数据分析功能又希望操作简便的用户。
213 2
数据团队必读:智能数据分析文档(DataV Note)五种高效工作模式
数据项目复杂,涉及代码、数据、运行环境等多部分。随着AI发展,数据科学团队面临挑战。协作式数据文档(如阿里云DataV Note)成为提升效率的关键工具。它支持跨角色协同、异构数据处理、多语言分析及高效沟通,帮助创建知识库,实现可重现的数据科学过程,并通过一键分享报告促进数据驱动决策。未来,大模型AI将进一步增强其功能,如智能绘图、总结探索、NLP2SQL/Python和AutoReport,为数据分析带来更多可能。
102 21
Pandas数据应用:电子商务数据分析
本文介绍如何使用 Pandas 进行电子商务数据分析,涵盖数据加载、清洗、预处理、分析与可视化。通过 `read_csv` 等函数加载数据,利用 `info()` 和 `describe()` 探索数据结构和统计信息。针对常见问题如缺失值、重复记录、异常值等,提供解决方案,如 `dropna()`、`drop_duplicates()` 和正则表达式处理。结合 Matplotlib 等库实现数据可视化,探讨内存不足和性能瓶颈的应对方法,并总结常见报错及解决策略,帮助提升电商企业的数据分析能力。
176 73
这可能是最适合探索式数据分析的工具
SPL(Structured Process Language)是一款结合了强大计算能力和灵活交互性的数据分析工具,特别适合探索式数据分析。它不仅支持分步执行和实时查看每步结果,还提供了丰富的表格数据计算类库,简化复杂运算。与Excel相比,SPL在处理复杂逻辑时更加简洁高效;相较于SQL和Python,SPL具备更好的交互性和更直观的操作体验。通过SPL的XLL插件,用户可以在Excel环境中直接使用SPL的强大功能,充分发挥两者优势。SPL开源免费,是探索式数据分析的理想选择。
基于机器学习的数据分析:PLC采集的生产数据预测设备故障模型
本文介绍如何利用Python和Scikit-learn构建基于PLC数据的设备故障预测模型。通过实时采集温度、振动、电流等参数,进行数据预处理和特征提取,选择合适的机器学习模型(如随机森林、XGBoost),并优化模型性能。文章还分享了边缘计算部署方案及常见问题排查,强调模型预测应结合定期维护,确保系统稳定运行。
75 0
Pandas数据应用:医疗数据分析
Pandas是Python中强大的数据操作和分析库,广泛应用于医疗数据分析。本文介绍了使用Pandas进行医疗数据分析的常见问题及解决方案,涵盖数据导入、预处理、清洗、转换、可视化等方面。通过解决文件路径错误、编码不匹配、缺失值处理、异常值识别、分类变量编码等问题,结合Matplotlib等工具实现数据可视化,并提供了解决常见报错的方法。掌握这些技巧可以提高医疗数据分析的效率和准确性。
103 22
利用数据分析工具评估特定业务场景下扩缩容操作对性能的影响
通过以上数据分析工具的运用,可以深入挖掘数据背后的信息,准确评估特定业务场景下扩缩容操作对 PolarDB Serverless 性能的影响。同时,这些分析结果还可以为后续的优化和决策提供有力的支持,确保业务系统在不断变化的环境中保持良好的性能表现。
107 48
数据集中存在大量的重复值,会对后续的数据分析和处理产生什么影响?
数据集中存在大量重复值可能会对后续的数据分析和处理产生多方面的负面影响
231 56
AI助理

你好,我是AI助理

可以解答问题、推荐解决方案等