Spark StringIndexer返回空数据集-问答-阿里云开发者社区-阿里云

开发者社区> 问答> 正文

Spark StringIndexer返回空数据集

2019-12-29 19:30:51 391 1

在一个特定列上进行转换后,Apache Spark StringIndexerModel返回一个空数据集。我正在使用成人数据集:http://mlr.cs.umass.edu/ml/datasets/Adult

步骤1:创建StringIndexerModel并将其保存在本地

StringIndexerModel model = new StringIndexer().setInputCol(column).setOutputCol("label").setHandleInvalid("skip").setStringOrderType("alphabetAsc").fit(originalDataset);
model.write().save(filelocation);

步骤2:读取索引器模型并转换新数据集

StringIndexerModel model = StringIndexerModel.read().load(filelocation);
newDataset = model.transform(newDataset).drop(column).withColumnRenamed("label", column);

新数据集:

+---+------------+------------+----------+-------------+------+--------------+-------------------+--------------+----------------+-----+--------------+----+-----------------+
|age|capital gain|capital loss|education |education num|fnlgwt|hours per week|marital status     |native country|occupation      |race |relationship  |sex |workclass        |
+---+------------+------------+----------+-------------+------+--------------+-------------------+--------------+----------------+-----+--------------+----+-----------------+
|39 |2174        |0           | Bachelors|13           |77516 |40            | Never-married     | United-States| Adm-clerical   |White| Not-in-family|Male| State-gov       |
|50 |0           |0           | Bachelors|13           |83311 |13            | Married-civ-spouse| United-States| Exec-managerial|White| Husband      |Male| Self-emp-not-inc|
+---+------------+------------+----------+-------------+------+--------------+-------------------+--------------+----------------+-----+--------------+----+-----------------+

正确的输出:

Column: education | File Location: localFolder/stringIndex/education
Labels: [ 10th,  11th,  12th,  1st-4th,  5th-6th,  7th-8th,  9th,  Assoc-acdm,  Assoc-voc,  Bachelors,  Doctorate,  HS-grad,  Masters,  Preschool,  Prof-school,  Some-college]
+---+------------+------------+-------------+------+--------------+-------------------+--------------+----------------+-----+--------------+----+-----------------+---------+
|age|capital gain|capital loss|education num|fnlgwt|hours per week|marital status     |native country|occupation      |race |relationship  |sex |workclass        |education|
+---+------------+------------+-------------+------+--------------+-------------------+--------------+----------------+-----+--------------+----+-----------------+---------+
|39 |2174        |0           |13           |77516 |40            | Never-married     | United-States| Adm-clerical   |White| Not-in-family|Male| State-gov       |9.0      |
|50 |0           |0           |13           |83311 |13            | Married-civ-spouse| United-States| Exec-managerial|White| Husband      |Male| Self-emp-not-inc|9.0      |
+---+------------+------------+-------------+------+--------------+-------------------+--------------+----------------+-----+--------------+----+-----------------+---------+

Column: marital status | File Location: localFolder/stringIndex/marital status
Labels: [ Divorced,  Married-AF-spouse,  Married-civ-spouse,  Married-spouse-absent,  Never-married,  Separated,  Widowed]
+---+------------+------------+-------------+------+--------------+--------------+----------------+-----+--------------+----+-----------------+---------+--------------+
|age|capital gain|capital loss|education num|fnlgwt|hours per week|native country|occupation      |race |relationship  |sex |workclass        |education|marital status|
+---+------------+------------+-------------+------+--------------+--------------+----------------+-----+--------------+----+-----------------+---------+--------------+
|39 |2174        |0           |13           |77516 |40            | United-States| Adm-clerical   |White| Not-in-family|Male| State-gov       |9.0      |4.0           |
|50 |0           |0           |13           |83311 |13            | United-States| Exec-managerial|White| Husband      |Male| Self-emp-not-inc|9.0      |2.0           |
+---+------------+------------+-------------+------+--------------+--------------+----------------+-----+--------------+----+-----------------+---------+--------------+

Column: native country | File Location: localFolder/stringIndex/native country
Labels: [ ?,  Cambodia,  Canada,  China,  Columbia,  Cuba,  Dominican-Republic,  Ecuador,  El-Salvador,  England,  France,  Germany,  Greece,  Guatemala,  Haiti,  Holand-Netherlands,  Honduras,  Hong,  Hungary,  India,  Iran,  Ireland,  Italy,  Jamaica,  Japan,  Laos,  Mexico,  Nicaragua,  Outlying-US(Guam-USVI-etc),  Peru,  Philippines,  Poland,  Portugal,  Puerto-Rico,  Scotland,  South,  Taiwan,  Thailand,  Trinadad&Tobago,  United-States,  Vietnam,  Yugoslavia]
+---+------------+------------+-------------+------+--------------+----------------+-----+--------------+----+-----------------+---------+--------------+--------------+
|age|capital gain|capital loss|education num|fnlgwt|hours per week|occupation      |race |relationship  |sex |workclass        |education|marital status|native country|
+---+------------+------------+-------------+------+--------------+----------------+-----+--------------+----+-----------------+---------+--------------+--------------+
|39 |2174        |0           |13           |77516 |40            | Adm-clerical   |White| Not-in-family|Male| State-gov       |9.0      |4.0           |39.0          |
|50 |0           |0           |13           |83311 |13            | Exec-managerial|White| Husband      |Male| Self-emp-not-inc|9.0      |2.0           |39.0          |
+---+------------+------------+-------------+------+--------------+----------------+-----+--------------+----+-----------------+---------+--------------+--------------+

Column: occupation | File Location: localFolder/stringIndex/occupation
Labels: [ ?,  Adm-clerical,  Armed-Forces,  Craft-repair,  Exec-managerial,  Farming-fishing,  Handlers-cleaners,  Machine-op-inspct,  Other-service,  Priv-house-serv,  Prof-specialty,  Protective-serv,  Sales,  Tech-support,  Transport-moving]
+---+------------+------------+-------------+------+--------------+-----+--------------+----+-----------------+---------+--------------+--------------+----------+
|age|capital gain|capital loss|education num|fnlgwt|hours per week|race |relationship  |sex |workclass        |education|marital status|native country|occupation|
+---+------------+------------+-------------+------+--------------+-----+--------------+----+-----------------+---------+--------------+--------------+----------+
|39 |2174        |0           |13           |77516 |40            |White| Not-in-family|Male| State-gov       |9.0      |4.0           |39.0          |1.0       |
|50 |0           |0           |13           |83311 |13            |White| Husband      |Male| Self-emp-not-inc|9.0      |2.0           |39.0          |4.0       |
+---+------------+------------+-------------+------+--------------+-----+--------------+----+-----------------+---------+--------------+--------------+----------+

输出错误:除此型号外,所有其他型号均正常工作

Column: race | File Location: localFolder/stringIndex/race
Labels: [ Amer-Indian-Eskimo,  Asian-Pac-Islander,  Black,  Other,  White]
+---+------------+------------+-------------+------+--------------+------------+---+---------+---------+--------------+--------------+----------+----+
|age|capital gain|capital loss|education num|fnlgwt|hours per week|relationship|sex|workclass|education|marital status|native country|occupation|race|
+---+------------+------------+-------------+------+--------------+------------+---+---------+---------+--------------+--------------+----------+----+
+---+------------+------------+-------------+------+--------------+------------+---+---------+---------+--------------+--------------+----------+----
取消 提交回答
全部回答(1)
  • 几许相思几点泪
    2019-12-29 19:31:03

    原来,新数据集的数据不正确。值前应有空格。

    添加空格使' White'我获得正确的输出。

    0 0
相关问答

4

回答

Spark 【问答合集】

社区小助手 2019-05-29 14:13:40 127147浏览量 回答数 4

10

回答

【精品问答合集】Hbase热门问答

hbase小能手 2019-05-29 14:37:26 120823浏览量 回答数 10

145

回答

【新手入门】云服务器linux使用手册

fanyue88888 2012-11-26 17:14:18 157701浏览量 回答数 145

8

回答

OceanBase 使用动画(持续更新)

mq4096 2019-02-20 17:16:36 337017浏览量 回答数 8

5

回答

java.lang.UnsupportedOperationException: This is supposed to be overridden by subclasses.

迷茫君 2019-07-16 09:26:11 118874浏览量 回答数 5

249

回答

阿里云LNAMP(Linux + Nginx + Apache + MySQL + PHP)环境一键安装脚本

云代维 2014-02-14 15:26:06 305641浏览量 回答数 249

97

回答

Redhat/CentOS一键安装web环境全攻略

xiaofanqie 2011-08-11 14:51:38 123706浏览量 回答数 97

27

回答

阿里云开源软件镜像站点上线啦!!

qilu 2014-01-06 18:14:06 96096浏览量 回答数 27

11

回答

【精品问答合集】MongoDB热门问答

李博 bluemind 2019-05-29 16:50:19 121332浏览量 回答数 11

37

回答

【云服务器教程征集令】这个六一阿里云邀您来吐槽!

fanyue88888 2012-06-01 18:55:41 113985浏览量 回答数 37
0
文章
505
问答
问答排行榜
最热
最新
相关电子书
更多
《2021云上架构与运维峰会演讲合集》
立即下载
《零基础CSS入门教程》
立即下载
《零基础HTML入门教程》
立即下载