根据指导文档,我写了一个脚本,如下:
{
"order":{
"hops":[
{
"from":"Reader",
"to":"Writer"
}
]
},
"setting":{
"errorLimit":{
"record":"1000000"
},
"speed":{
"concurrent":1,
"dmu":1,
"throttle":false
}
},
"steps":[
{
"category":"reader",
"name":"Reader",
"parameter":{
"column":[
"*"
],
"datasource":"BigDataEast",
"partition":["datatype='ali'"],
"table":"dwd_user_map"
},
"stepType":"odps"
},
{
"category":"writer",
"name":"Writer",
"parameter":{
"column":[
{
"name":"col1",
"type":"string"
},
{
"name":"col2",
"type":"string"
},
{
"name":"col3",
"type":"string"
},
{
"name":"col4",
"type":"string"
},
{
"name":"col5",
"type":"string"
},
{
"name":"col6",
"type":"string"
},
{
"name":"col7",
"type":"string"
},
{
"name":"col8",
"type":"string"
},
{
"name":"col9",
"type":"string"
},
{
"name":"col10",
"type":"string"
},
{
"name":"col11",
"type":"string"
},
{
"name":"col12",
"type":"string"
},
{
"name":"col13",
"type":"string"
}
],
"compress":"gzip",
"datasource":"hadoop_test",
"encoding":"UTF-8",
"fieldDelimiter":"\n",
"fileName":"test1",
"fileType":"text",
"path":"/cdm",
"writeMode":"append"
},
"stepType":"hdfs"
}
],
"type":"job",
"version":"2.0"
}
但是运行时候报了一个很莫名其妙的错误
Caused by: org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /szb_data__873b21de_085c_44cd_954a_fbaf386cec38/test1.gz could only be replicated to 0 nodes instead of minReplication (=1). There are3 datanode(s) running and 3 node(s) are excluded in this operation.
事实上我的DataNode完全正常,我通过代码将本地文件上传到HDFS也成功了。说明hdfs正常,有没有大神指导下这个是什么错误。