MapReduce关于类型转换报错记录
0. 写在前面
- 实验环境：Ubuntu Kylin16.04
- Hadoop版本：2.7.2
- IDE：Eclipse3.8
1. 程序代码
Mapper端
importorg.apache.hadoop.io.IntWritable; importorg.apache.hadoop.io.LongWritable; importorg.apache.hadoop.io.Text; importorg.apache.hadoop.mapreduce.Mapper; publicstaticclassWordCountMapperextendsMapper<LongWritable, Text, Text, IntWritable> { protectedvoidmap(LongWritablekey, Textvalue, Contextcontext) throwsIOException, InterruptedException { String[] strs=value.toString().split(" "); for (Stringstr : strs) { context.write(newText(str), newIntWritable(1)); } } }
Reducer端
importorg.apache.hadoop.io.IntWritable; importorg.apache.hadoop.io.LongWritable; importorg.apache.hadoop.io.Text; importorg.apache.hadoop.mapreduce.Reducer; publicstaticclassWordCountReduceextendsReducer<Text, IntWritable, Text, IntWritable> { protectedvoidreduce(Textkey, Iterable<IntWritable>values, Contextcontext) throwsIOException, InterruptedException { intsum=0; for (IntWritableval : values) { System.out.println("<"+key+","+val+">"); sum+=val.get(); } context.write(key, newIntWritable(sum)); } }
Driver端
importjava.io.IOException; importorg.apache.hadoop.conf.Configuration; importorg.apache.hadoop.fs.Path; importorg.apache.hadoop.io.IntWritable; importorg.apache.hadoop.io.LongWritable; importorg.apache.hadoop.io.Text; importorg.apache.hadoop.mapreduce.Job; importorg.apache.hadoop.mapreduce.Mapper; importorg.apache.hadoop.mapreduce.Reducer; importorg.apache.hadoop.mapreduce.lib.input.FileInputFormat; importorg.apache.hadoop.mapreduce.lib.output.FileOutputFormat; publicclassTxtCntDemo { publicstaticvoidmain(String[] args) throwsException { args=newString[] { "/input", "/output"}; Configurationconf=newConfiguration(); Jobjob=Job.getInstance(conf);/job.setJarByClass(TxtCntDemo.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class); job.setMapperClass(WordCountMap.class); job.setReducerClass(WordCountReduce.class); job.setPartitionerClass(MyPartitioner.class); job.setNumReduceTasks(4); FileInputFormat.addInputPath(job, newPath(args[0])); FileOutputFormat.setOutputPath(job, newPath(args[1])); job.waitForCompletion(true); } }
「错误描述」
java.lang.Exception: java.lang.ClassCastException: org.apache.hadoop.io.Text cannot be cast to org.apache.hadoop.io.IntWritable java.lang.Exception: java.lang.ClassCastException: org.apache.hadoop.io.LongWritable cannot be cast to org.apache.hadoop.io.IntWritable
mapper、reducer、driver分开成3个文件，报Text不可转换成IntWritable，还有LongWritable不能转换成IntWritable的错误
关于第二个错误：Mapper端执行时，key的默认输入是LongWritable类型，把LongWritable类型强行转换成Text类型自然就Error了。
java.lang.Exception: java.lang.ClassCastException: org.apache.hadoop.io.Text cannot be cast to org.apache.hadoop.io.IntWritable at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522) Caused by: java.lang.ClassCastException: org.apache.hadoop.io.Text cannot be cast to org.apache.hadoop.io.IntWritable at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:715) at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89) at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112) at cn.mr.WordCountMapper.map(WordCountMapper.java:15) at cn.mr.WordCountMapper.map(WordCountMapper.java:1) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
但是同样的代码mapper、reducer、driver直接放在一个文件下就顺利执行MR得出结果
这个属实给我整不会了 😢 😢
2. 参考
记录一下
