E-MapReduce Hadoop Streaming是什么?

python 写hadoop streaming作业


mapper代码如下

2. #!/usr/bin/env python
3. import sys
5. for line in sys.stdin:
6.     line = line.strip()
7.     words = line.split()
8.     for word in words:
9.         print '%s\t%s' % (word, 1)

reducer代码如下

1. #!/usr/bin/env python
2. from operator import itemgetter
3. import sys
5. current_word = None
6. current_count = 0
7. word = None
9. for line in sys.stdin:
10.     line = line.strip()
12.     word, count = line.split('\t', 1)
14.     try:
15.         count = int(count)
16.     except ValueError:
17.         continue
19.     if current_word == word:
20.         current_count += count
21.     else:
22.         if current_word:
23.             print '%s\t%s' % (current_word, current_count)
24.         current_count = count
25.         current_word = word
27. if current_word == word:
28.     print '%s\t%s' % (current_word, current_count)

假设mapper代码保存在/home/hadoop/mapper.py， reducer代码保存在/home/hadoop/reducer.py ，输入路径为hdfs文件系统的/tmp/input，输出路径为hdfs文件系统的/tmp/output。则在E-MapReduce集群上提交下面的hadoop命令
hadoop jar/usr/lib/hadoop-current/share/hadoop/tools/lib/hadoop-streaming-*.jar -file/home/hadoop/mapper.py -mapper mapper.py -file /home/hadoop/reducer.py -reducerreducer.py -input /tmp/hosts -output /tmp/output