E-MapReduce Hadoop Streaming是什么?

python 写hadoop streaming作业


mapper代码如下

2. [backcolor=transparent]#!/usr/bin/env python
3. [backcolor=transparent]import[backcolor=transparent] sys
5. [backcolor=transparent]for[backcolor=transparent] line [backcolor=transparent]in[backcolor=transparent] sys[backcolor=transparent].[backcolor=transparent]stdin[backcolor=transparent]:
6. [backcolor=transparent]    line [backcolor=transparent]=[backcolor=transparent] line[backcolor=transparent].[backcolor=transparent]strip[backcolor=transparent]()
7. [backcolor=transparent]    words [backcolor=transparent]=[backcolor=transparent] line[backcolor=transparent].[backcolor=transparent]split[backcolor=transparent]()
8. [backcolor=transparent]    [backcolor=transparent]for[backcolor=transparent] word [backcolor=transparent]in[backcolor=transparent] words[backcolor=transparent]:
9. [backcolor=transparent]        [backcolor=transparent]print[backcolor=transparent] [backcolor=transparent]'%s\t%s'[backcolor=transparent] [backcolor=transparent]%[backcolor=transparent] [backcolor=transparent]([backcolor=transparent]word[backcolor=transparent],[backcolor=transparent] [backcolor=transparent]1[backcolor=transparent])

1. [backcolor=transparent]#!/usr/bin/env python
2. [backcolor=transparent]from[backcolor=transparent] [backcolor=transparent]operator[backcolor=transparent] [backcolor=transparent]import[backcolor=transparent] itemgetter
3. [backcolor=transparent]import[backcolor=transparent] sys
5. [backcolor=transparent]current_word [backcolor=transparent]=[backcolor=transparent] [backcolor=transparent]None
6. [backcolor=transparent]current_count [backcolor=transparent]=[backcolor=transparent] [backcolor=transparent]0
7. [backcolor=transparent]word [backcolor=transparent]=[backcolor=transparent] [backcolor=transparent]None
9. [backcolor=transparent]for[backcolor=transparent] line [backcolor=transparent]in[backcolor=transparent] sys[backcolor=transparent].[backcolor=transparent]stdin[backcolor=transparent]:
10. [backcolor=transparent]    line [backcolor=transparent]=[backcolor=transparent] line[backcolor=transparent].[backcolor=transparent]strip[backcolor=transparent]()
12. [backcolor=transparent]    word[backcolor=transparent],[backcolor=transparent] count [backcolor=transparent]=[backcolor=transparent] line[backcolor=transparent].[backcolor=transparent]split[backcolor=transparent]([backcolor=transparent]'\t'[backcolor=transparent],[backcolor=transparent] [backcolor=transparent]1[backcolor=transparent])
14. [backcolor=transparent]    [backcolor=transparent]try[backcolor=transparent]:
15. [backcolor=transparent]        count [backcolor=transparent]=[backcolor=transparent] [backcolor=transparent]int[backcolor=transparent]([backcolor=transparent]count[backcolor=transparent])
16. [backcolor=transparent]    [backcolor=transparent]except[backcolor=transparent] [backcolor=transparent]ValueError[backcolor=transparent]:
17. [backcolor=transparent]        [backcolor=transparent]continue
19. [backcolor=transparent]    [backcolor=transparent]if[backcolor=transparent] current_word [backcolor=transparent]==[backcolor=transparent] word[backcolor=transparent]:
20. [backcolor=transparent]        current_count [backcolor=transparent]+=[backcolor=transparent] count
21. [backcolor=transparent]    [backcolor=transparent]else[backcolor=transparent]:
22. [backcolor=transparent]        [backcolor=transparent]if[backcolor=transparent] current_word[backcolor=transparent]:
23. [backcolor=transparent]            [backcolor=transparent]print[backcolor=transparent] [backcolor=transparent]'%s\t%s'[backcolor=transparent] [backcolor=transparent]%[backcolor=transparent] [backcolor=transparent]([backcolor=transparent]current_word[backcolor=transparent],[backcolor=transparent] current_count[backcolor=transparent])
24. [backcolor=transparent]        current_count [backcolor=transparent]=[backcolor=transparent] count
25. [backcolor=transparent]        current_word [backcolor=transparent]=[backcolor=transparent] word
27. [backcolor=transparent]if[backcolor=transparent] current_word [backcolor=transparent]==[backcolor=transparent] word[backcolor=transparent]:
28. [backcolor=transparent]    [backcolor=transparent]print[backcolor=transparent] [backcolor=transparent]'%s\t%s'[backcolor=transparent] [backcolor=transparent]%[backcolor=transparent] [backcolor=transparent]([backcolor=transparent]current_word[backcolor=transparent],[backcolor=transparent] current_count[backcolor=transparent])