E-MapReduce软件配置是什么?

软件配置的作用


Hadoop、Hive、Pig 等软件含有大量的配置，当需要对其软件配置进行修改时，就可以使用软件配置功能来实现。例如，HDFS 服务器的服务线程数目 dfs.namenode.handler.count 默认是 10，假设要加大到 50；HDFS 的文件块的大小 dfs.blocksize 默认是 128MB，假设系统都是小文件，想要改小到 64MB。
目前这个操作只能在集群启动的时候执行一次。

如何使用

1. 
   登录阿里云 E-MapReduce 控制台集群列表。
2. 
   在上方选择所在的地域（Region），所创建集群将会在对应的Region内。
3. 
   单击[backcolor=transparent]创建集群，即会进入创建集群的操作界面。
4. 
   在创建集群的软件配置这一步中可以看到所有包含的软件以及对应的版本。若想修改集群的配置，可以通过软件配置（可选）框选择相应的 json 格式配置文件，对集群的默认参数进行覆盖或添加。json 文件的样例内容如下[backcolor=transparent]{
5. [backcolor=transparent] [backcolor=transparent]"configurations"[backcolor=transparent]:[backcolor=transparent] [backcolor=transparent][
6. [backcolor=transparent]     [backcolor=transparent]{
7. [backcolor=transparent]         [backcolor=transparent]"classification"[backcolor=transparent]:[backcolor=transparent] [backcolor=transparent]"core-site"[backcolor=transparent],
8. [backcolor=transparent]         [backcolor=transparent]"properties"[backcolor=transparent]:[backcolor=transparent] [backcolor=transparent]{
9. [backcolor=transparent]             [backcolor=transparent]"fs.trash.interval"[backcolor=transparent]:[backcolor=transparent] [backcolor=transparent]"61"
10. [backcolor=transparent]         [backcolor=transparent]}
11. [backcolor=transparent]     [backcolor=transparent]},
12. [backcolor=transparent]     [backcolor=transparent]{
13. [backcolor=transparent]         [backcolor=transparent]"classification"[backcolor=transparent]:[backcolor=transparent] [backcolor=transparent]"hadoop-log4j"[backcolor=transparent],
14. [backcolor=transparent]         [backcolor=transparent]"properties"[backcolor=transparent]:[backcolor=transparent] [backcolor=transparent]{
15. [backcolor=transparent]             [backcolor=transparent]"hadoop.log.file"[backcolor=transparent]:[backcolor=transparent] [backcolor=transparent]"hadoop1.log"[backcolor=transparent],
16. [backcolor=transparent]             [backcolor=transparent]"hadoop.root.logger"[backcolor=transparent]:[backcolor=transparent] [backcolor=transparent]"INFO"[backcolor=transparent],
17. [backcolor=transparent]             [backcolor=transparent]"a.b.c"[backcolor=transparent]:[backcolor=transparent] [backcolor=transparent]"ABC"
18. [backcolor=transparent]         [backcolor=transparent]}
19. [backcolor=transparent]     [backcolor=transparent]},
20. [backcolor=transparent]     [backcolor=transparent]{
21. [backcolor=transparent]         [backcolor=transparent]"classification"[backcolor=transparent]:[backcolor=transparent] [backcolor=transparent]"hdfs-site"[backcolor=transparent],
22. [backcolor=transparent]         [backcolor=transparent]"properties"[backcolor=transparent]:[backcolor=transparent] [backcolor=transparent]{
23. [backcolor=transparent]             [backcolor=transparent]"dfs.namenode.handler.count"[backcolor=transparent]:[backcolor=transparent] [backcolor=transparent]"12"
24. [backcolor=transparent]         [backcolor=transparent]}
25. [backcolor=transparent]     [backcolor=transparent]},
26. [backcolor=transparent]     [backcolor=transparent]{
27. [backcolor=transparent]         [backcolor=transparent]"classification"[backcolor=transparent]:[backcolor=transparent] [backcolor=transparent]"mapred-site"[backcolor=transparent],
28. [backcolor=transparent]         [backcolor=transparent]"properties"[backcolor=transparent]:[backcolor=transparent] [backcolor=transparent]{
29. [backcolor=transparent]             [backcolor=transparent]"mapreduce.task.io.sort.mb"[backcolor=transparent]:[backcolor=transparent] [backcolor=transparent]"201"
30. [backcolor=transparent]         [backcolor=transparent]}
31. [backcolor=transparent]     [backcolor=transparent]},
32. [backcolor=transparent]     [backcolor=transparent]{
33. [backcolor=transparent]         [backcolor=transparent]"classification"[backcolor=transparent]:[backcolor=transparent] [backcolor=transparent]"yarn-site"[backcolor=transparent],
34. [backcolor=transparent]         [backcolor=transparent]"properties"[backcolor=transparent]:[backcolor=transparent] [backcolor=transparent]{
35. [backcolor=transparent]             [backcolor=transparent]"hadoop.security.groups.cache.secs"[backcolor=transparent]:[backcolor=transparent] [backcolor=transparent]"251"[backcolor=transparent],
36. [backcolor=transparent]             [backcolor=transparent]"yarn.nodemanager.remote-app-log-dir"[backcolor=transparent]:[backcolor=transparent] [backcolor=transparent]"/tmp/logs1"
37. [backcolor=transparent]         [backcolor=transparent]}
38. [backcolor=transparent]     [backcolor=transparent]},
39. [backcolor=transparent]     [backcolor=transparent]{
40. [backcolor=transparent]         [backcolor=transparent]"classification"[backcolor=transparent]:[backcolor=transparent] [backcolor=transparent]"httpsfs-site"[backcolor=transparent],
41. [backcolor=transparent]         [backcolor=transparent]"properties"[backcolor=transparent]:[backcolor=transparent] [backcolor=transparent]{
42. [backcolor=transparent]             [backcolor=transparent]"a.b.c.d"[backcolor=transparent]:[backcolor=transparent] [backcolor=transparent]"200"
43. [backcolor=transparent]         [backcolor=transparent]}
44. [backcolor=transparent]     [backcolor=transparent]},
45. [backcolor=transparent]     [backcolor=transparent]{
46. [backcolor=transparent]         [backcolor=transparent]"classification"[backcolor=transparent]:[backcolor=transparent] [backcolor=transparent]"capacity-scheduler"[backcolor=transparent],
47. [backcolor=transparent]         [backcolor=transparent]"properties"[backcolor=transparent]:[backcolor=transparent] [backcolor=transparent]{
48. [backcolor=transparent]             [backcolor=transparent]"yarn.scheduler.capacity.maximum-am-resource-percent"[backcolor=transparent]:[backcolor=transparent] [backcolor=transparent]"0.2"
49. [backcolor=transparent]         [backcolor=transparent]}
50. [backcolor=transparent]     [backcolor=transparent]},
51. [backcolor=transparent]     [backcolor=transparent]{
52. [backcolor=transparent]         [backcolor=transparent]"classification"[backcolor=transparent]:[backcolor=transparent] [backcolor=transparent]"hadoop-env"[backcolor=transparent],
53. [backcolor=transparent]         [backcolor=transparent]"properties"[backcolor=transparent]:[backcolor=transparent] [backcolor=transparent]{
54. [backcolor=transparent]             [backcolor=transparent]"BC"[backcolor=transparent]:[backcolor=transparent]"CD"
55. [backcolor=transparent]         [backcolor=transparent]},
56. [backcolor=transparent]         [backcolor=transparent]"configurations"[backcolor=transparent]:[
57. [backcolor=transparent]             [backcolor=transparent]{
58. [backcolor=transparent]                 [backcolor=transparent]"classification"[backcolor=transparent]:[backcolor=transparent]"export"[backcolor=transparent],
59. [backcolor=transparent]                 [backcolor=transparent]"properties"[backcolor=transparent]:[backcolor=transparent] [backcolor=transparent]{
60. [backcolor=transparent]                     [backcolor=transparent]"AB"[backcolor=transparent]:[backcolor=transparent]"${BC}"[backcolor=transparent],
61. [backcolor=transparent]                     [backcolor=transparent]"HADOOP_CLIENT_OPTS"[backcolor=transparent]:[backcolor=transparent]"\"-Xmx512m -Xms512m $HADOOP_CLIENT_OPTS\""
62. [backcolor=transparent]                 [backcolor=transparent]}
63. [backcolor=transparent]             [backcolor=transparent]}
64. [backcolor=transparent]         [backcolor=transparent]]
65. [backcolor=transparent]     [backcolor=transparent]},
66. [backcolor=transparent]     [backcolor=transparent]{
67. [backcolor=transparent]         [backcolor=transparent]"classification"[backcolor=transparent]:[backcolor=transparent] [backcolor=transparent]"httpfs-env"[backcolor=transparent],
68. [backcolor=transparent]         [backcolor=transparent]"properties"[backcolor=transparent]:[backcolor=transparent] [backcolor=transparent]{
69. [backcolor=transparent]         [backcolor=transparent]},
70. [backcolor=transparent]         [backcolor=transparent]"configurations"[backcolor=transparent]:[
71. [backcolor=transparent]             [backcolor=transparent]{
72. [backcolor=transparent]                 [backcolor=transparent]"classification"[backcolor=transparent]:[backcolor=transparent]"export"[backcolor=transparent],
73. [backcolor=transparent]                 [backcolor=transparent]"properties"[backcolor=transparent]:[backcolor=transparent] [backcolor=transparent]{
74. [backcolor=transparent]                     [backcolor=transparent]"HTTPFS_SSL_KEYSTORE_PASS"[backcolor=transparent]:[backcolor=transparent]"passwd"
75. [backcolor=transparent]                 [backcolor=transparent]}
76. [backcolor=transparent]             [backcolor=transparent]}
77. [backcolor=transparent]         [backcolor=transparent]]
78. [backcolor=transparent]     [backcolor=transparent]},
79. [backcolor=transparent]     [backcolor=transparent]{
80. [backcolor=transparent]         [backcolor=transparent]"classification"[backcolor=transparent]:[backcolor=transparent] [backcolor=transparent]"mapred-env"[backcolor=transparent],
81. [backcolor=transparent]         [backcolor=transparent]"properties"[backcolor=transparent]:[backcolor=transparent] [backcolor=transparent]{
82. [backcolor=transparent]         [backcolor=transparent]},
83. [backcolor=transparent]         [backcolor=transparent]"configurations"[backcolor=transparent]:[
84. [backcolor=transparent]             [backcolor=transparent]{
85. [backcolor=transparent]                 [backcolor=transparent]"classification"[backcolor=transparent]:[backcolor=transparent]"export"[backcolor=transparent],
86. [backcolor=transparent]                 [backcolor=transparent]"properties"[backcolor=transparent]:[backcolor=transparent] [backcolor=transparent]{
87. [backcolor=transparent]                     [backcolor=transparent]"HADOOP_JOB_HISTORYSERVER_HEAPSIZE"[backcolor=transparent]:[backcolor=transparent]"1001"
88. [backcolor=transparent]                 [backcolor=transparent]}
89. [backcolor=transparent]             [backcolor=transparent]}
90. [backcolor=transparent]         [backcolor=transparent]]
91. [backcolor=transparent]     [backcolor=transparent]},
92. [backcolor=transparent]     [backcolor=transparent]{
93. [backcolor=transparent]         [backcolor=transparent]"classification"[backcolor=transparent]:[backcolor=transparent] [backcolor=transparent]"yarn-env"[backcolor=transparent],
94. [backcolor=transparent]         [backcolor=transparent]"properties"[backcolor=transparent]:[backcolor=transparent] [backcolor=transparent]{
95. [backcolor=transparent]         [backcolor=transparent]},
96. [backcolor=transparent]         [backcolor=transparent]"configurations"[backcolor=transparent]:[
97. [backcolor=transparent]             [backcolor=transparent]{
98. [backcolor=transparent]                 [backcolor=transparent]"classification"[backcolor=transparent]:[backcolor=transparent]"export"[backcolor=transparent],
99. [backcolor=transparent]                 [backcolor=transparent]"properties"[backcolor=transparent]:[backcolor=transparent] [backcolor=transparent]{
100. [backcolor=transparent]                     [backcolor=transparent]"HADOOP_YARN_USER"[backcolor=transparent]:[backcolor=transparent]"${HADOOP_YARN_USER:-yarn1}"
101. [backcolor=transparent]                 [backcolor=transparent]}
102. [backcolor=transparent]             [backcolor=transparent]}
103. [backcolor=transparent]         [backcolor=transparent]]
104. [backcolor=transparent]     [backcolor=transparent]},
105. [backcolor=transparent]     [backcolor=transparent]{
106. [backcolor=transparent]         [backcolor=transparent]"classification"[backcolor=transparent]:[backcolor=transparent] [backcolor=transparent]"pig"[backcolor=transparent],
107. [backcolor=transparent]         [backcolor=transparent]"properties"[backcolor=transparent]:[backcolor=transparent] [backcolor=transparent]{
108. [backcolor=transparent]             [backcolor=transparent]"pig.tez.auto.parallelism"[backcolor=transparent]:[backcolor=transparent] [backcolor=transparent]"false"
109. [backcolor=transparent]         [backcolor=transparent]}
110. [backcolor=transparent]     [backcolor=transparent]},
111. [backcolor=transparent]     [backcolor=transparent]{
112. [backcolor=transparent]         [backcolor=transparent]"classification"[backcolor=transparent]:[backcolor=transparent] [backcolor=transparent]"pig-log4j"[backcolor=transparent],
113. [backcolor=transparent]         [backcolor=transparent]"properties"[backcolor=transparent]:[backcolor=transparent] [backcolor=transparent]{
114. [backcolor=transparent]             [backcolor=transparent]"log4j.logger.org.apache.pig"[backcolor=transparent]:[backcolor=transparent] [backcolor=transparent]"error, A"
115. [backcolor=transparent]         [backcolor=transparent]}
116. [backcolor=transparent]     [backcolor=transparent]},
117. [backcolor=transparent]     [backcolor=transparent]{
118. [backcolor=transparent]         [backcolor=transparent]"classification"[backcolor=transparent]:[backcolor=transparent] [backcolor=transparent]"hive-env"[backcolor=transparent],
119. [backcolor=transparent]         [backcolor=transparent]"properties"[backcolor=transparent]:[backcolor=transparent] [backcolor=transparent]{
120. [backcolor=transparent]             [backcolor=transparent]"BC"[backcolor=transparent]:[backcolor=transparent]"CD"
121. [backcolor=transparent]         [backcolor=transparent]},
122. [backcolor=transparent]         [backcolor=transparent]"configurations"[backcolor=transparent]:[
123. [backcolor=transparent]             [backcolor=transparent]{
124. [backcolor=transparent]                 [backcolor=transparent]"classification"[backcolor=transparent]:[backcolor=transparent]"export"[backcolor=transparent],
125. [backcolor=transparent]                 [backcolor=transparent]"properties"[backcolor=transparent]:[backcolor=transparent] [backcolor=transparent]{
126. [backcolor=transparent]                     [backcolor=transparent]"AB"[backcolor=transparent]:[backcolor=transparent]"${BC}"[backcolor=transparent],
127. [backcolor=transparent]                     [backcolor=transparent]"HADOOP_CLIENT_OPTS1"[backcolor=transparent]:[backcolor=transparent]"\"-Xmx512m -Xms512m $HADOOP_CLIENT_OPTS1\""
128. [backcolor=transparent]                 [backcolor=transparent]}
129. [backcolor=transparent]             [backcolor=transparent]}
130. [backcolor=transparent]         [backcolor=transparent]]
131. [backcolor=transparent]     [backcolor=transparent]},
132. [backcolor=transparent]     [backcolor=transparent]{
133. [backcolor=transparent]         [backcolor=transparent]"classification"[backcolor=transparent]:[backcolor=transparent] [backcolor=transparent]"hive-site"[backcolor=transparent],
134. [backcolor=transparent]         [backcolor=transparent]"properties"[backcolor=transparent]:[backcolor=transparent] [backcolor=transparent]{
135. [backcolor=transparent]             [backcolor=transparent]"hive.tez.java.opts"[backcolor=transparent]:[backcolor=transparent] [backcolor=transparent]"-Xmx3900m"
136. [backcolor=transparent]         [backcolor=transparent]}
137. [backcolor=transparent]     [backcolor=transparent]},
138. [backcolor=transparent]     [backcolor=transparent]{
139. [backcolor=transparent]         [backcolor=transparent]"classification"[backcolor=transparent]:[backcolor=transparent] [backcolor=transparent]"hive-exec-log4j"[backcolor=transparent],
140. [backcolor=transparent]         [backcolor=transparent]"properties"[backcolor=transparent]:[backcolor=transparent] [backcolor=transparent]{
141. [backcolor=transparent]             [backcolor=transparent]"log4j.logger.org.apache.zookeeper.ClientCnxnSocketNIO"[backcolor=transparent]:[backcolor=transparent] [backcolor=transparent]"INFO,FA"
142. [backcolor=transparent]         [backcolor=transparent]}
143. [backcolor=transparent]     [backcolor=transparent]},
144. [backcolor=transparent]     [backcolor=transparent]{
145. [backcolor=transparent]         [backcolor=transparent]"classification"[backcolor=transparent]:[backcolor=transparent] [backcolor=transparent]"hive-log4j"[backcolor=transparent],
146. [backcolor=transparent]         [backcolor=transparent]"properties"[backcolor=transparent]:[backcolor=transparent] [backcolor=transparent]{
147. [backcolor=transparent]             [backcolor=transparent]"log4j.logger.org.apache.zookeeper.server.NIOServerCnxn"[backcolor=transparent]:[backcolor=transparent] [backcolor=transparent]"INFO,DRFA"
148. [backcolor=transparent]         [backcolor=transparent]}
149. [backcolor=transparent]     [backcolor=transparent]}
150. [backcolor=transparent] [backcolor=transparent]]
151. [backcolor=transparent]}

classification 参数指定要修改的配置文件，properties 参数放置要修改的 key value 键值对，默认配置文件有对应的 key 有则只覆盖 value，没有则添加对应的 key value 键值对。
配置文件与 classification 的对应关系如下列表格所示：
[backcolor=transparent]Hadoop

Filename	classification
core-site.xml	core-site
log4j.properties	hadoop-log4j
hdfs-site.xml	hdfs-site
mapred-site.xml	mapred-site
yarn-site.xml	yarn-site
httpsfs-site.xml	httpsfs-site
capacity-scheduler.xml	capacity-scheduler
hadoop-env.sh	hadoop-env
httpfs-env.sh	httpfs-env
mapred-env.sh	mapred-env
yarn-env.sh	yarn-env

[backcolor=transparent]Pig

Filename	classification
pig.properties	pig
log4j.properties	pig-log4j

[backcolor=transparent]Hive

Filename	classification
hive-env.sh	hive-env
hive-site.xml	hive-site
hive-exec-log4j.properties	hive-exec-log4j
hive-log4j.properties	hive-log4j

core-site 这类扁平的 xml 文件只有一层，配置都放在 properties 里。而 hadoop-en v这类 sh 文件可能有两层结构，可以通过嵌套 configurations 的方式来设置，请参见示例里 hadoop-env 的部分，为 export 的 HADOOP_CLIENT_OPTS 属性添加了 -Xmx512m -Xms512m 的设置。
设置好后，确认后单击[backcolor=transparent]下一步。