起因
我使用的是 hadoop-3.1.3 版本,在进行基准测试时,执行以下代码;
[root@tcloud mapreduce]# hadoop jar ./hadoop-mapreduce-client-jobclient-3.1.3-tests.jar \
TestDFSIO \
-write \
-nrFiles 10 \
-size 10kb \
-resFile home/hadoop/tmp/TestDFSIO.log
问题一【Retrying connect to server】
无法连接 ResourceManager:
2021-08-16 14:25:30,253 INFO client.RMProxy:
Connecting to ResourceManager at tcloud/xxx.xx.x.x:8032
2021-08-16 14:25:31,339 INFO ipc.Client: Retrying connect to server:
tcloud/xxx.xx.x.x:8032. Already tried 0 time(s);
retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
解决:
原因是未启动yarn,启动即可:
[root@tcloud mapreduce]# start-yarn.sh
问题二【InvalidResourceRequestException】
无效的资源请求,请求的资源 type=[memory-mb] < 0或大于允许的最大分配【这是使用基础配置云服务器的尴尬,由于服务器是1核心2G内存,所以配置memory-mb时配的低,造成请求资源大于允许的最大内存】:
2021-08-16 15:22:10,535 INFO mapreduce.JobSubmitter:
Cleaning up the staging area
/tmp/hadoop-yarn/staging/root/.staging/job_1629098481992_0001
java.io.IOException: org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException:
Invalid resource request, requested resource type=[memory-mb] < 0
or greater than maximum allowed allocation. Requested resource=<memory:1536, vCores:1>,
maximum allowed allocation=<memory:256, vCores:4>,
please note that maximum allowed allocation is calculated by scheduler based on maximum
resource of registered NodeManagers,
which might be less than configured maximum allocation=<memory:256, vCores:4>
解决:
修改 ${HADOOP_HOME}/etc/hadoop/yarn-site.xml 文件:
# 停掉yarn
[root@tcloud hadoop]# stop-yarn.sh
# 修改配置文件
[root@tcloud hadoop]# vim ./yarn-site.xml
# 启动yarn
[root@tcloud hadoop]# start-yarn.sh
<!-- 调整内存大小即可 -->
<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>256</value>
</property>
<property>
<name>yarn.scheduler.maximum-allocation-mb</name>
<value>1540</value>
</property>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>1540</value>
</property>
未解决的问题 但不影响执行
2021-08-16 15:10:55,499 INFO conf.Configuration: resource-types.xml not found
2021-08-16 15:10:55,500 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.