Hadoop【问题记录 03】【ipc.Client: Retrying connect to server:xxx/:8032+InvalidResourceRequestException】解决

简介: 【4月更文挑战第2天】Hadoop【问题记录 03】【ipc.Client: Retrying connect to server:xxx/:8032+InvalidResourceRequestException】解决

起因

我使用的是 hadoop-3.1.3 版本,在进行基准测试时,执行以下代码;

[root@tcloud mapreduce]# hadoop jar ./hadoop-mapreduce-client-jobclient-3.1.3-tests.jar \
 TestDFSIO \
 -write \
 -nrFiles 10 \
 -size 10kb \
 -resFile home/hadoop/tmp/TestDFSIO.log

问题一【Retrying connect to server】

无法连接 ResourceManager:

2021-08-16 14:25:30,253 INFO client.RMProxy:
 Connecting to ResourceManager at tcloud/xxx.xx.x.x:8032

2021-08-16 14:25:31,339 INFO ipc.Client: Retrying connect to server: 
tcloud/xxx.xx.x.x:8032. Already tried 0 time(s); 
retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, 
sleepTime=1000 MILLISECONDS)

解决:

原因是未启动yarn,启动即可:

[root@tcloud mapreduce]# start-yarn.sh

问题二【InvalidResourceRequestException】

无效的资源请求,请求的资源 type=[memory-mb] < 0或大于允许的最大分配【这是使用基础配置云服务器的尴尬,由于服务器是1核心2G内存,所以配置memory-mb时配的低,造成请求资源大于允许的最大内存】:

2021-08-16 15:22:10,535 INFO mapreduce.JobSubmitter: 
Cleaning up the staging area 
/tmp/hadoop-yarn/staging/root/.staging/job_1629098481992_0001

java.io.IOException: org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException:
Invalid resource request, requested resource type=[memory-mb] < 0 
or greater than maximum allowed allocation. Requested resource=<memory:1536, vCores:1>, 
maximum allowed allocation=<memory:256, vCores:4>, 
please note that maximum allowed allocation is calculated by scheduler based on maximum 
resource of registered NodeManagers, 
which might be less than configured maximum allocation=<memory:256, vCores:4>

解决:

修改 ${HADOOP_HOME}/etc/hadoop/yarn-site.xml 文件:

# 停掉yarn
[root@tcloud hadoop]# stop-yarn.sh
# 修改配置文件
[root@tcloud hadoop]# vim ./yarn-site.xml
# 启动yarn
[root@tcloud hadoop]# start-yarn.sh
<!-- 调整内存大小即可 -->
 <property>
        <name>yarn.scheduler.minimum-allocation-mb</name>
        <value>256</value>
  </property>
  <property>
        <name>yarn.scheduler.maximum-allocation-mb</name>
        <value>1540</value>
  </property>
 <property>
        <name>yarn.nodemanager.resource.memory-mb</name>
        <value>1540</value>
  </property>

未解决的问题 但不影响执行

2021-08-16 15:10:55,499 INFO conf.Configuration: resource-types.xml not found
2021-08-16 15:10:55,500 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
目录
相关文章
|
9月前
|
分布式计算 资源调度 Hadoop
Hadoop【问题记录 02】【hadoop-3.1.3 单机版】ResourceManager无法启动NodeManager启动后过自动关闭 javax/activation/DataSource
【4月更文挑战第2天】Hadoop【问题记录 02】【hadoop-3.1.3 单机版】ResourceManager无法启动NodeManager启动后过自动关闭 javax/activation/DataSource
284 2
|
9月前
|
分布式计算 安全 Hadoop
HBase Shell-org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet 已解决
在HBase Shell遇到错误时,检查Hadoop非安全模式:`hdfs dfsadmin -safemode get`。问题解决在于`hbase-site.xml`中添加配置:Zookeeper客户端端口设为2181和预写日志提供者设为filesystem。
234 6
|
9月前
|
分布式计算 Hadoop
Hadoop【问题记录 01】【hadoop-3.1.3 单机版】【Attempting to operate on hdfs namenode as root】
【4月更文挑战第2天】Hadoop【问题记录 01】【hadoop-3.1.3 单机版】【Attempting to operate on hdfs namenode as root】
120 0
|
人工智能 安全 大数据
HBase启动报错:ERROR: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet
欢迎关注大数据和人工智能技术文章发布的微信公众号:清研学堂,在这里你可以学到夜白(作者笔名)精心整理的笔记,让我们每天进步一点点,让优秀成为一种习惯! 今天进入hbase shell中输入命令报错:ERROR: org.
6194 0
|
分布式计算 Hadoop 程序员
Apache Hadoop 启动报错:masternode:ssh: connect to host master port 22: Connection timed out 总结
Apache Hadoop 启动报错:masternode:ssh: connect to host master port 22: Connection timed out 总结
466 0
Apache Hadoop 启动报错:masternode:ssh: connect to host master port 22: Connection timed out 总结
|
分布式计算 Hadoop Java
|
存储 分布式计算 数据库
|
分布式计算 Hadoop
org.apache.hadoop.ipc.Client: Retrying connect to server异常的解决
<p><span style="color:rgb(51,51,51); line-height:22px; text-indent:26px; background-color:rgb(255,255,255)"><span style="font-family:SimSun; font-size:14px"><strong>检查发现是DataNode一直连接不到NameNode。</s
4327 0
|
4月前
|
分布式计算 Kubernetes Hadoop
大数据-82 Spark 集群模式启动、集群架构、集群管理器 Spark的HelloWorld + Hadoop + HDFS
大数据-82 Spark 集群模式启动、集群架构、集群管理器 Spark的HelloWorld + Hadoop + HDFS
241 6

相关实验场景

更多