使用YARN命令管理Hadoop作业

简介: 本文介绍了如何使用YARN命令来管理Hadoop作业,包括查看作业列表、检查作业状态、杀死作业、获取作业日志以及检查节点和队列状态等操作。

作者:尹正杰
版权声明:原创作品,谢绝转载!否则将追究法律责任。

一.YARN命令概述

  如下图所示,Hadoop的yarn命令可以管理YARN的各个方面。虽然ResourceManager Web UI非常有用,但当不能使用Web UI时,就需要使用命令来处理问题。

  Hadoop的yarn命令具有广泛的使用范围,从某种意义上来说,它可以帮助我们管理大量的Hadoop任务,例如获取和杀死正在运行的应用程序,获取作业和守护程序日志,甚至管理ResourceManager的上下线。

  可以使用yarn命令从命令行监视和管理应用程序。例如,可以使用"yarn application -list running"命令查看正在运行的作业的状态。

  同样,可以使用"yarn application -kill <Application ID>"命令从命令行正常删除正在运行的应用程序,而不是使用Linux kill命令杀死进程。

  最好用的yarn命令是application子命令,它允许快速查看某个时间点的集群时间点的使用情况。

  如下图所示,yarn top命令提供了有关"正在运行(running)","已提交(submitted)","待定(pending)","已完成(completed)","已终止(killed)","失败(failed)"的摘要信息,还显示为集群的足有作业分配的内存量和CPU内核量。

  本篇博客的重点是监控和管理集群,我们需要了解如何使用yarn命令监控YARN集群。

  通过该命令可以查看基本作业信息并执行其他任务,例如"yarn application -kill application_1604218163813_0003"来杀死Application ID为"application_1604218163813_0003"正在运行的Job。

  使用"yarn application"命令可以执行以下管理任务:
    (1)列出集群中运行的应用程序;
    (2)杀死正在运行的应用程序;
    (3)获取正在运行的应用程序的状态;

二.查看YARN应用程序

1>."yarn application -list"命令

  如下图所示,可以使用"yarn application -list"命令,立即查看集群中的所有应用程序(我们称为作业,YARN称为应用)。

[root@hadoop101.yinzhengjie.com ~]# yarn application -list 
20/11/01 17:06:30 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20/11/01 17:06:30 INFO client.RMProxy: Connecting to ResourceManager at hadoop101.yinzhengjie.com/172.200.6.101:8032
Total number of applications (application-types: [], states: [SUBMITTED, ACCEPTED, RUNNING] and tags: []):2
                Application-Id        Application-Name        Application-Type          User         Queue                 State           Final-State           Progress                 
          Tracking-URLapplication_1604218163813_0007              word count               MAPREDUCE          root    root.yinzhengjie.operation.op_queue01               RUNNING             UNDEFINED         
       50%    http://hadoop104.yinzhengjie.com:8705application_1604218163813_0008              word count               MAPREDUCE          root    root.yinzhengjie.development              ACCEPTED             UNDEFINED                 
0%                                    N/A[root@hadoop101.yinzhengjie.com ~]# 
[root@hadoop101.yinzhengjie.com ~]#

[root@hadoop101.yinzhengjie.com ~]# yarn application -list

2>."yarn application -list -appStates "命令

  使用"yarn application -list"命令可以检索所有作业的列表,而不考虑它们的状态。

  作业可以具有以下状态:
    "所有(ALL)","新的(NEW)","新的保存(NEW_SAVING)","提交(SUBMITTED)","接受(ACCEPTED)","正在运行(RUNNING)","已完成(FINISHED)","FAILED(失败)"和"被杀死的(KILLED)"。

[root@hadoop101.yinzhengjie.com ~]# yarn application -list -appStates all
20/11/01 17:17:23 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20/11/01 17:17:24 INFO client.RMProxy: Connecting to ResourceManager at hadoop101.yinzhengjie.com/172.200.6.101:8032
Total number of applications (application-types: [], states: [NEW, NEW_SAVING, SUBMITTED, ACCEPTED, RUNNING, FINISHED, FAILED, KILLED] and tags: []):8
                Application-Id        Application-Name        Application-Type          User         Queue                 State           Final-State           Progress                 
          Tracking-URL
application_1604218163813_0007              word count               MAPREDUCE          root    root.yinzhengjie.operation.op_queue01              FINISHED             SUCCEEDED         
      100%    http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0007
application_1604218163813_0008              word count               MAPREDUCE          root    root.yinzhengjie.development              FINISHED             SUCCEEDED               
    100%    http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0008
application_1604218163813_0003              word count               MAPREDUCE          root    root.yinzhengjie.operation.op_queue01              KILLED                KILLED         
    100%    http://hadoop101.yinzhengjie.com:8088/cluster/app/application_1604218163813_0003
application_1604218163813_0004              word count               MAPREDUCE          root    root.yinzhengjie.operation.op_queue01              FINISHED             SUCCEEDED         
    100%    http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0004
application_1604218163813_0005              word count               MAPREDUCE          root    root.yinzhengjie.operation.op_queue01              FINISHED             SUCCEEDED         
    100%    http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0005
application_1604218163813_0006              word count               MAPREDUCE          root    root.yinzhengjie.operation.op_queue01              FINISHED             SUCCEEDED         
    100%    http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0006
application_1604218163813_0001              word count               MAPREDUCE          root    root.yinzhengjie.operation.op_queue01              FINISHED             SUCCEEDED         
    100%    http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0001
application_1604218163813_0002              word count               MAPREDUCE          root    root.yinzhengjie.operation.op_queue01              FINISHED             SUCCEEDED         
    100%    http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0002
[root@hadoop101.yinzhengjie.com ~]#

[root@hadoop101.yinzhengjie.com ~]# yarn application -list -appStates all

[root@hadoop101.yinzhengjie.com ~]# yarn application -list -appStates all
20/11/01 17:17:23 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20/11/01 17:17:24 INFO client.RMProxy: Connecting to ResourceManager at hadoop101.yinzhengjie.com/172.200.6.101:8032
Total number of applications (application-types: [], states: [NEW, NEW_SAVING, SUBMITTED, ACCEPTED, RUNNING, FINISHED, FAILED, KILLED] and tags: []):8
                Application-Id        Application-Name        Application-Type          User         Queue                 State           Final-State           Progress                 
          Tracking-URL
application_1604218163813_0007              word count               MAPREDUCE          root    root.yinzhengjie.operation.op_queue01              FINISHED             SUCCEEDED         
      100%    http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0007
application_1604218163813_0008              word count               MAPREDUCE          root    root.yinzhengjie.development              FINISHED             SUCCEEDED               
    100%    http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0008
application_1604218163813_0003              word count               MAPREDUCE          root    root.yinzhengjie.operation.op_queue01              KILLED                KILLED         
    100%    http://hadoop101.yinzhengjie.com:8088/cluster/app/application_1604218163813_0003
application_1604218163813_0004              word count               MAPREDUCE          root    root.yinzhengjie.operation.op_queue01              FINISHED             SUCCEEDED         
    100%    http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0004
application_1604218163813_0005              word count               MAPREDUCE          root    root.yinzhengjie.operation.op_queue01              FINISHED             SUCCEEDED         
    100%    http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0005
application_1604218163813_0006              word count               MAPREDUCE          root    root.yinzhengjie.operation.op_queue01              FINISHED             SUCCEEDED         
    100%    http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0006
application_1604218163813_0001              word count               MAPREDUCE          root    root.yinzhengjie.operation.op_queue01              FINISHED             SUCCEEDED         
    100%    http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0001
application_1604218163813_0002              word count               MAPREDUCE          root    root.yinzhengjie.operation.op_queue01              FINISHED             SUCCEEDED         
    100%    http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0002
[root@hadoop101.yinzhengjie.com ~]# 
[root@hadoop101.yinzhengjie.com ~]# yarn application -list -appStates killed
20/11/01 17:17:27 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20/11/01 17:17:27 INFO client.RMProxy: Connecting to ResourceManager at hadoop101.yinzhengjie.com/172.200.6.101:8032
Total number of applications (application-types: [], states: [KILLED] and tags: []):1
                Application-Id        Application-Name        Application-Type          User         Queue                 State           Final-State           Progress                 
          Tracking-URL
application_1604218163813_0003              word count               MAPREDUCE          root    root.yinzhengjie.operation.op_queue01                KILLED                KILLED         
    100%    http://hadoop101.yinzhengjie.com:8088/cluster/app/application_1604218163813_0003
[root@hadoop101.yinzhengjie.com ~]#

[root@hadoop101.yinzhengjie.com ~]# yarn application -list -appStates killed

[root@hadoop101.yinzhengjie.com ~]# yarn application -list -appStates all
20/11/01 17:17:23 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20/11/01 17:17:24 INFO client.RMProxy: Connecting to ResourceManager at hadoop101.yinzhengjie.com/172.200.6.101:8032
Total number of applications (application-types: [], states: [NEW, NEW_SAVING, SUBMITTED, ACCEPTED, RUNNING, FINISHED, FAILED, KILLED] and tags: []):8
                Application-Id        Application-Name        Application-Type          User         Queue                 State           Final-State           Progress                 
          Tracking-URL
application_1604218163813_0007              word count               MAPREDUCE          root    root.yinzhengjie.operation.op_queue01              FINISHED             SUCCEEDED         
      100%    http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0007
application_1604218163813_0008              word count               MAPREDUCE          root    root.yinzhengjie.development              FINISHED             SUCCEEDED               
    100%    http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0008
application_1604218163813_0003              word count               MAPREDUCE          root    root.yinzhengjie.operation.op_queue01              KILLED                KILLED         
    100%    http://hadoop101.yinzhengjie.com:8088/cluster/app/application_1604218163813_0003
application_1604218163813_0004              word count               MAPREDUCE          root    root.yinzhengjie.operation.op_queue01              FINISHED             SUCCEEDED         
    100%    http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0004
application_1604218163813_0005              word count               MAPREDUCE          root    root.yinzhengjie.operation.op_queue01              FINISHED             SUCCEEDED         
    100%    http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0005
application_1604218163813_0006              word count               MAPREDUCE          root    root.yinzhengjie.operation.op_queue01              FINISHED             SUCCEEDED         
    100%    http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0006
application_1604218163813_0001              word count               MAPREDUCE          root    root.yinzhengjie.operation.op_queue01              FINISHED             SUCCEEDED         
    100%    http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0001
application_1604218163813_0002              word count               MAPREDUCE          root    root.yinzhengjie.operation.op_queue01              FINISHED             SUCCEEDED         
    100%    http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0002
[root@hadoop101.yinzhengjie.com ~]# 
[root@hadoop101.yinzhengjie.com ~]# yarn application -list -appStates failed
20/11/01 17:17:32 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20/11/01 17:17:32 INFO client.RMProxy: Connecting to ResourceManager at hadoop101.yinzhengjie.com/172.200.6.101:8032
Total number of applications (application-types: [], states: [FAILED] and tags: []):0
                Application-Id        Application-Name        Application-Type          User         Queue                 State           Final-State           Progress                 
          Tracking-URL
[root@hadoop101.yinzhengjie.com ~]#

[root@hadoop101.yinzhengjie.com ~]# yarn application -list -appStates failed

[root@hadoop101.yinzhengjie.com ~]# yarn application -list -appStates all
20/11/01 17:17:23 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20/11/01 17:17:24 INFO client.RMProxy: Connecting to ResourceManager at hadoop101.yinzhengjie.com/172.200.6.101:8032
Total number of applications (application-types: [], states: [NEW, NEW_SAVING, SUBMITTED, ACCEPTED, RUNNING, FINISHED, FAILED, KILLED] and tags: []):8
                Application-Id        Application-Name        Application-Type          User         Queue                 State           Final-State           Progress                 
          Tracking-URL
application_1604218163813_0007              word count               MAPREDUCE          root    root.yinzhengjie.operation.op_queue01              FINISHED             SUCCEEDED         
      100%    http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0007
application_1604218163813_0008              word count               MAPREDUCE          root    root.yinzhengjie.development              FINISHED             SUCCEEDED               
    100%    http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0008
application_1604218163813_0003              word count               MAPREDUCE          root    root.yinzhengjie.operation.op_queue01              KILLED                KILLED         
    100%    http://hadoop101.yinzhengjie.com:8088/cluster/app/application_1604218163813_0003
application_1604218163813_0004              word count               MAPREDUCE          root    root.yinzhengjie.operation.op_queue01              FINISHED             SUCCEEDED         
    100%    http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0004
application_1604218163813_0005              word count               MAPREDUCE          root    root.yinzhengjie.operation.op_queue01              FINISHED             SUCCEEDED         
    100%    http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0005
application_1604218163813_0006              word count               MAPREDUCE          root    root.yinzhengjie.operation.op_queue01              FINISHED             SUCCEEDED         
    100%    http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0006
application_1604218163813_0001              word count               MAPREDUCE          root    root.yinzhengjie.operation.op_queue01              FINISHED             SUCCEEDED         
    100%    http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0001
application_1604218163813_0002              word count               MAPREDUCE          root    root.yinzhengjie.operation.op_queue01              FINISHED             SUCCEEDED         
    100%    http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0002
[root@hadoop101.yinzhengjie.com ~]# 
[root@hadoop101.yinzhengjie.com ~]# yarn application -list -appStates finished
20/11/01 17:25:04 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20/11/01 17:25:04 INFO client.RMProxy: Connecting to ResourceManager at hadoop101.yinzhengjie.com/172.200.6.101:8032
Total number of applications (application-types: [], states: [FINISHED] and tags: []):7
                Application-Id        Application-Name        Application-Type          User         Queue                 State           Final-State           Progress                 
          Tracking-URL
application_1604218163813_0007              word count               MAPREDUCE          root    root.yinzhengjie.operation.op_queue01              FINISHED             SUCCEEDED         
      100%    http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0007
application_1604218163813_0008              word count               MAPREDUCE          root    root.yinzhengjie.development              FINISHED             SUCCEEDED               
      100%    http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0008
application_1604218163813_0004              word count               MAPREDUCE          root    root.yinzhengjie.operation.op_queue01              FINISHED             SUCCEEDED         
      100%    http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0004
application_1604218163813_0005              word count               MAPREDUCE          root    root.yinzhengjie.operation.op_queue01              FINISHED             SUCCEEDED         
      100%    http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0005
application_1604218163813_0006              word count               MAPREDUCE          root    root.yinzhengjie.operation.op_queue01              FINISHED             SUCCEEDED         
      100%    http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0006
application_1604218163813_0001              word count               MAPREDUCE          root    root.yinzhengjie.operation.op_queue01              FINISHED             SUCCEEDED         
      100%    http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0001
application_1604218163813_0002              word count               MAPREDUCE          root    root.yinzhengjie.operation.op_queue01              FINISHED             SUCCEEDED         
      100%    http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0002
[root@hadoop101.yinzhengjie.com ~]#

[root@hadoop101.yinzhengjie.com ~]# yarn application -list -appStates finished

三.检查应用程序的状态

1>.通过"yarn application -status "查看特定作业的状态

[root@hadoop101.yinzhengjie.com ~]# yarn application -list -appStates all
20/11/01 17:17:23 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20/11/01 17:17:24 INFO client.RMProxy: Connecting to ResourceManager at hadoop101.yinzhengjie.com/172.200.6.101:8032
Total number of applications (application-types: [], states: [NEW, NEW_SAVING, SUBMITTED, ACCEPTED, RUNNING, FINISHED, FAILED, KILLED] and tags: []):8
                Application-Id        Application-Name        Application-Type          User         Queue                 State           Final-State           Progress                 
          Tracking-URL
application_1604218163813_0007              word count               MAPREDUCE          root    root.yinzhengjie.operation.op_queue01              FINISHED             SUCCEEDED         
      100%    http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0007
application_1604218163813_0008              word count               MAPREDUCE          root    root.yinzhengjie.development              FINISHED             SUCCEEDED               
    100%    http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0008
application_1604218163813_0003              word count               MAPREDUCE          root    root.yinzhengjie.operation.op_queue01              KILLED                KILLED         
    100%    http://hadoop101.yinzhengjie.com:8088/cluster/app/application_1604218163813_0003
application_1604218163813_0004              word count               MAPREDUCE          root    root.yinzhengjie.operation.op_queue01              FINISHED             SUCCEEDED         
    100%    http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0004
application_1604218163813_0005              word count               MAPREDUCE          root    root.yinzhengjie.operation.op_queue01              FINISHED             SUCCEEDED         
    100%    http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0005
application_1604218163813_0006              word count               MAPREDUCE          root    root.yinzhengjie.operation.op_queue01              FINISHED             SUCCEEDED         
    100%    http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0006
application_1604218163813_0001              word count               MAPREDUCE          root    root.yinzhengjie.operation.op_queue01              FINISHED             SUCCEEDED         
    100%    http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0001
application_1604218163813_0002              word count               MAPREDUCE          root    root.yinzhengjie.operation.op_queue01              FINISHED             SUCCEEDED         
    100%    http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0002
[root@hadoop101.yinzhengjie.com ~]# 
[root@hadoop101.yinzhengjie.com ~]# yarn application -status application_1604218163813_0008
20/11/01 17:33:54 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20/11/01 17:33:54 INFO client.RMProxy: Connecting to ResourceManager at hadoop101.yinzhengjie.com/172.200.6.101:8032
20/11/01 17:33:55 INFO conf.Configuration: resource-types.xml not found
20/11/01 17:33:55 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
20/11/01 17:33:55 INFO resource.ResourceUtils: Adding resource type - name = memory-mb, units = Mi, type = COUNTABLE
20/11/01 17:33:55 INFO resource.ResourceUtils: Adding resource type - name = vcores, units = , type = COUNTABLE
Application Report : 
    Application-Id : application_1604218163813_0008
    Application-Name : word count
    Application-Type : MAPREDUCE
    User : root
    Queue : root.yinzhengjie.development
    Application Priority : 0
    Start-Time : 1604221581251
    Finish-Time : 1604221620578
    Progress : 100%
    State : FINISHED
    Final-State : SUCCEEDED
    Tracking-URL : http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0008
    RPC Port : 2890
    AM Host : hadoop103.yinzhengjie.com
    Aggregate Resource Allocation : 58152 MB-seconds, 27 vcore-seconds
    Aggregate Resource Preempted : 0 MB-seconds, 0 vcore-seconds
    Log Aggregation Status : SUCCEEDED
    Diagnostics : 
    Unmanaged Application : false
    Application Node Label Expression : <Not set>
    AM container Node Label Expression : <DEFAULT_PARTITION>
    TimeoutType : LIFETIME    ExpiryTime : UNLIMITED    RemainingTime : -1seconds

[root@hadoop101.yinzhengjie.com ~]#

[root@hadoop101.yinzhengjie.com ~]# yarn application -status application_1604218163813_0008

2>."yarn applicationattempt -list "命令

[root@hadoop101.yinzhengjie.com ~]# yarn application -list -appStates all
20/11/01 17:17:23 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20/11/01 17:17:24 INFO client.RMProxy: Connecting to ResourceManager at hadoop101.yinzhengjie.com/172.200.6.101:8032
Total number of applications (application-types: [], states: [NEW, NEW_SAVING, SUBMITTED, ACCEPTED, RUNNING, FINISHED, FAILED, KILLED] and tags: []):8
                Application-Id        Application-Name        Application-Type          User         Queue                 State           Final-State           Progress                 
          Tracking-URL
application_1604218163813_0007              word count               MAPREDUCE          root    root.yinzhengjie.operation.op_queue01              FINISHED             SUCCEEDED         
      100%    http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0007
application_1604218163813_0008              word count               MAPREDUCE          root    root.yinzhengjie.development              FINISHED             SUCCEEDED               
    100%    http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0008
application_1604218163813_0003              word count               MAPREDUCE          root    root.yinzhengjie.operation.op_queue01              KILLED                KILLED         
    100%    http://hadoop101.yinzhengjie.com:8088/cluster/app/application_1604218163813_0003
application_1604218163813_0004              word count               MAPREDUCE          root    root.yinzhengjie.operation.op_queue01              FINISHED             SUCCEEDED         
    100%    http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0004
application_1604218163813_0005              word count               MAPREDUCE          root    root.yinzhengjie.operation.op_queue01              FINISHED             SUCCEEDED         
    100%    http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0005
application_1604218163813_0006              word count               MAPREDUCE          root    root.yinzhengjie.operation.op_queue01              FINISHED             SUCCEEDED         
    100%    http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0006
application_1604218163813_0001              word count               MAPREDUCE          root    root.yinzhengjie.operation.op_queue01              FINISHED             SUCCEEDED         
    100%    http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0001
application_1604218163813_0002              word count               MAPREDUCE          root    root.yinzhengjie.operation.op_queue01              FINISHED             SUCCEEDED         
    100%    http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0002
[root@hadoop101.yinzhengjie.com ~]# 
[root@hadoop101.yinzhengjie.com ~]# yarn applicationattempt -list application_1604218163813_0008
20/11/01 17:44:51 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20/11/01 17:44:51 INFO client.RMProxy: Connecting to ResourceManager at hadoop101.yinzhengjie.com/172.200.6.101:8032
Total number of application attempts :1
         ApplicationAttempt-Id                   State                        AM-Container-Id                           Tracking-URL
appattempt_1604218163813_0008_000001                FINISHED    container_1604218163813_0008_01_000001    http://hadoop101.yinzhengjie.com:8088/proxy/application_1604218163813_0008/
[root@hadoop101.yinzhengjie.com ~]#

[root@hadoop101.yinzhengjie.com ~]# yarn applicationattempt -list application_1604218163813_0008

3>."yarn applicationattempt -status "命令

[root@hadoop101.yinzhengjie.com ~]# yarn applicationattempt -list application_1604218163813_0008 
20/11/01 17:51:51 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20/11/01 17:51:51 INFO client.RMProxy: Connecting to ResourceManager at hadoop101.yinzhengjie.com/172.200.6.101:8032
Total number of application attempts :1
         ApplicationAttempt-Id                   State                        AM-Container-Id                           Tracking-URL
appattempt_1604218163813_0008_000001                FINISHED    container_1604218163813_0008_01_000001    http://hadoop101.yinzhengjie.com:8088/proxy/application_1604218163813_0008/
[root@hadoop101.yinzhengjie.com ~]# 
[root@hadoop101.yinzhengjie.com ~]# 
[root@hadoop101.yinzhengjie.com ~]# yarn applicationattempt  -status appattempt_1604218163813_0008_000001
20/11/01 17:52:02 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20/11/01 17:52:02 INFO client.RMProxy: Connecting to ResourceManager at hadoop101.yinzhengjie.com/172.200.6.101:8032
Application Attempt Report : 
    ApplicationAttempt-Id : appattempt_1604218163813_0008_000001
    State : FINISHED
    AMContainer : container_1604218163813_0008_01_000001
    Tracking-URL : http://hadoop101.yinzhengjie.com:8088/proxy/application_1604218163813_0008/
    RPC Port : 2890
    AM Host : hadoop103.yinzhengjie.com
    Diagnostics : 
[root@hadoop101.yinzhengjie.com ~]# 
[root@hadoop101.yinzhengjie.com ~]#

[root@hadoop101.yinzhengjie.com ~]# yarn applicationattempt -status appattempt_1604218163813_0008_000001

四.kill正在执行的作业("yarn application -kill ")

[root@hadoop101.yinzhengjie.com ~]# yarn application -list
20/11/01 17:59:08 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20/11/01 17:59:08 INFO client.RMProxy: Connecting to ResourceManager at hadoop101.yinzhengjie.com/172.200.6.101:8032
Total number of applications (application-types: [], states: [SUBMITTED, ACCEPTED, RUNNING] and tags: []):1
                Application-Id        Application-Name        Application-Type          User         Queue                 State           Final-State           Progress                 
          Tracking-URL
application_1604218163813_0013              word count               MAPREDUCE          root    root.yinzhengjie.testing               RUNNING             UNDEFINED                 
5%    http://hadoop104.yinzhengjie.com:2692
[root@hadoop101.yinzhengjie.com ~]# 
[root@hadoop101.yinzhengjie.com ~]# yarn application -kill application_1604218163813_0013
20/11/01 17:59:17 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20/11/01 17:59:18 INFO client.RMProxy: Connecting to ResourceManager at hadoop101.yinzhengjie.com/172.200.6.101:8032
Killing application application_1604218163813_0013
20/11/01 17:59:18 INFO impl.YarnClientImpl: Killed application application_1604218163813_0013
[root@hadoop101.yinzhengjie.com ~]# 
[root@hadoop101.yinzhengjie.com ~]# yarn application -list
20/11/01 17:59:22 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20/11/01 17:59:22 INFO client.RMProxy: Connecting to ResourceManager at hadoop101.yinzhengjie.com/172.200.6.101:8032
Total number of applications (application-types: [], states: [SUBMITTED, ACCEPTED, RUNNING] and tags: []):0
                Application-Id        Application-Name        Application-Type          User         Queue                 State           Final-State           Progress                 
          Tracking-URL
[root@hadoop101.yinzhengjie.com ~]# 
[root@hadoop101.yinzhengjie.com ~]# 
[root@hadoop101.yinzhengjie.com ~]# yarn application -list -appStates killed
20/11/01 17:59:27 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20/11/01 17:59:27 INFO client.RMProxy: Connecting to ResourceManager at hadoop101.yinzhengjie.com/172.200.6.101:8032
Total number of applications (application-types: [], states: [KILLED] and tags: []):2
                Application-Id        Application-Name        Application-Type          User         Queue                 State           Final-State           Progress                 
          Tracking-URL
application_1604218163813_0013              word count               MAPREDUCE          root    root.yinzhengjie.testing                KILLED                KILLED               10
0%    http://hadoop101.yinzhengjie.com:8088/cluster/app/application_1604218163813_0013
application_1604218163813_0003              word count               MAPREDUCE          root    root.yinzhengjie.operation.op_queue01                KILLED                KILLED         
      100%    http://hadoop101.yinzhengjie.com:8088/cluster/app/application_1604218163813_0003
[root@hadoop101.yinzhengjie.com ~]#

[root@hadoop101.yinzhengjie.com ~]# yarn application -kill application_1604218163813_0013

五.检查节点状态

[root@hadoop101.yinzhengjie.com ~]# yarn node -all -list
20/11/01 18:09:42 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20/11/01 18:09:42 INFO client.RMProxy: Connecting to ResourceManager at hadoop101.yinzhengjie.com/172.200.6.101:8032
Total Nodes:3
         Node-Id         Node-State    Node-Http-Address    Number-of-Running-Containers
hadoop104.yinzhengjie.com:6666            RUNNING    hadoop104.yinzhengjie.com:8042                               0
hadoop102.yinzhengjie.com:6666            RUNNING    hadoop102.yinzhengjie.com:8042                               0
hadoop103.yinzhengjie.com:6666            RUNNING    hadoop103.yinzhengjie.com:8042                               0
[root@hadoop101.yinzhengjie.com ~]# 
[root@hadoop101.yinzhengjie.com ~]#

[root@hadoop101.yinzhengjie.com ~]# yarn node -all -list

六.检查YARN的队列状态

[root@hadoop101.yinzhengjie.com ~]# yarn queue -status yinzhengjie
20/11/01 18:15:47 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20/11/01 18:15:48 INFO client.RMProxy: Connecting to ResourceManager at hadoop101.yinzhengjie.com/172.200.6.101:8032
Queue Information : 
Queue Name : root.yinzhengjie
    State : RUNNING
    Capacity : .0%
    Current Capacity : .0%
    Maximum Capacity : -100.0%
    Default Node Label expression : <DEFAULT_PARTITION>
    Accessible Node Labels : 
[root@hadoop101.yinzhengjie.com ~]#

[root@hadoop101.yinzhengjie.com ~]# yarn queue -status yinzhengjie

七.获取作业的日志

[root@hadoop101.yinzhengjie.com ~]# ll
total 0
[root@hadoop101.yinzhengjie.com ~]# 
[root@hadoop101.yinzhengjie.com ~]# yarn logs -applicationId application_1604218163813_0008  > application_1604218163813_0008.log
20/11/01 18:27:09 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20/11/01 18:27:09 INFO client.RMProxy: Connecting to ResourceManager at hadoop101.yinzhengjie.com/172.200.6.101:8032
[root@hadoop101.yinzhengjie.com ~]# 
[root@hadoop101.yinzhengjie.com ~]# ll
total 140
-rw-r--r-- 1 root root 140715 Nov  1 18:27 application_1604218163813_0008.log
[root@hadoop101.yinzhengjie.com ~]# 
[root@hadoop101.yinzhengjie.com ~]# 
[root@hadoop101.yinzhengjie.com ~]# ll -h
total 140K
-rw-r--r-- 1 root root 138K Nov  1 18:27 application_1604218163813_0008.log
[root@hadoop101.yinzhengjie.com ~]#

八.YARN管理命令

1>.节点的上线与下线

2>.高可用ResourceManager

3>.其他

目录
相关文章
|
1月前
|
资源调度 分布式计算 Hadoop
YARN(Hadoop操作系统)的架构
本文详细解释了YARN(Hadoop操作系统)的架构,包括其主要组件如ResourceManager、NodeManager和ApplicationMaster的作用以及它们如何协同工作来管理Hadoop集群中的资源和调度作业。
37 3
YARN(Hadoop操作系统)的架构
|
2月前
|
分布式计算 资源调度 Hadoop
Hadoop入门基础(五):Hadoop 常用 Shell 命令一网打尽,提升你的大数据技能!
Hadoop入门基础(五):Hadoop 常用 Shell 命令一网打尽,提升你的大数据技能!
|
2月前
|
资源调度 分布式计算 算法
【揭秘Yarn调度秘籍】打破资源分配的枷锁,Hadoop Yarn权重调度全攻略!
【8月更文挑战第24天】在大数据处理领域,Hadoop Yarn 是一种关键的作业调度与集群资源管理工具。它支持多种调度器以适应不同需求,默认采用FIFO调度器,但可通过引入基于权重的调度算法来提高资源利用率。该算法根据作业或用户的权重值决定资源分配比例,权重高的可获得更多计算资源,特别适合多用户共享环境。管理员需在Yarn配置文件中启用特定调度器(如CapacityScheduler),并通过设置队列权重来实现资源的动态调整。合理配置权重有助于避免资源浪费,确保集群高效运行,满足不同用户需求。
34 3
|
2月前
|
XML 分布式计算 监控
详细指南:在Hadoop中配置Oozie作业
【8月更文挑战第31天】
33 0
|
2月前
|
SQL 资源调度 数据处理
实时计算 Flink版产品使用问题之-s参数在yarn-session.sh命令中是否有效
实时计算Flink版作为一种强大的流处理和批处理统一的计算框架,广泛应用于各种需要实时数据处理和分析的场景。实时计算Flink版通常结合SQL接口、DataStream API、以及与上下游数据源和存储系统的丰富连接器,提供了一套全面的解决方案,以应对各种实时计算需求。其低延迟、高吞吐、容错性强的特点,使其成为众多企业和组织实时数据处理首选的技术平台。以下是实时计算Flink版的一些典型使用合集。
|
3月前
|
分布式计算 Hadoop
hadoop格式化HDFS的命令
【7月更文挑战第21天】
240 5
|
3月前
|
分布式计算 资源调度 Hadoop
Hadoop执行格式化命令
【7月更文挑战第20天】
66 1
|
3月前
|
SQL 缓存 资源调度
实时计算 Flink版产品使用问题之在Flink on Yarn模式下,如何对job作业进行指标监控
实时计算Flink版作为一种强大的流处理和批处理统一的计算框架,广泛应用于各种需要实时数据处理和分析的场景。实时计算Flink版通常结合SQL接口、DataStream API、以及与上下游数据源和存储系统的丰富连接器,提供了一套全面的解决方案,以应对各种实时计算需求。其低延迟、高吞吐、容错性强的特点,使其成为众多企业和组织实时数据处理首选的技术平台。以下是实时计算Flink版的一些典型使用合集。
|
3月前
|
分布式计算 Hadoop
Hadoop配置作业使用Combiner
【7月更文挑战第7天】
28 4
|
4月前
|
分布式计算 Hadoop Java

相关实验场景

更多
下一篇
无影云桌面