大数据平台 CDP 中如何配置 hive 作业的 YARN 队列以确保SLA?

#用户在客户端指定的yarn 队列,会被服务端的放置规则覆盖，YARN RM 相关日志：
2022-09-21 09:39:01,306 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAppManager: Placed application with ID application_1663663279733_0012 in queue: dap, original submission queue was: root.dap2
#用户在客户端指定的yarn 队列，会被服务端的放置规则覆盖，yarn 相关源码
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager#copyPlacementQueueToSubmissionContext：
  private void copyPlacementQueueToSubmissionContext(
      ApplicationPlacementContext placementContext,
      ApplicationSubmissionContext context) {
    // Set the queue from the placement in the ApplicationSubmissionContext
    // Placement rule are only considered for new applications
    if (placementContext != null && !StringUtils.equalsIgnoreCase(
        context.getQueue(), placementContext.getQueue())) {
      LOG.info("Placed application with ID " + context.getApplicationId() +
          " in queue: " + placementContext.getQueue() +
          ", original submission queue was: " + context.getQueue());
      context.setQueue(placementContext.getQueue());
    }
  }
# 参数 yarn.scheduler.capacity.queue-mappings-override.enable 改为 true说明：
This function is used to specify whether the user specified queues can be overridden, this is a Boolean value and the default value is false.
If a queue mapping is present, will it override the value specified by the user? This can be used by administrators to place jobs in queues that are different than the one specified by the user.

#yarn
yarn.resourcemanager.application-tag-based-placement.enable: Whether to enable application placement based on user ID passed via application tags. When it is enabled, userid=<userId> pattern will be checked and if found, the application will be placed onto the found user's queue, if the original user has enough rights on the passed user's queue.
yarn.resourcemanager.application-tag-based-placement.username.whitelist: Comma separated list of users who can use the application tag based placement, if it is enabled.
#tez
tez.am.resource.memory.mb
tez.am.resource.cpu.vcores
tez.task.resource.memory.mb
tez.task.resource.cpu.vcores
tez.session.client.timeout.secs: default 2 mins. Time (in seconds) to wait for AM to come up when trying to submit a DAG from the client. Only relevant in session mode. If the cluster is busy and cannot launch the AM then this timeout may be hit. In those case, using non-session mode is recommended if applicable. Otherwise increase the timeout (set to -1 for infinity. Not recommended)
tez.session.am.dag.submit.timeout.secs: default 5 mins.Time (in seconds) for which the Tez Application Master should wait for a DAG to be submitted before shutting down.Only relevant in session mode. Any negative value will disable this check and allow the AM to hang around forever in idle mode.
tez.am.mode.session: Boolean value. Execution mode for the Tez application. True implies session mode. If the client code is written according to best practices then the same code can execute in either mode based on this configuration. Session mode is more aggressive in reserving execution resources and is typically used for interactive applications where multiple DAGs are submitted in quick succession by the same user. For long running applications, one-off executions, batch jobs etc non-session mode is recommended. If session mode is enabled then container reuse is recommended.
tez.queue.name: String value. The queue name for all jobs being submitted from a given client.
tez.local.mode: Boolean value. Enable local mode execution in Tez. Enables tasks to run in the same process as the app master. Primarily used for debugging.
#hive
hive.server2.enable.doAs
hive.tez.container.size
hive.tez.cpu.vcores
hive.server2.tez.default.queues：A list of comma separated values corresponding to YARN queues of the same name. When HiveServer2 is launched in Tez mode, this configuration needs to be set for multiple Tez sessions to run in parallel on the cluster.
hive.server2.tez.initialize.default.sessions: default true.This flag is used in HiveServer2 to enable a user to use HiveServer2 without turning on Tez for HiveServer2. The user could potentially want to run queries over Tez without the pool of sessions.
hive.server2.tez.sessions.init.threads: If hive.server2.tez.initialize.default.sessions is enabled, the maximum number of threads to use to initialize the default sessions.
hive.server2.tez.queue.access.check: Whether to check user access to explicitly specified YARN queues. yarn.resourcemanager.webapp.address must be configured to use this.
hive.server2.tez.sessions.custom.queue.allowed：default true. Whether Tez session pool should allow submitting queries to custom queues. The options are true, false (error out), ignore (accept the query but ignore the queue setting).
hive.server2.tez.session.lifetime: default 162h. The lifetime of the Tez sessions launched by HS2 when default sessions are enabled.Set to 0 to disable session expiration
hive.server2.tez.sessions.per.default.queue
hive.server2.tez.interactive.queue
hive.server2.tez.sessions.restricted.configs
CDP: tez.queue.name/mapreduce.job.queuename/mapred.job.queue.name
CDH: mapreduce.job.queuename/mapred.job.queue.name/mapred.job.queuename