+关注继续查看

# Changing Hive Dynamic Partition Limits

Symptoms:

Hive enforces limits on the number of dynamic partitions that it creates.  The default is 100 dynamic partitions per node, with a total (default) limit of 1000 dynamic partitions across all nodes. However, this can be tuned.  If your job tries to create too many dynamic partitions, you may see a stack trace similar to the following:

Diagnostic Messages for this Task:Error: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveFatalException: [Error 20004]: Fatal error occurred when node tried to create too many dynamic partitions. The maximum number of dynamic partitions is controlled by hive.exec.max.dynamic.partitions and hive.exec.max.dynamic.partitions.pernode. Maximum was set to: 100at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:283)at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:444)at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)at org.apache.hadoop.mapred.YarnChild\$2.run(YarnChild.java:167)at java.security.AccessController.doPrivileged(Native Method)at javax.security.auth.Subject.doAs(Subject.java:415)at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)Caused by: org.apache.hadoop.hive.ql.metadata.HiveFatalException: [Error 20004]: Fatal error occurred when node tried to create too many dynamic partitions. The maximum number of dynamic partitions is controlled by hive.exec.max.dynamic.partitions and hive.exec.max.dynamic.partitions.pernode. Maximum was set to: 100at org.apache.hadoop.hive.ql.exec.FileSinkOperator.getDynOutPaths(FileSinkOperator.java:747)at org.apache.hadoop.hive.ql.exec.FileSinkOperator.startGroup(FileSinkOperator.java:829)at org.apache.hadoop.hive.ql.exec.Operator.defaultStartGroup(Operator.java:498)at org.apache.hadoop.hive.ql.exec.Operator.startGroup(Operator.java:521)at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:232)... 7 moreFAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTaskMapReduce Jobs Launched:

Cause:

Your Hive job tried to create too many dynamic partitions on a node, exceeding the current limits. The number of dynamic partitions is controlled by the settings forhive.exec.max.dynamic.partitions and hive.exec.max.dynamic.partitions.pernode.  The default is 100 dynamic partitions on a single node, with a limit of 1000 overall.

Solution:

Set the values for hive.exec.max.dynamic.partitions and hive.exec.max.dynamic.partitions.pernode to higher values.  For example:

SET hive.exec.dynamic.partition=true;SET hive.exec.max.dynamic.partitions=2048;SET hive.exec.max.dynamic.partitions.pernode=256;

hive Illegal Operation state transition from CLOSED to ERROR的处理

1752 0

1426 0

1203 0
hive 异常 （Attempt to do update or delete on table terminal that does not use an ）
hive 异常 （Attempt to do update or delete on table terminal that does not use an ）   hive > delete from terminal where 1=1 ;  出现以下    ...
1626 0
DataXceiver error processing unknown operation src: /127.0.0.1:36479 dst: /127.0.0.1:50010处理

1641 0
+关注
@dailidong@

373

0

JS零基础入门教程（上册）