【spark系列2】spark 合并github (pull request)pr 的正确姿势-阿里云开发者社区

【spark系列2】spark 合并github (pull request)pr 的正确姿势

2022-05-28 134

版权

本文内容由阿里云实名注册用户自发贡献，版权归原作者所有，阿里云开发者社区不拥有其著作权，亦不承担相应法律责任。具体规则请查看《阿里云开发者社区用户服务协议》和《阿里云开发者社区知识产权保护指引》。如果您发现本社区中有涉嫌抄袭的内容，填写侵权投诉表单进行举报，一经查实，本社区将立刻删除涉嫌侵权内容。

简介： 【spark系列2】spark 合并github (pull request)pr 的正确姿势

最近在做内部spark版本升级的工作，涉及到github 上合并pr的工作，具体的是spark 2.x升级到spark 3.0.1时兼容hdfs cdh-2.6.0-5.13.1，报编译错误

[INFO] Compiling 25 Scala sources to /Users/libinsong/Documents/codes/tongdun/spark-3.0/resource-managers/yarn/target/scala-2.12/classes ...
[ERROR] [Error] resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:298: value setRolledLogsIncludePattern is not a member of org.apache.hadoop.yarn.api.records.LogAggregationContext
[ERROR] [Error] resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:300: value setRolledLogsExcludePattern is not a member of org.apache.hadoop.yarn.api.records.LogAggregationContext
[ERROR] [Error] resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:551: not found: value isLocalUri
[ERROR] [Error] resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:1367: not found: value isLocalUri
[ERROR] four errors found

具体的解决方法github spark pr 已经给出了解决方法，修改相应的代码就行,但是仅仅是修改的话，我们可以选择一种更加优雅的方式（以git cherry-pick方式），

现在简单的分享一下：

直接找到setRolledLogsIncludePattern一行，

 sparkConf.get(ROLLED_LOG_INCLUDE_PATTERN).foreach { includePattern =>
      try {
        val logAggregationContext = Records.newRecord(classOf[LogAggregationContext])
        logAggregationContext.setRolledLogsIncludePattern(includePattern)
        sparkConf.get(ROLLED_LOG_EXCLUDE_PATTERN).foreach { excludePattern =>
          logAggregationContext.setRolledLogsExcludePattern(excludePattern)
        }
        appContext.setLogAggregationContext(logAggregationContext)
      } catch {
        case NonFatal(e) =>
          logWarning(s"Ignoring ${ROLLED_LOG_INCLUDE_PATTERN.key} because the version of YARN " +
            "does not support it", e)
      }
    }
    appContext.setUnmanagedAM(isClientUnmanagedAMEnabled)
    sparkConf.get(APPLICATION_PRIORITY).foreach { appPriority =>
      appContext.setPriority(Priority.newInstance(appPriority))
    }
    appContext
  }