Spark - ReturnStatementInClosureException: Return statements aren‘t allowed in Spark closures

简介: Spark 使用 RDD 调用 Filter 函数时,dirver 端卡住,报错 ReturnStatementInClosureException: Return statements aren't allowed in Spark closures,即闭包内无法使用 return 函数。

 一.引言

Spark 使用 RDD 调用 Filter 函数时,dirver 端卡住,报错 ReturnStatementInClosureException: Return statements aren't allowed in Spark closures,即闭包内无法使用 return 函数:

image.gif编辑

二.使用场景

使用 rdd.filter 方法过滤 id 时使用了 return 方法,导致上述报错:

rdd.filter(arr => {
       val id = arr(0)
       val l = id.length()
       if (l <= 8) return false
       if (id.startsWith("1")) {
           true
       } else {
           false
       }
     })

image.gif

修改方案:

闭包函数内不使用 return 关键字即可

rdd.filter(arr => {
        val id = arr(0)
        val l = id.length()
        if (l <= 8) {
            false
        } else if (id.startsWith("1")) {
            true
        } else {
            false
        }
    })

image.gif


目录
相关文章
|
7月前
|
Java 数据库连接 mybatis
解决Error querying database. Cause: org.apache.ibatis.executor.ExecutorException: No constructor found
解决Error querying database. Cause: org.apache.ibatis.executor.ExecutorException: No constructor found
335 1
|
7月前
|
XML Java 数据库连接
org.apache.ibatis.binding.BindingException: Invalid bound statement (not found): com.example.forum.d
org.apache.ibatis.binding.BindingException: Invalid bound statement (not found): com.example.forum.d
69 1
|
7月前
|
分布式计算 安全 Java
Spark 编译出现 InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty
Spark 编译出现 InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty
188 0
|
XML Java 数据库连接
【异常解决】解决org.apache.ibatis.binding.BindingException: Invalid bound statement (not found)问题
【异常解决】解决org.apache.ibatis.binding.BindingException: Invalid bound statement (not found)问题
236 0
org.apache.ibatis.binding.BindingException: Invalid bound statement (not found)
org.apache.ibatis.binding.BindingException: Invalid bound statement (not found)
128 0
params argument given to the optimizer should be an iterable
params argument given to the optimizer should be an iterable
192 0
params argument given to the optimizer should be an iterable
|
分布式计算 Java Spark
Optimizing Spark job parameters
Optimizing Spark job parameters
265 0
|
XML Java 数据库连接
org.apache.ibatis.binding.BindingException: Invalid bound statement (not found):
org.apache.ibatis.binding.BindingException: Invalid bound statement (not found):
359 1
org.apache.ibatis.binding.BindingException: Invalid bound statement (not found)(解决)
org.apache.ibatis.binding.BindingException: Invalid bound statement (not found)(解决)
org.apache.ibatis.binding.BindingException: Invalid bound statement (not found)(解决)
|
XML Java 数据库连接
org.apache.ibatis.binding.BindingException: Invalid bound statement (not found):
org.apache.ibatis.binding.BindingException: Invalid bound statement (not found): 问题处理
org.apache.ibatis.binding.BindingException: Invalid bound statement (not found):