flattern scala数组类型列到多列-问答-阿里云开发者社区-阿里云

开发者社区> 问答> 正文

flattern scala数组类型列到多列

社区小助手 2018-12-19 16:55:14 1343

他们是否有可能在Scala DF中展平阵列?

正如我所知,使用列并选择filed.a可行,但我不想手动指定它们。

df.printSchema()
|-- client_version: string (nullable = true)
|-- filed: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- a: string (nullable = true)
| | |-- b: string (nullable = true)
| | |-- c: string (nullable = true)
| | |-- d: string (nullable = true)
最后的df

df.printSchema()

 |-- client_version: string (nullable = true)
 |-- filed_a: string (nullable = true)
 |-- filed_b: string (nullable = true)
 |-- filed_c: string (nullable = true)
 |-- filed_d: string (nullable = true)
c++数组类
分享到
取消 提交回答
全部回答(1)
  • 社区小助手
    2019-07-17 23:23:03

    您可以使用blast平展ArrayType列,并将嵌套的结构元素名称映射到所需的顶级列名,如下所示:
    import org.apache.spark.sql.functions._

    case class S(a: String, b: String, c: String, d: String)

    val df = Seq(
    ("1.0", Seq(S("a1", "b1", "c1", "d1"))),
    ("2.0", Seq(S("a2", "b2", "c2", "d2"), S("a3", "b3", "c3", "d3")))
    ).toDF("client_version", "filed")

    df.printSchema
    // root
    // |-- client_version: string (nullable = true)
    // |-- filed: array (nullable = true)
    // | |-- element: struct (containsNull = true)
    // | | |-- a: string (nullable = true)
    // | | |-- b: string (nullable = true)
    // | | |-- c: string (nullable = true)
    // | | |-- d: string (nullable = true)

    val dfFlattened = df.withColumn("filed_element", explode($"filed"))

    val structElements = dfFlattened.select($"filed_element.*").columns

    val dfResult = dfFlattened.select( col("client_version") +: structElements.map(

    c => col(s"filed_element.$c").as(s"filed_$c")

    ): _*
    )

    dfResult.show
    // +--------------+-------+-------+-------+-------+
    // |client_version|filed_a|filed_b|filed_c|filed_d|
    // +--------------+-------+-------+-------+-------+
    // | 1.0| a1| b1| c1| d1|
    // | 2.0| a2| b2| c2| d2|
    // | 2.0| a3| b3| c3| d3|
    // +--------------+-------+-------+-------+-------+

    dfResult.printSchema
    // root
    // |-- client_version: string (nullable = true)
    // |-- filed_a: string (nullable = true)
    // |-- filed_b: string (nullable = true)
    // |-- filed_c: string (nullable = true)
    // |-- filed_d: string (nullable = true)


    用于explode通过添加更多行来展平数组,然后select使用*符号将struct列重新置于顶部。

    import org.apache.spark.sql.functions.{collect_list, explode, struct}
    import spark.implicits._

    val df = Seq(("1", "a", "a", "a"),
    ("1", "b", "b", "b"),
    ("2", "a", "a", "a"),
    ("2", "b", "b", "b"),
    ("2", "c", "c", "c"),
    ("3", "a", "a","a")).toDF("idx", "A", "B", "C")
    .groupBy(("idx"))
    .agg(collect_list(struct("A", "B", "C")).as("nested_col"))

    df.printSchema()
    // root
    // |-- idx: string (nullable = true)
    // |-- nested_col: array (nullable = true)
    // | |-- element: struct (containsNull = true)
    // | | |-- A: string (nullable = true)
    // | | |-- B: string (nullable = true)
    // | | |-- C: string (nullable = true)

    df.show
    // +---+--------------------+
    // |idx| nested_col|
    // +---+--------------------+
    // | 3| [[a, a, a]]|
    // | 1|[[a, a, a], [b, b...|
    // | 2|[[a, a, a], [b, b...|
    // +---+--------------------+

    val dfExploded = df.withColumn("exploded", explode($"nested_col")).drop("nested_col")

    dfExploded.show
    // +---+---------+
    // |idx| exploded|
    // +---+---------+
    // | 3|[a, a, a]|
    // | 1|[a, a, a]|
    // | 1|[b, b, b]|
    // | 2|[a, a, a]|
    // | 2|[b, b, b]|
    // | 2|[c, c, c]|
    // +---+---------+

    val finalDF = dfExploded.select("idx", "exploded.*")

    finalDF.show
    // +---+---+---+---+
    // |idx| A| B| C|
    // +---+---+---+---+
    // | 3| a| a| a|
    // | 1| a| a| a|
    // | 1| b| b| b|
    // | 2| a| a| a|
    // | 2| b| b| b|
    // | 2| c| c| c|
    // +---+---+---+---+

    0 0
开发与运维
使用钉钉扫一扫加入圈子
+ 订阅

集结各类场景实战经验,助你开发运维畅行无忧

相似问题
最新问题
推荐课程