以json / text格式的Apache Spark DAG可视化

可以转换成节点-连接形式的JSON数据，然后使用dagre-d3来可视化展示。这样可视化后就能看出问题了。

使用.explain运算符，它将逻辑和（启用扩展标志）物理计划打印到控制台。

val records = spark.
readStream.
format("rate").
load

scala> records.explain

== Physical Plan ==
StreamingRelation rate, [timestamp#0, value#1L]

scala> records.explain(extended = true)

== Parsed Logical Plan ==
StreamingRelation DataSource(org.apache.spark.sql.SparkSession@4071aa13,rate,List(),None,List(),None,Map(),None), rate, [timestamp#0, value#1L]

== Analyzed Logical Plan ==
timestamp: timestamp, value: bigint
StreamingRelation DataSource(org.apache.spark.sql.SparkSession@4071aa13,rate,List(),None,List(),None,Map(),None), rate, [timestamp#0, value#1L]

== Optimized Logical Plan ==
StreamingRelation DataSource(org.apache.spark.sql.SparkSession@4071aa13,rate,List(),None,List(),None,Map(),None), rate, [timestamp#0, value#1L]

== Physical Plan ==
StreamingRelation rate, [timestamp#0, value#1L]
物理计划是DAG的转换，因此它可以帮助您。

这个链接可能会有所帮助：

https://jaceklaskowski.gitbooks.io/spark-structured-streaming/spark-sql-streaming-Dataset-explain.html