以下代码引发“检测到INNER联接的笛卡尔积”异常：

first_df = spark.createDataFrame([{"first_id": "1"}, {"first_id": "1"}, {"first_id": "1"}, ])
second_df = spark.createDataFrame([{"some_value": "????"}, ])

second_df = second_df.withColumn("second_id", F.lit("1"))

If the next line is uncommented, then the JOIN is working fine.

second_df.persist()

result_df = first_df.join(second_df,

                      first_df.first_id == second_df.second_id,
                      'inner')

data = result_df.collect()

result_df.explain()
并告诉我逻辑计划如下所示：

Filter (first_id#0 = 1)
+- LogicalRDD [first_id#0], false
and
Project [some_value#2, 1 AS second_id#4]
+- LogicalRDD [some_value#2], false
Join condition is missing or trivial.
Use the CROSS JOIN syntax to allow cartesian products between these relations.;
看起来有一个原因，当RuleExecutor应用名为CheckCartesianProducts的优化规则集时，这些逻辑计划的JOIN条件中不存在列（请参阅https://github.com/apache/spark/blob/v2.3.0/sql/ catalyst / src / main / scala / org / apache / spark / sql / catalyst / optimizer / Optimizer.scala＃L1114）。

但是，如果我在JOIN之前使用“persist”方法，它的工作原理和物理计划是：

*(3) SortMergeJoin [first_id#0], [second_id#4], Inner
:- *(1) Sort [first_id#0 ASC NULLS FIRST], false, 0
: +- Exchange hashpartitioning(first_id#0, 10)
: +- Scan ExistingRDD[first_id#0]
+- *(2) Sort [second_id#4 ASC NULLS FIRST], false, 0
+- Exchange hashpartitioning(second_id#4, 10)

  +- InMemoryTableScan [some_value#2, second_id#4]
        +- InMemoryRelation [some_value#2, second_id#4], true, 10000, StorageLevel(disk, memory, 1 replicas)
              +- *(1) Project [some_value#2, 1 AS second_id#4]
                 +- Scan ExistingRDD[some_value#2]

所以，可能有人可以解释内部导致这样的结果，因为持久化数据框架看起来并不像解决方案。