此解决方案使用

instr找到JSON curly braces的出现{and}
substr 获取JSON curly braces之间的子串（JSON文本）
然后它使用from_json定义预期JSON结构的模式。

from pyspark.sql.functions import from_json, instr
from pyspark.sql.types import *

Expected JSON schema

schema = StructType([StructField("timestmap", TimestampType()),

                 StructField("name", StringType())])

Filtering and parsing

parsed = df.select(from_json(

            df.value.substr(instr(df.value, '{'), instr(df.value, '}')), 
            schema).alias("json"))

Don't know if it's possible to do it in one step ...

parsed = parsed.select(F.col("json.timestmap").alias("timestmap"),

                   F.col("json.name").alias("name"))

parsed.printSchema()
parsed.show()
结果是

root
|-- timestmap: timestamp (nullable = true)
|-- name: string (nullable = true)

示例文本文件random.txt是

some random non json stuff here {"timestmap":21212121, "name":"John"}
some other random non json stuff here {"timestmap":21212122, "name":"Doe"}