193 DStream相关操作 - Output Operations on DStreams

简介: 193 DStream相关操作 - Output Operations on DStreams

Output Operations可以将DStream的数据输出到外部的数据库或文件系统,当某个Output Operations原语被调用时(与RDD的Action相同),streaming程序才会开始真正的计算过程。

Output Operation Meaning
print() Prints the first ten elements of every batch of data in a DStream on the driver node running the streaming application. This is useful for development and debugging.
saveAsTextFiles(prefix, [suffix]) Save this DStream’s contents as text files. The file name at each batch interval is generated based on prefix and suffix: “prefix-TIME_IN_MS[.suffix]”.
saveAsObjectFiles(prefix, [suffix]) Save this DStream’s contents as SequenceFiles of serialized Java objects. The file name at each batch interval is generated based on prefix and suffix: “prefix-TIME_IN_MS[.suffix]”.
saveAsHadoopFiles(prefix, [suffix]) Save this DStream’s contents as Hadoop files. The file name at each batch interval is generated based on prefix and suffix: “prefix-TIME_IN_MS[.suffix]”.
foreachRDD(func) The most generic output operator that applies a function, func, to each RDD generated from the stream. This function should push the data in each RDD to an external system, such as saving the RDD to files, or writing it over the network to a database. Note that the function func is executed in the driver process running the streaming application, and will usually have RDD actions in it that will force the computation of the streaming RDDs.
目录
相关文章
|
2月前
|
分布式计算 JavaScript 前端开发
Stream学习笔记(二)map与reduce
Stream学习笔记(二)map与reduce
|
5月前
|
机器学习/深度学习 分布式计算 API
192 DStream相关操作 - Transformations on DStreams
192 DStream相关操作 - Transformations on DStreams
15 0
|
分布式计算 算法 大数据
Rdd 算子_转换_sample | 学习笔记
快速学习 Rdd 算子_转换_sample
125 0
Rdd 算子_转换_sample | 学习笔记
|
消息中间件 SQL 分布式计算
Structured_Sink_Foreach | 学习笔记
快速学习 Structured_Sink_Foreach
99 0
Structured_Sink_Foreach | 学习笔记
|
存储 测试技术 索引
ES中数据流Data streams详解
ES中数据流Data streams详解
436 0
ES中数据流Data streams详解
|
分布式计算 Java 5G
spark异常:missing an output location for shuffle 0
spark异常:missing an output location for shuffle 0
381 0
|
分布式计算
RDD的 transformations 和 actions 总结
RDD的transformations和actions 两个RDD:一个RDD包含 {1, 2, 3} , 另一个RDD包含{3, 4, 5}
81 0
RDD的 transformations 和 actions 总结