3.4 Watermark 的引入
watermark 的引入很简单,对于乱序数据,最常见的引用方式如下:
dataStream.assignTimestampsAndWatermarks( new BoundedOutOfOrdernessTimestampExtractor[SensorReading](Time.millisecond s(1000)) { override def extractTimestamp(element: SensorReading): Long = { element.timestamp * 1000 } } )
Event Time 的使用一定要指定数据源中的时间戳。否则程序无法知道事件的事件时间是什么(数据源里的数据没有时间戳的话,就只能使用Processing Time 了)。
我们看到上面的例子中创建了一个看起来有点复杂的类,这个类实现的其实就是分配时间戳的接口。Flink 暴露了 TimestampAssigner 接口供我们实现,使我们可以自定义如何从事件数据中抽取时间戳。
val env = StreamExecutionEnvironment.getExecutionEnvironment // 从调用时刻开始给 env 创建的每一个 stream 追加时间特性 env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime) val readings: DataStream[SensorReading] = env .addSource(new SensorSource) .assignTimestampsAndWatermarks(new MyAssigner())
MyAssigner 有两种类型
AssignerWithPeriodicWatermarks
AssignerWithPunctuatedWatermarks
以上两个接口都继承自 TimestampAssigner。
Assigner with periodic watermarks
周期性的生成 watermark:系统会周期性的将 watermark 插入到流中(水位线也是一种特殊的事件!)。默认周期是 200 毫秒。可以使用
ExecutionConfig.setAutoWatermarkInterval()方法进行设置。
val env = StreamExecutionEnvironment.getExecutionEnvironment env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime) // 每隔 5 秒产生一个 watermark env.getConfig.setAutoWatermarkInterval(5000)
产生 watermark 的逻辑:每隔 5 秒钟,Flink 会调用
AssignerWithPeriodicWatermarks 的 getCurrentWatermark()方法。
如果方法返回一个时间戳大于之前水位的时间戳,新的 watermark 会被插入到流中。这个检查保证了水位线是单调递增的。如果方法返回的时间戳小于等于之前水位的时间戳,则不会产生新的 watermark。
例子,自定义一个周期性的时间戳抽取:
class PeriodicAssigner extends AssignerWithPeriodicWatermarks[SensorReading] { val bound: Long = 60 * 1000 // 延时为 1 分钟 var maxTs: Long = Long.MinValue // 观察到的最大时间戳 override def getCurrentWatermark: Watermark = { new Watermark(maxTs - bound) } override def extractTimestamp(r: SensorReading, previousTS: Long) = { maxTs = maxTs.max(r.timestamp) r.timestamp } }
Assigner with punctuated watermarks
间断式地生成 watermark。和周期性生成的方式不同,这种方式不是固定时间的,而是可以根据需要对每条数据进行筛选和处理。直接上代码来举个例子,我们只给sensor_1 的传感器的数据流插入 watermark:
class PunctuatedAssigner extends AssignerWithPunctuatedWatermarks[SensorReading] { val bound: Long = 60 * 1000 override def checkAndGetNextWatermark(r: SensorReading, extractedTS: Long): Watermark = { if (r.id == "sensor_1") { new Watermark(extractedTS - bound) } else { null } } override def extractTimestamp(r: SensorReading, previousTS: Long): Long = { r.timestamp } }
3.5 Watermark 的设定
在 Flink 中,watermark 由应用程序开发人员生成,这通常需要对相应的领域有一定的了解
如果 watermark 设置的延迟太久,收到结果的速度可能就会很慢,解决办法是在水位线到达之前输出一个近似结果
而如果 watermark 到达得太早,则可能收到错误结果,不过 Flink 处理迟到数据的机制可以解决这个问题
四、EvnetTime 在 window 中的使用
4.1 滚动窗口(TumblingEventTimeWindows)
def main(args: Array[String]): Unit = { // 环境 val env: StreamExecutionEnvironment = StreamExecutionEnvironment.getExecutionEnvironment env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime) env.setParallelism(1) val dstream: DataStream[String] = env.socketTextStream("localhost",7777) val textWithTsDstream: DataStream[(String, Long, Int)] = dstream.map { text => val arr: Array[String] = text.split(" ") (arr(0), arr(1).toLong, 1) } val textWithEventTimeDstream: DataStream[(String, Long, Int)] = textWithTsDstream.assignTimestampsAndWatermarks(new BoundedOutOfOrdernessTimestampExtractor[(String, Long, Int)](Time.milliseconds(1000)) { override def extractTimestamp(element: (String, Long, Int)): Long = { return element._2 } }) val textKeyStream: KeyedStream[(String, Long, Int), Tuple] = textWithEventTimeDstream.keyBy(0) textKeyStream.print("textkey:") val windowStream: WindowedStream[(String, Long, Int), Tuple, TimeWindow] = textKeyStream.window(TumblingEventTimeWindows.of(Time.seconds(2))) val groupDstream: DataStream[mutable.HashSet[Long]] = windowStream.fold(new mutable.HashSet[Long]()) { case (set, (key, ts, count)) => set += ts } groupDstream.print("window::::").setParallelism(1) env.execute() } }
结果是按照 Event Time 的时间窗口计算得出的,而无关系统的时间(包括输入的快慢)。
4.2 滑动窗口(SlidingEventTimeWindows)
def main(args: Array[String]): Unit = { // 环境 val env: StreamExecutionEnvironment = StreamExecutionEnvironment.getExecutionEnvironment env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime) env.setParallelism(1) val dstream: DataStream[String] = env.socketTextStream("localhost",7777) val textWithTsDstream: DataStream[(String, Long, Int)] = dstream.map { text => val arr: Array[String] = text.split(" ") (arr(0), arr(1).toLong, 1) } val textWithEventTimeDstream: DataStream[(String, Long, Int)] = textWithTsDstream.assignTimestampsAndWatermarks(new BoundedOutOfOrdernessTimestampExtractor[(String, Long, Int)](Time.milliseconds(1000)) { override def extractTimestamp(element: (String, Long, Int)): Long = { return element._2 } }) val textKeyStream: KeyedStream[(String, Long, Int), Tuple] = textWithEventTimeDstream.keyBy(0) textKeyStream.print("textkey:") val windowStream: WindowedStream[(String, Long, Int), Tuple, TimeWindow] = textKeyStream.window(SlidingEventTimeWindows.of(Time.seconds(2),Time.millis econds(500))) val groupDstream: DataStream[mutable.HashSet[Long]] = windowStream.fold(new mutable.HashSet[Long]()) { case (set, (key, ts, count)) => set += ts } groupDstream.print("window::::").setParallelism(1) env.execute() }
4.3 会话窗口(EventTimeSessionWindows)
相邻两次数据的 EventTime 的时间差超过指定的时间间隔就会触发执行。如果加入 Watermark, 会在符合窗口触发的情况下进行延迟。到达延迟水位再进行窗口触发。
def main(args: Array[String]): Unit = { // 环境 val env: StreamExecutionEnvironment = StreamExecutionEnvironment.getExecutionEnvironment env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime) env.setParallelism(1) val dstream: DataStream[String] = env.socketTextStream("localhost",7777) val textWithTsDstream: DataStream[(String, Long, Int)] = dstream.map { text => val arr: Array[String] = text.split(" ") (arr(0), arr(1).toLong, 1) } val textWithEventTimeDstream: DataStream[(String, Long, Int)] = textWithTsDstream.assignTimestampsAndWatermarks(new BoundedOutOfOrdernessTimestampExtractor[(String, Long, Int)](Time.milliseconds(1000)) { override def extractTimestamp(element: (String, Long, Int)): Long = { return element._2 } }) val textKeyStream: KeyedStream[(String, Long, Int), Tuple] = textWithEventTimeDstream.keyBy(0) textKeyStream.print("textkey:") val windowStream: WindowedStream[(String, Long, Int), Tuple, TimeWindow] = textKeyStream.window(EventTimeSessionWindows.withGap(Time.milliseconds(500) ) ) windowStream.reduce((text1,text2)=> ( text1._1,0L,text1._3+text2._3) ) .map(_._3).print("windows:::").setParallelism(1) env.execute() }