开发者社区> 问答> 正文

请问在blink里应该如何定义rowtime呢?

我使用blink planner来定义了下面的表 CREATE TABLE requests( rowtime TIMESTAMP, requestId VARCHAR, algoExtent ROW(mAdId VARCHAR)) with ( 'connector.type' = 'kafka', 'connector.version' = 'universal', 'connector.topic' = 'test_request', 'connector.startup-mode' = 'latest-offset', 'connector.properties.0.key' = 'zookeeper.connect', 'connector.properties.0.value' = '10.107.116.42:2181', 'connector.properties.1.key' = 'bootstrap.servers', 'connector.properties.1.value' = '10.107.116.42:9092', 'connector.properties.2.key' = 'group.id', 'connector.properties.2.value' = 'test_request', 'update-mode' = 'append','format.type' = 'json', 'format.derive-schema' = 'true', 'schema.0.rowtime.timestamps.type' = 'from-field', 'schema.0.rowtime.timestamps.from' = 'sqlTimestamp', 'schema.0.rowtime.watermarks.type' = 'periodic-ascending') 然后kafka里消息的格式如下 {"requestId": "rrrr","algoExtent": {"duration": 12,"adType ": "FEED_568_320","mAdId": "1910141050233527", "sqlTimestamp":"2019-10-17 19:08:01" }} 但是运行时报错 Caused by: org.apache.flink.streaming.runtime.tasks.ExceptionInChainedOperatorException: Could not forward element to next operator at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:654) at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:612) at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:592) at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:727) at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:705) at org.apache.flink.streaming.api.operators.StreamSourceContexts$ManualWatermarkContext.processAndCollectWithTimestamp(StreamSourceContexts.java:310) at org.apache.flink.streaming.api.operators.StreamSourceContexts$WatermarkContext.collectWithTimestamp(StreamSourceContexts.java:409) at org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher.emitRecordWithTimestamp(AbstractFetcher.java:398) at org.apache.flink.streaming.connectors.kafka.internal.KafkaFetcher.emitRecord(KafkaFetcher.java:185) at org.apache.flink.streaming.connectors.kafka.internal.KafkaFetcher.runFetchLoop(KafkaFetcher.java:150) at org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.run(FlinkKafkaConsumerBase.java:715) at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:100) at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:63) at org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:202) Caused by: org.apache.flink.streaming.runtime.tasks.ExceptionInChainedOperatorException: Could not forward element to next operator at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:654) at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:612) at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:592) at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:727) at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:705) at SourceConversion$4.processElement(Unknown Source) at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:637) ... 13 more Caused by: java.lang.NullPointerException at org.apache.flink.table.dataformat.GenericRow.getLong(GenericRow.java:58) at org.apache.flink.table.planner.plan.nodes.physical.stream.PeriodicWatermarkAssignerWrapper.extractTimestamp(StreamExecTableSourceScan.scala:202) at org.apache.flink.table.planner.plan.nodes.physical.stream.PeriodicWatermarkAssignerWrapper.extractTimestamp(StreamExecTableSourceScan.scala:194) at org.apache.flink.streaming.runtime.operators.TimestampsAndPeriodicWatermarksOperator.processElement(TimestampsAndPeriodicWatermarksOperator.java:64) at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:637) ... 19 more 请问在blink里应该如何定义rowtime呢?*来自志愿者整理的flink邮件归档

展开
收起
彗星halation 2021-12-08 10:15:35 741 0
1 条回答
写回答
取消 提交回答
  • 应该是你的 sqlTimestamp 字段中有 null 的数据,在去取 ts 的时候报 NPE 了。 目前 watermark assigner 要求每条数据的 ts 都是有值的。*来自志愿者整理的flink邮件归档

    2021-12-08 10:55:19
    赞同 展开评论 打赏
问答排行榜
最热
最新

相关电子书

更多
低代码开发师(初级)实战教程 立即下载
冬季实战营第三期:MySQL数据库进阶实战 立即下载
阿里巴巴DevOps 最佳实践手册 立即下载