问题一:flinksql报错Could not find any factory for identifier 'elasticsearch-6' that implements 'org.apache.flink.table.factories.DynamicTableSinkFactory' in the classpath. hello 我在使用flinksql连接器时当我将flink-sql-connector-elasticsearch6_2.11-1.11.1.jar放在lib下,程序正常执行,但是当我在pom中进行配置时会产生如下报错,同样的问题会产生在hbase、jdbc的connector中,请问下这可能是什么造成的
flinksql报错Could not find any factory for identifier 'elasticsearch-6' that implements 'org.apache.flink.table.factories.DynamicTableSinkFactory' in the classpath. hello 我在使用flinksql连接器时当我将flink-sql-connector-elasticsearch6_2.11-1.11.1.jar放在lib下,程序正常执行,但是当我在pom中进行配置时会产生如下报错,同样的问题会产生在hbase、jdbc的connector中,请问下这可能是什么造成的 org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: Unable to create a sink for writing table 'default_catalog.default_database.cloud_behavior_sink'.
Table options are:
'connector'='elasticsearch-6' 'document-type'='cdbp' 'hosts'='http://10.2.11.116:9200;http://10.2.11.117:9200;http://10.2.11.118:9200;http://10.2.11.119:9200' 'index'='flink_sql_test' 'sink.bulk-flush.max-actions'='100' at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:302) at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:198) at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:149) at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:699) at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:232) at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:916) at org.apache.flink.client.cli.CliFrontend.lambdamainmainmain10(CliFrontend.java:992) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917) at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41) at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:992) Caused by: org.apache.flink.table.api.ValidationException: Unable to create a sink for writing table 'default_catalog.default_database.cloud_behavior_sink'.
Table options are:
'connector'='elasticsearch-6' 'document-type'='cdbp' 'hosts'='http://10.2.11.116:9200;http://10.2.11.117:9200;http://10.2.11.118:9200;http://10.2.11.119:9200' 'index'='flink_sql_test' 'sink.bulk-flush.max-actions'='100' at org.apache.flink.table.factories.FactoryUtil.createTableSink(FactoryUtil.java:164) at org.apache.flink.table.planner.delegation.PlannerBase.getTableSink(PlannerBase.scala:344) at org.apache.flink.table.planner.delegation.PlannerBase.translateToRel(PlannerBase.scala:204) at org.apache.flink.table.planner.delegation.PlannerBase
anonfun$1.apply(PlannerBase.scala:163)atorg.apache.flink.table.planner.delegation.PlannerBaseanonfun$1.apply(PlannerBase.scala:163)atorg.apache.flink.table.planner.delegation.PlannerBase
anonfun$1.apply(PlannerBase.scala:163) at org.apache.flink.table.planner.delegation.PlannerBaseanonfun1.apply(PlannerBase.scala:163)atscala.collection.TraversableLike1.apply(PlannerBase.scala:163)atscala.collection.TraversableLike1.apply(PlannerBase.scala:163) at scala.collection.TraversableLikeanonfunanonfunanonfunmap1.apply(TraversableLike.scala:234)atscala.collection.TraversableLike1.apply(TraversableLike.scala:234)atscala.collection.TraversableLike1.apply(TraversableLike.scala:234) at scala.collection.TraversableLikeanonfunanonfunanonfunmap1.apply(TraversableLike.scala:234)atscala.collection.Iterator1.apply(TraversableLike.scala:234)atscala.collection.Iterator1.apply(TraversableLike.scala:234) at scala.collection.Iteratorclass.foreach(Iterator.scala:891) at scala.collection.AbstractIterator.foreach(Iterator.scala:1334) at scala.collection.IterableLikeclass.foreach(IterableLike.scala:72)atscala.collection.AbstractIterable.foreach(Iterable.scala:54)atscala.collection.TraversableLikeclass.foreach(IterableLike.scala:72)atscala.collection.AbstractIterable.foreach(Iterable.scala:54)atscala.collection.TraversableLikeclass.foreach(IterableLike.scala:72) at scala.collection.AbstractIterable.foreach(Iterable.scala:54) at scala.collection.TraversableLikeclass.map(TraversableLike.scala:234) at scala.collection.AbstractTraversable.map(Traversable.scala:104) at org.apache.flink.table.planner.delegation.PlannerBase.translate(PlannerBase.scala:163) at org.apache.flink.table.api.internal.TableEnvironmentImpl.translate(TableEnvironmentImpl.java:1264) at org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:700) at org.apache.flink.table.api.internal.TableEnvironmentImpl.executeOperation(TableEnvironmentImpl.java:787) at org.apache.flink.table.api.internal.TableEnvironmentImpl.executeSql(TableEnvironmentImpl.java:690) at com.intsig.flink.streaming.streaming_project.common.sql_submit.SqlSubmit.callDML(SqlSubmit.java:97) at com.intsig.flink.streaming.streaming_project.common.sql_submit.SqlSubmit.callCommand(SqlSubmit.java:72) at com.intsig.flink.streaming.streaming_project.common.sql_submit.SqlSubmit.run(SqlSubmit.java:53) at com.intsig.flink.streaming.streaming_project.common.sql_submit.SqlSubmit.main(SqlSubmit.java:24) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:288) ... 11 more Caused by: org.apache.flink.table.api.ValidationException: Cannot discover a connector using option ''connector'='elasticsearch-6''. at org.apache.flink.table.factories.FactoryUtil.getDynamicTableFactory(FactoryUtil.java:329) at org.apache.flink.table.factories.FactoryUtil.createTableSink(FactoryUtil.java:157) ... 37 more Caused by: org.apache.flink.table.api.ValidationException: Could not find any factory for identifier 'elasticsearch-6' that implements 'org.apache.flink.table.factories.DynamicTableSinkFactory' in the classpath.
Available factory identifiers are:
blackhole hbase-1.4 jdbc kafka print at org.apache.flink.table.factories.FactoryUtil.discoverFactory(FactoryUtil.java:240) at org.apache.flink.table.factories.FactoryUtil.getDynamicTableFactory(FactoryUtil.java:326) ... 38 more*来自志愿者整理的flink
参考回答:
Flink的class加载原则是child first,所以,尽量避免在pom中自己引入flink相关的依赖,避免跟Flink集群环境造成冲突,建议将安装包放在lib下,由flink去加载。
关于本问题的更多回答可点击原文查看:https://developer.aliyun.com/ask/370178?spm=a2c6h.14164896.0.0.42a563bfbbfLHp
问题二:flinksql例子建表的,没有报错就自动结束FAILED了是正常的嘛?
flinksql例子建表的
没有报错就自动结束FAILED了是正常的嘛?
参考回答:
sql也分批和流,如果是批结束且是finish是正常的
关于本问题的更多回答可点击原文查看:https://developer.aliyun.com/ask/500240?spm=a2c6h.14164896.0.0.42a563bfbbfLHp
问题三:使用flinksql mongodb-cdc执行报错怎么办?
使用flinksql mongodb-cdc执行报错unable to create a source for reading table,这个怎么办?
参考回答:
这个错误可能是由于以下原因导致的:
- 数据库或集合名称错误:确保你在SQL查询中使用的数据库和集合名称与实际情况相匹配。
- 连接字符串错误:确保你在配置CDC连接器时正确指定了MongoDB连接字符串,并且已经在Flink SQL中进行了正确的配置。
- 缺少MongoDB驱动程序:如果你没有在Flink的lib目录下放置MongoDB驱动程序,可能会遇到此错误。请确保你已经添加了MongoDB驱动程序,以便FlinkSQL能够与MongoDB数据库建立连接。
- CDC版本不兼容:如果你使用的Flink版本和MongoDB CDC版本不兼容,也可能会出现此错误。请尝试使用兼容的版本进行测试。
你可以通过查看具体的错误日志,排除以上可能的原因,并采取相应的措施解决问题。
关于本问题的更多回答可点击原文查看:https://developer.aliyun.com/ask/494471?spm=a2c6h.14164896.0.0.42a563bfbbfLHp
问题四:启动flinksql,然后随便执行一个sql报错,为什么?
启动flinksql,然后随便执行一个sql报错,为什么?
参考回答:
先看一下43268这个端口有没有打开
关于本问题的更多回答可点击原文查看:https://developer.aliyun.com/ask/482512?spm=a2c6h.14164896.0.0.42a563bfbbfLHp
问题五:大佬们Flink 用SQL API sink到 Doris, 中间不做处理,直接CDC 到Doris. job不报错, Doris表里没数据, 这有可能是什么原因呢?
大佬们Flink 用SQL API sink到 Doris, 中间不做处理,直接CDC 到Doris. job不报错, Doris表里没数据, 这有可能是什么原因呢?
参考回答:
打开doris的严格模式
关于本问题的更多回答可点击原文查看:https://developer.aliyun.com/ask/500428?spm=a2c6h.14164896.0.0.42a563bfbbfLHp