问题背景
最近在使用 Create table 备份表 as select * from 源表的方式进行备份数据时,发现备份表是acid表,使用spark sql读取该表的时候,出现如下异常:
they have the following capabilities: CONNECTORREAD,HIVEFULLACIDREAD,HIVEFULLACIDWRITE,HIVEMANAGESTATS,HIVECACHEINVALIDATE,CONNECTORWRITE. This table may be a Hive-managed ACID table, or require some other capability that Spark currently does not implement; at com.quantex.MainRun.main(MainRun.java:61) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:665)
解决方案
既然生成的是一个acid表,那么生成一个非acid表,是不是就可以解决这个问题了呢?
于是查询相关资料发现,在hive中是可以生成非acid表的,即在一个session中设置参数即可解决:
set hive.create.as.insert.only=false; set hive.create.as.acid=false; create table 备份表 as select * from 源表;