暂无个人介绍
set hive.compute.query.using.stats=false ; 然后再跑一下。
"When set to true Hive will answer a few queries like count(1) purely using stats\n" +
"stored in metastore. For basic stats collection turn on the config hive.stats.autogather to true.\n" +
"For more advanced stats collection need to run analyze table queries."
这个比较hack了
SparkSQL本身是支持StorageHandler,需要提供相关jar包。
访问HiveOnHBase需要如下jar包:
/usr/lib/hbase-current/lib/hbase-server-1.1.1.jar
/usr/lib/hbase-current/lib/hbase-common-1.1.1.jar
/usr/lib/hbase-current/lib/hbase-client-1.1.1.jar
/usr/lib/hbase-current/lib/hbase-protocol-1.1.1.jar
/usr/lib/hive-current/lib/hive-hbase-handler-2.3.3.jar
需要将上述jar包添加到spark,有两种方式:
a)通过--jars参数来添加
如:spark-sql --jars /usr/lib/hbase-current/lib/hbase-server-1.1.1.jar,/usr/lib/hbase-current/lib/hbase-common-1.1.1.jar,/usr/lib/hbase-current/lib/hbase-client-1.1.1.jar,/usr/lib/hbase-current/lib/hbase-protocol-1.1.1.jar,/usr/lib/hive-current/lib/hive-hbase-handler-2.3.3.jar
b)spark-defaults.conf里面配置
spark.executor.extraClassPath /opt/apps/extra-jars/*:/usr/lib/hbase-current/lib/hbase-server-1.1.1.jar:/usr/lib/hive-current/lib/hive-hbase-handler-2.3.3.jar:/usr/lib/hbase-current/lib/hbase-common-1.1.1.jar:/usr/lib/hbase-current/lib/hbase-client-1.1.1.jar:/usr/lib/hbase-current/lib/hbase-protocol-1.1.1.jar
spark.driver.extraClassPath /opt/apps/extra-jars/*:/usr/lib/hbase-current/lib/hbase-server-1.1.1.jar:/usr/lib/hive-current/lib/hive-hbase-handler-2.3.3.jar:/usr/lib/hbase-current/lib/hbase-common-1.1.1.jar:/usr/lib/hbase-current/lib/hbase-client-1.1.1.jar:/usr/lib/hbase-current/lib/hbase-protocol-1.1.1.jar
备注:
EMR-3.13.0以及以下版本,使用SparkSQL insert 数据到HiveOnHBase表的时候会出异常:
java.lang.ClassCastException: org.apache.hadoop.hive.hbase.HiveHBaseTableOutputFormat cannot be cast to org.apache.hadoop.hive.ql.io.HiveOutputFormat
您好 您在hive-site.xml里面设置了engine为tez
然后/etc/ecm/spark-conf/下面有个hive-site.xml软链接,直接链接到了/etc/ecm/hive-conf/hive-site.xml
所以可以这样
你把/etc/ecm/spark-conf/下面的hive-site.xml软链接删除,然后从/etc/ecm/hive-conf/hive-site.xml拷贝一份过来,然后把拷贝过来的这个里面hive.execution.engine改成mr