Hadoop Hive与Hbase关系 整合

本文測试步骤主要參考自：http://running.iteye.com/blog/898399 
当然。这边博文也是依照官网的步骤来的：http://wiki.apache.org/hadoop/Hive/HBaseIntegration 
1. 拷贝hbase-0.90.5.jar和zookeeper-3.3.2.jar到hive/lib下。 
注意：怎样hive/lib下已经存在这两个文件的其它版本号（比如zookeeper-3.3.1.jar），建议删除后使用hbase下的相关版本号。 

2. 改动hive/conf下hive-site.xml文件。在底部加入例如以下内容：

[html] view plaincopy
<!--  
<property>  
  <name>hive.exec.scratchdir</name>   
  <value>/usr/local/hive/tmp</value>   

</property>   
-->  
  
<property>   
  <name>hive.querylog.location</name>   
  <value>/usr/local/hive/logs</value>   
</property>   
  
<property>  
  <name>hive.aux.jars.path</name>   
  <value>file:///usr/local/hive/lib/hive-hbase-handler-0.8.0.jar,file:///usr/local/hive/lib/hbase-0.90.5.jar,file:///usr/local/hive/lib/zookeeper-3.3.2.jar</value>  

</property>  
注意：假设hive-site.xml不存在则自行创建，或者把hive-default.xml.template文件改名后使用。 
详细请參见：http://blog.csdn.net/kunshan_shenbin/article/details/7210020 

3. 拷贝hbase-0.90.5.jar到全部hadoop节点(包含master)的hadoop/lib下。

4. 拷贝hbase/conf下的hbase-site.xml文件到全部hadoop节点(包含master)的hadoop/conf下。

注意。hbase-site.xml文件配置信息參照：http://blog.csdn.net/kunshan_shenbin/article/details/7209990
注意，假设3,4两步跳过的话。执行hive时非常可能出现例如以下错误：

[html] view plaincopy
org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to connect to ZooKeeper but the connection closes immediately.   
This could be a sign that the server has too many connections (30 is the default). Consider inspecting your ZK server logs for that error and   
then make sure you are reusing HBaseConfiguration as often as you can. See HTable's javadoc for more information. at org.apache.hadoop.  
hbase.zookeeper.ZooKeeperWatcher. 
參考：http://blog.sina.com.cn/s/blog_410d18710100vlbq.html 

如今能够尝试启动Hive了。 
单节点启动：

集群启动：

> bin/hive -hiveconf hbase.zookeeper.quorum=slave

怎样hive-site.xml文件里没有配置hive.aux.jars.path，则能够依照例如以下方式启动。

> bin/hive --auxpath /usr/local/hive/lib/hive-hbase-handler-0.8.0.jar, /usr/local/hive/lib/hbase-0.90.5.jar, /usr/local/hive/lib/zookeeper-3.3.2.jar -hiveconf hbase.zookeeper.quorum=slave

接下来能够做一些測试了。

1.创建hbase识别的数据库： 
[sql] view plaincopy 
CREATE TABLE hbase_table_1(key int, value string)  
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'  
WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf1:val")  
TBLPROPERTIES ("hbase.table.name" = "xyz");  
hbase.table.name 定义在hbase的table名称 
hbase.columns.mapping 定义在hbase的列族 
2.使用sql导入数据 
a) 新建hive的数据表 
[sql] view plaincopy 
<span><span></span></span>hive> CREATE TABLE pokes (foo INT, bar STRING);  
b) 批量插入数据 
[sql] view plaincopy

hive> LOAD DATA LOCAL INPATH './examples/files/kv1.txt' OVERWRITE INTO TABLE 

pokes;  
c) 使用sql导入hbase_table_1 
[sql] view plaincopy 
hive> INSERT OVERWRITE TABLE hbase_table_1 SELECT * FROM pokes WHERE foo=86;  
3. 查看数据 
[sql] view plaincopy 
hive> select * from  hbase_table_1;  
这时能够登录Hbase去查看数据了. 
> /usr/local/hbase/bin/hbase shell 
hbase(main):001:0> describe 'xyz'   
hbase(main):002:0> scan 'xyz'   
hbase(main):003:0> put 'xyz','100','cf1:val','www.360buy.com' 
这时在Hive中能够看到刚才在Hbase中插入的数据了。

hive> select * from hbase_table_1 
4. hive訪问已经存在的hbase 
使用CREATE EXTERNAL TABLE 
[sql] view plaincopy 
CREATE EXTERNAL TABLE hbase_table_2(key int, value string)  
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'  
WITH SERDEPROPERTIES ("hbase.columns.mapping" = "cf1:val")  
TBLPROPERTIES("hbase.table.name" = "some_existing_table");  


多列和多列族（Multiple Columns and Families） 
1．创建数据库 
Java代码  
CREATE TABLE hbase_table_2(key int, value1 string, value2 int, value3 int)   
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'  
WITH SERDEPROPERTIES (  
"hbase.columns.mapping" = ":key,a:b,a:c,d:e"  
);  

2.插入数据 
Java代码  
INSERT OVERWRITE TABLE hbase_table_2 SELECT foo, bar, foo+1, foo+2   
FROM pokes WHERE foo=98 OR foo=100;  


这个有3个hive的列（value1和value2，value3），2个hbase的列族（a，d） 
Hive的2列（value1和value2）相应1个hbase的列族（a。在hbase的列名称b，c）。hive的另外1列（value3）相应列（e）位于列族（d）

3.登录hbase查看结构 
Java代码 

hbase(main):003:0> describe "hbase_table_2"  
DESCRIPTION                                                             ENABLED                                 
 {NAME => 'hbase_table_2', FAMILIES => [{NAME => 'a', COMPRESSION => 'N true                                    
 ONE', VERSIONS => '3', TTL => '2147483647', BLOCKSIZE => '65536', IN_M                                         
 EMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'd', COMPRESSION =>                                          
 'NONE', VERSIONS => '3', TTL => '2147483647', BLOCKSIZE => '65536', IN                                         
 _MEMORY => 'false', BLOCKCACHE => 'true'}]}                                                                    
1 row(s) in 1.0630 seconds 
4.查看hbase的数据 
Java代码 

hbase(main):004:0> scan 'hbase_table_2'  
ROW                          COLUMN+CELL                                                                        
 100                         column=a:b, timestamp=1297695262015, value=val_100                                 
 100                         column=a:c, timestamp=1297695262015, value=101                                     
 100                         column=d:e, timestamp=1297695262015, value=102                                     
 98                          column=a:b, timestamp=1297695242675, value=val_98                                  
 98                          column=a:c, timestamp=1297695242675, value=99                                      
 98                          column=d:e, timestamp=1297695242675, value=100                                     
2 row(s) in 0.0380 seconds 

5.在hive中查看 
Java代码 

hive> select * from hbase_table_2;  
OK  
100     val_100 101     102  
98      val_98  99      100  
Time taken: 3.238 seconds  
參考资料： 
http://running.iteye.com/blog/898399 
http://heipark.iteye.com/blog/1150648 
http://www.javabloger.com/article/apache-hadoop-hive-hbase-integration.html

本文转自mfrbuaa博客园博客，原文链接：http://www.cnblogs.com/mfrbuaa/p/5094133.html，如需转载请自行联系原作者