一 前言
Bottled Water是Confluent公司开发的一款可以将postgresql数据库转换为kafka events的工具。主要用于监视PG的数据增删改的变化，将变化的数据实时推送到kafka消息队列，其他关心数据变化的平台只要监听kafka的消息，就能得到改变的数据。具体参考德哥的博客《实时数据交换平台 - BottledWater-pg with confluent》，本文具体的安装步骤也参考了《Bottled Water: 实时集成postgresql与kafka》。
二 安装准备
安装完成PostgreSQL，参考《Centos7安装PostgreSQL9.6》；
安装完成Kafka，参考《Kafka部署》；
此外还依赖avro-c，Jansson，libcurl，librdkafka
三 安装步骤bottledwater-pg
3.1 安装libjansson
wget http://www.digip.org/jansson/releases/jansson-2.9.tar.bz2
tar -jxvf jansson-2.9.tar.bz2
cd jansson-2.9
./configure --prefix=/home/free/jansson
make
make install
export PKG_CONFIG_PATH=/home/free/jansson/lib/pkgconfig:$PKG_CONFIG_PATH
pkg-config --cflags --libs jansson
-I/home/free/jansson/include -L/home/free/jansson/lib -ljansson
3.1 安装avro
wget http://mirrors.hust.edu.cn/apache/avro/avro-1.8.1/avro-src-1.8.1.tar.gz
tar -zxvf avro-src-1.8.1.tar.gz
cd avro-src-1.8.1/lang/c
mkdir build
cd build
cmake .. -DCMAKE_INSTALL_PREFIX=/home/free/avro -DCMAKE_BUILD_TYPE=Release -DTHREADSAFE=true
make
make test
make install
export PKG_CONFIG_PATH=/home/free/avro/lib/pkgconfig:$PKG_CONFIG_PATH
在make这一步骤报错如下：
opt/avro-src-1.8.1/lang/c/src/schema.c:31:21: fatal error: jansson.h: No such file or directory
#include "jansson.h"
^
compilation terminated.
make[2]: *** [src/CMakeFiles/avro-shared.dir/schema.o] Error 1
make[1]: *** [src/CMakeFiles/avro-shared.dir/all] Error 2
make: *** [all] Error 2
这一步骤应该是编译缺少了头文件，先加上去，然后重新编译。
export C_INCLUDE_PATH=/home/free/jansson/include:$C_INCLUDE_PATH
3.3 安装libcurl
wget https://curl.haxx.se/download/curl-7.54.0.tar.gz
tar -zxvf curl-7.54.0.tar.gz
cd curl-7.54.0
./configure --prefix=/home/free/curl
make
make install
export PKG_CONFIG_PATH=/home/free/curl/lib/pkgconfig:$PKG_CONFIG_PATH
3.4 安装librdkafka
git clone https://github.com/edenhill/librdkafka
cd librdkafka
./configure --prefix=/home/free/librdkafka
#假如报错如下：
#[root@bogon librdkafka]# ./configure --prefix=/home/freegis/librdkafka
#-bash: ./configure: Permission denied
#赋予执行权限
#[root@bogon librdkafka]# chmod 777 configure
make -j 32
make install
export PKG_CONFIG_PATH=/home/free/librdkafka/lib/pkgconfig:$PKG_CONFIG_PATH
在make的时候，64位的linux会报错如下：
/bin/ld:librdkafka.lds:1: syntax errer in VERSION script
只需要在Makefile.config里面的WITH_LDS=y这一行注释掉重新make就可以了。
3.5 安装bottledwater-pg
在环境变量中新增如下依赖：
vi /etc/profile
#编辑内容如下
export CURLHOME=/home/freegis/curl
PATH=$PATH:$HOME/.local/bin:$HOME/bin:$GCCHOME/bin:$CURLHOME/bin
# wq!保存
source /etc/profile
下载安装文件并编译部署：
git clone https://github.com/confluentinc/bottledwater-pg
cd bottledwater-pg
#启用postgres的环境变量
source /home/postgres/.bashrc
make
make install
make可能报错如下：
ake[1]: Entering directory `/opt/bottledwater-pg-master/kafka'
gcc bottledwater.c -c -std=c99 -D_POSIX_C_SOURCE=200809L -I../client -I../ext -I/home/postgres/include -I/home/postgres/include/server -I/home/freegis/avro/include -DAVRO_1_8 -Wall -Wmissing-prototypes -Wpointer-arith -Wendif-labels -Wmissing-format-attribute -Wformat-security -o bottledwater.o
In file included from table_mapper.h:4:0,
from json.h:5,
from bottledwater.c:2:
registry.h:4:32: fatal error: librdkafka/rdkafka.h: No such file or directory
#include <librdkafka/rdkafka.h>
^
compilation terminated.
这种情况仍然是头文件找不到，为了麻烦，把用到的头文件都指定。
export C_INCLUDE_PATH=/home/free/jansson/include:$C_INCLUDE_PATH
export C_INCLUDE_PATH=/home/free/avro/include:$C_INCLUDE_PATH
export C_INCLUDE_PATH=/home/free/librdkafka/include:$C_INCLUDE_PATH
export C_INCLUDE_PATH=/home/free/curl/include:$C_INCLUDE_PATH
继续编译后报错：
/usr/bin/ld: cannot find -lrdkafka
详细如下：
缺失依赖.png
建立依赖软连接：
ln -s /home/free/librdkafka/lib/librdkafka.so /home/postgres/lib/librdkafka.so
检查安装情况
su - postgres
psql
create extension bottledwater;
如果创建扩展报错如下：
mcsas=# create extension bottledwater;
ERROR: could not load library "/home/postgres/lib/bottledwater.so": libavro.so.23.0.0: cannot open shared object file: No such file or directory
查看下bottledwater.so的依赖：
[postgres@bogon lib]$ ldd bottledwater.so
linux-vdso.so.1 => (0x00007fffc41fe000)
libavro.so.23.0.0 => /home/freegis/avro/lib/libavro.so.23.0.0 (0x00007f1f26769000)
libc.so.6 => /lib64/libc.so.6 (0x00007f1f263a8000)
libjansson.so.4 => /lib64/libjansson.so.4 (0x00007f1f2619c000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f1f25f7f000)
/lib64/ld-linux-x86-64.so.2 (0x00007f1f26bba000)
发现所有的依赖都正确。
后经大神提示，重启数据库，然后执行扩展。
pg_ctl restart
psql
\c mcsas
create extension bottledwater;
CREATE EXTENSION
