PostgreSQL DaaS设计注意 - schema与database的抉择
作者
digoal
日期
2016-10-12
标签
PostgreSQL , DaaS , 模板 , schema , database , apply delay , standby
背景
市面上有一些提供DaaS服务的厂商，例如heroKu，可能有上百万的数据库服务；
又比如提供PaaS平台的服务商，数据库也会有很多，同事这些数据库可能也是模板化的，这些厂商并不一定是为每个客户都新建一个数据库集群来满足数据库的需求。
很有可能是使用数据库或者schema来隔离不同用户的。
1. 例如将模板存储在模板数据库中，根据一个模板数据库创建新的数据库提供服务。
2. 有或者将模板存储在SQL文件中，使用SQL文件构建新的SCHEMA提供服务。
两种方式构建模板的差别
1. 数据库的方式隔离比较彻底，共用的资源较少。
可以实现存储的隔离。
可以实现connection的隔离。
可以实现auth的隔离。
可以实现权限的隔离。
但是数据库与数据库之间是无法直接访问的，需要的话通过DBLINK或者FDW插件，当然也可以应用层面跨库访问。
2. schema的方式，共用资源较多，可以同时操作不同的schema之间的对象，事务都是本地事务。
简单来说是有schema更便捷，但是权限隔离没有使用数据库那么彻底，可以从pg_class等系统表窥探到没有权限的对象的定义。
从生成效率来讲，使用数据库模板的方式会高很多，因为它只需要COPY DIR，产生的REDO很少，也不需要大量的变更元数据。
从删除效率来讲，差别也非常大，删除SCHEMA与创建schema一样，会产生大量的REDO，甚至会导致STANDBY剧烈的延迟，后面会有分析。而删除数据库很快，只产生少量的REDO。
本文将要给大家分析的就是两者在创建和删除时的大幅差异。
环境准备
用到两块PCI-E SSD，分别存放主库和备库。
主库监听5289,备库监听5290
配置
1. postgresql.conf
listen_addresses = '0.0.0.0'
port = 5289
max_connections = 100
unix_socket_directories = '.'
shared_buffers = 8GB
maintenance_work_mem = 512MB
dynamic_shared_memory_type = posix
bgwriter_delay = 10ms
bgwriter_lru_maxpages = 1000
bgwriter_lru_multiplier = 10.0
wal_level = hot_standby
synchronous_commit = off
wal_buffers = 128MB
wal_writer_delay = 10ms
checkpoint_segments = 256
checkpoint_timeout = 30min
checkpoint_completion_target = 0.0001
max_wal_senders = 10
wal_keep_segments = 512
hot_standby = on
max_standby_archive_delay = 300s
max_standby_streaming_delay = 300s
wal_receiver_status_interval = 1s
hot_standby_feedback = on
random_page_cost = 1.0
log_destination = 'csvlog'
logging_collector = on
log_truncate_on_rotation = on
log_checkpoints = on
log_connections = on
log_disconnections = on
log_error_verbosity = verbose
log_timezone = 'PRC'
autovacuum = on
log_autovacuum_min_duration = 0
autovacuum_naptime = 10s
datestyle = 'iso, mdy'
timezone = 'PRC'
lc_messages = 'C'
lc_monetary = 'C'
lc_numeric = 'C'
lc_time = 'C'
default_text_search_config = 'pg_catalog.english'
max_locks_per_transaction = 1000000
2. pg_hba.conf
local all all trust
host all all 127.0.0.1/32 trust
host all all ::1/128 trust
host replication postgres 127.0.0.1/32 trust
3. recovery.done
recovery_target_timeline = 'latest'
standby_mode = on
primary_conninfo = 'host=localhost port=5289 user=postgres'
创建备库
pg_basebackup -D /data01/digoal/pg_root5290 -F p -x -h 127.0.0.1 -p 5289 -U postgres
cd /data01/digoal/pg_root5290
mv recovery.done recovery.conf
vi postgresql.conf
port = 5290
pg_ctl start
准备schema
进入template1数据库，准备schema。
\c template1 postgres
主表建表语句如下，为了让schema尽量大一些，使用这种方法来建立。
create table test(
c0 serial unique check(c0>0),
c1 serial unique check(c1>0),
c2 serial unique check(c2>0),
c3 serial unique check(c3>0),
c4 serial unique check(c4>0),
c5 serial unique check(c5>0),
c6 serial unique check(c6>0),
c7 serial unique check(c7>0),
c8 serial unique check(c8>0),
c9 serial unique check(c9>0),
c10 serial unique check(c10>0),
c11 serial unique check(c11>0),
c12 serial unique check(c12>0),
c13 serial unique check(c13>0),
c14 serial unique check(c14>0),
c15 serial unique check(c15>0),
c16 serial unique check(c16>0),
c17 serial unique check(c17>0),
c18 serial unique check(c18>0),
c19 serial unique check(c19>0),
c20 serial unique check(c20>0),
c21 serial unique check(c21>0),
c22 serial unique check(c22>0),
c23 serial unique check(c23>0),
c24 serial unique check(c24>0),
c25 serial unique check(c25>0),
c26 serial unique check(c26>0),
c27 serial unique check(c27>0),
c28 serial unique check(c28>0),
c29 serial unique check(c29>0),
c30 serial unique check(c30>0),
c31 serial unique check(c31>0),
c32 serial unique check(c32>0),
c33 serial unique check(c33>0),
c34 serial unique check(c34>0),
c35 serial unique check(c35>0),
c36 serial unique check(c36>0),
c37 serial unique check(c37>0),
c38 serial unique check(c38>0),
c39 serial unique check(c39>0),
c40 serial unique check(c40>0),
c41 serial unique check(c41>0),
c42 serial unique check(c42>0),
c43 serial unique check(c43>0),
c44 serial unique check(c44>0),
c45 serial unique check(c45>0),
c46 serial unique check(c46>0),
c47 serial unique check(c47>0),
c48 serial unique check(c48>0),
c49 serial unique check(c49>0),
c50 serial unique check(c50>0),
c51 serial unique check(c51>0),
c52 serial unique check(c52>0),
c53 serial unique check(c53>0),
c54 serial unique check(c54>0),
c55 serial unique check(c55>0),
c56 serial unique check(c56>0),
c57 serial unique check(c57>0),
c58 serial unique check(c58>0),
c59 serial unique check(c59>0),
c60 serial unique check(c60>0),
c61 serial unique check(c61>0),
c62 serial unique check(c62>0),
c63 serial unique check(c63>0),
c64 serial unique check(c64>0),
c65 serial unique check(c65>0),
c66 serial unique check(c66>0),
c67 serial unique check(c67>0),
c68 serial unique check(c68>0),
c69 serial unique check(c69>0),
c70 serial unique check(c70>0),
c71 serial unique check(c71>0),
c72 serial unique check(c72>0),
c73 serial unique check(c73>0),
c74 serial unique check(c74>0),
c75 serial unique check(c75>0),
c76 serial unique check(c76>0),
c77 serial unique check(c77>0),
c78 serial unique check(c78>0),
c79 serial unique check(c79>0),
c80 serial unique check(c80>0),
c81 serial unique check(c81>0),
c82 serial unique check(c82>0),
c83 serial unique check(c83>0),
c84 serial unique check(c84>0),
c85 serial unique check(c85>0),
c86 serial unique check(c86>0),
c87 serial unique check(c87>0),
c88 serial unique check(c88>0),
c89 serial unique check(c89>0),
c90 serial unique check(c90>0),
c91 serial unique check(c91>0),
c92 serial unique check(c92>0),
c93 serial unique check(c93>0),
c94 serial unique check(c94>0),
c95 serial unique check(c95>0),
c96 serial unique check(c96>0),
c97 serial unique check(c97>0),
c98 serial unique check(c98>0),
c99 serial unique check(c99>0)
);
100个字段，每个字段都有一个约束。
在数据库元数据中，也会产生一大批系统记录，例如
每个表至少会新增的元数据(没算序列的，算序列还更多)
pg_class , 101条 (表+索引)
pg_attribute , 106条 (tableoid, cmax, cmin, xmax, xmin, ctid, 字段)
pg_constraint , 200条 (唯一, check各100个)
pg_depend , 401条 (表, 索引+唯一约束+check约束)(索引,唯一约束)
pg_index , 100条
同时还会产生很多数据文件，每个索引，表都会有一个数据文件，如果算上fork(vm, fsm, init)的话，就更多了。
使用test新建500张一样的表，会产生较多的元数据变动，同时会产生一堆数据文件。
do language plpgsql $$
declare
i int ;
begin
for i in 1..500 loop
execute 'create table test'||i||' (like test including all)';
end loop;
end;
$$;
建完表后，template1就变500多MB了。
template1=# \l+
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges | Size | Tablespace | Description
-----------+----------+----------+---------+-------+-----------------------+---------+------------+--------------------------------------------
postgres | postgres | UTF8 | C | C | | 1044 MB | pg_default | default administrative connection database
template0 | postgres | UTF8 | C | C | =c/postgres +| 6681 kB | pg_default | unmodifiable empty database
| | | | | postgres=CTc/postgres | | |
template1 | postgres | UTF8 | C | C | =c/postgres +| 624 MB | pg_default | default template for new databases
| | | | | postgres=CTc/postgres | | |
测试drop schema
以template1为模板创建新数据库
postgres=# create database db0 with template template1;
记录当前XLOG位点
postgres=# select pg_current_xlog_location();
-[ RECORD 1 ]------------+-----------
pg_current_xlog_location | 1/7394D08
删除schema
\c db0
drop schema public cascade;
记录当前XLOG位点
等待drop schema结束，并记录当前XLOG位点（很长一段时间后稳定（autovacuum）结束）
db0=# select pg_current_xlog_location();
-[ RECORD 1 ]------------+-----------
pg_current_xlog_location | 1/168E6EA8
监控延迟
在主库执行
\x
select
pg_size_pretty(pg_xlog_location_diff(pg_current_xlog_location(),sent_location)),
pg_size_pretty(pg_xlog_location_diff(pg_current_xlog_location(),write_location)),
pg_size_pretty(pg_xlog_location_diff(pg_current_xlog_location(),flush_location)),
pg_size_pretty(pg_xlog_location_diff(pg_current_xlog_location(),replay_location)),
* from pg_stat_replication ;
\watch 1
发现备库apply卡在一个REDO REC上很久，如果接下来主库又产生了大量的REDO，那么备库的apply就会延迟严重。
主机REDO发送是没有延迟的，也就是说REDO已经在备机那里了，但是还没有被apply。
-[ RECORD 1 ]----+------------------------------
pg_size_pretty | 0 bytes
pg_size_pretty | 0 bytes
pg_size_pretty | 0 bytes
pg_size_pretty | 56 MB -- 出现apply延迟
pid | 27375
usesysid | 10
usename | postgres
application_name | walreceiver
client_addr | 127.0.0.1
client_hostname |
client_port | 53164
backend_start | 2016-10-12 10:17:16.414473+08
backend_xmin | 2030
state | streaming
sent_location | 1/168E6EA8
write_location | 1/168E6EA8
flush_location | 1/168E6EA8
replay_location | 1/13151E28 -- 卡住
sync_priority | 0
sync_state | async
备机apply延迟严重的话，另外一个问题就是备机的xlog会占用较大的空间。
延迟分析
使用pg_xlogdump分析 "堵塞" apply的redo rec
pg_xlogdump -b 000000010000000100000013 000000010000000100000014 2>&1 |less
搜索1/13151E28
rmgr: Transaction len (rec/tot): 17680828/17680860, tx: 2029, lsn: 1/13151E28, prev 1/13151930, bkp: 0000, desc: commit: 2016-10-12 17:04:39.615288 CST; rels:
大量的文件位置
base/400932/199021 base/400932/199422 base/400932/199019 base/400932/199420 base/400932/199017 base/400932/199418 base/400932/199015 base/400932/199416 base/400932/199013
base/400932/199414 base/400932/199011 base/400932/199412 base/400932/199009 base/400932/199410
base/400932/199007 base/400932/199408 base/400932/199005 base/400932/199406 base/400932/199003 base/400932/199404 base/400932/199001 base/400932/199402 base/400932/198999
........
........
lcache 400523 snapshot 2608 relcache 400523 snapshot 2608 snapshot 2608 relcache 400730 relcache 400523 snapshot 2608 relcache 400523 snapshot 2608 relcache 400523 snapshot 2608 snapshot 2608 relcache 400728 relcache 400523 snapshot 2608
relcache 400523 snapshot 2608 relcache 400523 snapshot 2608 snapshot 2608 relcache 400726 relcache 400523 snapshot 2608 snapshot 2608 snapshot 2608
rmgr: Standby len (rec/tot): 24/ 56, tx: 0, lsn: 1/1423B310, prev 1/13151E28, bkp: 0000, desc: running xacts: nextXid 2030 latestCompletedXid 2029 oldestRunningXid 2030
这笔redo很大，十几MB
db0=# select pg_xlog_location_diff('1/1423B310', '1/13151E28');
-[ RECORD 1 ]---------+---------
pg_xlog_location_diff | 17732840
备库apply卡住的地方，跟踪备库startup进程(用于recovery的进程)在干什么
strace -p $pid
一堆的unlink
unlink("base/400932/307422") = 0
unlink("base/400932/307422.1") = -1 ENOENT (No such file or directory)
unlink("base/400932/307422_fsm") = -1 ENOENT (No such file or directory)
unlink("base/400932/307422_vm") = -1 ENOENT (No such file or directory)
unlink("base/400932/307422_init") = -1 ENOENT (No such file or directory)
unlink("base/400932/307420") = 0
unlink("base/400932/307420.1") = -1 ENOENT (No such file or directory)
unlink("base/400932/307420_fsm") = -1 ENOENT (No such file or directory)
unlink("base/400932/307420_vm") = -1 ENOENT (No such file or directory)
unlink("base/400932/307420_init") = -1 ENOENT (No such file or directory)
unlink("base/400932/307418") = 0
unlink("base/400932/307418.1") = -1 ENOENT (No such file or directory)
unlink("base/400932/307418_fsm") = -1 ENOENT (No such file or directory)
unlink("base/400932/307418_vm") = -1 ENOENT (No such file or directory)
unlink("base/400932/307418_init") = -1 ENOENT (No such file or directory)
unlink("base/400932/307416") = 0
unlink("base/400932/307416.1") = -1 ENOENT (No such file or directory)
unlink("base/400932/307416_fsm") = -1 ENOENT (No such file or directory)
unlink("base/400932/307416_vm") = -1 ENOENT (No such file or directory)
unlink("base/400932/307416_init") = -1 ENOENT (No such file or directory)
....
还有很多
查看一下template1下面有多少个文件，（200多个是系统自带的一些元表的数据文件）有50954多个文件。
unlink这些文件至少也要耗费10几分钟。
小结
1. drop schema 产生了多少redo
本例的测试用例，约17MB的REDO。
2. 为什么drop schema会导致standby apply的延迟严重
大量的文件操作，导致了apply的延迟。
测试基于database的DaaS
记录当前XLOG位点
postgres=# select pg_current_xlog_location();
pg_current_xlog_location
--------------------------
1/168EE5F8
(1 row)
以template1为模板创建新数据库
postgres=# create database db0 with template template1;
记录当前XLOG位点
postgres=# select pg_current_xlog_location();
pg_current_xlog_location
--------------------------
1/168F0640
(1 row)
创建数据库产生了多少REDO
postgres=# select pg_xlog_location_diff('1/168F0640', '1/168EE5F8');
-[ RECORD 1 ]---------+-----
pg_xlog_location_diff | 8264
删除database
postgres=# drop database db0;
DROP DATABASE
记录当前XLOG位点
postgres=# select pg_current_xlog_location();
pg_current_xlog_location
--------------------------
1/168F20E0
(1 row)
drop数据库产生了多少REDO
postgres=# select pg_xlog_location_diff('1/168F20E0','1/168F0640');
-[ RECORD 1 ]---------+-----
pg_xlog_location_diff | 6816
监控延迟
在主库执行
select
pg_size_pretty(pg_xlog_location_diff(pg_current_xlog_location(),sent_location)),
pg_size_pretty(pg_xlog_location_diff(pg_current_xlog_location(),write_location)),
pg_size_pretty(pg_xlog_location_diff(pg_current_xlog_location(),flush_location)),
pg_size_pretty(pg_xlog_location_diff(pg_current_xlog_location(),replay_location)),
* from pg_stat_replication ;
\watch 1
未发现延迟
-[ RECORD 1 ]----+------------------------------
pg_size_pretty | 0 bytes
pg_size_pretty | 0 bytes
pg_size_pretty | 0 bytes
pg_size_pretty | 0 bytes
pid | 27375
usesysid | 10
usename | postgres
application_name | walreceiver
client_addr | 127.0.0.1
client_hostname |
client_port | 53164
backend_start | 2016-10-12 10:17:16.414473+08
backend_xmin | 2046
state | streaming
sent_location | 1/168F20E0
write_location | 1/168F20E0
flush_location | 1/168F20E0
replay_location | 1/168F20E0
sync_priority | 0
sync_state | async
xlogdump分析
分析一下create 和 drop database产生的redo内容
pg_xlogdump -b 000000010000000100000016 000000010000000100000016 2>&1 |less
分析从1/168EE5F8到1/168F20E0的内容全部如下
rmgr: Standby len (rec/tot): 24/ 56, tx: 0, lsn: 1/168EE5F8, prev 1/168EE5A8, bkp: 0000, desc: running xacts: nextXid 2044 latestCompletedXid 2043 oldestRunningXid 2044
rmgr: Heap len (rec/tot): 21/ 6437, tx: 2044, lsn: 1/168EE630, prev 1/168EE5F8, bkp: 1000, desc: insert: rel 1664/0/12999; tid 0/24
backup bkp #0; rel 1664/0/12999; fork: main; block: 0; hole: offset: 120, length: 1832
rmgr: Btree len (rec/tot): 18/ 618, tx: 2044, lsn: 1/168EFF58, prev 1/168EE630, bkp: 1000, desc: insert: rel 1664/0/13001; tid 1/1
backup bkp #0; rel 1664/0/13001; fork: main; block: 1; hole: offset: 120, length: 7648
rmgr: Btree len (rec/tot): 18/ 594, tx: 2044, lsn: 1/168F01E0, prev 1/168EFF58, bkp: 1000, desc: insert: rel 1664/0/13002; tid 1/24
backup bkp #0; rel 1664/0/13002; fork: main; block: 1; hole: offset: 120, length: 7672
rmgr: Standby len (rec/tot): 28/ 60, tx: 0, lsn: 1/168F0438, prev 1/168F01E0, bkp: 0000, desc: running xacts: nextXid 2045 latestCompletedXid 2043 oldestRunningXid 2044; 1 xacts: 2044
rmgr: XLOG len (rec/tot): 72/ 104, tx: 0, lsn: 1/168F0478, prev 1/168F0438, bkp: 0000, desc: checkpoint: redo 1/168F0438; tli 1; prev tli 1; fpw true; xid 0/2045; oid 401408; multi 1; offset 0; oldest xid 1798 in DB 1; oldest multi 1 in DB 1; oldest running xid 2044; online
rmgr: Database len (rec/tot): 16/ 48, tx: 2044, lsn: 1/168F04E0, prev 1/168F0478, bkp: 0000, desc: create db: copy dir 1/1663 to 400934/1663
rmgr: Standby len (rec/tot): 28/ 60, tx: 0, lsn: 1/168F0510, prev 1/168F04E0, bkp: 0000, desc: running xacts: nextXid 2045 latestCompletedXid 2043 oldestRunningXid 2044; 1 xacts: 2044
rmgr: XLOG len (rec/tot): 72/ 104, tx: 0, lsn: 1/168F0550, prev 1/168F0510, bkp: 0000, desc: checkpoint: redo 1/168F0510; tli 1; prev tli 1; fpw true; xid 0/2045; oid 401408; multi 1; offset 0; oldest xid 1798 in DB 1; oldest multi 1 in DB 1; oldest running xid 2044; online
rmgr: Transaction len (rec/tot): 48/ 80, tx: 2044, lsn: 1/168F05B8, prev 1/168F0550, bkp: 0000, desc: commit: 2016-10-12 19:17:16.791771 CST; inval msgs: catcache 21
rmgr: Standby len (rec/tot): 24/ 56, tx: 0, lsn: 1/168F0608, prev 1/168F05B8, bkp: 0000, desc: running xacts: nextXid 2045 latestCompletedXid 2044 oldestRunningXid 2045
rmgr: Heap len (rec/tot): 26/ 6442, tx: 2045, lsn: 1/168F0640, prev 1/168F0608, bkp: 1000, desc: delete: rel 1664/0/12999; tid 0/24 KEYS_UPDATED
backup bkp #0; rel 1664/0/12999; fork: main; block: 0; hole: offset: 120, length: 1832
rmgr: Standby len (rec/tot): 28/ 60, tx: 0, lsn: 1/168F1F70, prev 1/168F0640, bkp: 0000, desc: running xacts: nextXid 2046 latestCompletedXid 2044 oldestRunningXid 2045; 1 xacts: 2045
rmgr: XLOG len (rec/tot): 72/ 104, tx: 0, lsn: 1/168F1FB0, prev 1/168F1F70, bkp: 0000, desc: checkpoint: redo 1/168F1F70; tli 1; prev tli 1; fpw true; xid 0/2046; oid 401408; multi 1; offset 0; oldest xid 1798 in DB 1; oldest multi 1 in DB 1; oldest running xid 2045; online
rmgr: Database len (rec/tot): 8/ 40, tx: 2045, lsn: 1/168F2030, prev 1/168F1FB0, bkp: 0000, desc: drop db: dir 400934/1663
rmgr: Transaction len (rec/tot): 48/ 80, tx: 2045, lsn: 1/168F2058, prev 1/168F2030, bkp: 0000, desc: commit: 2016-10-12 19:17:30.981401 CST; inval msgs: catcache 21
rmgr: Standby len (rec/tot): 24/ 56, tx: 0, lsn: 1/168F20A8, prev 1/168F2058, bkp: 0000, desc: running xacts: nextXid 2046 latestCompletedXid 2045 oldestRunningXid 2046
create 和 drop database并没有产生很多的日志，也没有那么多的文件操作。只有copy dir和drop dir。
文件操作少了，比drop schema快多了。
小结
1. schema和database在物理结构上的差别
database是以目录的形式组织在表空间的目录下的，而schema是以文件的形式在数据库的目录下的，没有再细分独立的目录。
所以在drop database时系统调用变得更简单，而drop schema需要挨个文件来。
2. schema和database在元数据上的差别
简单来说就是比擦屁股的动作, drop database擦屁股很快，因为元数据很少只影响pg_databases。
drop schema擦屁股就很烦了，要挨个清理pg_class, pg_attribute, 等等元表。 元表清理完还需要vacuum。
3. create 和 drop schema的文件操作很多，是一个个文件进行的，而且都会记录在REDO中，如果schema中有很对对象并且有很多文件的话，会非常慢。
4. create 和 drop database产生的日志少，系统调用也更少。
schema不建议作为daas的模板环境频繁(新增和删除时)使用，如果要频繁的创建和删除模板，建议使用database作为模板。
database作为模板的一个缺点是连接复用的问题，因为连接复用需要基于user+database，如果有很多DB的话，连接可能会消耗很多。
优化点
把schema放到database下，新增一个目录存放。删除的时候可以drop dir，但是清理元数据还是少不了的。
schema与其他schema之间的一些依赖关系也需要清理(可能涉及元数据的清理)。
