尚书中云：惟事事，乃其有备，有备无患。这教导我们做事一定要有准备，做事尚且如此，在企事业单位发展中处于基础地位的数据仓库软件在运行过程中，何尝不需要有备无患呢？
今天别的不表，主要来谈谈企业级数据仓库软件Deepgreen和Greenplum的高可用特性之一：计算节点镜像。
一、首先从理论上来讲，正常Segment节点和他的Mirror是分布在不同主机上的，以防止单点故障导致的数据库访问异常。当正常Segment节点出现故障时，Mirror节点可以自动接管Segment节点的服务，数据库仍然可以正常使用。这个过程对前台应用来说是透明的。
下面我们来看实操例子：
1.测试环境
本人笔记本电脑；
4核心，8G内存，1T硬盘
已安装1个主节点和2个计算节点，无镜像
2.首先查看集群状态：1 Master，2 Segments
## 执行命令：gpstate
20170628:06:26:51:003776 gpstart:flash:dgadmin-[INFO]:---------------------------
20170628:06:26:51:003776 gpstart:flash:dgadmin-[INFO]:-Master instance parameters
20170628:06:26:51:003776 gpstart:flash:dgadmin-[INFO]:---------------------------
20170628:06:26:51:003776 gpstart:flash:dgadmin-[INFO]:-Database = template1
20170628:06:26:51:003776 gpstart:flash:dgadmin-[INFO]:-Master Port = 15432
20170628:06:26:51:003776 gpstart:flash:dgadmin-[INFO]:-Master directory = /dgdata/master/dg-1
20170628:06:26:51:003776 gpstart:flash:dgadmin-[INFO]:-Timeout = 600 seconds
20170628:06:26:51:003776 gpstart:flash:dgadmin-[INFO]:-Master standby = Off
20170628:06:26:51:003776 gpstart:flash:dgadmin-[INFO]:---------------------------------------
20170628:06:26:51:003776 gpstart:flash:dgadmin-[INFO]:-Segment instances that will be started
20170628:06:26:51:003776 gpstart:flash:dgadmin-[INFO]:---------------------------------------
20170628:06:26:51:003776 gpstart:flash:dgadmin-[INFO]:- Host Datadir Port
20170628:06:26:51:003776 gpstart:flash:dgadmin-[INFO]:- flash /dgdata/primary/dg0 25432
20170628:06:26:51:003776 gpstart:flash:dgadmin-[INFO]:- flash /dgdata/primary/dg1 25433
3.创建镜像节点目录并添加节点
## 创建mirror目录
dgadmin@flash:~$ mkdir /dgdata/mirror## 执行命令：gpaddmirrors添加节点
dgadmin@flash:~$ gpaddmirrors
20170628:06:41:50:004294 gpaddmirrors:flash:dgadmin-[INFO]:-Starting gpaddmirrors with args:
20170628:06:41:50:004294 gpaddmirrors:flash:dgadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 4.3.99.00 build Deepgreen DB'
20170628:06:41:50:004294 gpaddmirrors:flash:dgadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.2.15 (Greenplum Database 4.3.99.00 build Deepgreen DB) on x86_64-unknown-linux-gnu, compiled by GCC gcc (Ubuntu 4.9.4-2ubuntu1~14.04.1) 4.9.4 compiled on May 18 2017 05:19:19'
20170628:06:41:50:004294 gpaddmirrors:flash:dgadmin-[INFO]:-Obtaining Segment details from master...
Enter mirror segment data directory location 1 of 2 >
/dgdata/mirror
Enter mirror segment data directory location 2 of 2 >
/dgdata/mirror
20170628:06:42:03:004294 gpaddmirrors:flash:dgadmin-[INFO]:-Greenplum Add Mirrors Parameters
20170628:06:42:03:004294 gpaddmirrors:flash:dgadmin-[INFO]:----------------------------------------------------------
20170628:06:42:03:004294 gpaddmirrors:flash:dgadmin-[INFO]:-Greenplum master data directory = /dgdata/master/dg-1
20170628:06:42:03:004294 gpaddmirrors:flash:dgadmin-[INFO]:-Greenplum master port = 15432
20170628:06:42:03:004294 gpaddmirrors:flash:dgadmin-[INFO]:-Parallel batch limit = 16
20170628:06:42:03:004294 gpaddmirrors:flash:dgadmin-[INFO]:----------------------------------------------------------
20170628:06:42:03:004294 gpaddmirrors:flash:dgadmin-[INFO]:-Mirror 1 of 2
20170628:06:42:03:004294 gpaddmirrors:flash:dgadmin-[INFO]:----------------------------------------------------------
20170628:06:42:03:004294 gpaddmirrors:flash:dgadmin-[INFO]:- Primary instance host = flash
20170628:06:42:03:004294 gpaddmirrors:flash:dgadmin-[INFO]:- Primary instance address = flash
20170628:06:42:03:004294 gpaddmirrors:flash:dgadmin-[INFO]:- Primary instance directory = /dgdata/primary/dg0
20170628:06:42:03:004294 gpaddmirrors:flash:dgadmin-[INFO]:- Primary instance port = 25432
20170628:06:42:03:004294 gpaddmirrors:flash:dgadmin-[INFO]:- Primary instance replication port = 28432
20170628:06:42:03:004294 gpaddmirrors:flash:dgadmin-[INFO]:- Mirror instance host = flash
20170628:06:42:03:004294 gpaddmirrors:flash:dgadmin-[INFO]:- Mirror instance address = flash
20170628:06:42:03:004294 gpaddmirrors:flash:dgadmin-[INFO]:- Mirror instance directory = /dgdata/mirror/dg0
20170628:06:42:03:004294 gpaddmirrors:flash:dgadmin-[INFO]:- Mirror instance port = 26432
20170628:06:42:03:004294 gpaddmirrors:flash:dgadmin-[INFO]:- Mirror instance replication port = 27432
20170628:06:42:03:004294 gpaddmirrors:flash:dgadmin-[INFO]:----------------------------------------------------------
20170628:06:42:03:004294 gpaddmirrors:flash:dgadmin-[INFO]:-Mirror 2 of 2
20170628:06:42:03:004294 gpaddmirrors:flash:dgadmin-[INFO]:----------------------------------------------------------
20170628:06:42:03:004294 gpaddmirrors:flash:dgadmin-[INFO]:- Primary instance host = flash
20170628:06:42:03:004294 gpaddmirrors:flash:dgadmin-[INFO]:- Primary instance address = flash
20170628:06:42:03:004294 gpaddmirrors:flash:dgadmin-[INFO]:- Primary instance directory = /dgdata/primary/dg1
20170628:06:42:03:004294 gpaddmirrors:flash:dgadmin-[INFO]:- Primary instance port = 25433
20170628:06:42:03:004294 gpaddmirrors:flash:dgadmin-[INFO]:- Primary instance replication port = 28433
20170628:06:42:03:004294 gpaddmirrors:flash:dgadmin-[INFO]:- Mirror instance host = flash
20170628:06:42:03:004294 gpaddmirrors:flash:dgadmin-[INFO]:- Mirror instance address = flash
20170628:06:42:03:004294 gpaddmirrors:flash:dgadmin-[INFO]:- Mirror instance directory = /dgdata/mirror/dg1
20170628:06:42:03:004294 gpaddmirrors:flash:dgadmin-[INFO]:- Mirror instance port = 26433
20170628:06:42:03:004294 gpaddmirrors:flash:dgadmin-[INFO]:- Mirror instance replication port = 27433
20170628:06:42:03:004294 gpaddmirrors:flash:dgadmin-[INFO]:----------------------------------------------------------
Continue with add mirrors procedure Yy|Nn (default=N):
> y
20170628:06:42:06:004294 gpaddmirrors:flash:dgadmin-[INFO]:-2 segment(s) to add
20170628:06:42:06:004294 gpaddmirrors:flash:dgadmin-[INFO]:-Building template directory
20170628:06:42:07:004294 gpaddmirrors:flash:dgadmin-[INFO]:-Validating remote directories
......
20170628:06:42:13:004294 gpaddmirrors:flash:dgadmin-[INFO]:-Copying template directory file
.
20170628:06:42:14:004294 gpaddmirrors:flash:dgadmin-[INFO]:-Configuring new segments
......
20170628:06:42:20:004294 gpaddmirrors:flash:dgadmin-[INFO]:-Cleaning files
.
20170628:06:42:21:004294 gpaddmirrors:flash:dgadmin-[INFO]:-Starting file move procedure for flash:/dgdata/mirror/dg0:content=0:dbid=4:mode=r:status=u
20170628:06:42:21:004294 gpaddmirrors:flash:dgadmin-[INFO]:-Starting file move procedure for flash:/dgdata/mirror/dg1:content=1:dbid=5:mode=r:status=u
20170628:06:42:21:004294 gpaddmirrors:flash:dgadmin-[INFO]:-Updating configuration with new mirrors
20170628:06:42:22:004294 gpaddmirrors:flash:dgadmin-[INFO]:-Updating mirrors
......
20170628:06:42:28:004294 gpaddmirrors:flash:dgadmin-[INFO]:-Starting mirrors
20170628:06:42:28:004294 gpaddmirrors:flash:dgadmin-[INFO]:-Commencing parallel primary and mirror segment instance startup, please wait...
.......
20170628:06:42:35:004294 gpaddmirrors:flash:dgadmin-[INFO]:-Process results...
20170628:06:42:35:004294 gpaddmirrors:flash:dgadmin-[INFO]:-Updating configuration to mark mirrors up
20170628:06:42:35:004294 gpaddmirrors:flash:dgadmin-[INFO]:-Updating primaries
20170628:06:42:35:004294 gpaddmirrors:flash:dgadmin-[INFO]:-Commencing parallel primary conversion of 2 segments, please wait...
......
20170628:06:42:41:004294 gpaddmirrors:flash:dgadmin-[INFO]:-Process results...
20170628:06:42:41:004294 gpaddmirrors:flash:dgadmin-[INFO]:-Done updating primaries
20170628:06:42:41:004294 gpaddmirrors:flash:dgadmin-[INFO]:-******************************************************************
20170628:06:42:41:004294 gpaddmirrors:flash:dgadmin-[INFO]:-Mirror segments have been added; data synchronization is in progress.
20170628:06:42:41:004294 gpaddmirrors:flash:dgadmin-[INFO]:-Data synchronization will continue in the background.
20170628:06:42:41:004294 gpaddmirrors:flash:dgadmin-[INFO]:-
20170628:06:42:41:004294 gpaddmirrors:flash:dgadmin-[INFO]:-Use gpstate -s to check the resynchronization progress.
20170628:06:42:41:004294 gpaddmirrors:flash:dgadmin-[INFO]:-*****************************************************************
4.重新查看集群状态：确定2个Segments都已经添加好Mirror
dgadmin@flash:~$ gpstate
20170628:06:59:19:005589 gpstate:flash:dgadmin-[INFO]:-Starting gpstate with args:
20170628:06:59:19:005589 gpstate:flash:dgadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 4.3.99.00 build Deepgreen DB'
20170628:06:59:20:005589 gpstate:flash:dgadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.2.15 (Greenplum Database 4.3.99.00 build Deepgreen DB) on x86_64-unknown-linux-gnu, compiled by GCC gcc (Ubuntu 4.9.4-2ubuntu1~14.04.1) 4.9.4 compiled on May 18 2017 05:19:19'
20170628:06:59:20:005589 gpstate:flash:dgadmin-[INFO]:-Obtaining Segment details from master...
20170628:06:59:20:005589 gpstate:flash:dgadmin-[INFO]:-Gathering data from segments...
....
20170628:06:59:24:005589 gpstate:flash:dgadmin-[INFO]:-Greenplum instance status summary
20170628:06:59:24:005589 gpstate:flash:dgadmin-[INFO]:-----------------------------------------------------
20170628:06:59:24:005589 gpstate:flash:dgadmin-[INFO]:- Master instance = Active
20170628:06:59:24:005589 gpstate:flash:dgadmin-[INFO]:- Master standby = No master standby configured
20170628:06:59:24:005589 gpstate:flash:dgadmin-[INFO]:- Total segment instance count from metadata = 4
20170628:06:59:24:005589 gpstate:flash:dgadmin-[INFO]:-----------------------------------------------------
20170628:06:59:24:005589 gpstate:flash:dgadmin-[INFO]:- Primary Segment Status
20170628:06:59:24:005589 gpstate:flash:dgadmin-[INFO]:-----------------------------------------------------
20170628:06:59:24:005589 gpstate:flash:dgadmin-[INFO]:- Total primary segments = 2
20170628:06:59:24:005589 gpstate:flash:dgadmin-[INFO]:- Total primary segment valid (at master) = 2
20170628:06:59:24:005589 gpstate:flash:dgadmin-[INFO]:- Total primary segment failures (at master) = 0
20170628:06:59:24:005589 gpstate:flash:dgadmin-[INFO]:- Total number of postmaster.pid files missing = 0
20170628:06:59:24:005589 gpstate:flash:dgadmin-[INFO]:- Total number of postmaster.pid files found = 2
20170628:06:59:24:005589 gpstate:flash:dgadmin-[INFO]:- Total number of postmaster.pid PIDs missing = 0
20170628:06:59:24:005589 gpstate:flash:dgadmin-[INFO]:- Total number of postmaster.pid PIDs found = 2
20170628:06:59:24:005589 gpstate:flash:dgadmin-[INFO]:- Total number of /tmp lock files missing = 0
20170628:06:59:24:005589 gpstate:flash:dgadmin-[INFO]:- Total number of /tmp lock files found = 2
20170628:06:59:24:005589 gpstate:flash:dgadmin-[INFO]:- Total number postmaster processes missing = 0
20170628:06:59:24:005589 gpstate:flash:dgadmin-[INFO]:- Total number postmaster processes found = 2
20170628:06:59:24:005589 gpstate:flash:dgadmin-[INFO]:-----------------------------------------------------
20170628:06:59:24:005589 gpstate:flash:dgadmin-[INFO]:- Mirror Segment Status
20170628:06:59:24:005589 gpstate:flash:dgadmin-[INFO]:-----------------------------------------------------
20170628:06:59:24:005589 gpstate:flash:dgadmin-[INFO]:- Total mirror segments = 2
20170628:06:59:24:005589 gpstate:flash:dgadmin-[INFO]:- Total mirror segment valid (at master) = 2
20170628:06:59:24:005589 gpstate:flash:dgadmin-[INFO]:- Total mirror segment failures (at master) = 0
20170628:06:59:24:005589 gpstate:flash:dgadmin-[INFO]:- Total number of postmaster.pid files missing = 0
20170628:06:59:24:005589 gpstate:flash:dgadmin-[INFO]:- Total number of postmaster.pid files found = 2
20170628:06:59:24:005589 gpstate:flash:dgadmin-[INFO]:- Total number of postmaster.pid PIDs missing = 0
20170628:06:59:24:005589 gpstate:flash:dgadmin-[INFO]:- Total number of postmaster.pid PIDs found = 2
20170628:06:59:24:005589 gpstate:flash:dgadmin-[INFO]:- Total number of /tmp lock files missing = 0
20170628:06:59:24:005589 gpstate:flash:dgadmin-[INFO]:- Total number of /tmp lock files found = 2
20170628:06:59:24:005589 gpstate:flash:dgadmin-[INFO]:- Total number postmaster processes missing = 0
20170628:06:59:24:005589 gpstate:flash:dgadmin-[INFO]:- Total number postmaster processes found = 2
20170628:06:59:24:005589 gpstate:flash:dgadmin-[INFO]:- Total number mirror segments acting as primary segments = 0
20170628:06:59:24:005589 gpstate:flash:dgadmin-[INFO]:- Total number mirror segments acting as mirror segments = 2
20170628:06:59:24:005589 gpstate:flash:dgadmin-[INFO]:-----------------------------------------------------
dgadmin@flash:~$
5.登录数据库进行查询测试
dgadmin@flash:~$ psql -d postgres
psql (8.2.15)
Type "help" for help.
postgres=# \c
You are now connected to database "postgres" as user "dgadmin".
postgres=# \l
List of databases
Name | Owner | Encoding | Access privileges
-----------+---------+----------+---------------------
postgres | dgadmin | UTF8 |
template0 | dgadmin | UTF8 | =c/dgadmin
: dgadmin=CTc/dgadmin
template1 | dgadmin | UTF8 | =c/dgadmin
: dgadmin=CTc/dgadmin
tpch | dgadmin | UTF8 |
tpch_1g | dgadmin | UTF8 |
(5 rows)
postgres=# \c tpch
You are now connected to database "tpch" as user "dgadmin".
tpch=# \dt
List of relations
Schema | Name | Type | Owner | Storage
--------+----------+-------+---------+----------------------
public | customer | table | dgadmin | append only columnar
public | lineitem | table | dgadmin | append only columnar
public | nation | table | dgadmin | append only columnar
public | orders | table | dgadmin | append only columnar
public | part | table | dgadmin | append only columnar
public | partsupp | table | dgadmin | append only columnar
public | region | table | dgadmin | append only columnar
public | supplier | table | dgadmin | append only columnar
(8 rows)
tpch=# select * from region;
r_regionkey | r_name | r_comment
-------------+---------------------------+---------------------------------------------------------------------------------------------------------------------
1 | AMERICA | hs use ironic, even requests. s
3 | EUROPE | ly final courts cajole furiously final excuse
0 | AFRICA | lar deposits. blithely final packages cajole. regular waters are final requests. regular accounts are according to
2 | ASIA | ges. thinly even pinto beans ca
4 | MIDDLE EAST | uickly special accounts cajole carefully blithely close requests. carefully final asymptotes haggle furiousl
(5 rows)
6.模拟实验：Segment1实例异常
## 查询dg1的后台进程号：3898
dgadmin@flash:~$ ps -ef | grep postgres
postgres 1089 1 0 Jun27 ? 00:00:01 /usr/lib/postgresql/9.3/bin/postgres -D /var/lib/postgresql/9.3/main -c config_file=/etc/postgresql/9.3/main/postgresql.conf
postgres 1099 1089 0 Jun27 ? 00:00:00 postgres: checkpointer process
postgres 1100 1089 0 Jun27 ? 00:00:00 postgres: writer process
postgres 1101 1089 0 Jun27 ? 00:00:00 postgres: wal writer process
postgres 1102 1089 0 Jun27 ? 00:00:00 postgres: autovacuum launcher process
postgres 1103 1089 0 Jun27 ? 00:00:02 postgres: stats collector process
dgadmin 3898 1 0 06:26 ? 00:00:00 /home/dgadmin/vitesse.deepgreendb.16.17.170518/bin/postgres -D /dgdata/primary/dg1 -p 25433 -b 3 -z 2 --silent-mode=true -i -M mirrorless -C 1
dgadmin 3899 1 0 06:26 ? 00:00:00 /home/dgadmin/vitesse.deepgreendb.16.17.170518/bin/postgres -D /dgdata/primary/dg0 -p 25432 -b 2 -z 2 --silent-mode=true -i -M mirrorless -C 0
dgadmin 3915 3898 0 06:26 ? 00:00:00 postgres: port 25433, sweeper process
dgadmin 3927 1 0 06:27 ? 00:00:00 /home/dgadmin/vitesse.deepgreendb.16.17.170518/bin/postgres -D /dgdata/master/dg-1 -p 15432 -b 1 -z 2 --silent-mode=true -i -M master -C -1 -x 0 -E
dgadmin 3928 3927 0 06:27 ? 00:00:00 postgres: port 15432, master logger process
dgadmin 3931 3927 0 06:27 ? 00:00:00 postgres: port 15432, stats collector process
dgadmin 3932 3927 0 06:27 ? 00:00:00 postgres: port 15432, writer process
dgadmin 3933 3927 0 06:27 ? 00:00:00 postgres: port 15432, checkpoint process
dgadmin 3934 3927 0 06:27 ? 00:00:00 postgres: port 15432, seqserver process
dgadmin 3935 3927 0 06:27 ? 00:00:00 postgres: port 15432, ftsprobe process
dgadmin 3936 3927 0 06:27 ? 00:00:00 postgres: port 15432, sweeper process
dgadmin 5012 1 0 06:42 ? 00:00:00 /home/dgadmin/vitesse.deepgreendb.16.17.170518/bin/postgres -D /dgdata/mirror/dg1 -p 26433 -b 5 -z 2 --silent-mode=true -i -M quiescent -C 1
dgadmin 5013 1 0 06:42 ? 00:00:00 /home/dgadmin/vitesse.deepgreendb.16.17.170518/bin/postgres -D /dgdata/mirror/dg0 -p 26432 -b 4 -z 2 --silent-mode=true -i -M quiescent -C 0
dgadmin 5014 5013 0 06:42 ? 00:00:00 postgres: port 26432, logger process
dgadmin 5015 5012 0 06:42 ? 00:00:00 postgres: port 26433, logger process
dgadmin 5022 5013 0 06:42 ? 00:00:00 postgres: port 26432, mirror process
dgadmin 5023 5012 0 06:42 ? 00:00:00 postgres: port 26433, mirror process
dgadmin 5106 5103 0 06:42 ? 00:00:00 postgres: port 25433, primary consumer ack process
dgadmin 5107 5103 0 06:42 ? 00:00:01 postgres: port 25433, primary recovery process
dgadmin 5108 5103 0 06:42 ? 00:00:00 postgres: port 25433, primary verification process
dgadmin 5114 3899 0 06:42 ? 00:00:01 postgres: port 25432, primary process
dgadmin 5115 5114 0 06:42 ? 00:00:00 postgres: port 25432, primary receiver ack process
dgadmin 5116 5114 1 06:42 ? 00:00:20 postgres: port 25432, primary sender process
dgadmin 5117 5114 0 06:42 ? 00:00:00 postgres: port 25432, primary consumer ack process
dgadmin 5118 5114 0 06:42 ? 00:00:01 postgres: port 25432, primary recovery process
dgadmin 5119 5114 0 06:42 ? 00:00:00 postgres: port 25432, primary verification process
dgadmin 5919 3733 0 07:08 pts/0 00:00:00 grep --color=auto postgres## 杀死进程，模拟进程异常
dgadmin@flash:~$ ps -ef | grep postgres
postgres 1089 1 0 Jun27 ? 00:00:01 /usr/lib/postgresql/9.3/bin/postgres -D /var/lib/postgresql/9.3/main -c config_file=/etc/postgresql/9.3/main/postgresql.conf
postgres 1099 1089 0 Jun27 ? 00:00:00 postgres: checkpointer process
dgadmin 3898 1 0 06:26 ? 00:00:00 /home/dgadmin/vitesse.deepgreendb.16.17.170518/bin/postgres -D /dgdata/primary/dg1 -p 25433 -b 3 -z 2 --silent-mode=true -i -M mirrorless -C 1
dgadmin 3899 1 0 06:26 ? 00:00:00 /home/dgadmin/vitesse.deepgreendb.16.17.170518/bin/postgres -D /dgdata/primary/dg0 -p 25432 -b 2 -z 2 --silent-mode=true -i -M mirrorless -C 0
dgadmin 3927 1 0 06:27 ? 00:00:00 /home/dgadmin/vitesse.deepgreendb.16.17.170518/bin/postgres -D /dgdata/master/dg-1 -p 15432 -b 1 -z 2 --silent-mode=true -i -M master -C -1 -x 0 -E
dgadmin 3936 3927 0 06:27 ? 00:00:00 postgres: port 15432, sweeper process
dgadmin 5012 1 0 06:42 ? 00:00:00 /home/dgadmin/vitesse.deepgreendb.16.17.170518/bin/postgres -D /dgdata/mirror/dg1 -p 26433 -b 5 -z 2 --silent-mode=true -i -M quiescent -C 1
dgadmin 5013 1 0 06:42 ? 00:00:00 /home/dgadmin/vitesse.deepgreendb.16.17.170518/bin/postgres -D /dgdata/mirror/dg0 -p 26432 -b 4 -z 2 --silent-mode=true -i -M quiescent -C 0
dgadmin 5919 3733 0 07:08 pts/0 00:00:00 grep --color=auto postgres## 杀死进程，模拟异常
dgadmin@flash:~$ kill -9 3898## 重新查看，dg1的进程3989已不存在
dgadmin@flash:~$ ps -ef | grep postgres
postgres 1089 1 0 Jun27 ? 00:00:01 /usr/lib/postgresql/9.3/bin/postgres -D /var/lib/postgresql/9.3/main -c config_file=/etc/postgresql/9.3/main/postgresql.conf
dgadmin 3899 1 0 06:26 ? 00:00:00 /home/dgadmin/vitesse.deepgreendb.16.17.170518/bin/postgres -D /dgdata/primary/dg0 -p 25432 -b 2 -z 2 --silent-mode=true -i -M mirrorless -C 0
dgadmin 3927 1 0 06:27 ? 00:00:00 /home/dgadmin/vitesse.deepgreendb.16.17.170518/bin/postgres -D /dgdata/master/dg-1 -p 15432 -b 1 -z 2 --silent-mode=true -i -M master -C -1 -x 0 -E
dgadmin 5012 1 0 06:42 ? 00:00:00 /home/dgadmin/vitesse.deepgreendb.16.17.170518/bin/postgres -D /dgdata/mirror/dg1 -p 26433 -b 5 -z 2 --silent-mode=true -i -M quiescent -C 1
dgadmin 5013 1 0 06:42 ? 00:00:00 /home/dgadmin/vitesse.deepgreendb.16.17.170518/bin/postgres -D /dgdata/mirror/dg0 -p 26432 -b 4 -z 2 --silent-mode=true -i -M quiescent -C 0
dgadmin 5014 5013 0 06:42 ? 00:00:00 postgres: port 26432, logger process
dgadmin 5119 5114 0 06:42 ? 00:00:00 postgres: port 25432, primary verification process
dgadmin 5939 3733 0 07:09 pts/0 00:00:00 grep --color=auto postgres
7.访问数据库测试连通性：仍然可以正常查询
dgadmin@flash:~$ psql -d tpch
psql (8.2.15)
Type "help" for help.
tpch=# select * from region;
r_regionkey | r_name | r_comment
-------------+---------------------------+---------------------------------------------------------------------------------------------------------------------
1 | AMERICA | hs use ironic, even requests. s
3 | EUROPE | ly final courts cajole furiously final excuse
0 | AFRICA | lar deposits. blithely final packages cajole. regular waters are final requests. regular accounts are according to
2 | ASIA | ges. thinly even pinto beans ca
4 | MIDDLE EAST | uickly special accounts cajole carefully blithely close requests. carefully final asymptotes haggle furiousl
(5 rows)
8.查看集群状态：显示有1个primary节点异常
dgadmin@flash:~$ gpstate
20170628:07:15:28:006079 gpstate:flash:dgadmin-[INFO]:-Starting gpstate with args:
20170628:07:15:28:006079 gpstate:flash:dgadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 4.3.99.00 build Deepgreen DB'
20170628:07:15:28:006079 gpstate:flash:dgadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.2.15 (Greenplum Database 4.3.99.00 build Deepgreen DB) on x86_64-unknown-linux-gnu, compiled by GCC gcc (Ubuntu 4.9.4-2ubuntu1~14.04.1) 4.9.4 compiled on May 18 2017 05:19:19'
20170628:07:15:28:006079 gpstate:flash:dgadmin-[INFO]:-Obtaining Segment details from master...
20170628:07:15:28:006079 gpstate:flash:dgadmin-[INFO]:-Gathering data from segments...
.
20170628:07:15:29:006079 gpstate:flash:dgadmin-[INFO]:-Greenplum instance status summary
20170628:07:15:29:006079 gpstate:flash:dgadmin-[INFO]:-----------------------------------------------------
20170628:07:15:29:006079 gpstate:flash:dgadmin-[INFO]:- Master instance = Active
20170628:07:15:29:006079 gpstate:flash:dgadmin-[INFO]:- Master standby = No master standby configured
20170628:07:15:29:006079 gpstate:flash:dgadmin-[INFO]:- Total segment instance count from metadata = 4
20170628:07:15:29:006079 gpstate:flash:dgadmin-[INFO]:-----------------------------------------------------
20170628:07:15:29:006079 gpstate:flash:dgadmin-[INFO]:- Primary Segment Status
20170628:07:15:29:006079 gpstate:flash:dgadmin-[INFO]:-----------------------------------------------------
20170628:07:15:29:006079 gpstate:flash:dgadmin-[INFO]:- Total primary segments = 2
20170628:07:15:29:006079 gpstate:flash:dgadmin-[INFO]:- Total primary segment valid (at master) = 1
20170628:07:15:29:006079 gpstate:flash:dgadmin-[WARNING]:-Total primary segment failures (at master) = 1 <<<<<<<<
20170628:07:15:29:006079 gpstate:flash:dgadmin-[INFO]:- Total number of postmaster.pid files missing = 0
20170628:07:15:29:006079 gpstate:flash:dgadmin-[INFO]:- Total number of postmaster.pid files found = 2
20170628:07:15:29:006079 gpstate:flash:dgadmin-[INFO]:- Total number of postmaster.pid PIDs missing = 0
20170628:07:15:29:006079 gpstate:flash:dgadmin-[INFO]:- Total number of postmaster.pid PIDs found = 2
20170628:07:15:29:006079 gpstate:flash:dgadmin-[INFO]:- Total number of /tmp lock files missing = 0
20170628:07:15:29:006079 gpstate:flash:dgadmin-[INFO]:- Total number of /tmp lock files found = 2
20170628:07:15:29:006079 gpstate:flash:dgadmin-[WARNING]:-Total number postmaster processes missing = 1 <<<<<<<<
20170628:07:15:29:006079 gpstate:flash:dgadmin-[INFO]:- Total number postmaster processes found = 1
20170628:07:15:29:006079 gpstate:flash:dgadmin-[INFO]:-----------------------------------------------------
20170628:07:15:29:006079 gpstate:flash:dgadmin-[INFO]:- Mirror Segment Status
20170628:07:15:29:006079 gpstate:flash:dgadmin-[INFO]:-----------------------------------------------------
20170628:07:15:29:006079 gpstate:flash:dgadmin-[INFO]:- Total mirror segments = 2
20170628:07:15:29:006079 gpstate:flash:dgadmin-[INFO]:- Total mirror segment valid (at master) = 2
20170628:07:15:29:006079 gpstate:flash:dgadmin-[INFO]:- Total mirror segment failures (at master) = 0
20170628:07:15:29:006079 gpstate:flash:dgadmin-[INFO]:- Total number of postmaster.pid files missing = 0
20170628:07:15:29:006079 gpstate:flash:dgadmin-[INFO]:- Total number of postmaster.pid files found = 2
20170628:07:15:29:006079 gpstate:flash:dgadmin-[INFO]:- Total number of postmaster.pid PIDs missing = 0
20170628:07:15:29:006079 gpstate:flash:dgadmin-[INFO]:- Total number of postmaster.pid PIDs found = 2
20170628:07:15:29:006079 gpstate:flash:dgadmin-[INFO]:- Total number of /tmp lock files missing = 0
20170628:07:15:29:006079 gpstate:flash:dgadmin-[INFO]:- Total number of /tmp lock files found = 2
20170628:07:15:29:006079 gpstate:flash:dgadmin-[INFO]:- Total number postmaster processes missing = 0
20170628:07:15:29:006079 gpstate:flash:dgadmin-[INFO]:- Total number postmaster processes found = 2
20170628:07:15:29:006079 gpstate:flash:dgadmin-[WARNING]:-Total number mirror segments acting as primary segments = 1 <<<<<<<<
20170628:07:15:29:006079 gpstate:flash:dgadmin-[INFO]:- Total number mirror segments acting as mirror segments = 1
20170628:07:15:29:006079 gpstate:flash:dgadmin-[INFO]:-----------------------------------------------------
dgadmin@flash:~$
至此本实验完成，可以看到，在Segment主实例异常情况下，Mirror会立即接管服务，不会对前台应用产生停机影响。
实验完成了，但是事情还没有结束哦，因为这是故障，所以故障产生后，需要进行修复，那么基于整个集群，我们都需要做什么呢？
要点一：及时恢复故障节点
要点二：虽然Mirror可以临时接管服务，保持服务的连续性，但是在实际生产过程中，由于节点及其Mirror的分散性，长期使用Mirror会导致数据分布不均匀，所以故障修复后，建议及时切换回原来的架构。
二、故障节点恢复
节点在故障以后，可以通过gprecoverseg命令恢复故障节点，如下：
dgadmin@flash:~$ gprecoverseg
20170628:22:01:34:001650 gprecoverseg:flash:dgadmin-[INFO]:-Starting gprecoverseg with args:
20170628:22:01:34:001650 gprecoverseg:flash:dgadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 4.3.99.00 build Deepgreen DB'
20170628:22:01:34:001650 gprecoverseg:flash:dgadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.2.15 (Greenplum Database 4.3.99.00 build Deepgreen DB) on x86_64-unknown-linux-gnu, compiled by GCC gcc (Ubuntu 4.9.4-2ubuntu1~14.04.1) 4.9.4 compiled on May 18 2017 05:19:19'
20170628:22:01:34:001650 gprecoverseg:flash:dgadmin-[INFO]:-Checking if segments are ready
20170628:22:01:34:001650 gprecoverseg:flash:dgadmin-[INFO]:-Obtaining Segment details from master...
20170628:22:01:35:001650 gprecoverseg:flash:dgadmin-[INFO]:-Obtaining Segment details from master...
20170628:22:01:35:001650 gprecoverseg:flash:dgadmin-[INFO]:-Greenplum instance recovery parameters
20170628:22:01:35:001650 gprecoverseg:flash:dgadmin-[INFO]:----------------------------------------------------------
20170628:22:01:35:001650 gprecoverseg:flash:dgadmin-[INFO]:-Recovery type = Standard
20170628:22:01:35:001650 gprecoverseg:flash:dgadmin-[INFO]:----------------------------------------------------------
20170628:22:01:35:001650 gprecoverseg:flash:dgadmin-[INFO]:-Recovery 1 of 1
20170628:22:01:35:001650 gprecoverseg:flash:dgadmin-[INFO]:----------------------------------------------------------
20170628:22:01:35:001650 gprecoverseg:flash:dgadmin-[INFO]:- Synchronization mode = Incremental
20170628:22:01:35:001650 gprecoverseg:flash:dgadmin-[INFO]:- Failed instance host = flash
20170628:22:01:35:001650 gprecoverseg:flash:dgadmin-[INFO]:- Failed instance address = flash
20170628:22:01:35:001650 gprecoverseg:flash:dgadmin-[INFO]:- Failed instance directory = /dgdata/primary/dg1
20170628:22:01:35:001650 gprecoverseg:flash:dgadmin-[INFO]:- Failed instance port = 25433
20170628:22:01:35:001650 gprecoverseg:flash:dgadmin-[INFO]:- Failed instance replication port = 28433
20170628:22:01:35:001650 gprecoverseg:flash:dgadmin-[INFO]:- Recovery Source instance host = flash
20170628:22:01:35:001650 gprecoverseg:flash:dgadmin-[INFO]:- Recovery Source instance address = flash
20170628:22:01:35:001650 gprecoverseg:flash:dgadmin-[INFO]:- Recovery Source instance directory = /dgdata/mirror/dg1
20170628:22:01:35:001650 gprecoverseg:flash:dgadmin-[INFO]:- Recovery Source instance port = 26433
20170628:22:01:35:001650 gprecoverseg:flash:dgadmin-[INFO]:- Recovery Source instance replication port = 27433
20170628:22:01:35:001650 gprecoverseg:flash:dgadmin-[INFO]:- Recovery Target = in-place
20170628:22:01:35:001650 gprecoverseg:flash:dgadmin-[INFO]:----------------------------------------------------------
Continue with segment recovery procedure Yy|Nn (default=N):
> y
20170628:22:01:38:001650 gprecoverseg:flash:dgadmin-[INFO]:-1 segment(s) to recover
20170628:22:01:38:001650 gprecoverseg:flash:dgadmin-[INFO]:-Ensuring 1 failed segment(s) are stopped
20170628:22:01:39:001650 gprecoverseg:flash:dgadmin-[INFO]:-Ensuring that shared memory is cleaned up for stopped segments
20170628:22:01:45:001650 gprecoverseg:flash:dgadmin-[INFO]:-Updating configuration with new mirrors
20170628:22:01:45:001650 gprecoverseg:flash:dgadmin-[INFO]:-Updating mirrors
......
20170628:22:01:51:001650 gprecoverseg:flash:dgadmin-[INFO]:-Starting mirrors
20170628:22:01:51:001650 gprecoverseg:flash:dgadmin-[INFO]:-Commencing parallel primary and mirror segment instance startup, please wait...
.......
20170628:22:01:58:001650 gprecoverseg:flash:dgadmin-[INFO]:-Process results...
20170628:22:01:58:001650 gprecoverseg:flash:dgadmin-[INFO]:-Updating configuration to mark mirrors up
20170628:22:01:58:001650 gprecoverseg:flash:dgadmin-[INFO]:-Updating primaries
20170628:22:01:58:001650 gprecoverseg:flash:dgadmin-[INFO]:-Commencing parallel primary conversion of 1 segments, please wait...
.......
20170628:22:02:05:001650 gprecoverseg:flash:dgadmin-[INFO]:-Process results...
20170628:22:02:05:001650 gprecoverseg:flash:dgadmin-[INFO]:-Done updating primaries
20170628:22:02:06:001650 gprecoverseg:flash:dgadmin-[INFO]:-******************************************************************
20170628:22:02:06:001650 gprecoverseg:flash:dgadmin-[INFO]:-Updating segments for resynchronization is completed.
20170628:22:02:06:001650 gprecoverseg:flash:dgadmin-[INFO]:-For segments updated successfully, resynchronization will continue in the background.
20170628:22:02:06:001650 gprecoverseg:flash:dgadmin-[INFO]:-
20170628:22:02:06:001650 gprecoverseg:flash:dgadmin-[INFO]:-Use gpstate -s to check the resynchronization progress.
20170628:22:02:06:001650 gprecoverseg:flash:dgadmin-[INFO]:-******************************************************************
查看集群状态
dgadmin@flash:~$ gpstate
20170628:22:19:45:003527 gpstate:flash:dgadmin-[INFO]:-Starting gpstate with args:
20170628:22:19:45:003527 gpstate:flash:dgadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 4.3.99.00 build Deepgreen DB'
20170628:22:19:45:003527 gpstate:flash:dgadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.2.15 (Greenplum Database 4.3.99.00 build Deepgreen DB) on x86_64-unknown-linux-gnu, compiled by GCC gcc (Ubuntu 4.9.4-2ubuntu1~14.04.1) 4.9.4 compiled on May 18 2017 05:19:19'
20170628:22:19:45:003527 gpstate:flash:dgadmin-[INFO]:-Obtaining Segment details from master...
20170628:22:19:45:003527 gpstate:flash:dgadmin-[INFO]:-Gathering data from segments...
.
20170628:22:19:46:003527 gpstate:flash:dgadmin-[INFO]:-Greenplum instance status summary
20170628:22:19:46:003527 gpstate:flash:dgadmin-[INFO]:-----------------------------------------------------
20170628:22:19:46:003527 gpstate:flash:dgadmin-[INFO]:- Master instance = Active
20170628:22:19:46:003527 gpstate:flash:dgadmin-[INFO]:- Master standby = No master standby configured
20170628:22:19:46:003527 gpstate:flash:dgadmin-[INFO]:- Total segment instance count from metadata = 4
20170628:22:19:46:003527 gpstate:flash:dgadmin-[INFO]:-----------------------------------------------------
20170628:22:19:46:003527 gpstate:flash:dgadmin-[INFO]:- Primary Segment Status
20170628:22:19:46:003527 gpstate:flash:dgadmin-[INFO]:-----------------------------------------------------
20170628:22:19:46:003527 gpstate:flash:dgadmin-[INFO]:- Total primary segments = 2
20170628:22:19:46:003527 gpstate:flash:dgadmin-[INFO]:- Total primary segment valid (at master) = 2
20170628:22:19:46:003527 gpstate:flash:dgadmin-[INFO]:- Total primary segment failures (at master) = 0
20170628:22:19:46:003527 gpstate:flash:dgadmin-[INFO]:- Total number of postmaster.pid files missing = 0
20170628:22:19:46:003527 gpstate:flash:dgadmin-[INFO]:- Total number of postmaster.pid files found = 2
20170628:22:19:46:003527 gpstate:flash:dgadmin-[INFO]:- Total number of postmaster.pid PIDs missing = 0
20170628:22:19:46:003527 gpstate:flash:dgadmin-[INFO]:- Total number of postmaster.pid PIDs found = 2
20170628:22:19:46:003527 gpstate:flash:dgadmin-[INFO]:- Total number of /tmp lock files missing = 0
20170628:22:19:46:003527 gpstate:flash:dgadmin-[INFO]:- Total number of /tmp lock files found = 2
20170628:22:19:46:003527 gpstate:flash:dgadmin-[INFO]:- Total number postmaster processes missing = 0
20170628:22:19:46:003527 gpstate:flash:dgadmin-[INFO]:- Total number postmaster processes found = 2
20170628:22:19:46:003527 gpstate:flash:dgadmin-[INFO]:-----------------------------------------------------
20170628:22:19:46:003527 gpstate:flash:dgadmin-[INFO]:- Mirror Segment Status
20170628:22:19:46:003527 gpstate:flash:dgadmin-[INFO]:-----------------------------------------------------
20170628:22:19:46:003527 gpstate:flash:dgadmin-[INFO]:- Total mirror segments = 2
20170628:22:19:46:003527 gpstate:flash:dgadmin-[INFO]:- Total mirror segment valid (at master) = 2
20170628:22:19:46:003527 gpstate:flash:dgadmin-[INFO]:- Total mirror segment failures (at master) = 0
20170628:22:19:46:003527 gpstate:flash:dgadmin-[INFO]:- Total number of postmaster.pid files missing = 0
20170628:22:19:46:003527 gpstate:flash:dgadmin-[INFO]:- Total number of postmaster.pid files found = 2
20170628:22:19:46:003527 gpstate:flash:dgadmin-[INFO]:- Total number of postmaster.pid PIDs missing = 0
20170628:22:19:46:003527 gpstate:flash:dgadmin-[INFO]:- Total number of postmaster.pid PIDs found = 2
20170628:22:19:46:003527 gpstate:flash:dgadmin-[INFO]:- Total number of /tmp lock files missing = 0
20170628:22:19:46:003527 gpstate:flash:dgadmin-[INFO]:- Total number of /tmp lock files found = 2
20170628:22:19:46:003527 gpstate:flash:dgadmin-[INFO]:- Total number postmaster processes missing = 0
20170628:22:19:46:003527 gpstate:flash:dgadmin-[INFO]:- Total number postmaster processes found = 2
20170628:22:19:46:003527 gpstate:flash:dgadmin-[WARNING]:-Total number mirror segments acting as primary segments = 1 <<<<<<<<
20170628:22:19:46:003527 gpstate:flash:dgadmin-[INFO]:- Total number mirror segments acting as mirror segments = 1
20170628:22:19:46:003527 gpstate:flash:dgadmin-[INFO]:-----------------------------------------------------
三、切换回原集群状态
从第二部分的最后状态代码也可以看出，目前是一个Mirror节点接管了Primary节点的服务，我们本节要讲Mirror的服务交还给Primary。
下面执行：gprecoverseg -r 命令进行节点切换，切换完成后执行gpstate查看状态，代码略。
另外，我们也可以在数据库里面通过字典表查看切换信息：
postgres=# select * from gp_configuration_history;
time | dbid | desc
-------------------------------+------+-----------------------------------------------------------------------------------------------
2017-06-28 06:42:21.789641+08 | 4 | gpaddmirrors: segment config for resync: inserted mirror segment configuration
2017-06-28 06:42:21.789641+08 | 5 | gpaddmirrors: segment config for resync: inserted mirror segment configuration
2017-06-28 06:42:21.789641+08 | 2 | gpaddmirrors: segment config for resync: segment mode, status, and replication port
2017-06-28 06:42:21.789641+08 | 3 | gpaddmirrors: segment config for resync: segment mode, status, and replication port
2017-06-28 06:42:35.327377+08 | 2 | gpaddmirrors: segment resync marking mirrors up and primaries resync: segment mode and status
2017-06-28 06:42:35.327377+08 | 3 | gpaddmirrors: segment resync marking mirrors up and primaries resync: segment mode and status
2017-06-28 06:42:35.327377+08 | 4 | gpaddmirrors: segment resync marking mirrors up and primaries resync: segment mode and status
2017-06-28 06:42:35.327377+08 | 5 | gpaddmirrors: segment resync marking mirrors up and primaries resync: segment mode and status
2017-06-28 06:56:03.98806+08 | 2 | FTS: changed segment to insync from resync.
2017-06-28 06:56:03.98806+08 | 4 | FTS: changed segment to insync from resync.
2017-06-28 06:56:04.045572+08 | 3 | FTS: changed segment to insync from resync.
2017-06-28 06:56:04.045572+08 | 5 | FTS: changed segment to insync from resync.
2017-06-28 07:10:07.745568+08 | 3 | FTS: content 1 fault marking status DOWN role m
2017-06-28 07:10:07.745849+08 | 5 | FTS: content 1 fault marking status UP mode: change-tracking role p
2017-06-28 22:01:45.341958+08 | 3 | gprecoverseg: segment config for resync: segment mode and status
2017-06-28 22:01:58.732622+08 | 3 | gprecoverseg: segment resync marking mirrors up and primaries resync: segment mode and status
2017-06-28 22:01:58.732622+08 | 5 | gprecoverseg: segment resync marking mirrors up and primaries resync: segment mode and status
2017-06-28 22:02:57.53962+08 | 5 | FTS: changed segment to insync from resync.
2017-06-28 22:02:57.53962+08 | 3 | FTS: changed segment to insync from resync.
2017-06-28 22:31:02.680013+08 | 5 | FTS: content 1 fault marking status DOWN role m
2017-06-28 22:31:02.680309+08 | 3 | FTS: content 1 fault marking status UP mode: change-tracking role p
2017-06-28 22:31:12.655695+08 | 5 | gprecoverseg: segment config for resync: segment mode and status
2017-06-28 22:31:25.848857+08 | 3 | gprecoverseg: segment resync marking mirrors up and primaries resync: segment mode and status
2017-06-28 22:31:25.848857+08 | 5 | gprecoverseg: segment resync marking mirrors up and primaries resync: segment mode and status
2017-06-28 22:31:58.752335+08 | 3 | FTS: changed segment to insync from resync.
2017-06-28 22:31:58.752335+08 | 5 | FTS: changed segment to insync from resync.
(26 rows)
这样，整个今天的分享就结束了，最后再啰嗦一句。从本文例子看出，数据库主/备切换相当的简单，gprecoverseg命令相当的智能，在Primary的主机出现故障之后，Mirror会自动切换为Primary，不影响数据库的正常工作，但是对监控不是很到位的系统来说，不建议使用这个功能，首先这个功能存在一定的BUG，其次，监控不到位，一旦发现切换，并不能及时发现，如果再有节点出现故障，可能对数据恢复造成影响，而且如果单个节点的数据量非常大的时候，gprecoverseg同步数据的过程将会很漫长。
最后，祝大家再Deepgreen & Greenplum的路上一去不复返^_^
同系列文章参考：
