Greenplum failed segment的恢复方法

简介:
当为segment节点配置了mirror时,如果segment节点挂了,FTS进程会监测到,并且自动将mirror节点切换为primary角色。
在数据库中可以查看到状态。下面表示优先角色为primary的节点已经down了。
postgres=# select * from gp_segment_configuration order by role,dbid;
 dbid | content | role | preferred_role | mode | status | port  |     hostname      |      address      | replication_port | san_mounts 
------+---------+------+----------------+------+--------+-------+-------------------+-------------------+------------------+------------
    2 |       0 | m    | p              | s    | d      | 40000 | digoal.sqa.zmf | digoal.sqa.zmf |            43000 | 
    3 |       1 | m    | p              | s    | d      | 40001 | digoal.sqa.zmf | digoal.sqa.zmf |            43001 | 
    4 |       2 | m    | p              | s    | d      | 40002 | digoal.sqa.zmf | digoal.sqa.zmf |            43002 | 
    5 |       3 | m    | p              | s    | d      | 40003 | digoal.sqa.zmf | digoal.sqa.zmf |            43003 | 
    6 |       4 | m    | p              | s    | d      | 40004 | digoal.sqa.zmf | digoal.sqa.zmf |            43004 | 
    7 |       5 | m    | p              | s    | d      | 40005 | digoal.sqa.zmf | digoal.sqa.zmf |            43005 | 
    8 |       6 | m    | p              | s    | d      | 40006 | digoal.sqa.zmf | digoal.sqa.zmf |            43006 | 
    9 |       7 | m    | p              | s    | d      | 40007 | digoal.sqa.zmf | digoal.sqa.zmf |            43007 | 
   10 |       8 | m    | p              | s    | d      | 40008 | digoal.sqa.zmf | digoal.sqa.zmf |            43008 | 
   11 |       9 | m    | p              | s    | d      | 40009 | digoal.sqa.zmf | digoal.sqa.zmf |            43009 | 
   12 |      10 | m    | p              | s    | d      | 40010 | digoal.sqa.zmf | digoal.sqa.zmf |            43010 | 
   13 |      11 | m    | p              | s    | d      | 40011 | digoal.sqa.zmf | digoal.sqa.zmf |            43011 | 
   14 |      12 | m    | p              | s    | d      | 40012 | digoal.sqa.zmf | digoal.sqa.zmf |            43012 | 
   15 |      13 | m    | p              | s    | d      | 40013 | digoal.sqa.zmf | digoal.sqa.zmf |            43013 | 
   16 |      14 | m    | p              | s    | d      | 40014 | digoal.sqa.zmf | digoal.sqa.zmf |            43014 | 
   17 |      15 | m    | p              | s    | d      | 40015 | digoal.sqa.zmf | digoal.sqa.zmf |            43015 | 
   18 |      16 | m    | p              | s    | d      | 40016 | digoal.sqa.zmf | digoal.sqa.zmf |            43016 | 
   19 |      17 | m    | p              | s    | d      | 40017 | digoal.sqa.zmf | digoal.sqa.zmf |            43017 | 
   20 |      18 | m    | p              | s    | d      | 40018 | digoal.sqa.zmf | digoal.sqa.zmf |            43018 | 
   21 |      19 | m    | p              | s    | d      | 40019 | digoal.sqa.zmf | digoal.sqa.zmf |            43019 | 
   22 |      20 | m    | p              | s    | d      | 40020 | digoal.sqa.zmf | digoal.sqa.zmf |            43020 | 
   23 |      21 | m    | p              | s    | d      | 40021 | digoal.sqa.zmf | digoal.sqa.zmf |            43021 | 
   24 |      22 | m    | p              | s    | d      | 50011 | digoal.sqa.zmf | digoal.sqa.zmf |            43022 | 
    1 |      -1 | p    | p              | s    | u      |  1921 | digoal.sqa.zmf | digoal.sqa.zmf |                  | 
   25 |       0 | p    | m              | c    | u      | 41000 | digoal.sqa.zmf | digoal.sqa.zmf |            42000 | 
   26 |       1 | p    | m              | c    | u      | 41001 | digoal.sqa.zmf | digoal.sqa.zmf |            42001 | 
   27 |       2 | p    | m              | c    | u      | 41002 | digoal.sqa.zmf | digoal.sqa.zmf |            42002 | 
   28 |       3 | p    | m              | c    | u      | 41003 | digoal.sqa.zmf | digoal.sqa.zmf |            42003 | 
   29 |       4 | p    | m              | c    | u      | 41004 | digoal.sqa.zmf | digoal.sqa.zmf |            42004 | 
   30 |       5 | p    | m              | c    | u      | 41005 | digoal.sqa.zmf | digoal.sqa.zmf |            42005 | 
   31 |       6 | p    | m              | c    | u      | 41006 | digoal.sqa.zmf | digoal.sqa.zmf |            42006 | 
   32 |       7 | p    | m              | c    | u      | 41007 | digoal.sqa.zmf | digoal.sqa.zmf |            42007 | 
   33 |       8 | p    | m              | c    | u      | 41008 | digoal.sqa.zmf | digoal.sqa.zmf |            42008 | 
   34 |       9 | p    | m              | c    | u      | 41009 | digoal.sqa.zmf | digoal.sqa.zmf |            42009 | 
   35 |      10 | p    | m              | c    | u      | 41010 | digoal.sqa.zmf | digoal.sqa.zmf |            42010 | 
   36 |      11 | p    | m              | c    | u      | 41011 | digoal.sqa.zmf | digoal.sqa.zmf |            42011 | 
   37 |      12 | p    | m              | c    | u      | 41012 | digoal.sqa.zmf | digoal.sqa.zmf |            42012 | 
   38 |      13 | p    | m              | c    | u      | 41013 | digoal.sqa.zmf | digoal.sqa.zmf |            42013 | 
   39 |      14 | p    | m              | c    | u      | 41014 | digoal.sqa.zmf | digoal.sqa.zmf |            42014 | 
   40 |      15 | p    | m              | c    | u      | 41015 | digoal.sqa.zmf | digoal.sqa.zmf |            42015 | 
   41 |      16 | p    | m              | c    | u      | 41016 | digoal.sqa.zmf | digoal.sqa.zmf |            42016 | 
   42 |      17 | p    | m              | c    | u      | 41017 | digoal.sqa.zmf | digoal.sqa.zmf |            42017 | 
   43 |      18 | p    | m              | c    | u      | 41018 | digoal.sqa.zmf | digoal.sqa.zmf |            42018 | 
   44 |      19 | p    | m              | c    | u      | 41019 | digoal.sqa.zmf | digoal.sqa.zmf |            42019 | 
   45 |      20 | p    | m              | c    | u      | 41020 | digoal.sqa.zmf | digoal.sqa.zmf |            42020 | 
   46 |      21 | p    | m              | c    | u      | 41021 | digoal.sqa.zmf | digoal.sqa.zmf |            42021 | 
   47 |      22 | p    | m              | c    | u      | 41022 | digoal.sqa.zmf | digoal.sqa.zmf |            42022 | 
(47 rows)
mode和status字段的含义。
mode = s, c, r 分别表示synced, change logging, resyning。
已同步,说明节点现在是mirror状态,并且已经完成同步。
跟踪块变更,说明节点现在是primary状态,正在产生块变更日志。
同步中,说明节点现在是mirror状态,并且还没有同步完成。

status = u, d 分布表示up 和 down。

preferred_role 表示角色优先为什么角色。
role表示当前为什么角色。

在这种状态下停库的话,会告诉你有几个节点已经DOWN了。
$gpstop -M fast -a

20151218:16:29:59:006401 gpstop:digoal:digoal-[INFO]:-----------------------------------------------------
20151218:16:29:59:006401 gpstop:digoal:digoal-[INFO]:-   Segments stopped successfully                              = 46
20151218:16:29:59:006401 gpstop:digoal:digoal-[INFO]:-   Segments with errors during stop                           = 0
20151218:16:29:59:006401 gpstop:digoal:digoal-[INFO]:-   
20151218:16:29:59:006401 gpstop:digoal:digoal-[WARNING]:-Segments that are currently marked down in configuration   = 23   <<<<<<<<
20151218:16:29:59:006401 gpstop:digoal:digoal-[INFO]:-            (stop was still attempted on these segments)

启动数据库会忽略这些DOWN的节点
$gpstart
20151218:16:32:53:097351 gpstart:digoal:digoal-[WARNING]:-Skipping startup of segment marked down in configuration: on digoal.sqa.zmf directory /disk1/digoal/gpdata/gpseg0 <<<<<
20151218:16:32:53:097351 gpstart:digoal:digoal-[WARNING]:-Skipping startup of segment marked down in configuration: on digoal.sqa.zmf directory /disk1/digoal/gpdata/gpseg1 <<<<<
20151218:16:32:53:097351 gpstart:digoal:digoal-[WARNING]:-Skipping startup of segment marked down in configuration: on digoal.sqa.zmf directory /disk1/digoal/gpdata/gpseg2 <<<<<
20151218:16:32:53:097351 gpstart:digoal:digoal-[WARNING]:-Skipping startup of segment marked down in configuration: on digoal.sqa.zmf directory /disk1/digoal/gpdata/gpseg3 <<<<<
20151218:16:32:53:097351 gpstart:digoal:digoal-[WARNING]:-Skipping startup of segment marked down in configuration: on digoal.sqa.zmf directory /disk1/digoal/gpdata/gpseg4 <<<<<
20151218:16:32:53:097351 gpstart:digoal:digoal-[WARNING]:-Skipping startup of segment marked down in configuration: on digoal.sqa.zmf directory /disk1/digoal/gpdata/gpseg5 <<<<<
20151218:16:32:53:097351 gpstart:digoal:digoal-[WARNING]:-Skipping startup of segment marked down in configuration: on digoal.sqa.zmf directory /disk1/digoal/gpdata/gpseg6 <<<<<
20151218:16:32:53:097351 gpstart:digoal:digoal-[WARNING]:-Skipping startup of segment marked down in configuration: on digoal.sqa.zmf directory /disk1/digoal/gpdata/gpseg7 <<<<<
20151218:16:32:53:097351 gpstart:digoal:digoal-[WARNING]:-Skipping startup of segment marked down in configuration: on digoal.sqa.zmf directory /disk1/digoal/gpdata/gpseg8 <<<<<
20151218:16:32:53:097351 gpstart:digoal:digoal-[WARNING]:-Skipping startup of segment marked down in configuration: on digoal.sqa.zmf directory /disk1/digoal/gpdata/gpseg9 <<<<<
20151218:16:32:53:097351 gpstart:digoal:digoal-[WARNING]:-Skipping startup of segment marked down in configuration: on digoal.sqa.zmf directory /disk1/digoal/gpdata/gpseg10 <<<<<
20151218:16:32:53:097351 gpstart:digoal:digoal-[WARNING]:-Skipping startup of segment marked down in configuration: on digoal.sqa.zmf directory /disk1/digoal/gpdata/gpseg11 <<<<<
20151218:16:32:53:097351 gpstart:digoal:digoal-[WARNING]:-Skipping startup of segment marked down in configuration: on digoal.sqa.zmf directory /disk1/digoal/gpdata/gpseg12 <<<<<
20151218:16:32:53:097351 gpstart:digoal:digoal-[WARNING]:-Skipping startup of segment marked down in configuration: on digoal.sqa.zmf directory /disk1/digoal/gpdata/gpseg13 <<<<<
20151218:16:32:53:097351 gpstart:digoal:digoal-[WARNING]:-Skipping startup of segment marked down in configuration: on digoal.sqa.zmf directory /disk1/digoal/gpdata/gpseg14 <<<<<
20151218:16:32:53:097351 gpstart:digoal:digoal-[WARNING]:-Skipping startup of segment marked down in configuration: on digoal.sqa.zmf directory /disk1/digoal/gpdata/gpseg15 <<<<<
20151218:16:32:53:097351 gpstart:digoal:digoal-[WARNING]:-Skipping startup of segment marked down in configuration: on digoal.sqa.zmf directory /disk1/digoal/gpdata/gpseg16 <<<<<
20151218:16:32:53:097351 gpstart:digoal:digoal-[WARNING]:-Skipping startup of segment marked down in configuration: on digoal.sqa.zmf directory /disk1/digoal/gpdata/gpseg17 <<<<<
20151218:16:32:53:097351 gpstart:digoal:digoal-[WARNING]:-Skipping startup of segment marked down in configuration: on digoal.sqa.zmf directory /disk1/digoal/gpdata/gpseg18 <<<<<
20151218:16:32:53:097351 gpstart:digoal:digoal-[WARNING]:-Skipping startup of segment marked down in configuration: on digoal.sqa.zmf directory /disk1/digoal/gpdata/gpseg19 <<<<<
20151218:16:32:53:097351 gpstart:digoal:digoal-[WARNING]:-Skipping startup of segment marked down in configuration: on digoal.sqa.zmf directory /disk1/digoal/gpdata/gpseg20 <<<<<
20151218:16:32:53:097351 gpstart:digoal:digoal-[WARNING]:-Skipping startup of segment marked down in configuration: on digoal.sqa.zmf directory /disk1/digoal/gpdata/gpseg21 <<<<<
20151218:16:32:53:097351 gpstart:digoal:digoal-[WARNING]:-Skipping startup of segment marked down in configuration: on digoal.sqa.zmf directory /disk1/digoal/gpdata/gpseg22 <<<<<
使用gpstate -m可以看到,当前mirror处于primary角色。
$gpstate -m
20151218:16:52:55:036249 gpstate:digoal:digoal-[INFO]:-Starting gpstate with args: -m
20151218:16:52:55:036249 gpstate:digoal:digoal-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 4.3.6.1 build 2'
20151218:16:52:55:036249 gpstate:digoal:digoal-[INFO]:-master Greenplum Version: 'PostgreSQL 8.2.15 (Greenplum Database 4.3.6.1 build 2) on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.2 compiled on Oct  1 2015 15:14:22'
20151218:16:52:55:036249 gpstate:digoal:digoal-[INFO]:-Obtaining Segment details from master...
20151218:16:52:55:036249 gpstate:digoal:digoal-[INFO]:--------------------------------------------------------------
20151218:16:52:55:036249 gpstate:digoal:digoal-[INFO]:--Current GPDB mirror list and status
20151218:16:52:55:036249 gpstate:digoal:digoal-[INFO]:--Type = Group
20151218:16:52:55:036249 gpstate:digoal:digoal-[INFO]:--------------------------------------------------------------
20151218:16:52:55:036249 gpstate:digoal:digoal-[INFO]:-   Mirror              Datadir                                 Port    Status              Data Status       
20151218:16:52:55:036249 gpstate:digoal:digoal-[INFO]:-   digoal.sqa.zmf   /disk1/digoal/gpdata_mirror/gpseg0    41000   Acting as Primary   Change Tracking
20151218:16:52:55:036249 gpstate:digoal:digoal-[INFO]:-   digoal.sqa.zmf   /disk1/digoal/gpdata_mirror/gpseg1    41001   Acting as Primary   Change Tracking
20151218:16:52:55:036249 gpstate:digoal:digoal-[INFO]:-   digoal.sqa.zmf   /disk1/digoal/gpdata_mirror/gpseg2    41002   Acting as Primary   Change Tracking
20151218:16:52:55:036249 gpstate:digoal:digoal-[INFO]:-   digoal.sqa.zmf   /disk1/digoal/gpdata_mirror/gpseg3    41003   Acting as Primary   Change Tracking
20151218:16:52:55:036249 gpstate:digoal:digoal-[INFO]:-   digoal.sqa.zmf   /disk1/digoal/gpdata_mirror/gpseg4    41004   Acting as Primary   Change Tracking
20151218:16:52:55:036249 gpstate:digoal:digoal-[INFO]:-   digoal.sqa.zmf   /disk1/digoal/gpdata_mirror/gpseg5    41005   Acting as Primary   Change Tracking
20151218:16:52:55:036249 gpstate:digoal:digoal-[INFO]:-   digoal.sqa.zmf   /disk1/digoal/gpdata_mirror/gpseg6    41006   Acting as Primary   Change Tracking
20151218:16:52:55:036249 gpstate:digoal:digoal-[INFO]:-   digoal.sqa.zmf   /disk1/digoal/gpdata_mirror/gpseg7    41007   Acting as Primary   Change Tracking
20151218:16:52:55:036249 gpstate:digoal:digoal-[INFO]:-   digoal.sqa.zmf   /disk1/digoal/gpdata_mirror/gpseg8    41008   Acting as Primary   Change Tracking
20151218:16:52:55:036249 gpstate:digoal:digoal-[INFO]:-   digoal.sqa.zmf   /disk1/digoal/gpdata_mirror/gpseg9    41009   Acting as Primary   Change Tracking
20151218:16:52:55:036249 gpstate:digoal:digoal-[INFO]:-   digoal.sqa.zmf   /disk1/digoal/gpdata_mirror/gpseg10   41010   Acting as Primary   Change Tracking
20151218:16:52:55:036249 gpstate:digoal:digoal-[INFO]:-   digoal.sqa.zmf   /disk1/digoal/gpdata_mirror/gpseg11   41011   Acting as Primary   Change Tracking
20151218:16:52:55:036249 gpstate:digoal:digoal-[INFO]:-   digoal.sqa.zmf   /disk1/digoal/gpdata_mirror/gpseg12   41012   Acting as Primary   Change Tracking
20151218:16:52:55:036249 gpstate:digoal:digoal-[INFO]:-   digoal.sqa.zmf   /disk1/digoal/gpdata_mirror/gpseg13   41013   Acting as Primary   Change Tracking
20151218:16:52:55:036249 gpstate:digoal:digoal-[INFO]:-   digoal.sqa.zmf   /disk1/digoal/gpdata_mirror/gpseg14   41014   Acting as Primary   Change Tracking
20151218:16:52:55:036249 gpstate:digoal:digoal-[INFO]:-   digoal.sqa.zmf   /disk1/digoal/gpdata_mirror/gpseg15   41015   Acting as Primary   Change Tracking
20151218:16:52:55:036249 gpstate:digoal:digoal-[INFO]:-   digoal.sqa.zmf   /disk1/digoal/gpdata_mirror/gpseg16   41016   Acting as Primary   Change Tracking
20151218:16:52:55:036249 gpstate:digoal:digoal-[INFO]:-   digoal.sqa.zmf   /disk1/digoal/gpdata_mirror/gpseg17   41017   Acting as Primary   Change Tracking
20151218:16:52:55:036249 gpstate:digoal:digoal-[INFO]:-   digoal.sqa.zmf   /disk1/digoal/gpdata_mirror/gpseg18   41018   Acting as Primary   Change Tracking
20151218:16:52:55:036249 gpstate:digoal:digoal-[INFO]:-   digoal.sqa.zmf   /disk1/digoal/gpdata_mirror/gpseg19   41019   Acting as Primary   Change Tracking
20151218:16:52:55:036249 gpstate:digoal:digoal-[INFO]:-   digoal.sqa.zmf   /disk1/digoal/gpdata_mirror/gpseg20   41020   Acting as Primary   Change Tracking
20151218:16:52:55:036249 gpstate:digoal:digoal-[INFO]:-   digoal.sqa.zmf   /disk1/digoal/gpdata_mirror/gpseg21   41021   Acting as Primary   Change Tracking
20151218:16:52:55:036249 gpstate:digoal:digoal-[INFO]:-   digoal.sqa.zmf   /disk1/digoal/gpdata_mirror/gpseg22   41022   Acting as Primary   Change Tracking
20151218:16:52:55:036249 gpstate:digoal:digoal-[INFO]:--------------------------------------------------------------
20151218:16:52:55:036249 gpstate:digoal:digoal-[WARNING]:-23 segment(s) configured as mirror(s) are acting as primaries
20151218:16:52:55:036249 gpstate:digoal:digoal-[WARNING]:-23 mirror segment(s) acting as primaries are in change tracking

进入主题,到底怎么恢复failed segment呢?
1. 首先要产生一个恢复配置文件
$gprecoverseg -o ./recov
20151218:17:02:03:034183 gprecoverseg:digoal:digoal-[INFO]:-Starting gprecoverseg with args: -o ./recov
20151218:17:02:03:034183 gprecoverseg:digoal:digoal-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 4.3.6.1 build 2'
20151218:17:02:03:034183 gprecoverseg:digoal:digoal-[INFO]:-master Greenplum Version: 'PostgreSQL 8.2.15 (Greenplum Database 4.3.6.1 build 2) on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.2 compiled on Oct  1 2015 15:14:22'
20151218:17:02:03:034183 gprecoverseg:digoal:digoal-[INFO]:-Checking if segments are ready
20151218:17:02:03:034183 gprecoverseg:digoal:digoal-[INFO]:-Obtaining Segment details from master...
20151218:17:02:04:034183 gprecoverseg:digoal:digoal-[INFO]:-Obtaining Segment details from master...
20151218:17:02:07:034183 gprecoverseg:digoal:digoal-[INFO]:-Configuration file output to ./recov successfully.
配置文件内容
$cat ./recov 
filespaceOrder=
digoal.sqa.zmf:40000:/disk1/digoal/gpdata/gpseg0
digoal.sqa.zmf:40001:/disk1/digoal/gpdata/gpseg1
digoal.sqa.zmf:40002:/disk1/digoal/gpdata/gpseg2
digoal.sqa.zmf:40003:/disk1/digoal/gpdata/gpseg3
digoal.sqa.zmf:40004:/disk1/digoal/gpdata/gpseg4
digoal.sqa.zmf:40005:/disk1/digoal/gpdata/gpseg5
digoal.sqa.zmf:40006:/disk1/digoal/gpdata/gpseg6
digoal.sqa.zmf:40007:/disk1/digoal/gpdata/gpseg7
digoal.sqa.zmf:40008:/disk1/digoal/gpdata/gpseg8
digoal.sqa.zmf:40009:/disk1/digoal/gpdata/gpseg9
digoal.sqa.zmf:40010:/disk1/digoal/gpdata/gpseg10
digoal.sqa.zmf:40011:/disk1/digoal/gpdata/gpseg11
digoal.sqa.zmf:40012:/disk1/digoal/gpdata/gpseg12
digoal.sqa.zmf:40013:/disk1/digoal/gpdata/gpseg13
digoal.sqa.zmf:40014:/disk1/digoal/gpdata/gpseg14
digoal.sqa.zmf:40015:/disk1/digoal/gpdata/gpseg15
digoal.sqa.zmf:40016:/disk1/digoal/gpdata/gpseg16
digoal.sqa.zmf:40017:/disk1/digoal/gpdata/gpseg17
digoal.sqa.zmf:40018:/disk1/digoal/gpdata/gpseg18
digoal.sqa.zmf:40019:/disk1/digoal/gpdata/gpseg19
digoal.sqa.zmf:40020:/disk1/digoal/gpdata/gpseg20
digoal.sqa.zmf:40021:/disk1/digoal/gpdata/gpseg21
digoal.sqa.zmf:50011:/disk1/digoal/gpdata/gpseg22

2. 然后使用这个配置文件进行恢复
$gprecoverseg -i ./recov
。。。
20151218:17:02:28:050151 gprecoverseg:digoal:digoal-[INFO]:-   Synchronization mode                        = Incremental
20151218:17:02:28:050151 gprecoverseg:digoal:digoal-[INFO]:-   Failed instance host                        = digoal.sqa.zmf
20151218:17:02:28:050151 gprecoverseg:digoal:digoal-[INFO]:-   Failed instance address                     = digoal.sqa.zmf
20151218:17:02:28:050151 gprecoverseg:digoal:digoal-[INFO]:-   Failed instance directory                   = /disk1/digoal/gpdata/gpseg22
20151218:17:02:28:050151 gprecoverseg:digoal:digoal-[INFO]:-   Failed instance port                        = 50011
20151218:17:02:28:050151 gprecoverseg:digoal:digoal-[INFO]:-   Failed instance replication port            = 43022
20151218:17:02:28:050151 gprecoverseg:digoal:digoal-[INFO]:-   Recovery Source instance host               = digoal.sqa.zmf
20151218:17:02:28:050151 gprecoverseg:digoal:digoal-[INFO]:-   Recovery Source instance address            = digoal.sqa.zmf
20151218:17:02:28:050151 gprecoverseg:digoal:digoal-[INFO]:-   Recovery Source instance directory          = /disk1/digoal/gpdata_mirror/gpseg22
20151218:17:02:28:050151 gprecoverseg:digoal:digoal-[INFO]:-   Recovery Source instance port               = 41022
20151218:17:02:28:050151 gprecoverseg:digoal:digoal-[INFO]:-   Recovery Source instance replication port   = 42022
20151218:17:02:28:050151 gprecoverseg:digoal:digoal-[INFO]:-   Recovery Target                             = in-place
20151218:17:02:28:050151 gprecoverseg:digoal:digoal-[INFO]:----------------------------------------------------------

Continue with segment recovery procedure Yy|Nn (default=N):
> y

恢复过程中,查看mirror状态。
$gpstate -m
...
20151218:17:03:00:072483 gpstate:digoal:digoal-[INFO]:-   Mirror              Datadir                                 Port    Status              Data Status       
20151218:17:03:00:072483 gpstate:digoal:digoal-[INFO]:-   digoal.sqa.zmf   /disk1/digoal/gpdata_mirror/gpseg0    41000   Acting as Primary   Resynchronizing
20151218:17:03:00:072483 gpstate:digoal:digoal-[INFO]:-   digoal.sqa.zmf   /disk1/digoal/gpdata_mirror/gpseg1    41001   Acting as Primary   Resynchronizing
20151218:17:03:00:072483 gpstate:digoal:digoal-[INFO]:-   digoal.sqa.zmf   /disk1/digoal/gpdata_mirror/gpseg2    41002   Acting as Primary   Resynchronizing
...
同步完成的状态如下  (Synchronized)
20151218:17:08:38:106220 gpstate:digoal:digoal-[INFO]:-   digoal.sqa.zmf   /disk1/digoal/gpdata_mirror/gpseg1    41001   Acting as Primary   Synchronized
20151218:17:08:38:106220 gpstate:digoal:digoal-[INFO]:-   digoal.sqa.zmf   /disk1/digoal/gpdata_mirror/gpseg2    41002   Acting as Primary   Synchronized
20151218:17:08:38:106220 gpstate:digoal:digoal-[INFO]:-   digoal.sqa.zmf   /disk1/digoal/gpdata_mirror/gpseg3    41003   Acting as Primary   Synchronized

3. 到上一步,数据库的主备就恢复了,但是还有一步,是可选的。
你要不要把primary , mirror角色对调一下,因为现在mirror和primary和优先角色是相反的。
如果要对调,使用以下命令,会停库来处理。
$gprecoverseg -r
20151218:17:12:16:120198 gprecoverseg:digoal:digoal-[INFO]:-Starting gprecoverseg with args: -r
20151218:17:12:16:120198 gprecoverseg:digoal:digoal-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 4.3.6.1 build 2'
20151218:17:12:16:120198 gprecoverseg:digoal:digoal-[INFO]:-master Greenplum Version: 'PostgreSQL 8.2.15 (Greenplum Database 4.3.6.1 build 2) on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.2 compiled on Oct  1 2015 15:14:22'
20151218:17:12:16:120198 gprecoverseg:digoal:digoal-[INFO]:-Checking if segments are ready
20151218:17:12:16:120198 gprecoverseg:digoal:digoal-[INFO]:-Obtaining Segment details from master...
20151218:17:12:17:120198 gprecoverseg:digoal:digoal-[INFO]:-Obtaining Segment details from master...
20151218:17:12:18:120198 gprecoverseg:digoal:digoal-[INFO]:-Greenplum instance recovery parameters
20151218:17:12:18:120198 gprecoverseg:digoal:digoal-[INFO]:----------------------------------------------------------
20151218:17:12:18:120198 gprecoverseg:digoal:digoal-[INFO]:-Recovery type              = Rebalance
20151218:17:12:18:120198 gprecoverseg:digoal:digoal-[INFO]:----------------------------------------------------------
20151218:17:12:18:120198 gprecoverseg:digoal:digoal-[INFO]:-Unbalanced segment 1 of 46
20151218:17:12:18:120198 gprecoverseg:digoal:digoal-[INFO]:----------------------------------------------------------
20151218:17:12:18:120198 gprecoverseg:digoal:digoal-[INFO]:-   Unbalanced instance host               = digoal.sqa.zmf
20151218:17:12:18:120198 gprecoverseg:digoal:digoal-[INFO]:-   Unbalanced instance address            = digoal.sqa.zmf
20151218:17:12:18:120198 gprecoverseg:digoal:digoal-[INFO]:-   Unbalanced instance directory          = /disk1/digoal/gpdata_mirror/gpseg0
20151218:17:12:18:120198 gprecoverseg:digoal:digoal-[INFO]:-   Unbalanced instance port               = 41000
20151218:17:12:18:120198 gprecoverseg:digoal:digoal-[INFO]:-   Unbalanced instance replication port   = 42000
20151218:17:12:18:120198 gprecoverseg:digoal:digoal-[INFO]:-   Balanced role                          = Mirror
20151218:17:12:18:120198 gprecoverseg:digoal:digoal-[INFO]:-   Current role                           = Primary
......
20151218:17:12:18:120198 gprecoverseg:digoal:digoal-[INFO]:----------------------------------------------------------
20151218:17:12:18:120198 gprecoverseg:digoal:digoal-[WARNING]:-This operation will cancel queries that are currently executing.
20151218:17:12:18:120198 gprecoverseg:digoal:digoal-[WARNING]:-Connections to the database however will not be interrupted.

Continue with segment rebalance procedure Yy|Nn (default=N):
> y
...
20151218:17:13:03:120198 gprecoverseg:digoal:digoal-[INFO]:-******************************************************************
20151218:17:13:03:120198 gprecoverseg:digoal:digoal-[INFO]:-The rebalance operation has completed successfully.
20151218:17:13:03:120198 gprecoverseg:digoal:digoal-[INFO]:-There is a resynchronization running in the background to bring all
20151218:17:13:03:120198 gprecoverseg:digoal:digoal-[INFO]:-segments in sync.
20151218:17:13:03:120198 gprecoverseg:digoal:digoal-[INFO]:-
20151218:17:13:03:120198 gprecoverseg:digoal:digoal-[INFO]:-Use gpstate -e to check the resynchronization progress.
20151218:17:13:03:120198 gprecoverseg:digoal:digoal-[INFO]:-******************************************************************
查看同步状态
$gpstate -m
。。。
20151218:17:16:09:093083 gpstate:digoal:digoal-[INFO]:--------------------------------------------------------------
20151218:17:16:09:093083 gpstate:digoal:digoal-[INFO]:-   Mirror              Datadir                                 Port    Status    Data Status    
20151218:17:16:09:093083 gpstate:digoal:digoal-[INFO]:-   digoal.sqa.zmf   /disk1/digoal/gpdata_mirror/gpseg0    41000   Passive   Synchronized
20151218:17:16:09:093083 gpstate:digoal:digoal-[INFO]:-   digoal.sqa.zmf   /disk1/digoal/gpdata_mirror/gpseg1    41001   Passive   Synchronized
。。。

查看segment配置,已经是synced和up的状态了。
postgres=# select * from gp_segment_configuration order by role,dbid;
 dbid | content | role | preferred_role | mode | status | port  |     hostname      |      address      | replication_port | san_mounts 
------+---------+------+----------------+------+--------+-------+-------------------+-------------------+------------------+------------
   25 |       0 | m    | m              | s    | u      | 41000 | digoal.sqa.zmf | digoal.sqa.zmf |            42000 | 
   26 |       1 | m    | m              | s    | u      | 41001 | digoal.sqa.zmf | digoal.sqa.zmf |            42001 | 
   27 |       2 | m    | m              | s    | u      | 41002 | digoal.sqa.zmf | digoal.sqa.zmf |            42002 | 
   28 |       3 | m    | m              | s    | u      | 41003 | digoal.sqa.zmf | digoal.sqa.zmf |            42003 | 
   29 |       4 | m    | m              | s    | u      | 41004 | digoal.sqa.zmf | digoal.sqa.zmf |            42004 | 
   30 |       5 | m    | m              | s    | u      | 41005 | digoal.sqa.zmf | digoal.sqa.zmf |            42005 | 
   31 |       6 | m    | m              | s    | u      | 41006 | digoal.sqa.zmf | digoal.sqa.zmf |            42006 | 
   32 |       7 | m    | m              | s    | u      | 41007 | digoal.sqa.zmf | digoal.sqa.zmf |            42007 | 
   33 |       8 | m    | m              | s    | u      | 41008 | digoal.sqa.zmf | digoal.sqa.zmf |            42008 | 
   34 |       9 | m    | m              | s    | u      | 41009 | digoal.sqa.zmf | digoal.sqa.zmf |            42009 | 
   35 |      10 | m    | m              | s    | u      | 41010 | digoal.sqa.zmf | digoal.sqa.zmf |            42010 | 
   36 |      11 | m    | m              | s    | u      | 41011 | digoal.sqa.zmf | digoal.sqa.zmf |            42011 | 
   37 |      12 | m    | m              | s    | u      | 41012 | digoal.sqa.zmf | digoal.sqa.zmf |            42012 | 
   38 |      13 | m    | m              | s    | u      | 41013 | digoal.sqa.zmf | digoal.sqa.zmf |            42013 | 
   39 |      14 | m    | m              | s    | u      | 41014 | digoal.sqa.zmf | digoal.sqa.zmf |            42014 | 
   40 |      15 | m    | m              | s    | u      | 41015 | digoal.sqa.zmf | digoal.sqa.zmf |            42015 | 
   41 |      16 | m    | m              | s    | u      | 41016 | digoal.sqa.zmf | digoal.sqa.zmf |            42016 | 
   42 |      17 | m    | m              | s    | u      | 41017 | digoal.sqa.zmf | digoal.sqa.zmf |            42017 | 
   43 |      18 | m    | m              | s    | u      | 41018 | digoal.sqa.zmf | digoal.sqa.zmf |            42018 | 
   44 |      19 | m    | m              | s    | u      | 41019 | digoal.sqa.zmf | digoal.sqa.zmf |            42019 | 
   45 |      20 | m    | m              | s    | u      | 41020 | digoal.sqa.zmf | digoal.sqa.zmf |            42020 | 
   46 |      21 | m    | m              | s    | u      | 41021 | digoal.sqa.zmf | digoal.sqa.zmf |            42021 | 
   47 |      22 | m    | m              | s    | u      | 41022 | digoal.sqa.zmf | digoal.sqa.zmf |            42022 | 
    1 |      -1 | p    | p              | s    | u      |  1921 | digoal.sqa.zmf | digoal.sqa.zmf |                  | 
    2 |       0 | p    | p              | s    | u      | 40000 | digoal.sqa.zmf | digoal.sqa.zmf |            43000 | 
    3 |       1 | p    | p              | s    | u      | 40001 | digoal.sqa.zmf | digoal.sqa.zmf |            43001 | 
    4 |       2 | p    | p              | s    | u      | 40002 | digoal.sqa.zmf | digoal.sqa.zmf |            43002 | 
    5 |       3 | p    | p              | s    | u      | 40003 | digoal.sqa.zmf | digoal.sqa.zmf |            43003 | 
    6 |       4 | p    | p              | s    | u      | 40004 | digoal.sqa.zmf | digoal.sqa.zmf |            43004 | 
    7 |       5 | p    | p              | s    | u      | 40005 | digoal.sqa.zmf | digoal.sqa.zmf |            43005 | 
    8 |       6 | p    | p              | s    | u      | 40006 | digoal.sqa.zmf | digoal.sqa.zmf |            43006 | 
    9 |       7 | p    | p              | s    | u      | 40007 | digoal.sqa.zmf | digoal.sqa.zmf |            43007 | 
   10 |       8 | p    | p              | s    | u      | 40008 | digoal.sqa.zmf | digoal.sqa.zmf |            43008 | 
   11 |       9 | p    | p              | s    | u      | 40009 | digoal.sqa.zmf | digoal.sqa.zmf |            43009 | 
   12 |      10 | p    | p              | s    | u      | 40010 | digoal.sqa.zmf | digoal.sqa.zmf |            43010 | 
   13 |      11 | p    | p              | s    | u      | 40011 | digoal.sqa.zmf | digoal.sqa.zmf |            43011 | 
   14 |      12 | p    | p              | s    | u      | 40012 | digoal.sqa.zmf | digoal.sqa.zmf |            43012 | 
   15 |      13 | p    | p              | s    | u      | 40013 | digoal.sqa.zmf | digoal.sqa.zmf |            43013 | 
   16 |      14 | p    | p              | s    | u      | 40014 | digoal.sqa.zmf | digoal.sqa.zmf |            43014 | 
   17 |      15 | p    | p              | s    | u      | 40015 | digoal.sqa.zmf | digoal.sqa.zmf |            43015 | 
   18 |      16 | p    | p              | s    | u      | 40016 | digoal.sqa.zmf | digoal.sqa.zmf |            43016 | 
   19 |      17 | p    | p              | s    | u      | 40017 | digoal.sqa.zmf | digoal.sqa.zmf |            43017 | 
   20 |      18 | p    | p              | s    | u      | 40018 | digoal.sqa.zmf | digoal.sqa.zmf |            43018 | 
   21 |      19 | p    | p              | s    | u      | 40019 | digoal.sqa.zmf | digoal.sqa.zmf |            43019 | 
   22 |      20 | p    | p              | s    | u      | 40020 | digoal.sqa.zmf | digoal.sqa.zmf |            43020 | 
   23 |      21 | p    | p              | s    | u      | 40021 | digoal.sqa.zmf | digoal.sqa.zmf |            43021 | 
   24 |      22 | p    | p              | s    | u      | 50011 | digoal.sqa.zmf | digoal.sqa.zmf |            43022 | 
(47 rows)

[参考]
gprecoverseg -h
目录
相关文章
|
6月前
|
关系型数据库 分布式数据库 PolarDB
PolarDB操作报错合集之执行drop操作报The consensus follower is not allowed to to do current operation错误,select可以执行,是什么导致的
在使用阿里云的PolarDB(包括PolarDB-X)时,用户可能会遇到各种操作报错。下面汇总了一些常见的报错情况及其可能的原因和解决办法:1.安装PolarDB-X报错、2.PolarDB安装后无法连接、3.PolarDB-X 使用rpm安装启动卡顿、4.PolarDB执行UPDATE/INSERT报错、5.DDL操作提示“Lock conflict”、6.数据集成时联通PolarDB报错、7.编译DN报错(RockyLinux)、8.CheckStorage报错(源数据库实例被删除)、9.嵌套事务错误(TDDL-4604)。
167 1
|
6月前
|
关系型数据库 MySQL 分布式数据库
PolarDB操作报错合集之遇到报错“Reading record is now unsafe on slave” ,该如何解决
在使用阿里云的PolarDB(包括PolarDB-X)时,用户可能会遇到各种操作报错。下面汇总了一些常见的报错情况及其可能的原因和解决办法:1.安装PolarDB-X报错、2.PolarDB安装后无法连接、3.PolarDB-X 使用rpm安装启动卡顿、4.PolarDB执行UPDATE/INSERT报错、5.DDL操作提示“Lock conflict”、6.数据集成时联通PolarDB报错、7.编译DN报错(RockyLinux)、8.CheckStorage报错(源数据库实例被删除)、9.嵌套事务错误(TDDL-4604)。
167 0
|
存储 关系型数据库 MySQL
mysql 数据库无法启动(Ignoring the redo log due to missing MLOG_CHECKPOINT between the checkpoint .... and)
数据库机器的CPU和主板都换了,重新开机,发现mysql数据库无法启动!
427 0
|
关系型数据库 数据库 PostgreSQL
PG异常无法启动的问题:could not read file "pg_logical/replorigin_checkpoint": Success
问题描述 新安装不久的PostgreSQL数据库,断电后重启,查看日志如下 2019-01-08 08:44:19.989 UTC [7493] LOG: database system was interrupted; last known up at 2018-12-24 10:56:28 UTC 2019-01-08 08:44:19.
3426 0
|
关系型数据库 数据库
【DB吐槽大会】第25期 - PG 不支持物理Partial Standby
大家好,这里是DB吐槽大会,第25期 - PG 不支持物理Partial Standby
|
SQL Oracle 关系型数据库
xDB Replication Server - PostgreSQL, Oracle, SQL Server, PPAS 全量、增量(redo log based, or trigger based)同步(支持single-master, mult-master同步, 支持DDL)
xDB Replication Server - PostgreSQL, Oracle, SQL Server, PPAS 全量、增量(redo log based, or trigger based)同步(支持single-master, mult-master同步, 支持DDL)
1001 0
|
SQL
MySQL:简单记录删除binary log的接口和O_DIRECT不会用到REDO
一、栈帧 #0 my_delete (name=0x7ffff0fa0490 "./binlog.000005", MyFlags=0) at /root/softm/percona-server-5.
903 0
|
Oracle 关系型数据库 数据库
oracle的增量检查点与block buffer
通过dump工具分析oracle的增量检查点机制
2553 0