Greenplum 恢复segment节点之后，节点一直处于down状态

背景

模拟节点故障的情况，将节点对应的进程kill掉之后，primary和mirror发生切换

gp_sydb=# select dbid,content, role, preferred_role, mode, status,datadir from gp_segment_configuration  where content=5;
 dbid | content | role | preferred_role | mode | status |               datadir
------+---------+------+----------------+------+--------+--------------------------------------
    7 |       5 | m    | p              | n    | d      | /data6/greenplum/data/primary/gpseg5
   22 |       5 | p    | m              | n    | u      | /data7/greenplum/data/mirror/gpseg5

恢复

gprecoverseg -o recov
gpadmin-[INFO]:-Configuration file output to recov successfully

查看recov文件

[gpadmin@tdxy-paas-test5 ~]$ more recov
tdxy-paas-test6|6002|/data6/greenplum/data/primary/gpseg5

执行恢复

[gpadmin@tdxy-paas-test5 ~]$ gprecoverseg -i ./recov -F
20200324:20:56:46:035342 gprecoverseg:tdxy-paas-test5:gpadmin-[INFO]:-Starting gprecoverseg with args: -i ./recov -F
20200324:20:56:46:035342 gprecoverseg:tdxy-paas-test5:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 6.1.0 build commit:6788ca8c13b2bd6e8976ccffea07313cbab30560'
20200324:20:56:46:035342 gprecoverseg:tdxy-paas-test5:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 9.4.24 (Greenplum Database 6.1.0 build commit:6788ca8c13b2bd6e8976ccffea07313cbab30560) on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 6.4.0, 64-bit compiled on Nov  1 2019 22:06:07'
20200324:20:56:46:035342 gprecoverseg:tdxy-paas-test5:gpadmin-[INFO]:-Obtaining Segment details from master...
20200324:20:56:47:035342 gprecoverseg:tdxy-paas-test5:gpadmin-[INFO]:-Heap checksum setting is consistent between master and the segments that are candidates for recoverseg
20200324:20:56:47:035342 gprecoverseg:tdxy-paas-test5:gpadmin-[INFO]:-Greenplum instance recovery parameters
20200324:20:56:47:035342 gprecoverseg:tdxy-paas-test5:gpadmin-[INFO]:----------------------------------------------------------
20200324:20:56:47:035342 gprecoverseg:tdxy-paas-test5:gpadmin-[INFO]:-Recovery from configuration -i option supplied
20200324:20:56:47:035342 gprecoverseg:tdxy-paas-test5:gpadmin-[INFO]:----------------------------------------------------------
20200324:20:56:47:035342 gprecoverseg:tdxy-paas-test5:gpadmin-[INFO]:-Recovery 1 of 1
20200324:20:56:47:035342 gprecoverseg:tdxy-paas-test5:gpadmin-[INFO]:----------------------------------------------------------
20200324:20:56:47:035342 gprecoverseg:tdxy-paas-test5:gpadmin-[INFO]:-   Synchronization mode                 = Full
20200324:20:56:47:035342 gprecoverseg:tdxy-paas-test5:gpadmin-[INFO]:-   Failed instance host                 = tdxy-paas-test6
20200324:20:56:47:035342 gprecoverseg:tdxy-paas-test5:gpadmin-[INFO]:-   Failed instance address              = tdxy-paas-test6
20200324:20:56:47:035342 gprecoverseg:tdxy-paas-test5:gpadmin-[INFO]:-   Failed instance directory            = /data6/greenplum/data/primary/gpseg5
20200324:20:56:47:035342 gprecoverseg:tdxy-paas-test5:gpadmin-[INFO]:-   Failed instance port                 = 6002
20200324:20:56:47:035342 gprecoverseg:tdxy-paas-test5:gpadmin-[INFO]:-   Recovery Source instance host        = tdxy-paas-test11
20200324:20:56:47:035342 gprecoverseg:tdxy-paas-test5:gpadmin-[INFO]:-   Recovery Source instance address     = tdxy-paas-test11
20200324:20:56:47:035342 gprecoverseg:tdxy-paas-test5:gpadmin-[INFO]:-   Recovery Source instance directory   = /data7/greenplum/data/mirror/gpseg5
20200324:20:56:47:035342 gprecoverseg:tdxy-paas-test5:gpadmin-[INFO]:-   Recovery Source instance port        = 7002
20200324:20:56:47:035342 gprecoverseg:tdxy-paas-test5:gpadmin-[INFO]:-   Recovery Target                      = in-place
20200324:20:56:47:035342 gprecoverseg:tdxy-paas-test5:gpadmin-[INFO]:----------------------------------------------------------

Continue with segment recovery procedure Yy|Nn (default=N):
> Y
20200324:20:56:49:035342 gprecoverseg:tdxy-paas-test5:gpadmin-[INFO]:-1 segment(s) to recover
20200324:20:56:49:035342 gprecoverseg:tdxy-paas-test5:gpadmin-[INFO]:-Ensuring 1 failed segment(s) are stopped
20200324:20:56:50:035342 gprecoverseg:tdxy-paas-test5:gpadmin-[INFO]:-26166: /data6/greenplum/data/primary/gpseg5
20200324:20:56:51:035342 gprecoverseg:tdxy-paas-test5:gpadmin-[INFO]:-Ensuring that shared memory is cleaned up for stopped segments
20200324:20:56:51:035342 gprecoverseg:tdxy-paas-test5:gpadmin-[INFO]:-Validating remote directories
20200324:20:56:52:035342 gprecoverseg:tdxy-paas-test5:gpadmin-[INFO]:-Configuring new segments
tdxy-paas-test6 (dbid 7): pg_basebackup: base backup completed
20200324:20:56:58:035342 gprecoverseg:tdxy-paas-test5:gpadmin-[INFO]:-Updating configuration with new mirrors
20200324:20:56:58:035342 gprecoverseg:tdxy-paas-test5:gpadmin-[INFO]:-Updating mirrors
20200324:20:56:58:035342 gprecoverseg:tdxy-paas-test5:gpadmin-[INFO]:-Starting mirrors
20200324:20:56:58:035342 gprecoverseg:tdxy-paas-test5:gpadmin-[INFO]:-era is 4d5a66a317e25141_200324202911
20200324:20:56:58:035342 gprecoverseg:tdxy-paas-test5:gpadmin-[INFO]:-Commencing parallel segment instance startup, please wait...
..
20200324:20:57:01:035342 gprecoverseg:tdxy-paas-test5:gpadmin-[INFO]:-Process results...
20200324:20:57:01:035342 gprecoverseg:tdxy-paas-test5:gpadmin-[INFO]:-Triggering FTS probe
20200324:20:57:06:035342 gprecoverseg:tdxy-paas-test5:gpadmin-[INFO]:-******************************************************************
20200324:20:57:06:035342 gprecoverseg:tdxy-paas-test5:gpadmin-[INFO]:-Updating segments for streaming is completed.
20200324:20:57:06:035342 gprecoverseg:tdxy-paas-test5:gpadmin-[INFO]:-For segments updated successfully, streaming will continue in the background.
20200324:20:57:06:035342 gprecoverseg:tdxy-paas-test5:gpadmin-[INFO]:-Use  gpstate -s  to check the streaming progress.
20200324:20:57:06:035342 gprecoverseg:tdxy-paas-test5:gpadmin-[INFO]:-

查看mirror状态

gpstart, gplogfilter[gpadmin@tdxy-paas-test5 ~]$ gpstate -m
20200324:20:57:52:035619 gpstate:tdxy-paas-test5:gpadmin-[INFO]:-Starting gpstate with args: -m
20200324:20:57:52:035619 gpstate:tdxy-paas-test5:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 6.1.0 build commit:6788ca8c13b2bd6e8976ccffea07313cbab30560'
20200324:20:57:52:035619 gpstate:tdxy-paas-test5:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 9.4.24 (Greenplum Database 6.1.0 build commit:6788ca8c13b2bd6e8976ccffea07313cbab30560) on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 6.4.0, 64-bit compiled on Nov  1 2019 22:06:07'
20200324:20:57:52:035619 gpstate:tdxy-paas-test5:gpadmin-[INFO]:-Obtaining Segment details from master...
20200324:20:57:52:035619 gpstate:tdxy-paas-test5:gpadmin-[INFO]:--------------------------------------------------------------
20200324:20:57:52:035619 gpstate:tdxy-paas-test5:gpadmin-[INFO]:--Current GPDB mirror list and status
20200324:20:57:52:035619 gpstate:tdxy-paas-test5:gpadmin-[INFO]:--Type = Spread
20200324:20:57:52:035619 gpstate:tdxy-paas-test5:gpadmin-[INFO]:--------------------------------------------------------------
20200324:20:57:52:035619 gpstate:tdxy-paas-test5:gpadmin-[INFO]:-   Mirror             Datadir                                Port   Status              Data Status
20200324:20:57:52:035619 gpstate:tdxy-paas-test5:gpadmin-[INFO]:-   tdxy-paas-test6    /data3/greenplum/data/mirror/gpseg0    7000   Passive             Synchronized
20200324:20:57:52:035619 gpstate:tdxy-paas-test5:gpadmin-[INFO]:-   tdxy-paas-test7    /data5/greenplum/data/mirror/gpseg1    7001   Passive             Synchronized
20200324:20:57:52:035619 gpstate:tdxy-paas-test5:gpadmin-[INFO]:-   tdxy-paas-test8    /data7/greenplum/data/mirror/gpseg2    7002   Passive             Synchronized
20200324:20:57:52:035619 gpstate:tdxy-paas-test5:gpadmin-[INFO]:-   tdxy-paas-test7    /data3/greenplum/data/mirror/gpseg3    7000   Passive             Synchronized
20200324:20:57:52:035619 gpstate:tdxy-paas-test5:gpadmin-[INFO]:-   tdxy-paas-test8    /data5/greenplum/data/mirror/gpseg4    7001   Passive             Synchronized
20200324:20:57:52:035619 gpstate:tdxy-paas-test5:gpadmin-[INFO]:-   tdxy-paas-test11   /data7/greenplum/data/mirror/gpseg5    7002   Acting as Primary   Not In Sync
20200324:20:57:52:035619 gpstate:tdxy-paas-test5:gpadmin-[INFO]:-   tdxy-paas-test8    /data3/greenplum/data/mirror/gpseg6    7000   Passive             Synchronized
20200324:20:57:52:035619 gpstate:tdxy-paas-test5:gpadmin-[INFO]:-   tdxy-paas-test11   /data5/greenplum/data/mirror/gpseg7    7001   Passive             Synchronized
20200324:20:57:52:035619 gpstate:tdxy-paas-test5:gpadmin-[INFO]:-   tdxy-paas-test6    /data7/greenplum/data/mirror/gpseg8    7002   Passive             Synchronized
20200324:20:57:52:035619 gpstate:tdxy-paas-test5:gpadmin-[INFO]:-   tdxy-paas-test11   /data3/greenplum/data/mirror/gpseg9    7000   Passive             Synchronized
20200324:20:57:52:035619 gpstate:tdxy-paas-test5:gpadmin-[INFO]:-   tdxy-paas-test6    /data5/greenplum/data/mirror/gpseg10   7001   Passive             Synchronized
20200324:20:57:52:035619 gpstate:tdxy-paas-test5:gpadmin-[INFO]:-   tdxy-paas-test7    /data7/greenplum/data/mirror/gpseg11   7002   Passive             Synchronized
20200324:20:57:52:035619 gpstate:tdxy-paas-test5:gpadmin-[INFO]:--------------------------------------------------------------
20200324:20:57:52:035619 gpstate:tdxy-paas-test5:gpadmin-[WARNING]:-1 segment(s) configured as mirror(s) are acting as primaries
20200324:20:57:52:035619 gpstate:tdxy-paas-test5:gpadmin-[WARNING]:-1 mirror segment(s) acting as primaries are not synchronized

等了好久

20200324:20:58:34:035798 gpstate:tdxy-paas-test5:gpadmin-[INFO]:-----------------------------------------------------
20200324:20:58:34:035798 gpstate:tdxy-paas-test5:gpadmin-[INFO]:-   Segment Info
20200324:20:58:34:035798 gpstate:tdxy-paas-test5:gpadmin-[INFO]:-      Hostname                          = tdxy-paas-test6
20200324:20:58:34:035798 gpstate:tdxy-paas-test5:gpadmin-[INFO]:-      Address                           = tdxy-paas-test6
20200324:20:58:34:035798 gpstate:tdxy-paas-test5:gpadmin-[INFO]:-      Datadir                           = /data6/greenplum/data/primary/gpseg5
20200324:20:58:34:035798 gpstate:tdxy-paas-test5:gpadmin-[INFO]:-      Port                              = 6002
20200324:20:58:34:035798 gpstate:tdxy-paas-test5:gpadmin-[INFO]:-   Mirroring Info
20200324:20:58:34:035798 gpstate:tdxy-paas-test5:gpadmin-[INFO]:-      Current role                      = Mirror
20200324:20:58:34:035798 gpstate:tdxy-paas-test5:gpadmin-[INFO]:-      Preferred role                    = Primary
20200324:20:58:34:035798 gpstate:tdxy-paas-test5:gpadmin-[INFO]:-      Mirror status                     = Streaming
20200324:20:58:34:035798 gpstate:tdxy-paas-test5:gpadmin-[INFO]:-   Replication Info
20200324:20:58:34:035798 gpstate:tdxy-paas-test5:gpadmin-[INFO]:-      WAL Sent Location                 = 1/3C054FB0
20200324:20:58:34:035798 gpstate:tdxy-paas-test5:gpadmin-[INFO]:-      WAL Flush Location                = 1/3C054FB0
20200324:20:58:34:035798 gpstate:tdxy-paas-test5:gpadmin-[INFO]:-      WAL Replay Location               = 1/3C054FB0
20200324:20:58:34:035798 gpstate:tdxy-paas-test5:gpadmin-[INFO]:-   Status
20200324:20:58:34:035798 gpstate:tdxy-paas-test5:gpadmin-[INFO]:-      PID                               = 28614
20200324:20:58:34:035798 gpstate:tdxy-paas-test5:gpadmin-[WARNING]:-   Configuration reports status as   = Down                                    <<<<<<<<
20200324:20:58:34:035798 gpstate:tdxy-paas-test5:gpadmin-[WARNING]:-   Segment status                    = Down in configuration                   <<<<<<<<

GPCC状态一直是：

该节点一直还是处于down的状态，很奇怪。请问大家有遇到过类似的问题吗？谢谢