ORA-16766: Redo Apply is stopped

一、环境描述

Oracle 12c
aix 7.1

二、问题
MRP进程无法启动，报错ORA-16766

1.查看数据库状态

DGMGRL> ** show database verbose orcldg;**

Database - orcldg

  Role:               PHYSICAL STANDBY
  Intended State:     APPLY-ON
  Transport Lag:      3 days 12 hours 25 minutes 24 seconds (computed 75 seconds ago)
  Apply Lag:          3 days 12 hours 25 minutes 41 seconds (computed 75 seconds ago)
  Average Apply Rate: 361.00 KByte/s
  Active Apply Rate:  (unknown)
  Maximum Apply Rate: 340.77 MByte/s
  Real Time Query:    OFF
  Instance(s):
    orcldg

  Database Error(s):
  **  ORA-16766: Redo Apply is stopped**

  Database Warning(s):
    ORA-16857: standby disconnected from redo source for longer than specified threshold

2.尝试启动MRP进程

DGMGRL>  edit database 'orcldg' set state='apply-on';  
Succeeded.
DGMGRL> 

3.查看恢复进程

SQL> ** select process,status from v$managed_standby;**

PROCESS                              STATUS
------------------------------------ ------------------------------------------------
ARCH                                 CLOSING
ARCH                                 CONNECTED
ARCH                                 CLOSING
ARCH                                 CLOSING
RFS                                  IDLE

4.查看日志

SQL>** select message from v$dataguard_status;**

MESSAGE
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Creating archive destination file : +DATADG (1463121 blocks)
Creating archive destination file : +DATADG (1479835 blocks)
Creating archive destination file : +DATADG (1494118 blocks)
Creating archive destination file : +DATADG (1505856 blocks)
Creating archive destination file : +DATADG (1480255 blocks)
...............
Creating archive destination file : +DATADG (1489779 blocks)
Creating archive destination file : +DATADG (1486756 blocks)
Creating archive destination file : +DATADG (1444692 blocks)
Creating archive destination file : +DATADG (844053 blocks)

MESSAGE
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Media Recovery Log +DATADG/COREDG/ARCHIVELOG/2017_11_28/thread_1_seq_1770.1375.961232949
Media Recovery Log +DATADG/COREDG/ARCHIVELOG/2017_11_28/thread_2_seq_1163.1201.961232061
MRP0: Background Media Recovery terminated with error 1237
Managed Standby Recovery not using Real Time Apply
MRP0: Background Media Recovery process shutdown
Attempt to start background Managed Standby Recovery process
MRP0: Background Managed Standby Recovery process started
Managed Standby Recovery starting Real Time Apply
Media Recovery Log +DATADG/COREDG/ARCHIVELOG/2017_11_28/thread_1_seq_1770.1375.961232949
Media Recovery Log +DATADG/COREDG/ARCHIVELOG/2017_11_28/thread_2_seq_1163.1201.961232061
MRP0: Background Media Recovery terminated with error 1237

MESSAGE
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Managed Standby Recovery not using Real Time Apply
MRP0: Background Media Recovery process shutdown
Creating archive destination file : +DATADG (1685417 blocks)

5.无法创建文件，查看磁盘组空间

ASMCMD> lsdg
State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  NORMAL  N         512   4096  1048576    153600   153412            51200           51106              0             N  CRSDG/
MOUNTED  EXTERN  N         512   4096  1048576   1024000      128                0            ** 128**              0             N  DATADG/
ASMCMD> 

三、处理方法
删除部分归档日志，释放空间，然后启动MRP进程

rman target /
delete archivelog all;

DGMGRL>**  edit database 'orcldg' set state='apply-on'; ** 
Succeeded.
DGMGRL>