Friday, September 16, 2011

Background Managed Standby Recovery process not detected

We have had 2 instances where the dg broker did not start MAnaged Standby Recovery of the physical standby database. The first time was after a switchover and the next time was after hardware maintenance. The physical standby came up successfully and the dg broker started up, but Managed Standby Recovery never started. On both occasions I noticed these entries in the alert.log file:

...
Mon Sep 12 14:03:32 2011
Starting Data Guard Broker (DMON)
..
OCISessionBegin with PasswordVerifier succeeded
ALTER SYSTEM SET ..
ALTER SYSTEM SET ..
ALTER SYSTEM SET ..
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE  THROUGH ALL SWITCHOVER DISCONNECT  USING CURRENT LOGFILE
...
Mon Sep 12 14:08:06 2011
Background MRP initialization phase wait timeout   **
Background Managed Standby Recovery process not detected  **
Completed: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE  THROUGH ALL SWITCHOVER DISCONNECT  USING CURRENT LOGFILE


I bounced the database a few times but the same error persisted. The only way I could solve this was to “kick start” the recovery by manually recovering a few archivelogs:

SQL>ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE DISCONNECT FROM SESSION;
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE DISCONNECT FROM SESSION
*
ERROR at line 1:
ORA-01153: an incompatible media recovery is active

But
SQL>ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL;
only hangs

shutdown immediate also only hangs
shutdown abort did it.


SQL> startup mount

Start manual recovery before the DG Broker starts Managed Standby Recovery

SQL> RECOVER standby DATABASE  UNTIL CANCEL;

ORA-00279: change 10662691458828 generated at 09/13/2011 01:18:04 needed for thread 1
ORA-00289: suggestion : +PCASDGF/abcprd/archivelog/2011_09_13/thread_1_seq_82723.528.742958295
ORA-00280: change 10662691458828 for thread 1 is in sequence #82723


Specify log: {=suggested | filename | AUTO | CANCEL}
ORA-00279: change 10662691458829 generated at 09/13/2011 01:18:13 needed for thread 1
ORA-00289: suggestion : +PCASDGF/abcprd/archivelog/2011_09_13/thread_1_seq_82724.2112.742961677
ORA-00280: change 10662691458829 for thread 1 is in sequence #82724
ORA-00278: log file '+PCASDGF/abcprd/archivelog/2011_09_13/thread_1_seq_82723.528.742958295' no longer needed for this recovery


Specify log: {=suggested | filename | AUTO | CANCEL}

Hit enter, to manually recover a few logs and then

ORA-00279: change 10663204118583 generated at 09/13/2011 07:00:27 needed for thread 1
ORA-00289: suggestion : +PCASDGF/abcprd/archivelog/2011_09_13/thread_1_seq_82745.2365.742980297
ORA-00280: change 10663204118583 for thread 1 is in sequence #82745
ORA-00278: log file '+PCASDGF/abcprd/archivelog/2011_09_13/thread_1_seq_82744.2938.742978831' no longer needed for this recovery


Specify log: {=suggested | filename | AUTO | CANCEL}
CANCEL
Media recovery cancelled.

SQL>ALTER DATABASE RECOVER MANAGED STANDBY DATABASE THROUGH ALL SWITCHOVER DISCONNECT  USING CURRENT LOGFILE

Database altered.


Just to be sure I did another bounce a few minutes later and this time the DG Broker did everything it was supposed to and Managed Standby Recovery was started automatically.

6 comments:

Anonymous said...

This didn't work until I disabled and recreated Block Change Tracking - on the Standby (11.2.0.2.3) DB.

Regards,
Vin.

Anonymous said...

I have follow Vin advice and all works fine now. Also 11.2.0.2.3 DB RAC. Looks like a BUG

Regards,
Tomo

Unknown said...

Thanks for the feedback Vin & Tomo

Anonymous said...

Thanks for posting this solution. Had the same issue on DB 11.1.0.7.8.

PD said...

It helped (in the middle of the night!) - Thanks!

Anonymous said...

Thanks this worked!!!