Background Managed Standby Recovery process not detected

We have had 2 instances where the dg broker did not start MAnaged Standby Recovery of the physical standby database. The first time was after a switchover and the next time was after hardware maintenance. The physical standby came up successfully and the dg broker started up, but Managed Standby Recovery never started. On both occasions I noticed these entries in the alert.log file:

...

Mon Sep 12 14:03:32 2011

Starting Data Guard Broker (DMON)

..

OCISessionBegin with PasswordVerifier succeeded

ALTER SYSTEM SET ..

ALTER SYSTEM SET ..

ALTER SYSTEM SET ..

ALTER DATABASE RECOVER MANAGED STANDBY DATABASE  THROUGH ALL SWITCHOVER DISCONNECT  USING CURRENT LOGFILE

...

Mon Sep 12 14:08:06 2011

Background MRP initialization phase wait timeout   **

Background Managed Standby Recovery process not detected  **

Completed: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE  THROUGH ALL SWITCHOVER DISCONNECT  USING CURRENT LOGFILE

I bounced the database a few times but the same error persisted. The only way I could solve this was to “kick start” the recovery by manually recovering a few archivelogs:

SQL>ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE DISCONNECT FROM SESSION;

ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE DISCONNECT FROM SESSION

*

ERROR at line 1:

ORA-01153: an incompatible media recovery is active

But

SQL>ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL;

only hangs

shutdown immediate also only hangs

shutdown abort did it.

SQL> startup mount

Start manual recovery before the DG Broker starts Managed Standby Recovery

SQL> RECOVER standby DATABASE  UNTIL CANCEL;

ORA-00279: change 10662691458828 generated at 09/13/2011 01:18:04 needed for thread 1

ORA-00289: suggestion : +PCASDGF/abcprd/archivelog/2011_09_13/thread_1_seq_82723.528.742958295

ORA-00280: change 10662691458828 for thread 1 is in sequence #82723

Specify log: {=suggested | filename | AUTO | CANCEL}

ORA-00279: change 10662691458829 generated at 09/13/2011 01:18:13 needed for thread 1

ORA-00289: suggestion : +PCASDGF/abcprd/archivelog/2011_09_13/thread_1_seq_82724.2112.742961677

ORA-00280: change 10662691458829 for thread 1 is in sequence #82724

ORA-00278: log file '+PCASDGF/abcprd/archivelog/2011_09_13/thread_1_seq_82723.528.742958295' no longer needed for this recovery

Specify log: {=suggested | filename | AUTO | CANCEL}

Hit enter, to manually recover a few logs and then

ORA-00279: change 10663204118583 generated at 09/13/2011 07:00:27 needed for thread 1

ORA-00289: suggestion : +PCASDGF/abcprd/archivelog/2011_09_13/thread_1_seq_82745.2365.742980297

ORA-00280: change 10663204118583 for thread 1 is in sequence #82745

ORA-00278: log file '+PCASDGF/abcprd/archivelog/2011_09_13/thread_1_seq_82744.2938.742978831' no longer needed for this recovery

Specify log: {=suggested | filename | AUTO | CANCEL}

CANCEL

Media recovery cancelled.

SQL>ALTER DATABASE RECOVER MANAGED STANDBY DATABASE THROUGH ALL SWITCHOVER DISCONNECT  USING CURRENT LOGFILE

Database altered.

Just to be sure I did another bounce a few minutes later and this time the DG Broker did everything it was supposed to and Managed Standby Recovery was started automatically.