Wednesday, March 6, 2019

ORACLE RAC : TERMINATING THE INSTANCE DUE TO ERROR 304

After refreshing my QA database using RMAN DUPLICATE, my instance startup was failing with below error


USER (ospid: 60897): terminating the instance due to error 304
Instance terminated by USER, pid = 60897
Wed Mar 06 02:14:34 2019
Starting ORACLE instance (normal)

Looking into the spfile, I noticed the the database was pulling wrong instance_number and thread numbers even though the DB configuration was correct


$ srvctl config database -d oradb
Database unique name: oradb
Database name: oradb
Oracle home: /opt/app/oradb/oracle/product/11.2.0.4
Oracle user: oradb
Spfile: +oradb_DATA/oradb/spfileoradb.ora
Domain: db.abc.com
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: oradb
Database instances: oradb1,oradb2
Disk Groups: oradb_DATA,oradb_FRA,oradb_REDO1,oradb_REDO2
Mount point paths: 
Services: oradb_1_2.db.abc.com,oradb_2_1.db.abc.com
Type: RAC
Database is administrator managed

From the pfile I created from the current spfile I could see 

*.instance_number=2
*.thread=2

To resolve this, bring down the complete database and just start the failing instance, in our case instance 1


srvctl start instance -d oradb -i oradb1

Once the instance is started, login to SQL and run below

SQL> alter system set instance_number=1 scope=spfile sid='oradb1';

System altered.

SQL> alter system set thread=1 scope=spfile sid='oradb1';

System altered.

SQL> alter system set undo_tablespace='UNDO01' sid='oradb1';

System altered.

shutdown the instance and start the complete database

SQL> shutdown immediate

srvctl start database -d oradb


Hope this resolves your issue.