In sweet memories of my ever loving brother "kutty thambi " ARUN KUMAR

Tuesday, August 31, 2010

restore the server parameter file(spfile) from RMAN backup

1.Connect to target Database with RMAN.

i)If you have lost your spfile and dont have pfile then, follow the steps to recover spfile using RMAN.

RMAN>CONNECT TARGET /

ii)If your database is not up and you don't use recovery catalog then use,
RMAN>CONNECT TARGET /
RMAN>SET DBID=3148849783

2)Start the instance with dummy parameter file.
RMAN>STARTUP FORCE NOMOUNT

Example:

3)Restore server parameter file.
To restore in default location,
RMAN> RESTORE SPFILE FROM AUTOBACKUP;

To restore in another location,
RMAN> RESTORE SPFILE TO 'new_location' FROM AUTOBACKUP;
RMAN> restore spfile to '/tmp/spfilejay.ora' from autobackup;

If you want to restore to a pfile then use,
RMAN> restore spfile to pfile '/tmp/initjay.ora';

4)Restore Spfile from control file autobackup;
RMAN> run
2> { set controlfile autobackup format for device type disk to '/u01/app/oracle/backup/JAY_%F';
3> restore spfile from autobackup;
4> }

Note after restore spfile from RMAN shutdown and startup the database
else you will receive the error message
ORA-00205: error in identifying control file, check alert log for more info


1. Restore spfile from autobackup;
RMAN> set dbid=3148849783

executing command: SET DBID

RMAN> startup force nomount;

startup failed: ORA-01078: failure in processing system parameters
LRM-00109: could not open parameter file '/u01/app/oracle/product/10.2.0/db_1/dbs/initjay.ora'

starting Oracle instance without parameter file for retrival of spfile
Oracle instance started

Total System Global Area 159383552 bytes

Fixed Size 1218268 bytes
Variable Size 54528292 bytes
Database Buffers 100663296 bytes
Redo Buffers 2973696 bytes

RMAN> restore spfile from autobackup;

2. Restore spfile to non-default location
RMAN> restore spfile to '/tmp/spfilejay.ora' from autobackup;

Starting restore at 31-AUG-10
using channel ORA_DISK_1

recovery area destination: /u01/app/oracle/flash_recovery_area
database name (or database unique name) used for search: JAY
channel ORA_DISK_1: no autobackups found in the recovery area
channel ORA_DISK_1: looking for autobackup on day: 20100831
channel ORA_DISK_1: autobackup found: /u01/app/oracle/backup/JAY_c-3148849783-20100831-00
channel ORA_DISK_1: SPFILE restore from autobackup complete
Finished restore at 31-AUG-10

3. If you want to restore to a pfile from restored spfile

RMAN> restore spfile to pfile '/tmp/initjay.ora';

Starting restore at 31-AUG-10
allocated channel: ORA_DISK_1
channel ORA_DISK_1: sid=139 devtype=DISK

channel ORA_DISK_1: starting datafile backupset restore
channel ORA_DISK_1: restoring SPFILE to PFILE
output filename=/tmp/initjay.ora
channel ORA_DISK_1: reading from backup piece /u01/app/oracle/backup/JAY_c-3148849783-20100831-00
channel ORA_DISK_1: restored backup piece 1
piece handle=/u01/app/oracle/backup/JAY_c-3148849783-20100831-00 tag=TAG20100831T174354
channel ORA_DISK_1: restore complete, elapsed time: 00:00:04
Finished restore at 31-AUG-10

4.Restore spfile from Controlfile AutoBackup
RMAN> set dbid=3148849783

executing command: SET DBID

RMAN> startup force nomount;

startup failed: ORA-01078: failure in processing system parameters
LRM-00109: could not open parameter file '/u01/app/oracle/product/10.2.0/db_1/dbs/initjay.ora'

starting Oracle instance without parameter file for retrival of spfile
Oracle instance started

Total System Global Area 159383552 bytes

Fixed Size 1218268 bytes
Variable Size 54528292 bytes
Database Buffers 100663296 bytes
Redo Buffers 2973696 bytes

RMAN> run
2> { set controlfile autobackup format for device type disk to '/u01/app/oracle/backup/JAY_%F';
3> restore spfile from autobackup; 4> }

executing command: SET CONTROLFILE AUTOBACKUP FORMAT

Starting restore at 31-AUG-10
using channel ORA_DISK_1

channel ORA_DISK_1: looking for autobackup on day: 20100831
channel ORA_DISK_1: autobackup found: /u01/app/oracle/backup/JAY_c-3148849783-20100831-00
channel ORA_DISK_1: SPFILE restore from autobackup complete
Finished restore at 31-AUG-10




Reference:
Oracle® Database Backup and Recovery Basics 10g Release 2 (10.2)

ORA-00283: ORA-00314: ORA-00312: Error and solution

While trying to restore controlfile from backup , while recovery i got the error

ORA-00283: recovery session canceled due to errors
ORA-00314: log 2 of thread 1, expected sequence# 2 doesn't match 11
ORA-00312: online log 2 thread 1: '/u01/app/oracle/oradata/jay/redo02.log'

Reason:

In any case, the ArchiveLog must be backed up. Else, a RESTORE alone cannot do a RECOVER.

If your database backup did not include the ArchiveLogs, then the backup you created does not have the Redo information that Oracle must apply to the Database Backup. That is why you got the "unknown log'.
Also, if the controlfile backup is before the archivelog backup, the controlfile, even when restored, is not aware of the archivelogs in the backup created subsequent to it.
RMAN can still do a RECOVER , implicitly using the "BACKUP CONTROLFILE" and doing a rollforward but it needs to have to restore the ArchiveLog first -- and the information about which Backupset contains the ArchiveLog is not available to it. You would need to CATALOG the ArchiveLog BackupSet and then restore the archivelogs from there.
(If you use an RMAN Recovery Catalog database, then of course, the Catalog has information about the ArchiveLogs and the BackupSets containing the ArchiveLogs so RMAN queries the Catalog to identify the BackupSets and extracts the necessary ArchiveLogs from the Backupsets).

Solution:
SQL> archive log list;
Database log mode              Archive Mode
Automatic archival             Enabled
Archive destination            USE_DB_RECOVERY_FILE_DEST
Oldest online log sequence     11
Next log sequence to archive   13
Current log sequence           13
SQL> ! rm -rf /u01/app/oracle/oradata/jay/control*

SQL> shu immediate
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> startup
ORACLE instance started.

Total System Global Area  444596224 bytes
Fixed Size                  1219904 bytes
Variable Size             188744384 bytes
Database Buffers          251658240 bytes
Redo Buffers                2973696 bytes
ORA-00205: error in identifying control file, check alert log for more info


[oracle@rac1 ~]$ rlrman target /

Recovery Manager: Release 10.2.0.1.0 - Production on Tue Aug 31 11:44:44 2010

Copyright (c) 1982, 2005, Oracle.  All rights reserved.

connected to target database: jay (not mounted)

RMAN> restore controlfile from '/u01/control01.ctl';

Starting restore at 31-AUG-10
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: sid=155 devtype=DISK

channel ORA_DISK_1: copied control file copy
output filename=/u01/app/oracle/oradata/jay/control01.ctl
output filename=/u01/app/oracle/oradata/jay/control02.ctl
output filename=/u01/app/oracle/oradata/jay/control03.ctl
Finished restore at 31-AUG-10

RMAN> alter database mount;

database mounted
released channel: ORA_DISK_1

RMAN> alter database open resetlogs;

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of alter db command at 08/31/2010 11:45:45
ORA-01139: RESETLOGS option only valid after an incomplete database recovery

RMAN> alter database open;

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of alter db command at 08/31/2010 11:45:51
ORA-01122: database file 1 failed verification check
ORA-01110: data file 1: '/u01/app/oracle/oradata/jay/system01.dbf'
ORA-01207: file is more recent than control file - old control file

RMAN> recover database;

Starting recover at 31-AUG-10
allocated channel: ORA_DISK_1
channel ORA_DISK_1: sid=155 devtype=DISK
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of recover command at 08/31/2010 11:46:04
RMAN-06094: datafile 1 must be restored

RMAN> restore database;

Starting restore at 31-AUG-10
using channel ORA_DISK_1

channel ORA_DISK_1: starting datafile backupset restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
restoring datafile 00002 to /u01/app/oracle/oradata/jay/undotbs01.dbf
restoring datafile 00003 to /u01/app/oracle/oradata/jay/sysaux01.dbf
restoring datafile 00004 to /u01/app/oracle/oradata/jay/users01.dbf
channel ORA_DISK_1: reading from backup piece /u01/app/oracle/backup/jay/1mlmljvq_1_1
channel ORA_DISK_1: restored backup piece 1
piece handle=/u01/app/oracle/backup/jay/1mlmljvq_1_1 tag=TAG20100830T185905
channel ORA_DISK_1: restore complete, elapsed time: 00:00:46
channel ORA_DISK_1: starting datafile backupset restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
restoring datafile 00001 to /u01/app/oracle/oradata/jay/system01.dbf
channel ORA_DISK_1: reading from backup piece /u01/app/oracle/backup/jay/1vlmncgc_1_1
channel ORA_DISK_1: restored backup piece 1
piece handle=/u01/app/oracle/backup/jay/1vlmncgc_1_1 tag=TAG20100831T110340
channel ORA_DISK_1: restore complete, elapsed time: 00:01:37
Finished restore at 31-AUG-10

RMAN> recover database;

Starting recover at 31-AUG-10
using channel ORA_DISK_1

starting media recovery

archive log thread 1 sequence 2 is already on disk as file /u01/app/oracle/flash_recovery_area/JAY/archivelog/2010_08_30/o1_mf_1_2_67qdks9h_.arc
archive log thread 1 sequence 3 is already on disk as file /u01/app/oracle/flash_recovery_area/JAY/archivelog/2010_08_30/o1_mf_1_3_67qfn02l_.arc
archive log thread 1 sequence 4 is already on disk as file /u01/app/oracle/flash_recovery_area/JAY/archivelog/2010_08_30/o1_mf_1_4_67qfn0cq_.arc
archive log thread 1 sequence 5 is already on disk as file /u01/app/oracle/flash_recovery_area/JAY/archivelog/2010_08_30/o1_mf_1_5_67qfmzj5_.arc
archive log thread 1 sequence 1 is already on disk as file /u01/app/oracle/flash_recovery_area/JAY/archivelog/2010_08_30/o1_mf_1_1_67qgpfo4_.arc
archive log thread 1 sequence 2 is already on disk as file /u01/app/oracle/flash_recovery_area/JAY/archivelog/2010_08_31/o1_mf_1_2_67s357q5_.arc
archive log thread 1 sequence 3 is already on disk as file /u01/app/oracle/flash_recovery_area/JAY/archivelog/2010_08_31/o1_mf_1_3_67s357v5_.arc
archive log thread 1 sequence 4 is already on disk as file /u01/app/oracle/flash_recovery_area/JAY/archivelog/2010_08_31/o1_mf_1_4_67s357hd_.arc
archive log thread 1 sequence 1 is already on disk as file /u01/app/oracle/flash_recovery_area/JAY/archivelog/2010_08_31/o1_mf_1_1_67s44djv_.arc
archive log thread 1 sequence 2 is already on disk as file /u01/app/oracle/flash_recovery_area/JAY/archivelog/2010_08_31/o1_mf_1_2_67s44dr5_.arc
archive log thread 1 sequence 3 is already on disk as file /u01/app/oracle/flash_recovery_area/JAY/archivelog/2010_08_31/o1_mf_1_3_67s44dwq_.arc
archive log thread 1 sequence 1 is already on disk as file /u01/app/oracle/flash_recovery_area/JAY/archivelog/2010_08_31/o1_mf_1_1_67s4jbd0_.arc
archive log thread 1 sequence 2 is already on disk as file /u01/app/oracle/flash_recovery_area/JAY/archivelog/2010_08_31/o1_mf_1_2_67s4jc79_.arc
archive log thread 1 sequence 3 is already on disk as file /u01/app/oracle/flash_recovery_area/JAY/archivelog/2010_08_31/o1_mf_1_3_67s4jgmp_.arc
archive log thread 1 sequence 4 is already on disk as file /u01/app/oracle/flash_recovery_area/JAY/archivelog/2010_08_31/o1_mf_1_4_67s4jgsl_.arc
archive log thread 1 sequence 5 is already on disk as file /u01/app/oracle/flash_recovery_area/JAY/archivelog/2010_08_31/o1_mf_1_5_67s4jmr8_.arc
archive log thread 1 sequence 6 is already on disk as file /u01/app/oracle/flash_recovery_area/JAY/archivelog/2010_08_31/o1_mf_1_6_67s4jmvt_.arc
archive log thread 1 sequence 7 is already on disk as file /u01/app/oracle/flash_recovery_area/JAY/archivelog/2010_08_31/o1_mf_1_7_67s4jrxw_.arc
archive log thread 1 sequence 8 is already on disk as file /u01/app/oracle/flash_recovery_area/JAY/archivelog/2010_08_31/o1_mf_1_8_67s4js5b_.arc
archive log thread 1 sequence 9 is already on disk as file /u01/app/oracle/flash_recovery_area/JAY/archivelog/2010_08_31/o1_mf_1_9_67s633xq_.arc
archive log thread 1 sequence 10 is already on disk as file /u01/app/oracle/flash_recovery_area/JAY/archivelog/2010_08_31/o1_mf_1_10_67s6340m_.arc
archive log thread 1 sequence 11 is already on disk as file /u01/app/oracle/flash_recovery_area/JAY/archivelog/2010_08_31/o1_mf_1_11_67s633qq_.arc
archive log thread 1 sequence 1 is already on disk as file /u01/app/oracle/flash_recovery_area/JAY/archivelog/2010_08_31/o1_mf_1_1_67s6698c_.arc
archive log thread 1 sequence 2 is already on disk as file /u01/app/oracle/flash_recovery_area/JAY/archivelog/2010_08_31/o1_mf_1_2_67s66btj_.arc
archive log thread 1 sequence 3 is already on disk as file /u01/app/oracle/flash_recovery_area/JAY/archivelog/2010_08_31/o1_mf_1_3_67s66jb0_.arc
archive log filename=/u01/app/oracle/flash_recovery_area/JAY/archivelog/2010_08_30/o1_mf_1_2_67qdks9h_.arc thread=1 sequence=2
archive log filename=/u01/app/oracle/flash_recovery_area/JAY/archivelog/2010_08_30/o1_mf_1_3_67qfn02l_.arc thread=1 sequence=3
archive log filename=/u01/app/oracle/flash_recovery_area/JAY/archivelog/2010_08_30/o1_mf_1_4_67qfn0cq_.arc thread=1 sequence=4
archive log filename=/u01/app/oracle/flash_recovery_area/JAY/archivelog/2010_08_30/o1_mf_1_5_67qfmzj5_.arc thread=1 sequence=5
archive log filename=/u01/app/oracle/flash_recovery_area/JAY/archivelog/2010_08_30/o1_mf_1_1_67qgpfo4_.arc thread=1 sequence=1
archive log filename=/u01/app/oracle/flash_recovery_area/JAY/archivelog/2010_08_31/o1_mf_1_2_67s357q5_.arc thread=1 sequence=2
archive log filename=/u01/app/oracle/flash_recovery_area/JAY/archivelog/2010_08_31/o1_mf_1_3_67s357v5_.arc thread=1 sequence=3
archive log filename=/u01/app/oracle/flash_recovery_area/JAY/archivelog/2010_08_31/o1_mf_1_4_67s357hd_.arc thread=1 sequence=4
archive log filename=/u01/app/oracle/flash_recovery_area/JAY/archivelog/2010_08_31/o1_mf_1_1_67s44djv_.arc thread=1 sequence=1
archive log filename=/u01/app/oracle/flash_recovery_area/JAY/archivelog/2010_08_31/o1_mf_1_2_67s44dr5_.arc thread=1 sequence=2
archive log filename=/u01/app/oracle/flash_recovery_area/JAY/archivelog/2010_08_31/o1_mf_1_3_67s44dwq_.arc thread=1 sequence=3
archive log filename=/u01/app/oracle/flash_recovery_area/JAY/archivelog/2010_08_31/o1_mf_1_1_67s4jbd0_.arc thread=1 sequence=1
archive log filename=/u01/app/oracle/flash_recovery_area/JAY/archivelog/2010_08_31/o1_mf_1_2_67s4jc79_.arc thread=1 sequence=2
archive log filename=/u01/app/oracle/flash_recovery_area/JAY/archivelog/2010_08_31/o1_mf_1_3_67s4jgmp_.arc thread=1 sequence=3
archive log filename=/u01/app/oracle/flash_recovery_area/JAY/archivelog/2010_08_31/o1_mf_1_4_67s4jgsl_.arc thread=1 sequence=4
archive log filename=/u01/app/oracle/flash_recovery_area/JAY/archivelog/2010_08_31/o1_mf_1_5_67s4jmr8_.arc thread=1 sequence=5
archive log filename=/u01/app/oracle/flash_recovery_area/JAY/archivelog/2010_08_31/o1_mf_1_6_67s4jmvt_.arc thread=1 sequence=6
archive log filename=/u01/app/oracle/flash_recovery_area/JAY/archivelog/2010_08_31/o1_mf_1_7_67s4jrxw_.arc thread=1 sequence=7
archive log filename=/u01/app/oracle/flash_recovery_area/JAY/archivelog/2010_08_31/o1_mf_1_8_67s4js5b_.arc thread=1 sequence=8
archive log filename=/u01/app/oracle/flash_recovery_area/JAY/archivelog/2010_08_31/o1_mf_1_9_67s633xq_.arc thread=1 sequence=9
archive log filename=/u01/app/oracle/flash_recovery_area/JAY/archivelog/2010_08_31/o1_mf_1_10_67s6340m_.arc thread=1 sequence=10
archive log filename=/u01/app/oracle/flash_recovery_area/JAY/archivelog/2010_08_31/o1_mf_1_11_67s633qq_.arc thread=1 sequence=11
archive log filename=/u01/app/oracle/flash_recovery_area/JAY/archivelog/2010_08_31/o1_mf_1_1_67s6698c_.arc thread=1 sequence=1
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of recover command at 08/31/2010 11:53:18
ORA-00283: recovery session canceled due to errors
RMAN-11003: failure during parse/execution of SQL statement: alter database recover logfile '/u01/app/oracle/flash_recovery_area/JAY/archivelog/2010_08_31/o1_mf_1_1_67s6698c_.arc'
ORA-00283: recovery session canceled due to errors
ORA-00314: log 2 of thread 1, expected sequence# 2 doesn't match 11
ORA-00312: online log 2 thread 1: '/u01/app/oracle/oradata/jay/redo02.log'

RMAN> exit


Recovery Manager complete.

SQL> shu immediate
ORA-01109: database not open


Database dismounted.
ORACLE instance shut down.
SQL> startup
ORACLE instance started.

Total System Global Area  444596224 bytes
Fixed Size                  1219904 bytes
Variable Size             188744384 bytes
Database Buffers          251658240 bytes
Redo Buffers                2973696 bytes
Database mounted.
ORA-01113: file 1 needs media recovery
ORA-01110: data file 1: '/u01/app/oracle/oradata/jay/system01.dbf'


SQL> alter database clear logfile '/u01/app/oracle/oradata/jay/redo02.log'; 
Database altered.


SQL> alter database open; 
alter database open
*
ERROR at line 1:
ORA-01113: file 1 needs media recovery
ORA-01110: data file 1: '/u01/app/oracle/oradata/jay/system01.dbf'


SQL>  recover database until cancel;
ORA-00279: change 696770 generated at 08/31/2010 11:23:13 needed for thread 1
ORA-00289: suggestion :
/u01/app/oracle/flash_recovery_area/JAY/archivelog/2010_08_31/o1_mf_1_2_%u_.arc
ORA-00280: change 696770 for thread 1 is in sequence #2


Specify log: {=suggested | filename | AUTO | CANCEL}
cancel
Media recovery cancelled.
SQL> alter database open resetlogs;

Database altered.

SQL> archive log list;
Database log mode              Archive Mode
Automatic archival             Enabled
Archive destination            USE_DB_RECOVERY_FILE_DEST
Oldest online log sequence     1
Next log sequence to archive   1
Current log sequence           1
SQL>

Saturday, August 28, 2010

Disaster recovery using RMAN Demo


As part of disaster recovery exercise or to test the validity of a RMAN backup,  a full restore and recovery of databases can be performed on scratch or test servers utilising the production RMAN backups which have been restored from the tape backups on these test or scratch servers.

This note will illustrate the above procedure by detailing the steps required to restore the backup of a production database 
1.  same server with same location
2.  Different server with same location

The following assumptions are made in this note:

1.The RMAN backups have been restored from tape backups to the same backup location on the test server as the production server where the backup was originally taken
2.The identical directory structure as is present on production has been created on the test server. This will apply to not only the location of the database files (data, control files, redo log files), but also to the bdump,cdump, udump and adump locations. 
3.Controlfile autobackup has been enabled. This is important.

#Overview

Restore the spfile from the autobackup
Restore the controlfile from the autobackup
Restore the data files
Recover by applying archived redo log files
Open the database with resetlogs

1.Restore the SPFILE

[oracle@rac1 backup]$ rman target /

Recovery Manager: Release 10.2.0.1.0 - Production on Sat Aug 28 03:12:17 2010

Copyright (c) 1982, 2005, Oracle.  All rights reserved.

connected to target database (not started)

RMAN> set dbid=3148849783 
executing command: SET DBID

RMAN> startup force nomount; 
startup failed: ORA-01078: failure in processing system parameters
LRM-00109: could not open parameter file '/u01/app/oracle/product/10.2.0/db_1/dbs/initjay.ora'

starting Oracle instance without parameter file for retrival of spfile
Oracle instance started

Total System Global Area     159383552 bytes

Fixed Size                     1218268 bytes
Variable Size                 54528292 bytes
Database Buffers             100663296 bytes
Redo Buffers                   2973696 bytes

RMAN> run 
2> { set controlfile autobackup format for device type disk to '/u01/app/oracle/backup/JAY_%F';
3> RESTORE SPFILE FROM AUTOBACKUP; 
4> } 
executing command: SET CONTROLFILE AUTOBACKUP FORMAT
using target database control file instead of recovery catalog

Starting restore at 28-AUG-10
allocated channel: ORA_DISK_1
channel ORA_DISK_1: sid=36 devtype=DISK

channel ORA_DISK_1: looking for autobackup on day: 20100828
channel ORA_DISK_1: autobackup found: /u01/app/oracle/backup/JAY_c-3148849783-20100828-03
channel ORA_DISK_1: SPFILE restore from autobackup complete
Finished restore at 28-AUG-10

RMAN>
RMAN> shutdown immediate

Oracle instance shut down

2.Restore the Control File

RMAN> startup nomount

connected to target database (not started)
Oracle instance started

Total System Global Area     444596224 bytes

Fixed Size                     1219904 bytes
Variable Size                130024128 bytes
Database Buffers             310378496 bytes
Redo Buffers                   2973696 bytes

RMAN> set dbid=3148849783 
executing command: SET DBID

RMAN> run
 2> { set controlfile autobackup format for device type disk to '/u01/app/oracle/backup/JAY_%F';
3> RESTORE controlfile FROM AUTOBACKUP; 
4> }

executing command: SET CONTROLFILE AUTOBACKUP FORMAT

Starting restore at 28-AUG-10
allocated channel: ORA_DISK_1
channel ORA_DISK_1: sid=156 devtype=DISK

recovery area destination: /u01/app/oracle/flash_recovery_area
database name (or database unique name) used for search: JAY
channel ORA_DISK_1: autobackup found in the recovery area
channel ORA_DISK_1: autobackup found: /u01/app/oracle/flash_recovery_area/JAY/autobackup/2010_08_28/o1_mf_s_728189319_67jbxhyh_.bkp
channel ORA_DISK_1: control file restore from autobackup complete
output filename=/u01/app/oracle/oradata/jay/control01.ctl
output filename=/u01/app/oracle/oradata/jay/control02.ctl
output filename=/u01/app/oracle/oradata/jay/control03.ctl
Finished restore at 28-AUG-10


3.Restore the database


RMAN> alter database mount;

database mounted
released channel: ORA_DISK_1

RMAN> restore database;

Starting restore at 28-AUG-10
Starting implicit crosscheck backup at 28-AUG-10
allocated channel: ORA_DISK_1
channel ORA_DISK_1: sid=156 devtype=DISK
Crosschecked 8 objects
Finished implicit crosscheck backup at 28-AUG-10

Starting implicit crosscheck copy at 28-AUG-10
using channel ORA_DISK_1
Finished implicit crosscheck copy at 28-AUG-10

searching for all files in the recovery area
cataloging files...
cataloging done

List of Cataloged Files
=======================
File Name: /u01/app/oracle/flash_recovery_area/JAY/autobackup/2010_08_28/o1_mf_s_728189319_67jbxhyh_.bkp
File Name: /u01/app/oracle/flash_recovery_area/JAY/archivelog/2010_08_28/o1_mf_1_26_67jc8po5_.arc
File Name: /u01/app/oracle/flash_recovery_area/JAY/archivelog/2010_08_28/o1_mf_1_24_67jc5rko_.arc
File Name: /u01/app/oracle/flash_recovery_area/JAY/archivelog/2010_08_28/o1_mf_1_27_67jc90og_.arc
File Name: /u01/app/oracle/flash_recovery_area/JAY/archivelog/2010_08_28/o1_mf_1_25_67jc6knx_.arc
File Name: /u01/app/oracle/flash_recovery_area/JAY/archivelog/2010_08_28/o1_mf_1_23_67jc4rpk_.arc

using channel ORA_DISK_1

channel ORA_DISK_1: starting datafile backupset restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
restoring datafile 00001 to /u01/app/oracle/oradata/jay/system01.dbf
restoring datafile 00002 to /u01/app/oracle/oradata/jay/undotbs01.dbf
restoring datafile 00003 to /u01/app/oracle/oradata/jay/sysaux01.dbf
restoring datafile 00004 to /u01/app/oracle/oradata/jay/users01.dbf
channel ORA_DISK_1: reading from backup piece /u01/app/oracle/backup/JAY_0mlmei9b_1_1.bckp
channel ORA_DISK_1: restored backup piece 1
piece handle=/u01/app/oracle/backup/JAY_0mlmei9b_1_1.bckp tag=TAG20100828T024706channel ORA_DISK_1: restore complete, elapsed time: 00:01:47
Finished restore at 28-AUG-10

4.Recover the database

RMAN> recover database;
Starting recover at 28-AUG-10
using channel ORA_DISK_1

starting media recovery

archive log thread 1 sequence 22 is already on disk as file /u01/app/oracle/flash_recovery_area/JAY/archivelog/2010_08_28/o1_mf_1_22_67jbxb7n_.arc
archive log thread 1 sequence 23 is already on disk as file /u01/app/oracle/flash_recovery_area/JAY/archivelog/2010_08_28/o1_mf_1_23_67jc4rpk_.arc
archive log thread 1 sequence 24 is already on disk as file /u01/app/oracle/flash_recovery_area/JAY/archivelog/2010_08_28/o1_mf_1_24_67jc5rko_.arc
archive log thread 1 sequence 25 is already on disk as file /u01/app/oracle/flash_recovery_area/JAY/archivelog/2010_08_28/o1_mf_1_25_67jc6knx_.arc
archive log thread 1 sequence 26 is already on disk as file /u01/app/oracle/flash_recovery_area/JAY/archivelog/2010_08_28/o1_mf_1_26_67jc8po5_.arc
archive log thread 1 sequence 27 is already on disk as file /u01/app/oracle/flash_recovery_area/JAY/archivelog/2010_08_28/o1_mf_1_27_67jc90og_.arc
archive log thread 1 sequence 28 is already on disk as file /u01/app/oracle/oradata/jay/redo03.log
archive log filename=/u01/app/oracle/flash_recovery_area/JAY/archivelog/2010_08_28/o1_mf_1_22_67jbxb7n_.arc thread=1 sequence=22
archive log filename=/u01/app/oracle/flash_recovery_area/JAY/archivelog/2010_08_28/o1_mf_1_23_67jc4rpk_.arc thread=1 sequence=23
archive log filename=/u01/app/oracle/flash_recovery_area/JAY/archivelog/2010_08_28/o1_mf_1_24_67jc5rko_.arc thread=1 sequence=24
archive log filename=/u01/app/oracle/flash_recovery_area/JAY/archivelog/2010_08_28/o1_mf_1_25_67jc6knx_.arc thread=1 sequence=25
archive log filename=/u01/app/oracle/flash_recovery_area/JAY/archivelog/2010_08_28/o1_mf_1_26_67jc8po5_.arc thread=1 sequence=26
archive log filename=/u01/app/oracle/flash_recovery_area/JAY/archivelog/2010_08_28/o1_mf_1_27_67jc90og_.arc thread=1 sequence=27
archive log filename=/u01/app/oracle/oradata/jay/redo03.log thread=1 sequence=28media recovery complete, elapsed time: 00:00:06
Finished recover at 28-AUG-10 
 
 The recovery will fail at a point where it cannot restore any more archived redo log
files.
I had full backup so no error else it gives error , no problem just exit from rman relogin 
and open the database with resetlog.
 
RMAN> sql 'alter database open resetlogs';
sql statement: alter database open resetlogs

RMAN> exit


Recovery Manager complete.

5.Verify the database
[oracle@rac1 backup]$ sqlplus '/as sysdba'

SQL*Plus: Release 10.2.0.1.0 - Production on Sat Aug 28 03:22:35 2010

Copyright (c) 1982, 2005, Oracle.  All rights reserved.


Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - Production
With the Partitioning and OLAP options

SQL> archive log list;
Database log mode              Archive Mode
Automatic archival             Enabled
Archive destination            USE_DB_RECOVERY_FILE_DEST
Oldest online log sequence     0
Next log sequence to archive   1
Current log sequence           1
SQL> select name,dbid from v$database;

NAME            DBID
--------- ----------
JAY       3148849783

SQL>


DBCA hangs at 85%

cause:

kernel BUG at kernel/exit.c:904!
kernel/exit.c:904!
solution:

Upgraded the Kernel to version RHEL4.7

also
the same solution for the Enterprise Manager Agent Starts on RHEL 4

refer metalink Doc ID: 729543.1
Linux Crashes when Enterprise Manager Agent Starts on RHEL 4 Update 6 and 7

reference:
http://forums.oracle.com/forums/thread.jspa?threadID=871254
http://newappsdba.blogspot.com/2009/06/kernel-bug-and-oracle-enterprise-linux.html

Friday, August 27, 2010

Transparent Application Failover TAF Demo

Transparent Application Failover TAF Demo

here, i demonstrated the demo of TAF , i concentrate only the practical demo, for theoretical part refer the oracle documentation.

1. BASIC METHOD
ADD THE ENTRIES TO THE RAC DATABASE NODE1 tnsnames.ora file
CDBS_TAF =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = racnode1-vip.localdomain)(PORT = 1521))
    (ADDRESS = (PROTOCOL = TCP)(HOST = racnode2-vip.localdomain)(PORT = 1521))
    (LOAD_BALANCE = yes)
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = cdbs)
      (FAILOVER_MODE=
      (TYPE=SELECT)
      (METHOD=BASIC)
     )
    )
  )
Make sure all the resources are up and running.
root@racnode1 bin]# ./crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------
ora....s1.inst application    ONLINE    ONLINE    racnode1
ora....s2.inst application    ONLINE    ONLINE    racnode2
ora....bs1.srv application    ONLINE    ONLINE    racnode1
ora....bs2.srv application    ONLINE    ONLINE    racnode2
ora...._srv.cs application    ONLINE    ONLINE    racnode2
ora.cdbs.db    application    ONLINE    ONLINE    racnode2
ora....SM1.asm application    ONLINE    ONLINE    racnode1
ora....E1.lsnr application    ONLINE    ONLINE    racnode1
ora....de1.gsd application    ONLINE    ONLINE    racnode1
ora....de1.ons application    ONLINE    ONLINE    racnode1
ora....de1.vip application    ONLINE    ONLINE    racnode1
ora....SM2.asm application    ONLINE    ONLINE    racnode2
ora....E2.lsnr application    ONLINE    ONLINE    racnode2
ora....de2.gsd application    ONLINE    ONLINE    racnode2
ora....de2.ons application    ONLINE    ONLINE    racnode2
ora....de2.vip application    ONLINE    ONLINE    racnode2

[oracle@racnode1 admin]$ sqlplus system/oracle@CDBS_TAF

SQL*Plus: Release 10.2.0.1.0 - Production on Thu Aug 26 17:36:45 2010

Copyright (c) 1982, 2005, Oracle.  All rights reserved.


Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - Production
With the Partitioning, Real Application Clusters, OLAP and Data Mining options

SQL> show user;
USER is "SYSTEM"

SQL> col osuser format a10
SQL> col username format a10
SQL> col failover_type format a15
SQL> col failover_method format a18

SQL> select username,osuser,failover_method,failover_type from v$session
  2  where username='SYSTEM';

USERNAME   OSUSER     FAILOVER_METHOD    FAILOVER_TYPE
---------- ---------- ------------------ ---------------
SYSTEM     oracle     BASIC              SELECT


SQL> select instance_name from v$instance;

INSTANCE_NAME
----------------
cdbs1

SQL> select count(*) from demotb;

  COUNT(*)
----------
       954

SQL> select * from demotb;

run the long running query


simultaneously open another terminal and give shutdown abort instance cdbs1

SQL> shu abort
ORACLE instance shut down.

but the query runs smoothly without any interruption after the completion of the query, verify the instance name

SQL>select instance_name from v$instance;

INSTANCE_NAME
----------------
cdbs2

automatically it switch over to the instance cdbs2 , because of this user can access the database and no failure in queries. users can access without interruption. 
USER will not receive any error message while running query.

2. PRECONNECT METHOD
Add the entries to the tnsnames.ora file on both nodes.
CDBS1_TAF =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = racnode1-vip.localdomain)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = cdbs)
      (INSTANCE_NAME = cdbs1)
      (FAILOVER_MODE=
      (BACKUP=CDBS2_TAF)
      (TYPE=SELECT)
      (METHOD=PRECONNECT)
     )
    )
  )


CDBS2_TAF =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = racnode2-vip.localdomain)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = cdbs)
      (INSTANCE_NAME = cdbs2)
      (FAILOVER_MODE=
      (BACKUP=CDBS1_TAF)
      (TYPE=SELECT)
      (METHOD=PRECONNECT)
     )
    )
  )

connect to the database on node1
[oracle@racnode1 ~]$ sqlplus system/oracle@CDBS1_TAF

SQL*Plus: Release 10.2.0.1.0 - Production on Fri Aug 27 12:23:11 2010

Copyright (c) 1982, 2005, Oracle.  All rights reserved.


Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - Production
With the Partitioning, Real Application Clusters, OLAP and Data Mining options

SQL> SELECT INSTANCE_NAME FROM V$INSTANCE;

INSTANCE_NAME
----------------
cdbs1

SQL> COL USERNAME FOR a10
SQL> col osuser for a10
SQL> col failover_type format a15
SQL> col failover_method for a18
SQL> select username,failover_type,failover_method from v$session
  2  where username='SYSTEM';

USERNAME   FAILOVER_TYPE   FAILOVER_METHOD
---------- --------------- ------------------
SYSTEM     SELECT          PRECONNECT

SQL> select count(*) from demo;

  COUNT(*)
----------
     49783

run the long running query.

SQL> select * from demo;

simultaneously issue shutdown abort in new terminal
SQL> shu abort;
ORACLE instance shut down.
SQL>

the query runs smoothly, user will not receive any kind of error messages.
then check the instance name in node1 machine.
SQL> select instance_name from v$instance;

INSTANCE_NAME
----------------
cdbs2

SQL> select username,osuser,failover_type,failover_method from v$session
  2  where username='SYSTEM';

USERNAME   OSUSER     FAILOVER_TYPE   FAILOVER_METHOD
---------- ---------- --------------- ------------------
SYSTEM     oracle     SELECT          PRECONNECT

SQL>

checking node2 machine before and after TAF.
Before TAF
SQL> select username,osuser,failover_type,failover_method from v$session
  2  where username='SYSTEM';

USERNAME                       OSUSER                         FAILOVER_TYPE
------------------------------ ------------------------------ -------------
FAILOVER_M
----------
SYSTEM                         oracle                         NONE
NONE


SQL> select instance_name from v$instance;

INSTANCE_NAME
----------------
cdbs2

SQL> select instance_name from v$instance;

INSTANCE_NAME
----------------
cdbs2

After TAF.
SQL> select username,osuser,failover_type,failover_method from v$session
  2  where username='SYSTEM';

USERNAME                       OSUSER                         FAILOVER_TYPE
------------------------------ ------------------------------ -------------
FAILOVER_M
----------
SYSTEM                         oracle                         SELECT
PRECONNECT


SQL>


Hope, this will helps you to understand the TAF.

for theoritical part please refer oracle documentation.

Regards,
RAJESHKUMAR G

Thursday, August 26, 2010

[PRKC-1044 : Failed to check remote command execution setup for node

[PRKC-1044 : Failed to check remote command execution setup for node racnode1.localdomain using shells /usr/bin/ssh and /usr/bin/rsh Permission denied.]]

solution:
Configure SSH between two nodes

#

Log in as the software owner (in this example, the oracle user).

note: dont use su - oracle , logout and login as oracle user.
make sure you login as oracle user.

#

To ensure that you are logged in as the Oracle user, and that the user ID matches the expected user ID you have assigned to the Oracle user, enter the commands id. Ensure that Oracle user group and user and the terminal window process group and user IDs are identical.
example: check it on both nodes
[oracle@racnode1 ~]$ id 
uid=500(oracle) gid=500(oinstall) groups=500(oinstall),501(dba)
[oracle@racnode2 ~]$ id 
uid=500(oracle) gid=500(oinstall) groups=500(oinstall),501(dba)
#
While logged in as oracle perform the following on each of the nodes of the RAC:

1. cd $HOME
2. mkdir ~/.ssh
3. chmod 700 ~/.ssh
4. /usr/bin/ssh-keygen -t rsa
5. /usr/bin/ssh-keygen -t dsa


On Node 1:

1. cd $HOME/.ssh
2. cat id_rsa.pub >> authorized_keys
3. cat id_dsa.pub >> authorized_keys
4. Copy the authorized_keys file to the node 2. scp authorized_keys racnode2:/home/oracle/.ssh


On Node 2:

1. cd $HOME/.ssh
2. cat id_rsa.pub >> authorized_keys
3. cat id_dsa.pub >> authorized_keys
4. scp authorized_keys racnode1:/home/oracle/.ssh

On each node, enter the following commands to start the SSH agent, and to load the SSH keys into memory:

$ exec /usr/bin/ssh-agent $SHELL
$ /usr/bin/ssh-add

At the prompt, enter the pass phrase for each key that you generated.

* Now perform a ssh between all the nodes including the node-priv hostnames as well. Check to make sure that ssh is configured well without prompting for the password (on both the nodes):
#
Can you make sure your sshd_config file as following entries uncommented?


X11Forwarding yes

X11DisplayOffset 10

X11UseLocalhost yes


If not then edit and uncomment them. Restart the ssh once you make these changes.

Restart OpenSSH server

Type the following command:

#/etc/init.d/ssh restart


Add the entries to the .bashrc file as oracle user
#

if [ -t 0 ]; then
stty intr ^C
fi

#

Try creating a symbolic link:

ln -s /usr/bin/ssh /usr/local/bin/ssh
ln -s /usr/bin/scp /usr/local/bin/scp

After you set the dsa and rsa keys in to the authorized_keys file.

From NODE 1

ssh racnode1 date
ssh racnode2 date

ssh racnode1.localdomain date
ssh racnode2.localdomain date

ssh racnode1-priv date
ssh racnode2-priv date

ssh racnode1-priv.localdomain date
ssh racnode2-priv.localdomain date

From NODE 2

ssh racnode1 date
ssh racnode2 date

ssh racnode1.localdomain date
ssh racnode2.localdomain date

ssh racnode1-priv date
ssh racnode2-priv date

ssh racnode1-priv.localdomain date
ssh racnode2-priv.localdomain date


Hope, This will resolve your issue.

references:
http://download.oracle.com/docs/cd/B28359_01/install.111/b28263/prelinux.htm#BABJBAEB

Saturday, August 21, 2010

ORA-01122: ORA-01110: ORA-01206: After RMAN duplicate in source database

ORA-01122: database file 1 failed verification check
ORA-01110: data file 1: '+DATA/demodb/datafile/system.296.726853295'
ORA-01206: file is not part of this database - wrong database id

Reason:

RMAN Duplicate command updates source database, datafile headers.


Solution:

Restore and Recover the database with RMAN Backup.

to avoid this error add the following entries in pfile(clone/duplicate database)
db_file_name_convert='+DATA/DEMODB/DATAFILE/','/u01/app/oracle/oradata/dupdb/'
log_file_name_convert='+DATA/DEMODB/ONLINELOG/','/u01/app/oracle/oradata/dupdb/'

or set new name command in rman script.


[oracle@rac2 ~]$ . oraenv
ORACLE_SID = [+ASM2] ? demodb
[oracle@rac2 ~]$ rlsqlplus

SQL*Plus: Release 10.2.0.1.0 - Production on Sat Aug 21 14:32:31 2010

Copyright (c) 1982, 2005, Oracle. All rights reserved.

Enter user-name: sys as sysdba
Enter password:

Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - Production
With the Partitioning, Real Application Clusters and OLAP options

SQL> startup
ORA-01081: cannot start already-running ORACLE - shut it down first
SQL> select status from v$instance;

STATUS
------------
MOUNTED

SQL> alter database open;
alter database open
*
ERROR at line 1:
ORA-01122: database file 1 failed verification check
ORA-01110: data file 1: '+DATA/demodb/datafile/system.296.726853295'
ORA-01206: file is not part of this database - wrong database id

oracle@rac2 backup]$ . oraenv
ORACLE_SID = [demodb] ? demodb
[oracle@rac2 backup]$ rman target /

Recovery Manager: Release 10.2.0.1.0 - Production on Sat Aug 21 14:34:47 2010

Copyright (c) 1982, 2005, Oracle. All rights reserved.

connected to target database: DEMODB (DBID=3723121283, not open)

RMAN> restore database
2> ;

Starting restore at 21-AUG-10
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: sid=154 devtype=DISK

channel ORA_DISK_1: starting datafile backupset restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
restoring datafile 00001 to +DATA/demodb/datafile/system.296.726853295
restoring datafile 00002 to +DATA/demodb/datafile/undotbs1.298.726853331
restoring datafile 00003 to +DATA/demodb/datafile/sysaux.297.726853315
restoring datafile 00004 to +DATA/demodb/datafile/users.299.726853339
channel ORA_DISK_1: reading from backup piece /u01/app/oracle/backup/DEMODB_02ll6n2k_1_1.bkp
channel ORA_DISK_1: restored backup piece 1
piece handle=/u01/app/oracle/backup/DEMODB_02ll6n2k_1_1.bkp tag=TAG20100813T000329
channel ORA_DISK_1: restore complete, elapsed time: 00:02:38
Finished restore at 21-AUG-10

RMAN> recover database;

Starting recover at 21-AUG-10
using channel ORA_DISK_1

starting media recovery

archive log thread 1 sequence 3 is already on disk as file +DATA/demodb/archivelog/2010_08_13/thread_1_seq_3.311.726883477
archive log thread 1 sequence 4 is already on disk as file +DATA/demodb/archivelog/2010_08_13/thread_1_seq_4.312.726885007
archive log thread 1 sequence 5 is already on disk as file +DATA/demodb/archivelog/2010_08_13/thread_1_seq_5.313.726940211
archive log thread 1 sequence 6 is already on disk as file +DATA/demodb/archivelog/2010_08_13/thread_1_seq_6.314.726940893
archive log thread 1 sequence 7 is already on disk as file +DATA/demodb/archivelog/2010_08_13/thread_1_seq_7.315.726940901
archive log filename=+DATA/demodb/archivelog/2010_08_13/thread_1_seq_3.311.726883477 thread=1 sequence=3
archive log filename=+DATA/demodb/archivelog/2010_08_13/thread_1_seq_4.312.726885007 thread=1 sequence=4
archive log filename=+DATA/demodb/archivelog/2010_08_13/thread_1_seq_5.313.726940211 thread=1 sequence=5
media recovery complete, elapsed time: 00:00:39
Finished recover at 21-AUG-10

RMAN> exit


Recovery Manager complete.
[oracle@rac2 backup]$ . oraenv
ORACLE_SID = [demodb] ?
[oracle@rac2 backup]$ rlsqlplus

SQL*Plus: Release 10.2.0.1.0 - Production on Sat Aug 21 14:44:48 2010

Copyright (c) 1982, 2005, Oracle. All rights reserved.

Enter user-name: sys as sysdba
Enter password:

Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - Production
With the Partitioning, Real Application Clusters and OLAP options

SQL> alter database open resetlogs;
alter database open resetlogs
*
ERROR at line 1:
ORA-01139: RESETLOGS option only valid after an incomplete database recovery




SQL> alter database open;

Database altered.

SQL> archive log list;
Database log mode Archive Mode
Automatic archival Enabled
Archive destination USE_DB_RECOVERY_FILE_DEST
Oldest online log sequence 7
Next log sequence to archive 9
Current log sequence 9
SQL> select name from v$datafile;

NAME
--------------------------------------------------------------------------------
+DATA/demodb/datafile/system.296.726853295
+DATA/demodb/datafile/undotbs1.298.726853331
+DATA/demodb/datafile/sysaux.297.726853315
+DATA/demodb/datafile/users.299.726853339

SQL>

Friday, August 20, 2010

Recover Corrupt/Missing OCR and Voting Disk with No Backup

Recover Corrupt/Missing OCR and Voting Disk without Backup

It happens. Not very often, but it can happen. You are faced with a corrupt or missing Oracle Cluster Registry (OCR) and have no backup to recover from.

/u01/crs/oracle/product/10.2.0/crs/log/rac1/alertrac1.log

[client(20186)]CRS-1006:The OCR location /dev/raw/raw1 is inaccessible.
Details in /u01/crs/oracle/product/10.2.0/crs/log/rac1/client/ocrcheck_20186.log.

ocrcheck_20186.log file contents
2010-08-20 12:19:19.796: [ default][4143896256]a_init:7!: Backend init unsuccessful : [22]
2010-08-20 12:19:19.804: [OCRCHECK][4143896256]Failed to initialize OCR context:
[PROC-22: The OCR backend has an invalid format]
2010-08-20 12:19:19.804: [OCRCHECK][4143896256]Failed to initialize ocrchek2
2010-08-20 12:19:19.804: [OCRCHECK][4143896256]Exiting [status=failed].

[root@rac1 bin]# ./ocrcheck
PROT-601: Failed to initialize ocrcheck
[root@rac1 bin]# ./crs_stat -t
CRS-0184: Cannot communicate with the CRS daemon.

[root@rac1 bin]#
[root@rac1 bin]# ./crsctl query css votedisk
OCR initialization failed with invalid format: PROC-22: The OCR backend has an invalid format
[root@rac1 bin]#

[root@rac2 bin]# ./crs_stat -t
CRS-0184: Cannot communicate with the CRS daemon.

verify all the database are disconnect and shutdown

1.Execute rootdelete.sh from All Nodes.

The rootdelete.sh script can be found at $ORA_CRS_HOME/install/rootdelete.sh on all nodes in the cluster

[root@rac1 install]# pwd
/u01/crs/oracle/product/10.2.0/crs/install
[root@rac1 install]# ./rootdelete.sh
Shutting down Oracle Cluster Ready Services (CRS):
OCR initialization failed with invalid format: PROC-22: The OCR backend has an invalid format
Shutdown has begun. The daemons should exit soon.
Checking to see if Oracle CRS stack is down...
Oracle CRS stack is not running.
Oracle CRS stack is down now.
Removing script for Oracle Cluster Ready services
Updating ocr file for downgrade
Cleaning up SCR settings in '/etc/oracle/scls_scr'
[root@rac1 install]#

[root@rac2 install]# pwd
/u01/crs/oracle/product/10.2.0/crs/install
[root@rac2 install]# ./rootdelete.sh
Shutting down Oracle Cluster Ready Services (CRS):
OCR initialization failed with invalid format:
PROC-22: The OCR backend has an invalid format
Shutdown has begun. The daemons should exit soon.
Checking to see if Oracle CRS stack is down...
Oracle CRS stack is not running.
Oracle CRS stack is down now.
Removing script for Oracle Cluster Ready services
Updating ocr file for downgrade
Cleaning up SCR settings in '/etc/oracle/scls_scr'
[root@rac2 install]#

2.Run rootdeinstall.sh from the Primary Node.


[root@rac1 install]# ./rootdeinstall.sh

Removing contents from OCR device
2560+0 records in
2560+0 records out

3.Run root.sh from the Primary Node. (same node as above)

[root@rac1 install]# cd ..
[root@rac1 crs]# ./root.sh
WARNING: directory '/u01/crs/oracle/product/10.2.0' is not owned by root
WARNING: directory '/u01/crs/oracle/product' is not owned by root
WARNING: directory '/u01/crs/oracle' is not owned by root
WARNING: directory '/u01/crs' is not owned by root
WARNING: directory '/u01' is not owned by root
Checking to see if Oracle CRS stack is already configured

Setting the permissions on OCR backup directory
Setting up NS directories
Oracle Cluster Registry configuration upgraded successfully
WARNING: directory '/u01/crs/oracle/product/10.2.0' is not owned by root
WARNING: directory '/u01/crs/oracle/product' is not owned by root
WARNING: directory '/u01/crs/oracle' is not owned by root
WARNING: directory '/u01/crs' is not owned by root
WARNING: directory '/u01' is not owned by root
assigning default hostname rac1 for node 1.
assigning default hostname rac2 for node 2.
Successfully accumulated necessary OCR keys.
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node :
node 1: rac1 rac1-priv rac1
node 2: rac2 rac2-priv rac2
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Now formatting voting device: /dev/raw/raw2
Format of 1 voting devices complete.
Startup will be queued to init within 90 seconds.
Adding daemons to inittab
Expecting the CRS daemons to be up within 600 seconds.
CSS is active on these nodes.
rac1
CSS is inactive on these nodes.
rac2
Local node checking complete.
Run root.sh on remaining nodes to start CRS daemons.
[root@rac1 crs]#
[root@rac2 crs]# pwd
/u01/crs/oracle/product/10.2.0/crs
[root@rac2 crs]# ./root.sh
WARNING: directory '/u01/crs/oracle/product/10.2.0' is not owned by root
WARNING: directory '/u01/crs/oracle/product' is not owned by root
WARNING: directory '/u01/crs/oracle' is not owned by root
WARNING: directory '/u01/crs' is not owned by root
WARNING: directory '/u01' is not owned by root
Checking to see if Oracle CRS stack is already configured

Setting the permissions on OCR backup directory
Setting up NS directories
Oracle Cluster Registry configuration upgraded successfully
WARNING: directory '/u01/crs/oracle/product/10.2.0' is not owned by root
WARNING: directory '/u01/crs/oracle/product' is not owned by root
WARNING: directory '/u01/crs/oracle' is not owned by root
WARNING: directory '/u01/crs' is not owned by root
WARNING: directory '/u01' is not owned by root
clscfg: EXISTING configuration version 3 detected.
clscfg: version 3 is 10G Release 2.
assigning default hostname rac1 for node 1.
assigning default hostname rac2 for node 2.
Successfully accumulated necessary OCR keys.
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node :
node 1: rac1 rac1-priv rac1
node 2: rac2 rac2-priv rac2
clscfg: Arguments check out successfully.

NO KEYS WERE WRITTEN. Supply -force parameter to override.
-force is destructive and will destroy any previous cluster
configuration.
Oracle Cluster Registry for cluster has already been initialized
Startup will be queued to init within 90 seconds.
Adding daemons to inittab
Expecting the CRS daemons to be up within 600 seconds.
CSS is active on these nodes.
rac1
rac2
CSS is active on all nodes.
Waiting for the Oracle CRSD and EVMD to start
Waiting for the Oracle CRSD and EVMD to start
Waiting for the Oracle CRSD and EVMD to start
Waiting for the Oracle CRSD and EVMD to start
Waiting for the Oracle CRSD and EVMD to start
Waiting for the Oracle CRSD and EVMD to start
Waiting for the Oracle CRSD and EVMD to start
Oracle CRS stack installed and running under init(1M)
Running vipca(silent) for configuring nodeapps
The given interface(s), "eth0" is not public. Public interfaces should be used to configure virtual IPs.


Oracle 10.2.0.1 users should note that running root.sh on the last node will fail. Most notably is the silent mode VIPCA configuration failing because of BUG 4437727 in 10.2.0.1.

to workaround these errors.

[root@rac2 bin]# pwd
/u01/crs/oracle/product/10.2.0/crs/bin
[root@rac2 bin]# ./vipca &

When the "VIP Configuration Assistant" appears, this is how I answered the screen prompts:

Welcome: Click Next
Network interfaces: Select only the public interface - eth0
Virtual IPs for cluster nodes:
Node Name: rac1
IP Alias Name: rac1-vip.localdomain
IP Address: 192.168.1.111
Subnet Mask: 255.255.255.0

Node Name: rac2
IP Alias Name: rac2-vip.localdomain
IP Address: 192.168.1.112
Subnet Mask: 255.255.255.0

Summary: Click Finish
Configuration Assistant Progress Dialog: Click OK after configuration is complete.
Configuration Results: Click Exit


4.Configure Server-Side ONS using racgons.

CRS_home/bin/racgons add_config hostname1:port hostname2:port

[root@rac1 crs]# cd bin
[root@rac1 bin]# pwd
/u01/crs/oracle/product/10.2.0/crs/bin
[root@rac1 bin]# ./racgons add_config rac1:6200 rac2:6200
[root@rac1 bin]# ./onsctl ping
Number of onsconfiguration retrieved, numcfg = 2
onscfg[0]
{node = rac1, port = 6200}
Adding remote host rac1:6200
GETHOSTBYNAME(rac1): 2
onscfg[1]
{node = rac2, port = 6200}
Adding remote host rac2:6200
GETHOSTBYNAME(rac2): 2
ons is running ...
[root@rac1 bin]#

5.Configure Network Interfaces for Clusterware.
Log in as the owner of the Oracle Clusterware software which is typically the oracle user account and configure all network interfaces. The first step is to identify the current interfaces and IP addresses using oifcfg iflist.

[root@rac1 bin]# pwd
/u01/crs/oracle/product/10.2.0/crs/bin
[root@rac1 bin]# ./oifcfg iflist
eth0 192.168.1.0
eth1 192.168.2.0
[root@rac1 bin]# ./oifcfg setif -global eth0/192.168.1.0:public
[root@rac1 bin]# ./oifcfg setif -global eth1/192.168.2.0:cluster_interconnect
[root@rac1 bin]# ./oifcfg getif
eth0 192.168.1.0 global public
eth1 192.168.2.0 global cluster_interconnect
[root@rac1 bin]#

6.Add TNS Listener using NETCA.

As the Oracle Clusterware software owner (typically oracle), add a cluster TNS listener configuration to OCR using netca. This may give errors if the listener.ora contains the entries already. If this is the case, move the listener.ora to /tmp from the $ORACLE_HOME/network/admin or from the $TNS_ADMIN directory if the TNS_ADMIN environmental is defined and then run netca. Add all the listeners that were added during the original Oracle Clusterware software installation.

[oracle@rac2 admin]$ mv listener.ora /tmp/listener.ora.original

[oracle@rac1 admin]$ mv listener.ora /tmp/listener.ora.original
[oracle@rac1 admin]$ netca &

Screen Name Response
Select the Type of Oracle
Net Services Configuration
Select Cluster configuration
Select the nodes to configure Select all
Type of Configuration Select Listener configuration.
Listener Configuration
Next 6 Screens
The following screens are now like any other normal listener configuration. You can simply accept the default parameters for the next six screens:
What do you want to do: Add
Listener name: LISTENER
Selected protocols: TCP
Port number: 1521
Configure another listener: No
Listener configuration complete! [ Next ]
You will be returned to this Welcome (Type of Configuration) Screen.
Type of Configuration Select Naming Methods configuration.
Naming Methods Configuration The following screens are:
Selected Naming Methods: Local Naming
Naming Methods configuration complete! [ Next ]
You will be returned to this Welcome (Type of Configuration) Screen.
Type of Configuration Click Finish to exit the NETCA.

7.Add all Resources Back to OCR using srvctl



[root@rac1 bin]# ./crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora....C1.lsnr application ONLINE ONLINE rac1
ora.rac1.gsd application ONLINE ONLINE rac1
ora.rac1.ons application ONLINE ONLINE rac1
ora.rac1.vip application ONLINE ONLINE rac1
ora....C2.lsnr application ONLINE ONLINE rac2
ora.rac2.gsd application ONLINE ONLINE rac2
ora.rac2.ons application ONLINE ONLINE rac2
ora.rac2.vip application ONLINE ONLINE rac2

As a final step, log in as the Oracle Clusterware software owner (typically oracle) and add all resources back to the OCR using the srvctl command.

Please ensure that these commands are not run as the root user account.

Add ASM INSTANCE(S) to OCR:

srvctl add asm -n -i -o

[oracle@rac1 bin]$ pwd
/u01/crs/oracle/product/10.2.0/crs/bin
[oracle@rac1 bin]$ ./srvctl add asm -i +ASM1 -n rac1 -o /u01/app/oracle/product/10.2.0/db_1
[oracle@rac1 bin]$ ./srvctl add asm -i +ASM2 -n rac2 -o /u01/app/oracle/product/10.2.0/db_1
[oracle@rac1 bin]$





Add DATABASE to OCR:

srvctl add database -d -o

[oracle@rac1 bin]$ ./srvctl add database -d cdbs -o /u01/app/oracle/product/10.2.0/db_1

Add INSTANCE(S) to OCR:

srvctl add instance -d -i -n

[oracle@rac1 bin]$ ./srvctl add instance -d cdbs -i cdbs1 -n rac1
[oracle@rac1 bin]$ ./srvctl add instance -d cdbs -i cdbs2 -n rac2

Add SERVICE(S) to OCR:

srvctl add service -d -s -r -P

where TAF_policy is set to NONE, BASIC, or PRECONNECT

[oracle@rac1 bin]$ ./srvctl add service -d cdbs -s cdbs_srvc -r cdbs1,cdbs2 -P BASIC

After completing the steps above, the OCR should have been successfully recreated. Bring up all of the resources that were added to the OCR and run cluvfy to verify the cluster configuration.

[oracle@rac1 bin]$ ./crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora....s1.inst application OFFLINE OFFLINE
ora....s2.inst application OFFLINE OFFLINE
ora....bs1.srv application OFFLINE OFFLINE
ora....bs2.srv application OFFLINE OFFLINE
ora....srvc.cs application OFFLINE OFFLINE
ora.cdbs.db application OFFLINE OFFLINE
ora....SM1.asm application OFFLINE OFFLINE
ora....C1.lsnr application ONLINE ONLINE rac1
ora.rac1.gsd application ONLINE ONLINE rac1
ora.rac1.ons application ONLINE ONLINE rac1
ora.rac1.vip application ONLINE ONLINE rac1
ora....SM2.asm application OFFLINE OFFLINE
ora....C2.lsnr application ONLINE ONLINE rac2
ora.rac2.gsd application ONLINE ONLINE rac2
ora.rac2.ons application ONLINE ONLINE rac2
ora.rac2.vip application ONLINE ONLINE rac2

[oracle@rac1 bin]$ srvctl start asm -n rac1
[oracle@rac1 bin]$ srvctl start asm -n rac2
[oracle@rac1 bin]$ srvctl start database -d cdbs
[oracle@rac1 bin]$ srvctl start service -d cdbs
[oracle@rac1 bin]$ ./crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora....s1.inst application ONLINE ONLINE rac1
ora....s2.inst application ONLINE ONLINE rac2
ora....bs1.srv application ONLINE ONLINE rac1
ora....bs2.srv application ONLINE ONLINE rac2
ora....srvc.cs application ONLINE ONLINE rac1
ora.cdbs.db application ONLINE ONLINE rac2
ora....SM1.asm application ONLINE ONLINE rac1
ora....C1.lsnr application ONLINE ONLINE rac1
ora.rac1.gsd application ONLINE ONLINE rac1
ora.rac1.ons application ONLINE ONLINE rac1
ora.rac1.vip application ONLINE ONLINE rac1
ora....SM2.asm application ONLINE ONLINE rac2
ora....C2.lsnr application ONLINE ONLINE rac2
ora.rac2.gsd application ONLINE ONLINE rac2
ora.rac2.ons application ONLINE ONLINE rac2
ora.rac2.vip application ONLINE ONLINE rac2
[oracle@rac1 bin]$

[oracle@rac1 bin]$ cluvfy stage -post crsinst -n rac1,rac2

Performing post-checks for cluster services setup

Checking node reachability...
Node reachability check passed from node "rac1".


Checking user equivalence...
User equivalence check passed for user "oracle".

Checking Cluster manager integrity...


Checking CSS daemon...
Daemon status check passed for "CSS daemon".

Cluster manager integrity check passed.

Checking cluster integrity...


Cluster integrity check passed


Checking OCR integrity...

Checking the absence of a non-clustered configuration...
All nodes free of non-clustered, local-only configurations.

Uniqueness check for OCR device passed.

Checking the version of OCR...
OCR of correct Version "2" exists.

Checking data integrity of OCR...
Data integrity check for OCR passed.

OCR integrity check passed.

Checking CRS integrity...

Checking daemon liveness...
Liveness check passed for "CRS daemon".

Checking daemon liveness...
Liveness check passed for "CSS daemon".

Checking daemon liveness...
Liveness check passed for "EVM daemon".

Checking CRS health...
CRS health check passed.

CRS integrity check passed.

Checking node application existence...


Checking existence of VIP node application (required)
Check passed.

Checking existence of ONS node application (optional)
Check passed.

Checking existence of GSD node application (optional)
Check passed.


Post-check for cluster services setup was successful.
[oracle@rac1 bin]$




source and reference:
http://www.idevelopment.info/data/Oracle/DBA_tips/Oracle10gRAC/CLUSTER_70.shtml#Recover%20Corrupt/Missing%20OCR

Thursday, August 19, 2010

ORA-00439: feature not enabled: Real Application Clusters

Yesterday, i tried to upgrade my Clusterware with patch file p6810189_10204_Linux-x86.zip

due to some mistake the clusterware corrupted.

so i Reinstall CRS Without Disturbing Installed Oracle RDBMS Home

after successful installation of clusterware and add the existing resources to clusterware.

i started ASM with srvctl command, i got

ORA-00439: feature not enabled: Real Application Clusters

Because the Oracle Home disabled RAC, previously i disabled RAC to access single instance (non-cluster) database.

So, we should relink library to enable RAC on this Oracle Home

$ cd $ORACLE_HOME/rdbms/lib

$ make -f ins_rdbms.mk rac_on

$ make -f ins_rdbms.mk ioracle

[oracle@rac1 bin]$ ./srvctl start asm -n rac1
PRKS-1009 : Failed to start ASM instance "+ASM1" on node "rac1",
[PRKS-1009 : Failed to start ASM instance "+ASM1" on node "rac1", [rac1:ora.rac1.ASM1.asm:
rac1:ora.rac1.ASM1.asm:SQL*Plus: Release 10.2.0.1.0 - Production on Thu Aug 19 13:33:01 2010
rac1:ora.rac1.ASM1.asm:
rac1:ora.rac1.ASM1.asm:Copyright (c) 1982, 2005, Oracle. All rights reserved.
rac1:ora.rac1.ASM1.asm:
rac1:ora.rac1.ASM1.asm:Enter user-name: Connected to an idle instance.
rac1:ora.rac1.ASM1.asm:
rac1:ora.rac1.ASM1.asm:SQL> ORA-00439: feature not enabled: Real Application Clusters
rac1:ora.rac1.ASM1.asm:SQL> Disconnected
rac1:ora.rac1.ASM1.asm:
CRS-0215: Could not start resource 'ora.rac1.ASM1.asm'.]]
[PRKS-1009 : Failed to start ASM instance "+ASM1" on node "rac1", [rac1:ora.rac1.ASM1.asm:
rac1:ora.rac1.ASM1.asm:SQL*Plus: Release 10.2.0.1.0 - Production on Thu Aug 19 13:33:01 2010
rac1:ora.rac1.ASM1.asm:
rac1:ora.rac1.ASM1.asm:Copyright (c) 1982, 2005, Oracle. All rights reserved.
rac1:ora.rac1.ASM1.asm:
rac1:ora.rac1.ASM1.asm:Enter user-name: Connected to an idle instance.
rac1:ora.rac1.ASM1.asm:
rac1:ora.rac1.ASM1.asm:SQL> ORA-00439: feature not enabled: Real Application Clusters
rac1:ora.rac1.ASM1.asm:SQL> Disconnected
rac1:ora.rac1.ASM1.asm:
CRS-0215: Could not start resource 'ora.rac1.ASM1.asm'.]]


solution

[oracle@rac1 bin]$ cd $ORACLE_HOME/rdbms/lib
[oracle@rac1 lib]$ pwd
/u01/app/oracle/product/10.2.0/db_1/rdbms/lib
[oracle@rac1 lib]$ make -f ins_rdbms.mk rac_on
rm -f /u01/app/oracle/product/10.2.0/db_1/lib/libskgxp10.so
cp /u01/app/oracle/product/10.2.0/db_1/lib//libskgxpu.so /u01/app/oracle/product/10.2.0/db_1/lib/libskgxp10.so
cp /u01/app/oracle/product/10.2.0/db_1/lib/libskgxns.so /u01/app/oracle/product/10.2.0/db_1/lib/libskgxn2.so
/usr/bin/ar d /u01/app/oracle/product/10.2.0/db_1/rdbms/lib/libknlopt.a ksnkcs.o/usr/bin/ar cr /u01/app/oracle/product/10.2.0/db_1/rdbms/lib/libknlopt.a /u01/app/oracle/product/10.2.0/db_1/rdbms/lib/kcsm.o
[oracle@rac1 lib]$ make -f ins_rdbms.mk ioracle
chmod 755 /u01/app/oracle/product/10.2.0/db_1/bin

- Linking Oracle
rm -f /u01/app/oracle/product/10.2.0/db_1/rdbms/lib/oracle
gcc -o /u01/app/oracle/product/10.2.0/db_1/rdbms/lib/oracle -L/u01/app/oracle/product/10.2.0/db_1/rdbms/lib/ -L/u01/app/oracle/product/10.2.0/db_1/lib/ -L/u01/app/oracle/product/10.2.0/db_1/lib/stubs/ -L/usr/lib -lirc -Wl,-E `test -f /u01/app/oracle/product/10.2.0/db_1/rdbms/lib/sskgpsmti.o && echo /u01/app/oracle/product/10.2.0/db_1/rdbms/lib/sskgpsmti.o` /u01/app/oracle/product/10.2.0/db_1/rdbms/lib/opimai.o /u01/app/oracle/product/10.2.0/db_1/rdbms/lib/ssoraed.o /u01/app/oracle/product/10.2.0/db_1/rdbms/lib/ttcsoi.o /u01/app/oracle/product/10.2.0/db_1/rdbms/lib/defopt.o -Wl,--whole-archive -lperfsrv10 -Wl,--no-whole-archive /u01/app/oracle/product/10.2.0/db_1/lib/nautab.o /u01/app/oracle/product/10.2.0/db_1/lib/naeet.o /u01/app/oracle/product/10.2.0/db_1/lib/naect.o /u01/app/oracle/product/10.2.0/db_1/lib/naedhs.o /u01/app/oracle/product/10.2.0/db_1/rdbms/lib/config.o -lserver10 -lodm10 -lnnet10 -lskgxp10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lxml10 -lcore10 -lunls10 -lsnls10 -lnls10 -lcore10 -lnls10 -lhasgen10 -lcore10 -lskgxn2 -locr10 -locrb10 -locrutl10 -lhasgen10 -lcore10 -lskgxn2 -lclient10 -lvsn10 -lcommon10 -lgeneric10 -lknlopt `if /usr/bin/ar tv /u01/app/oracle/product/10.2.0/db_1/rdbms/lib/libknlopt.a | grep xsyeolap.o > /dev/null 2>&1 ; then echo "-loraolap10" ; fi` -lslax10 -lpls10 -lplp10 -lserver10 -lclient10 -lvsn10 -lcommon10 -lgeneric10 -lknlopt -lslax10 -lpls10 -lplp10 -ljox10 -lserver10 -lclsra10 -ldbcfg10 -locijdbcst10 -lwwg `cat /u01/app/oracle/product/10.2.0/db_1/lib/ldflags` -lnsslb10 -lncrypt10 -lnsgr10 -lnzjs10 -ln10 -lnnz10 -lnl10 -lnro10 `cat /u01/app/oracle/product/10.2.0/db_1/lib/ldflags` -lnsslb10 -lncrypt10 -lnsgr10 -lnzjs10 -ln10 -lnnz10 -lnl10 -lmm -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lxml10 -lcore10 -lunls10 -lsnls10 -lnls10 -lcore10 -lnls10 `cat /u01/app/oracle/product/10.2.0/db_1/lib/ldflags` -lnsslb10 -lncrypt10 -lnsgr10 -lnzjs10 -ln10 -lnnz10 -lnl10 -lnro10 `cat /u01/app/oracle/product/10.2.0/db_1/lib/ldflags` -lnsslb10 -lncrypt10 -lnsgr10 -lnzjs10 -ln10 -lnnz10 -lnl10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lxml10 -lcore10 -lunls10 -lsnls10 -lnls10 -lcore10 -lnls10 `if /usr/bin/ar tv /u01/app/oracle/product/10.2.0/db_1/rdbms/lib/libknlopt.a | grep "kxmnsd.o" > /dev/null 2>&1 ; then echo " " ; else echo "-lordsdo10"; fi` -lctxc10 -lctx10 -lzx10 -lgx10 -lctx10 -lzx10 -lgx10 -lordimt10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lxml10 -lcore10 -lunls10 -lsnls10 -lnls10 -lcore10 -lnls10 -lsnls10 -lunls10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lxml10 -lcore10 -lunls10 -lsnls10 -lnls10 -lcore10 -lnls10 -laio `cat /u01/app/oracle/product/10.2.0/db_1/lib/sysliblist` -Wl,-rpath,/u01/app/oracle/product/10.2.0/db_1/lib -lm `cat /u01/app/oracle/product/10.2.0/db_1/lib/sysliblist` -ldl -lm -L/u01/app/oracle/product/10.2.0/db_1/lib
/u01/app/oracle/product/10.2.0/db_1/lib//libcore10.a(lcd.o)(.text+0x8e8): In function `lcdprm':
: warning: the `gets' function is dangerous and should not be used.
mv -f /u01/app/oracle/product/10.2.0/db_1/bin/oracle /u01/app/oracle/product/10.2.0/db_1/bin/oracleO
mv /u01/app/oracle/product/10.2.0/db_1/rdbms/lib/oracle /u01/app/oracle/product/10.2.0/db_1/bin/oracle
chmod 6751 /u01/app/oracle/product/10.2.0/db_1/bin/oracle
[oracle@rac1 lib]$

[oracle@rac2 ~]$ cd $ORACLE_HOME/rdbms/lib
[oracle@rac2 lib]$ make -f ins_rdbms.mk rac_on
rm -f /u01/app/oracle/product/10.2.0/db_1/lib/libskgxp10.so
cp /u01/app/oracle/product/10.2.0/db_1/lib//libskgxpu.so /u01/app/oracle/product/10.2.0/db_1/lib/libskgxp10.so
cp /u01/app/oracle/product/10.2.0/db_1/lib/libskgxns.so /u01/app/oracle/product/10.2.0/db_1/lib/libskgxn2.so
/usr/bin/ar d /u01/app/oracle/product/10.2.0/db_1/rdbms/lib/libknlopt.a ksnkcs.o/usr/bin/ar cr /u01/app/oracle/product/10.2.0/db_1/rdbms/lib/libknlopt.a /u01/app/oracle/product/10.2.0/db_1/rdbms/lib/kcsm.o
[oracle@rac2 lib]$ make -f ins_rdbms.mk ioracle
chmod 755 /u01/app/oracle/product/10.2.0/db_1/bin

- Linking Oracle
rm -f /u01/app/oracle/product/10.2.0/db_1/rdbms/lib/oracle
gcc -o /u01/app/oracle/product/10.2.0/db_1/rdbms/lib/oracle -L/u01/app/oracle/product/10.2.0/db_1/rdbms/lib/ -L/u01/app/oracle/product/10.2.0/db_1/lib/ -L/u01/app/oracle/product/10.2.0/db_1/lib/stubs/ -L/usr/lib -lirc -Wl,-E `test -f /u01/app/oracle/product/10.2.0/db_1/rdbms/lib/sskgpsmti.o && echo /u01/app/oracle/product/10.2.0/db_1/rdbms/lib/sskgpsmti.o` /u01/app/oracle/product/10.2.0/db_1/rdbms/lib/opimai.o /u01/app/oracle/product/10.2.0/db_1/rdbms/lib/ssoraed.o /u01/app/oracle/product/10.2.0/db_1/rdbms/lib/ttcsoi.o /u01/app/oracle/product/10.2.0/db_1/rdbms/lib/defopt.o -Wl,--whole-archive -lperfsrv10 -Wl,--no-whole-archive /u01/app/oracle/product/10.2.0/db_1/lib/nautab.o /u01/app/oracle/product/10.2.0/db_1/lib/naeet.o /u01/app/oracle/product/10.2.0/db_1/lib/naect.o /u01/app/oracle/product/10.2.0/db_1/lib/naedhs.o /u01/app/oracle/product/10.2.0/db_1/rdbms/lib/config.o -lserver10 -lodm10 -lnnet10 -lskgxp10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lxml10 -lcore10 -lunls10 -lsnls10 -lnls10 -lcore10 -lnls10 -lhasgen10 -lcore10 -lskgxn2 -locr10 -locrb10 -locrutl10 -lhasgen10 -lcore10 -lskgxn2 -lclient10 -lvsn10 -lcommon10 -lgeneric10 -lknlopt `if /usr/bin/ar tv /u01/app/oracle/product/10.2.0/db_1/rdbms/lib/libknlopt.a | grep xsyeolap.o > /dev/null 2>&1 ; then echo "-loraolap10" ; fi` -lslax10 -lpls10 -lplp10 -lserver10 -lclient10 -lvsn10 -lcommon10 -lgeneric10 -lknlopt -lslax10 -lpls10 -lplp10 -ljox10 -lserver10 -lclsra10 -ldbcfg10 -locijdbcst10 -lwwg `cat /u01/app/oracle/product/10.2.0/db_1/lib/ldflags` -lnsslb10 -lncrypt10 -lnsgr10 -lnzjs10 -ln10 -lnnz10 -lnl10 -lnro10 `cat /u01/app/oracle/product/10.2.0/db_1/lib/ldflags` -lnsslb10 -lncrypt10 -lnsgr10 -lnzjs10 -ln10 -lnnz10 -lnl10 -lmm -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lxml10 -lcore10 -lunls10 -lsnls10 -lnls10 -lcore10 -lnls10 `cat /u01/app/oracle/product/10.2.0/db_1/lib/ldflags` -lnsslb10 -lncrypt10 -lnsgr10 -lnzjs10 -ln10 -lnnz10 -lnl10 -lnro10 `cat /u01/app/oracle/product/10.2.0/db_1/lib/ldflags` -lnsslb10 -lncrypt10 -lnsgr10 -lnzjs10 -ln10 -lnnz10 -lnl10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lxml10 -lcore10 -lunls10 -lsnls10 -lnls10 -lcore10 -lnls10 `if /usr/bin/ar tv /u01/app/oracle/product/10.2.0/db_1/rdbms/lib/libknlopt.a | grep "kxmnsd.o" > /dev/null 2>&1 ; then echo " " ; else echo "-lordsdo10"; fi` -lctxc10 -lctx10 -lzx10 -lgx10 -lctx10 -lzx10 -lgx10 -lordimt10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lxml10 -lcore10 -lunls10 -lsnls10 -lnls10 -lcore10 -lnls10 -lsnls10 -lunls10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lxml10 -lcore10 -lunls10 -lsnls10 -lnls10 -lcore10 -lnls10 -laio `cat /u01/app/oracle/product/10.2.0/db_1/lib/sysliblist` -Wl,-rpath,/u01/app/oracle/product/10.2.0/db_1/lib -lm `cat /u01/app/oracle/product/10.2.0/db_1/lib/sysliblist` -ldl -lm -L/u01/app/oracle/product/10.2.0/db_1/lib
/u01/app/oracle/product/10.2.0/db_1/lib//libcore10.a(lcd.o)(.text+0x8e8): In function `lcdprm':
: warning: the `gets' function is dangerous and should not be used.
mv -f /u01/app/oracle/product/10.2.0/db_1/bin/oracle /u01/app/oracle/product/10.2.0/db_1/bin/oracleO
mv /u01/app/oracle/product/10.2.0/db_1/rdbms/lib/oracle /u01/app/oracle/product/10.2.0/db_1/bin/oracle
chmod 6751 /u01/app/oracle/product/10.2.0/db_1/bin/oracle
[oracle@rac2 lib]$

[oracle@rac1 bin]$ ./srvctl start asm -n rac1

[oracle@rac1 bin]$ ./srvctl start asm -n rac1

[oracle@rac1 bin]$ ./srvctl start asm -n rac2

[oracle@rac1 bin]$ ./srvctl start database -d cdbs

[oracle@rac1 bin]$ ./crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora....bs1.srv application ONLINE ONLINE rac1
ora....bs2.srv application ONLINE ONLINE rac2
ora....cdbs.cs application ONLINE ONLINE rac2
ora....s1.inst application ONLINE ONLINE rac1
ora....s2.inst application ONLINE ONLINE rac2
ora.cdbs.db application ONLINE ONLINE rac2
ora....SM1.asm application ONLINE ONLINE rac1
ora....C1.lsnr application ONLINE ONLINE rac1
ora.rac1.gsd application ONLINE ONLINE rac1
ora.rac1.ons application ONLINE ONLINE rac1
ora.rac1.vip application ONLINE ONLINE rac1
ora....SM2.asm application ONLINE ONLINE rac2
ora....C2.lsnr application ONLINE ONLINE rac2
ora.rac2.gsd application ONLINE ONLINE rac2
ora.rac2.ons application ONLINE ONLINE rac2
ora.rac2.vip application ONLINE ONLINE rac2
[oracle@rac1 bin]$





source and reference:
http://surachartopun.com/2008/11/ora-00439-feature-not-enabled-real.html

ORA-29702: error occurred in Cluster Group Service operation

# problem

SQL> startup;
ORA-29702: error occurred in Cluster Group Service operation

29702, 00000, "error occurred in Cluster Group Service operation"
// *Cause: An unexpected error occurred while performing a CGS operation.
// *Action: Verify that the LMON process is still active. Also,
// check the Oracle LMON trace files for errors.

# solution

cd $ORACLE_HOME/rdbms/lib
make -f ins_rdbms.mk rac_off
make -f ins_rdbms.mk ioracle


[oracle@rac1 ~]$ . oraenv
ORACLE_SID = [RAC1] ? jay
[oracle@rac1 ~]$ sqlplus '/as sysdba'

SQL*Plus: Release 10.2.0.1.0 - Production on Thu Aug 19 11:51:52 2010

Copyright (c) 1982, 2005, Oracle. All rights reserved.

Connected to an idle instance.

SQL> startup
ORA-29702: error occurred in Cluster Group Service operation


[oracle@rac1 ~]$ cd $ORACLE_HOME/rdbms/lib
[oracle@rac1 lib]$ pwd
/u01/app/oracle/product/10.2.0/db_1/rdbms/lib
[oracle@rac1 lib]$ make -f ins_rdbms.mk rac_off
rm -f /u01/app/oracle/product/10.2.0/db_1/lib/libskgxp10.so
cp /u01/app/oracle/product/10.2.0/db_1/lib//libskgxpd.so /u01/app/oracle/product/10.2.0/db_1/lib/libskgxp10.so
rm -f /u01/app/oracle/product/10.2.0/db_1/lib/libskgxn2.so
cp /u01/app/oracle/product/10.2.0/db_1/lib//libskgxns.so \
/u01/app/oracle/product/10.2.0/db_1/lib/libskgxn2.so
/usr/bin/ar d /u01/app/oracle/product/10.2.0/db_1/rdbms/lib/libknlopt.a kcsm.o
/usr/bin/ar cr /u01/app/oracle/product/10.2.0/db_1/rdbms/lib/libknlopt.a /u01/app/oracle/product/10.2.0/db_1/rdbms/lib/ksnkcs.o
[oracle@rac1 lib]$ make -f ins_rdbms.mk ioracle
chmod 755 /u01/app/oracle/product/10.2.0/db_1/bin

- Linking Oracle
rm -f /u01/app/oracle/product/10.2.0/db_1/rdbms/lib/oracle
gcc -o /u01/app/oracle/product/10.2.0/db_1/rdbms/lib/oracle -L/u01/app/oracle/product/10.2.0/db_1/rdbms/lib/ -L/u01/app/oracle/product/10.2.0/db_1/lib/ -L/u01/app/oracle/product/10.2.0/db_1/lib/stubs/ -L/usr/lib -lirc -Wl,-E `test -f /u01/app/oracle/product/10.2.0/db_1/rdbms/lib/sskgpsmti.o && echo /u01/app/oracle/product/10.2.0/db_1/rdbms/lib/sskgpsmti.o` /u01/app/oracle/product/10.2.0/db_1/rdbms/lib/opimai.o /u01/app/oracle/product/10.2.0/db_1/rdbms/lib/ssoraed.o /u01/app/oracle/product/10.2.0/db_1/rdbms/lib/ttcsoi.o /u01/app/oracle/product/10.2.0/db_1/rdbms/lib/defopt.o -Wl,--whole-archive -lperfsrv10 -Wl,--no-whole-archive /u01/app/oracle/product/10.2.0/db_1/lib/nautab.o /u01/app/oracle/product/10.2.0/db_1/lib/naeet.o /u01/app/oracle/product/10.2.0/db_1/lib/naect.o /u01/app/oracle/product/10.2.0/db_1/lib/naedhs.o /u01/app/oracle/product/10.2.0/db_1/rdbms/lib/config.o -lserver10 -lodm10 -lnnet10 -lskgxp10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lxml10 -lcore10 -lunls10 -lsnls10 -lnls10 -lcore10 -lnls10 -lhasgen10 -lcore10 -lskgxn2 -locr10 -locrb10 -locrutl10 -lhasgen10 -lcore10 -lskgxn2 -lclient10 -lvsn10 -lcommon10 -lgeneric10 -lknlopt `if /usr/bin/ar tv /u01/app/oracle/product/10.2.0/db_1/rdbms/lib/libknlopt.a | grep xsyeolap.o > /dev/null 2>&1 ; then echo "-loraolap10" ; fi` -lslax10 -lpls10 -lplp10 -lserver10 -lclient10 -lvsn10 -lcommon10 -lgeneric10 -lknlopt -lslax10 -lpls10 -lplp10 -ljox10 -lserver10 -lclsra10 -ldbcfg10 -locijdbcst10 -lwwg `cat /u01/app/oracle/product/10.2.0/db_1/lib/ldflags` -lnsslb10 -lncrypt10 -lnsgr10 -lnzjs10 -ln10 -lnnz10 -lnl10 -lnro10 `cat /u01/app/oracle/product/10.2.0/db_1/lib/ldflags` -lnsslb10 -lncrypt10 -lnsgr10 -lnzjs10 -ln10 -lnnz10 -lnl10 -lmm -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lxml10 -lcore10 -lunls10 -lsnls10 -lnls10 -lcore10 -lnls10 `cat /u01/app/oracle/product/10.2.0/db_1/lib/ldflags` -lnsslb10 -lncrypt10 -lnsgr10 -lnzjs10 -ln10 -lnnz10 -lnl10 -lnro10 `cat /u01/app/oracle/product/10.2.0/db_1/lib/ldflags` -lnsslb10 -lncrypt10 -lnsgr10 -lnzjs10 -ln10 -lnnz10 -lnl10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lxml10 -lcore10 -lunls10 -lsnls10 -lnls10 -lcore10 -lnls10 `if /usr/bin/ar tv /u01/app/oracle/product/10.2.0/db_1/rdbms/lib/libknlopt.a | grep "kxmnsd.o" > /dev/null 2>&1 ; then echo " " ; else echo "-lordsdo10"; fi` -lctxc10 -lctx10 -lzx10 -lgx10 -lctx10 -lzx10 -lgx10 -lordimt10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lxml10 -lcore10 -lunls10 -lsnls10 -lnls10 -lcore10 -lnls10 -lsnls10 -lunls10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lxml10 -lcore10 -lunls10 -lsnls10 -lnls10 -lcore10 -lnls10 -laio `cat /u01/app/oracle/product/10.2.0/db_1/lib/sysliblist` -Wl,-rpath,/u01/app/oracle/product/10.2.0/db_1/lib -lm `cat /u01/app/oracle/product/10.2.0/db_1/lib/sysliblist` -ldl -lm -L/u01/app/oracle/product/10.2.0/db_1/lib
/u01/app/oracle/product/10.2.0/db_1/lib//libcore10.a(lcd.o)(.text+0x8e8): In function `lcdprm':
: warning: the `gets' function is dangerous and should not be used.
mv -f /u01/app/oracle/product/10.2.0/db_1/bin/oracle /u01/app/oracle/product/10.2.0/db_1/bin/oracleO
mv /u01/app/oracle/product/10.2.0/db_1/rdbms/lib/oracle /u01/app/oracle/product/10.2.0/db_1/bin/oracle
chmod 6751 /u01/app/oracle/product/10.2.0/db_1/bin/oracle
[oracle@rac1 lib]$


SQL> startup
ORACLE instance started.

Total System Global Area 444596224 bytes
Fixed Size 1219904 bytes
Variable Size 130024128 bytes
Database Buffers 310378496 bytes
Redo Buffers 2973696 bytes
Database mounted.
Database opened.
SQL> select name from v$datafile;

NAME
--------------------------------------------------------------------------------/u01/app/oracle/oradata/jay/system01.dbf
/u01/app/oracle/oradata/jay/undotbs01.dbf
/u01/app/oracle/oradata/jay/sysaux01.dbf
/u01/app/oracle/oradata/jay/users01.dbf

SQL> select name from v$database;

NAME
---------
JAY




source and reference:
http://blog.contractoracle.com/2009/08/ora-29702-error-occurred-in-cluster.html

Friday, August 13, 2010

Duplicate/Clone/Convert ASM file system to NON-ASM Using RMAN

CLONING THE DATABASE FROM ASM FILE SYSTEM TO NON-ASM FILE SYSTEM:
Its similar like normal cloning, Here the ASM Filesystem database is DEMODB, I am going to clone the database and also convert the file system to Non-ASM Filesystem database(DUPDB).

SOURCE MACHINE NAME: RAC2 DATABASE NAME: DEMODB
TARGET MACHINE NAME: RAC1

1. Take RMAN Backup of ASM file sytem Database.

RMAN> configure channel device type disk format '/u01/app/oracle/backup/%U';

RMAN> backup full database format '/u01/app/oracle/backup/%d_%U.bkp' plus archivelog format '/u01/app/oracle/backup/%d_%U.bckp';

RMAN> BACKUP ARCHIVELOG ALL;

2. Configure listener.ora and tnsnames.ora file on both target and the source machines.

Target system RAC1 listener.ora file

SID_LIST_LISTENER_RAC1 =
(SID_LIST =
(SID_DESC =
(SID_NAME = PLSExtProc)
(ORACLE_HOME = /u01/app/oracle/product/10.2.0/db_1)
(PROGRAM = extproc)
)
(SID_DESC =
(GLOBAL_DBNAME = dupdb)
(ORACLE_HOME = /u01/app/oracle/product/10.2.0/db_1)
(SID_NAME = dupdb)
)

)


Tnsnames.ora file on both machines RAC1, RAC2

DEMODB =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = rac2-vip.localdomain)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = demodb)
)
)
DUPDB =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = rac1-vip.localdomain)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = dupdb)
)
)

3. Copy the init,ora and password file from source database to $ORACLE_HOME/dbs on target machine
and change the init.ora parameters essential to clone.

sample init.ora file of clone database dupdb

dupdb.__db_cache_size=310378496
dupdb.__java_pool_size=4194304
dupdb.__large_pool_size=4194304
dupdb.__shared_pool_size=121634816
dupdb.__streams_pool_size=0
*.audit_file_dest='/u01/app/oracle/admin/dupdb/adump'
*.background_dump_dest='/u01/app/oracle/admin/dupdb/bdump'
*.compatible='10.2.0.1.0'
*.core_dump_dest='/u01/app/oracle/admin/dupdb/cdump'
*.db_block_size=8192
*.db_domain=''
*.db_file_multiblock_read_count=16
*.db_name='dupdb'
*.db_recovery_file_dest='/u01/app/oracle/flash_recovery_area'
*.db_recovery_file_dest_size=2147483648
*.dispatchers='(PROTOCOL=TCP) (SERVICE=dupdbXDB)'
*.job_queue_processes=10
*.log_archive_format='%t_%s_%r.dbf'
*.open_cursors=300
*.pga_aggregate_target=147849216
*.processes=150
*.remote_login_passwordfile='EXCLUSIVE'
*.sga_target=444596224
*.undo_management='AUTO'
*.undo_tablespace='UNDOTBS1'
*.user_dump_dest='/u01/app/oracle/admin/dupdb/udump'

db_file_name_convert='+DATA/DEMODB/DATAFILE/','/u01/app/oracle/oradata/dupdb/'
log_file_name_convert='+DATA/DEMODB/ONLINELOG/','/u01/app/oracle/oradata/dupdb/'


note: add the controlfile location later when the rman duplicate scripts run, issue show parameter control_files in target DUPDB database.
it show the location add the location to the init parameter later.
control_files='/u01/app/oracle/flash_recovery_area/DUPDB/controlfile/o1_mf_66bjo21r_.ctl'

4. create the required directories for the bdump, cdump, adump and udump locations and also directories for database files.

5. Copy the backup files from the source to the target machine. Ensure they are located in same directory name as source machine.

6. STARTUP NOMOUNT the target database

[oracle@rac1 dupdb]$ . oraenv
ORACLE_SID = [dupdb] ?
[oracle@rac1 dupdb]$ sqlplus '/as sysdba'

SQL*Plus: Release 10.2.0.1.0 - Production on Fri Aug 13 18:28:22 2010

Copyright (c) 1982, 2005, Oracle. All rights reserved.

Connected to an idle instance.

SQL> startup nomount
ORACLE instance started.


7. RMAN command on the target machine:

[oracle@rac1 dupdb]$ rman target=sys/oracle@demodb auxiliary=/
RMAN> duplicate target database to dupdb nofilenamecheck;

8. Check the location of the datafiles and online redo log files

SQL> archive log list;
Database log mode Archive Mode
Automatic archival Enabled
Archive destination USE_DB_RECOVERY_FILE_DEST
Oldest online log sequence 0
Next log sequence to archive 1
Current log sequence 1
SQL> select name from v$datafile;

NAME
--------------------------------------------------------------------------------
/u01/app/oracle/oradata/dupdb/system.296.726853295
/u01/app/oracle/oradata/dupdb/undotbs1.298.726853331
/u01/app/oracle/oradata/dupdb/sysaux.297.726853315
/u01/app/oracle/oradata/dupdb/users.299.726853339

SQL> select member from v$logfile;

MEMBER
--------------------------------------------------------------------------------
/u01/app/oracle/oradata/dupdb/group_3.306.726853707
/u01/app/oracle/oradata/dupdb/group_3.307.726853719
/u01/app/oracle/oradata/dupdb/group_2.304.726853687
/u01/app/oracle/oradata/dupdb/group_2.305.726853695
/u01/app/oracle/oradata/dupdb/group_1.302.726853671
/u01/app/oracle/oradata/dupdb/group_1.303.726853681

6 rows selected.

SQL> show parameter control_files

NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
control_files string /u01/app/oracle/flash_recovery
_area/DUPDB/controlfile/o1_mf_
66bjo21r_.ctl
SQL>



hope, this will helps you.
 
Share/Bookmark