11g Grid Control: Using 'opmnctl startall | status | stopall' in OMS_HOME Fails with "opmnctl: ORACLE_INSTANCE is not set"

Symptoms

Trying to start the 11.1.0.1 Grid Control with opmnctl command fails with the following errors.

[oraoem@erputdb2 bin]$ ./opmnctl status
opmnctl: ORACLE_INSTANCE is not set.

opmnctl status requires that ORACLE_INSTANCE be correctly set in your
environment.



Cause

In the 11.1.0.1 Grid Control release, "opmnctl" is not a valid command for starting the OMS as the 11g OMS is deployed in Weblogic server unlike 10g version which is deployed on an OC4J.

opmnctl utility should be used to start the OMS only in the 10g versions of Grid Control nad not in the 11g Grid Control version.


Solution

Grid Control 11.1.0.1 uses "emctl" command to start Grid Control:

cd OMS_HOME/bin
./emctl start oms

This will start the Grid Control 11.1.0.1 OMS as well as starting the necessary Weblogic processes needed by the OMS.


11g Grid Control: Using 'opmnctl startall | status | stopall' in OMS_HOME Fails with "opmnctl: ORACLE_INSTANCE is not set" [ID 1160335.1]

EMD upload error: uploadXMLFiles skipped

Problem :

EMD upload error: uploadXMLFiles skipped : While emctl upload agent, got below error message.


emctl upload agent

EMD upload error: uploadXMLFiles skipped :: OMS version not checked yet. If this issue persists check trace files for ping to OMS related errors.



SQL> select target_name from mgmt_targets where target_type='oracle_emd';

TARGET_NAME
--------------------------------------------------------------------------------
testa.abcdiar.com:3872
probi.abcdiar.com:3872
bidev.abcdiar.com:3872
devdb.abcdiar.com:3872


SQL> exec mgmt_admin.cleanup_agent('testa.abcdiar.com:3872');

PL/SQL procedure successfully completed.


SQL> select target_name from mgmt_targets where target_type='oracle_emd';

TARGET_NAME
--------------------------------------------------------------------------------
probi.abcdiar.com:3872
bidev.abcdiar.com:3872
devdb.abcdiar.com:3872

Now testa removed.

Above syntax sucessfully completed, Please run below to agent running instance.

emctl clearstate agent
emctl secure agent
emctl start agent
emctl upload agent

Run below query, Now again testa host name added.

SQL> select target_name from mgmt_targets where target_type='oracle_emd';

TARGET_NAME
--------------------------------------------------------------------------------
testa.abcdiar.com:3872
probi.abcdiar.com:3872
bidev.abcdiar.com:3872
devdb.abcdiar.com:3872

Resolving ORA-00376 Error Encountered in database

Resolving ORA-00376 Error Encountered in database 


Thu Jun 13 07:32:24 2013 316193 [DM_SESSION_E_CURSOR_ERROR]error:  "A database error has occurred during the execution of a cursor ('ORA-00376: file 16 cannot be read at this time
ORA-01110: data file 16: '/u01/oracle/db/11.1.0/test/index02.dbf'
')."
Thu Jun 13 07:32:24 2013 448544 [DM_SESSION_E_CURSOR_ERROR]error:  "A database error has occurred during the execution of a cursor ('ORA-00376: file 16 cannot be read at this time
ORA-01110: data file 16: '/u01/oracle/db/11.1.0/test/index02.dbf'



select file#, status, name from v$datafile;

File# Status name

  16    Recover /u01/oracle/db/11.1.0/test/index02.dbf


Note : Above dbf file recover mode. Also offline.

============================
Solution

RMAN> recover datafile 16 preview;
recover datafile 16;

alter database datafile 16 online;
===================================


Please check the online backup. if backup is perfect, recover the above datafile.

[oracle@node1 trace]$ rman target /

Recovery Manager: Release 11.1.0.7.0 - Production on Thu Jun 13 10:29:53 2013

Copyright (c) 1982, 2007, Oracle. All rights reserved.

connected to target database: test(DBID=156985000006)

RMAN> recover datafile 16 preview;

Starting recover at 13-JUN-13
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=1953 instance=test1 device type=DISK
using channel ORA_DISK_1


List of Backup Sets
===================


BS Key Size Device Type Elapsed Time Completion Time
------- ---------- ----------- ------------ ---------------
3945 100.50M DISK 00:00:20 12-JUN-13
BP Key: 3945 Status: AVAILABLE Compressed: YES Tag: TAG20130612T220026
Piece Name: /backup/testbackup/testdaily/12062013/archive_of_test_3973_12_06_2013

List of Archived Logs in backup set 3945
Thrd Seq Low SCN Low Time Next SCN Next Time
---- ------- ---------- --------- ---------- ---------
2 3478 5970906464373 11-JUN-13 5970909574324 12-JUN-13
1 3960 5970909574318 12-JUN-13 5970909831663 12-JUN-13
2 3479 5970909574324 12-JUN-13 5970909851956 12-JUN-13
1 3961 5970909831663 12-JUN-13 5970909831667 12-JUN-13
2 3480 5970909851956 12-JUN-13 5970909914198 12-JUN-13
1 3962 5970909860407 12-JUN-13 5970909914022 12-JUN-13
1 3963 5970909914022 12-JUN-13 5970909914235 12-JUN-13
2 3481 5970909914198 12-JUN-13 5970909914239 12-JUN-13

BS Key Size Device Type Elapsed Time Completion Time
------- ---------- ----------- ------------ ---------------
3947 35.74M DISK 00:00:07 12-JUN-13
BP Key: 3947 Status: AVAILABLE Compressed: YES Tag: TAG20130612T222103
Piece Name: /backup/testbackup/testdaily/12062013/archive_of_test_3975_12_06_2013

List of Archived Logs in backup set 3947
Thrd Seq Low SCN Low Time Next SCN Next Time
---- ------- ---------- --------- ---------- ---------
1 3964 5970909914235 12-JUN-13 5970909927088 12-JUN-13
2 3482 5970909914239 12-JUN-13 5970909933136 12-JUN-13
1 3965 5970909927088 12-JUN-13 5970909933128 12-JUN-13
List of Archived Log Copies for database with db_unique_name test
=====================================================================

Key Thrd Seq S Low Time
------- ---- ------- - ---------
7423 1 3966 A 12-JUN-13
Name: /arch/oracle/11.1.0/test/test2/1_3966_774361419.arc

7424 1 3967 A 12-JUN-13
Name: /arch/oracle/11.1.0/test/test2/1_3967_774361419.arc

7425 1 3968 A 12-JUN-13
Name: /arch/oracle/11.1.0/test/test1/1_3968_774361419.arc

7428 1 3969 A 13-JUN-13
Name: /arch/oracle/11.1.0/test/test1/1_3969_774361419.arc

7426 2 3483 A 12-JUN-13
Name: /arch/oracle/11.1.0/test/test2/2_3483_774361419.arc

7427 2 3484 A 13-JUN-13
Name: /arch/oracle/11.1.0/test/test2/2_3484_774361419.arc

Media recovery start SCN is 5970909574318
Recovery must be done beyond SCN 281474976710655 to clear datafile fuzziness
Finished recover at 13-JUN-13



RMAN> recover datafile 16;

Starting recover at 13-JUN-13
using channel ORA_DISK_1

starting media recovery

archived log for thread 1 with sequence 3966 is already on disk as file /arch/oracle/11.1.0/test/test2/1_3966_774361419.arc
archived log for thread 1 with sequence 3967 is already on disk as file /arch/oracle/11.1.0/test/test2/1_3967_774361419.arc
archived log for thread 1 with sequence 3968 is already on disk as file /arch/oracle/11.1.0/test/test1/1_3968_774361419.arc
archived log for thread 1 with sequence 3969 is already on disk as file /arch/oracle/11.1.0/test/test1/1_3969_774361419.arc
archived log for thread 2 with sequence 3483 is already on disk as file /arch/oracle/11.1.0/test/test2/2_3483_774361419.arc
archived log for thread 2 with sequence 3484 is already on disk as file /arch/oracle/11.1.0/test/test2/2_3484_774361419.arc
channel ORA_DISK_1: starting archived log restore to default destination
channel ORA_DISK_1: restoring archived log
archived log thread=2 sequence=3478
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=3960
channel ORA_DISK_1: restoring archived log
archived log thread=2 sequence=3479
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=3961
channel ORA_DISK_1: restoring archived log
archived log thread=2 sequence=3480
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=3962
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=3963
channel ORA_DISK_1: restoring archived log
archived log thread=2 sequence=3481
channel ORA_DISK_1: reading from backup piece /backup/testbackup/testdaily/12062013/archive_of_test_3973_12_06_2013
channel ORA_DISK_1: piece handle=/backup/testbackup/testdaily/12062013/archive_of_test_3973_12_06_2013 tag=TAG20130612T220026
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:25
archived log file name=/arch/oracle/11.1.0/test/test1/1_3960_774361419.arc thread=1 sequence=3960
archived log file name=/arch/oracle/11.1.0/test/test1/2_3478_774361419.arc thread=2 sequence=3478
archived log file name=/arch/oracle/11.1.0/test/test1/2_3479_774361419.arc thread=2 sequence=3479
media recovery complete, elapsed time: 00:00:02
channel ORA_DISK_1: starting archived log restore to default destination
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=3964
channel ORA_DISK_1: restoring archived log
archived log thread=2 sequence=3482
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=3965
channel ORA_DISK_1: reading from backup piece /backup/testbackup/testdaily/12062013/archive_of_test_3975_12_06_2013
channel ORA_DISK_1: piece handle=/backup/testbackup/testdaily/12062013/archive_of_test_3975_12_06_2013 tag=TAG20130612T222103
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:15
Finished recover at 13-JUN-13

RMAN> sql 'alter database datafile 16 online';

sql statement: alter database datafile 16 online