Monday

R12 - Cloning from an RMAN backup using duplicate database

Since most DBA's are using rman for their backup strategy I thought I would put together the steps to clone from an rman backup. The steps you follow are pretty much the same as described in Appendix A: Recreating the database control files manually in Rapid Clone in Note 406982.1 - Cloning Oracle Applications Release 12 with Rapid Clone.

Here are the steps:
  1. Execute preclone on all tiers of the source system. This includes both the database and application tiers. (For this example, TEST is my source system.)

    For the database execute: $ORACLE_HOME/appsutil/scripts/<context>/adpreclone.pl dbTier
    Where context name is of the format <sid>_<hostname>

    For the application tier: $ADMIN_SCRIPTS_HOME/adpreclone.pl appsTier

  2. Prepare the files needed for the clone and copy them to the target server.
    • Take a FULL rman backup and copy the files to the target server and place them in the identical path. ie. if your rman backups go to /u01/backup on the source server, place them in /u01/backup on the destination server. To be safe, you may want to copy some of the archive files generated while the database was being backed up. Place them in an identical path on the target server as well.
    • Application Tier: tar up the application files and copy them to the destination server. The cloning document referenced above ask you to take a copy of the $APPL_TOP, $COMMON_TOP, $IAS_ORACLE_HOME and $ORACLE_HOME. Normally I just tar up the System Base Directory, which is the root directory for your application files.
    • Database Tier: tar up the database $ORACLE_HOME.

      ex. from a single tier system. The first tar file contains the application files and the second is the database $ORACLE_HOME

      [oratest@myserver TEST]$ pwd
      /u01/TEST
      [oratest@myserver TEST]$ ls
      apps db inst
      [oratest@myserver TEST]$ tar cvfzp TEST_apps_inst_myserver.tar.gz apps inst
      .
      .
      [oratest@myserver TEST]$ tar cvfzp TEST_dbhome_myserver.tar.gz db/tech_st

      Notice for the database $ORACLE_HOME I only added the db/tech_st directory to the archive. The reason is that the database files are under db/apps_st and we don't need those.
    • Copy the tar files to the destination server, create a directory for your new environment, for example /u01/DEV. (For the purpose of this article I will be using /u01/DEV as the system base for the target envrionment we are building and myserver is the server name.)
    • Extract each of the tar files with the command tar xvfzp

      Ex. tar xvfzp TEST_apps_inst_myserver.tar.gz

  3. Configure the target system.
    • On the database tier execute adcfgclone.pl with the dbTechStack parameter.

      For example. /u01/DEV/db/tech_st/10.2.0/appsutil/clone/bin/adcfgclone.pl dbTechStack

      By passing the dbTechStack parameter we are tell the script to configure only the necessary $ORACLE_HOME files such as the init file for the new environment, listener.ora, database environment settings file, etc. It will also start the listener.

      You will be prompted the standard post cloning questions such as the SID of the new environment, number of DATA_TOPS, Oracle Home location, port settings, etc.

      Once this is complete goto /u01/DEV/db/tech_st/10.2.0 and execute the environment settings file to make sure your environment is set correctly.

      [oradev@myserver 10.2.0] . ./DEV_myserver.env

  4. Duplicate the source database to the target.
    • In order to duplicate the source database you'll need to know the scn value to recover to. There are two wasy to do this. The first is to login to your rman catalog, find the Chk SCN of the files in the last backupset of your rman backup and add 1 to it.

      Ex. Output from a rman> List backups
      .
      .
      List of Datafiles in backup set 55729
      File LV Type Ckp SCN Ckp Time Name
      ---- -- ---- ---------- --------- ----
      7 1 Incr 5965309363843 15-JUN-09 /u02/TEST/db/apps_st/data/owad01.dbf
      .

      .
      So in this case the SCN we would be recovery to is 5965309363843 + 1 = 5965309363844.

      The other method is to login to the rman catalog via sqlplus and execute the following query:

      select max(absolute_fuzzy_change#)+1,
      max(checkpoint_change#)+1
      from rc_backup_datafile;


      Use which ever value is greater.
    • Modify the db_file_name_convert and log_file_name convert parameters in the target init file. Example:

      db_file_name_convert=('/u02/PROD/db/apps_st/data/', '/u02/DEV/db/apps_st/data/',
      '/u01/PROD/db/apps_st/data/', '/u02/DEV/db/apps_st/data/')

      log_file_name_convert=(/u02/PROD/db/apps_st/data/', '/u02/DEV/db/apps_st/data/',
      '/u01/PROD/db/apps_st/data/', '/u02/DEV/db/apps_st/data/')

    • Verify you can connect to source system from the target as sysdba. You will need to add a tns entry to the $TNS_ADMIN/tnsnames.ora file for the source system.
    • Duplicate the database. Before we use rman to duplicate the source database we need to start the target database in nomount mode.

      Start rman:

      rman target sys/<syspass>@TEST catalog rman/rman@RMAN auxiliary /

      If there are no connection errors duplicate the database with the following script:

      run {
      set until scn 5965309363844;
      allocate auxiliary channel ch1 type disk;
      allocate auxiliary channel ch2 type disk;
      duplicate target database to DEV }


      The most common errors at this point are connection errors to the source database and rman catalog. As well, if the log_file_name_convert and db_file_name_convert parameters are not set properly you will see errors. Fix the problems, login with rman again and re-execute the script.

      When the rman duplicate has finished the database will be open and ready to proceed with the next steps.

    • Execute the library update script:

      cd $ORACLE_HOME/appsutil/install/DEV_myserver where DEV_myserver is the <context_name> of the new environment.

      sqlplus "/ as sysdba"@adupdlib.sql

      If your on linux replace with so, HPUX with sl and for windows servers leave blank.

    • Configure the target database

      cd $ORACLE_HOME/appsutil/clone/bin/adcfgclone.pl dbconfig

      Where is $ORACLE_HOME/appsutil/DEV_myserver.xml

  5. Configure the application tier.

    cd /u01/DEV/apps/apps_st/comn/clone/bin
    perl adcfgclone.pl appsTier

    You will be prompted the standard cloning questions consisting of the system base directories, which services you want enabled, port pool, etc. Make sure you choose the same port pool as you did when configuring the database tier in step 3.

    Once that is finished, initialize your environment by executing

    . /u01/DEV/apps/apps_st/appl/APPSDEV_myserver.env


  6. Shutdown the application tier.

    cd $ADMIN_SCRIPTS_HOME
    ./adstpall.sh apps/<source apps pass>

  7. Login as apps to the database and execute:

    exec fnd_conc_clone.setup_clean;

    I don't believe this step is necessary but if you don't do this you will see references to your source environment in the FND_% tables. Every time you execute this procedure you need to run autoconfig on each of the tiers (db and application). We will get to that in a second.

  8. Change the apps password. Chances are you don't want to have the same apps password as the source database, so its best to change it now while the environment is down.

    With the apps tier environment initialized:

    FNDCPASS apps/<source apps pass> 0 Y system/<source system pass>> SYSTEM APPLSYS <new apps pass>

  9. Run autoconfig on both the db tier and application tier.

    db tier:
    cd $ORACLE_HOME/appsutil/scripts/DEV_myserver
    ./adautocfg.sh

    Application Tier
    cd $ADMIN_SCRIPTS_HOME
    ./adautocfg.sh

  10. If there are no errors with autoconfig start the application. Your already in the $ADMIN_SCRIPTS_HOME so just execute:

    ./adstrtal.sh apps/<new apps pass>

  11. Login to the application and perform any post cloning activities. You may want to override the work flow email address so that notifications goto a test/dev mailbox instead of users. We always change the colors and site_name profile options, etc. More details can be found in Section 3: Finishing tasks of the R12 cloning document referenced earlier.


Thats it, hopefully now you have successfully cloning an EBS environment using rman duplicate.

Tuesday

Kernel bug and Oracle Enterprise Linux Support

A few months ago I joined a new company embarking on a new R12 implementation project. From a DBA perspective I haven't noticed major day to day differences than 11i.. Patching, routine maintenance, etc are all pretty similar. One big difference that I have noticed is that it seems to be more stable. With 11i it seemed I was constantly applying one off patches but for this R12 implementation, I think I have applied 6 at most.

Last week we made the decision to go live... within 10 minutes production crashed! We could ping the server but could not login either remotely or at the console.. So a hard reset was in order. When the server came back up we found the following in /var/log/messages:

Jun 3 11:55:18 myserver kernel: kernel BUG at kernel/exit.c:904!
Jun 3 11:55:18 myserver kernel: invalid operand: 0000 [#1]
Jun 3 11:55:18 myserver kernel: SMP

A quick search on metalink turned up the following note:

Linux Crashes when Enterprise Manager Agent Starts on RHEL 4 Update 6 and 7
Doc ID: 729543.1

This note caught my eye immediately because we just started to setup grid control for this server in preparation for going live. We had also just purchased OEL support from Oracle, so we opened an SR with them to confirm, which they did.

Over the past few weeks we were debating which vendor to purchase linux support from. There were two camps, one which preferred Redhat and the other (dba's) Oracle. My argument was that Oracle would be more aware of linux based issues affecting their software. The other side argued that they weren't sure Oracle could deliver the same level of service as Redhat.

Fast forward to the kernel bug. It turns out that OEL customers are not affected because the OEL 4.7 kernel already contains the fix. I realize this is just one case but at least it adds some weight to my argument.

I won't be able to apply the kernel patch until the next maintenance window, so until then i'll have to monitor our production environment the old way, via scripts.

Its just a simple change right?

Putting a database in archivelog mode? Set the destination, the archive log format, shutdown immediate, startup mount, alter database archivelog, alter database open, alter system archive log current. Done.

Except, alter system archive log current failed:


ORA-19504: failed to create file "/u02/archive/ORCL/ORCL_1_1_686533687.dbf"
ORA-27040: file create error, unable to create file
Linux Error: 13: Permission denied



Huh, thats ok. This is an Oracle EBS environment and I created the archive destination directory with the applmgr user, not oracle. No problem, su - root, chown oracle:dba . , done.

alter system archive log current again, success!

Can't remember what query I issued next (its been a hazzy night) but I was greeted with the following error:


Errors in file /u01/oracle/ORCL/db/tech_st/10.2.0/admin/ORCL_myserver/bdump/orcl_smon_10547.trc:
ORA-00376: file 5 cannot be read at this time
ORA-01110: data file 5: '/u01/oracle/ORCL/db/apps_st/data/system05.dbf'



WTF? Just some background about this environment. I only cloned it the day before and I didn't have any backups. I was putting the database in archivelog mode so I could take backups.

This error typically means the datafile is offline, tho i'm not sure why. So I bring it online:


alter database datafile 5 online;
recover datafile 5;



But:


ORA-00283: recovery session canceled due to errors
ORA-00264: no recovery required


Hrmm.. ok, so I try to open the database:

SQL> alter database open;
alter database open
*
ERROR at line 1:
ORA-03113: end-of-file on communication channel



WTF? Check the alert log:


Errors in file /u01/oracle/ORCL/db/tech_st/10.2.0/admin/ORCL_rc1657/bdump/uat2_lgwr_11557.trc:
ORA-00600: internal error code, arguments: [3712], [1], [1], [1388],
[3796296244], [1388], [3796296241], []



Lovely. At this point I have 3 options:

1. Work with Oracle Support.
2. Rebuild the environment since I don't have any backups. Developers lose 1 day.
3. Back it up, research the error and see if I can find a solution.

I worked with Oracle support up until 6:30am, at which point I believe my analyst was finished for the day and went home. This is a UAT environment, so the SR is a P2. I wasn't expecting 24x7 support, so this was just bad timing.

I searched metalink and didn't find any solid hits. So I tried google. Not many hits, but I received one exact match, in chinese. Google graciously offered to translate the page for me and about the only useful text on the page was:

"Who came across the internal bug? 我用了alter database clear logfile group 后再重启动数据库就发生这个错误了。 I used the alter database clear logfile group restart the database after this error has occurred. "

At this point I had an hour left before testers started and only 2 options remaining. I figure I was going to have to rebuild the database anyways, so i'd try clearing the log files. I cleared the inactive groups without issue. The only group remaining was the current and if the clear logfile group command failed i'd have to proceed with the rebuild.

From the alert log:

alter database clear unarchived logfile group 1
Tue May 12 08:14:34 2009
Thread 1 advanced to log sequence 6 (preparation to clear logfile)
WARNING! CLEARING REDO LOG WHICH HAS NOT BEEN ARCHIVED. BACKUPS TAKEN
BEFORE 05/12/2009 08:14:34 (CHANGE 1) CANNOT BE USED FOR RECOVERY.
Clearing online log 1 of thread 1 sequence number 5
Tue May 12 08:17:34 2009
Completed: alter database clear unarchived logfile group 1



Success!!!

Alter database open; Success!!!


Thats my 6 hr simple change (most of that time was working with Oracle Support, taking a backup [only 5MB/s IO??!] of the environment before I played around with it, etc.) Now I need some sleep. You know your tired when you almost use dish washing liquid for cream in your coffee. Luckily an attentive coworker prevented disaster! Normally an all nighter doesn't leave me this drained but the previous night I only had 3 hrs sleep.

Wednesday

How to Change an Application Server User Password

The solution depends on if you are using Oracle Internet Directory (OID) or not. If you are using OID you can login to the OID Console or the oidadmin GUI to reset the password easily.

But what if your application server is stand alone? If your trying to reset the oc4jadmin superuser account then the only solution is manually modify the $ORACLE_HOME/j2ee/home/config/system-jazn-data.xml file.

Search the file for your the username and place the new password, prefixed with an exclamation point "!" between the <credentials> </credentials> delimiters. I am assuming that you have to prefix it with the ! so that Oracle knows how to handle the password. ie, that its not encrypted.

Restart the application server and verify that you can login.

How about regular user accounts tho? Surprisingly you have to follow the same process. I must have poked around in the Application Server Console for half an hour in disbelief that I couldn't reset a user password without knowing the old password.

If I missed something or if you know of another way to reset user accounts, please post a comment.

Monday

Oracle Waives Extended Support Fees

Here is some interesting news from Collaborate today. Oracle has waived some Extended Support fees for some product lines including E-Business Suite 11i10 and 10gR2. Good news for those that can't afford to upgrade at the moment due to the economy.

For Details Click Here.