Monday

R12 - Cloning from an RMAN backup using duplicate database

Since most DBA's are using rman for their backup strategy I thought I would put together the steps to clone from an rman backup. The steps you follow are pretty much the same as described in Appendix A: Recreating the database control files manually in Rapid Clone in Note 406982.1 - Cloning Oracle Applications Release 12 with Rapid Clone.

Here are the steps:
  1. Execute preclone on all tiers of the source system. This includes both the database and application tiers. (For this example, TEST is my source system.)

    For the database execute: $ORACLE_HOME/appsutil/scripts/<context>/adpreclone.pl dbTier
    Where context name is of the format <sid>_<hostname>

    For the application tier: $ADMIN_SCRIPTS_HOME/adpreclone.pl appsTier

  2. Prepare the files needed for the clone and copy them to the target server.
    • Take a FULL rman backup and copy the files to the target server and place them in the identical path. ie. if your rman backups go to /u01/backup on the source server, place them in /u01/backup on the destination server. To be safe, you may want to copy some of the archive files generated while the database was being backed up. Place them in an identical path on the target server as well.
    • Application Tier: tar up the application files and copy them to the destination server. The cloning document referenced above ask you to take a copy of the $APPL_TOP, $COMMON_TOP, $IAS_ORACLE_HOME and $ORACLE_HOME. Normally I just tar up the System Base Directory, which is the root directory for your application files.
    • Database Tier: tar up the database $ORACLE_HOME.

      ex. from a single tier system. The first tar file contains the application files and the second is the database $ORACLE_HOME

      [oratest@myserver TEST]$ pwd
      /u01/TEST
      [oratest@myserver TEST]$ ls
      apps db inst
      [oratest@myserver TEST]$ tar cvfzp TEST_apps_inst_myserver.tar.gz apps inst
      .
      .
      [oratest@myserver TEST]$ tar cvfzp TEST_dbhome_myserver.tar.gz db/tech_st

      Notice for the database $ORACLE_HOME I only added the db/tech_st directory to the archive. The reason is that the database files are under db/apps_st and we don't need those.
    • Copy the tar files to the destination server, create a directory for your new environment, for example /u01/DEV. (For the purpose of this article I will be using /u01/DEV as the system base for the target envrionment we are building and myserver is the server name.)
    • Extract each of the tar files with the command tar xvfzp

      Ex. tar xvfzp TEST_apps_inst_myserver.tar.gz

  3. Configure the target system.
    • On the database tier execute adcfgclone.pl with the dbTechStack parameter.

      For example. /u01/DEV/db/tech_st/10.2.0/appsutil/clone/bin/adcfgclone.pl dbTechStack

      By passing the dbTechStack parameter we are tell the script to configure only the necessary $ORACLE_HOME files such as the init file for the new environment, listener.ora, database environment settings file, etc. It will also start the listener.

      You will be prompted the standard post cloning questions such as the SID of the new environment, number of DATA_TOPS, Oracle Home location, port settings, etc.

      Once this is complete goto /u01/DEV/db/tech_st/10.2.0 and execute the environment settings file to make sure your environment is set correctly.

      [oradev@myserver 10.2.0] . ./DEV_myserver.env

  4. Duplicate the source database to the target.
    • In order to duplicate the source database you'll need to know the scn value to recover to. There are two wasy to do this. The first is to login to your rman catalog, find the Chk SCN of the files in the last backupset of your rman backup and add 1 to it.

      Ex. Output from a rman> List backups
      .
      .
      List of Datafiles in backup set 55729
      File LV Type Ckp SCN Ckp Time Name
      ---- -- ---- ---------- --------- ----
      7 1 Incr 5965309363843 15-JUN-09 /u02/TEST/db/apps_st/data/owad01.dbf
      .

      .
      So in this case the SCN we would be recovery to is 5965309363843 + 1 = 5965309363844.

      The other method is to login to the rman catalog via sqlplus and execute the following query:

      select max(absolute_fuzzy_change#)+1,
      max(checkpoint_change#)+1
      from rc_backup_datafile;


      Use which ever value is greater.
    • Modify the db_file_name_convert and log_file_name convert parameters in the target init file. Example:

      db_file_name_convert=('/u02/PROD/db/apps_st/data/', '/u02/DEV/db/apps_st/data/',
      '/u01/PROD/db/apps_st/data/', '/u02/DEV/db/apps_st/data/')

      log_file_name_convert=(/u02/PROD/db/apps_st/data/', '/u02/DEV/db/apps_st/data/',
      '/u01/PROD/db/apps_st/data/', '/u02/DEV/db/apps_st/data/')

    • Verify you can connect to source system from the target as sysdba. You will need to add a tns entry to the $TNS_ADMIN/tnsnames.ora file for the source system.
    • Duplicate the database. Before we use rman to duplicate the source database we need to start the target database in nomount mode.

      Start rman:

      rman target sys/<syspass>@TEST catalog rman/rman@RMAN auxiliary /

      If there are no connection errors duplicate the database with the following script:

      run {
      set until scn 5965309363844;
      allocate auxiliary channel ch1 type disk;
      allocate auxiliary channel ch2 type disk;
      duplicate target database to DEV }


      The most common errors at this point are connection errors to the source database and rman catalog. As well, if the log_file_name_convert and db_file_name_convert parameters are not set properly you will see errors. Fix the problems, login with rman again and re-execute the script.

      When the rman duplicate has finished the database will be open and ready to proceed with the next steps.

    • Execute the library update script:

      cd $ORACLE_HOME/appsutil/install/DEV_myserver where DEV_myserver is the <context_name> of the new environment.

      sqlplus "/ as sysdba"@adupdlib.sql

      If your on linux replace with so, HPUX with sl and for windows servers leave blank.

    • Configure the target database

      cd $ORACLE_HOME/appsutil/clone/bin/adcfgclone.pl dbconfig

      Where is $ORACLE_HOME/appsutil/DEV_myserver.xml

  5. Configure the application tier.

    cd /u01/DEV/apps/apps_st/comn/clone/bin
    perl adcfgclone.pl appsTier

    You will be prompted the standard cloning questions consisting of the system base directories, which services you want enabled, port pool, etc. Make sure you choose the same port pool as you did when configuring the database tier in step 3.

    Once that is finished, initialize your environment by executing

    . /u01/DEV/apps/apps_st/appl/APPSDEV_myserver.env


  6. Shutdown the application tier.

    cd $ADMIN_SCRIPTS_HOME
    ./adstpall.sh apps/<source apps pass>

  7. Login as apps to the database and execute:

    exec fnd_conc_clone.setup_clean;

    I don't believe this step is necessary but if you don't do this you will see references to your source environment in the FND_% tables. Every time you execute this procedure you need to run autoconfig on each of the tiers (db and application). We will get to that in a second.

  8. Change the apps password. Chances are you don't want to have the same apps password as the source database, so its best to change it now while the environment is down.

    With the apps tier environment initialized:

    FNDCPASS apps/<source apps pass> 0 Y system/<source system pass>> SYSTEM APPLSYS <new apps pass>

  9. Run autoconfig on both the db tier and application tier.

    db tier:
    cd $ORACLE_HOME/appsutil/scripts/DEV_myserver
    ./adautocfg.sh

    Application Tier
    cd $ADMIN_SCRIPTS_HOME
    ./adautocfg.sh

  10. If there are no errors with autoconfig start the application. Your already in the $ADMIN_SCRIPTS_HOME so just execute:

    ./adstrtal.sh apps/<new apps pass>

  11. Login to the application and perform any post cloning activities. You may want to override the work flow email address so that notifications goto a test/dev mailbox instead of users. We always change the colors and site_name profile options, etc. More details can be found in Section 3: Finishing tasks of the R12 cloning document referenced earlier.


Thats it, hopefully now you have successfully cloning an EBS environment using rman duplicate.

Tuesday

Kernel bug and Oracle Enterprise Linux Support

A few months ago I joined a new company embarking on a new R12 implementation project. From a DBA perspective I haven't noticed major day to day differences than 11i.. Patching, routine maintenance, etc are all pretty similar. One big difference that I have noticed is that it seems to be more stable. With 11i it seemed I was constantly applying one off patches but for this R12 implementation, I think I have applied 6 at most.

Last week we made the decision to go live... within 10 minutes production crashed! We could ping the server but could not login either remotely or at the console.. So a hard reset was in order. When the server came back up we found the following in /var/log/messages:

Jun 3 11:55:18 myserver kernel: kernel BUG at kernel/exit.c:904!
Jun 3 11:55:18 myserver kernel: invalid operand: 0000 [#1]
Jun 3 11:55:18 myserver kernel: SMP

A quick search on metalink turned up the following note:

Linux Crashes when Enterprise Manager Agent Starts on RHEL 4 Update 6 and 7
Doc ID: 729543.1

This note caught my eye immediately because we just started to setup grid control for this server in preparation for going live. We had also just purchased OEL support from Oracle, so we opened an SR with them to confirm, which they did.

Over the past few weeks we were debating which vendor to purchase linux support from. There were two camps, one which preferred Redhat and the other (dba's) Oracle. My argument was that Oracle would be more aware of linux based issues affecting their software. The other side argued that they weren't sure Oracle could deliver the same level of service as Redhat.

Fast forward to the kernel bug. It turns out that OEL customers are not affected because the OEL 4.7 kernel already contains the fix. I realize this is just one case but at least it adds some weight to my argument.

I won't be able to apply the kernel patch until the next maintenance window, so until then i'll have to monitor our production environment the old way, via scripts.