Oracle Apps Blog

Monday

R12 Cloning: adcrdb.sh missing from file system.

This weekend I had to build an R12 environment. We use Oracle VM in our development environment so this made the who process much easier. Before I would have to setup a new server, install the OS, patch it, put in the Oracle pre-req's then clone. Now I just need to run pre-clone, shutdown the environment, copy it, rename/reconfigure the vm and run post clone.

When I ran adcfgclone.pl dbTier tho I hit the error described in the following note:

On Release 12 adcfgclone is failing with AC-00423, RC-50014 adcrdb.sh missing from file system [ID 549872.1]

The solution is to copy adcrdb.sh from the source system and place it under $ORACLE_HOME/appsutil/template

However, when I executed adcfgclone.pl again I hit the same error. adcrdb.sh would get deleted from the template directory.

The solution was to run autoconfig, preclone and recopy the db files and software over.

Well, at least I didn't have to build a new server.. Saved me a little time anyways.

If your having cloning issues you may want to take a look at the other issues I encountered as well. Just click on the EBS Cloning label on the right side.

Sunday

R12 Cloning: Can't login as apps

This past week I was cloning a DEV environment from PROD. I noticed that the adcfgclone.pl script appeared to be hanging. Looking at the logs it was stuck on:

Attempting to connect to database using :

jdbc:oracle:thin:@myserver:1553:DEV,APPS
DbSid :DEV
DbHost :myserver
DomainName:mydomain.com
DbPort :1553

So the first thing I tried was logging into the database using tns, ex: sqlplus apps/apps@DEV. This also hung. However, other accounts such as dbsnmp, etc were fine.

Using Grid Control I noticed there was a lot of contention in the database:

Drilling down into concurrency it was very easy to see that the issue was related to library cache locks.

Looking at the sessions involved I found that they were related to our SOA server and other integrated applications. Since I am in the process of cloning this database from PROD, the databases are identical, down to the account passwords. So all of these connection attempts were using the incorrect password. I reset the passwords to their appropriate DEV values a little later in the cloning process.

I did a quick search on Metalink and this note jumped out at me:

LIBRARY CACHE LOCKS DUE TO INVALID LOGIN ATTEMPTS [ID 1309738.1]

The recommendation is to set the following event:

EVENT="28401 TRACE NAME CONTEXT FOREVER, LEVEL 1"

I set this parameter, bounced the database, reran adcfgclone.pl and it ran with no issues.

So, just an FYI. If your cloning an EBS environment and have alot of integrated applications/processes that you may run into this issue until you either reset the account passwords or set this event.

Monday

Mac OSX: Disable Java 7

As most of you are probably aware if you are running Java 7 on your desktop there is a bug which will prevent forms from launching:

Steven Chan covers the certifications and the workarounds on his blog.

https://blogs.oracle.com/stevenChan/entry/java_jre_1_7_0

My Mac was updated to Java 7 a short period of time ago and of course I had issues opening forms. I'm late to the game so i'm sure most people have already hit this issue. I'm just documenting it here as a personal reference for when I need to back the fix out.

There are two solutions to this problem:

1. Patch your EBS environment as per Steven's article above.
2. Disable Java 7 on your Mac.

I choose to do number 2 as I needed to login ASAP and didn't have time to wait for patches to get through QA.

To disable Java 7 and re-enable Java 6 follow the steps on Apple's Support site:

http://support.apple.com/kb/HT5559?viewlocale=en_US

While I was trying* to track down when my system was updated to Java 7, I found the following site from Oracle which provides alot of information for Mac users of Java:

http://www.java.com/en/download/faq/java_mac.xml

* I'm not sure how my Mac was updated to 7. I'm assuming it was when our tech support upgraded my Mac to Mountain Lion and updated it.

Friday

Portal Upgrade 11.1.1.3 to 11.1.1.6

A few months ago we upgraded our Fusion Middleware Portal environment from 11.1.1.3 to 11.1.1.6.   The upgrade went pretty smoothly, especially compared to our 11.1.1.1 to 11.1.1.3 upgrade.   It seems that Oracle has replaced alot of the manual post-upgrade tasks with scripts.   It seemed much easier this time around.

Below is the upgrade.   I'll post a followup shortly with some of the issues I encountered and how they were resolved.

1. Shutdown Portal
   a. $FUSION_HOME/PORTALasinst_1/bin/opmnctl stopall;

   b. $DOMAIN_HOME/bin/stopManagedWebLogic.sh WLS_PORTAL
        $DOMAIN_HOME/bin/stopWebLogic.sh

2. Backup application and Database

3. If you haven't already download the patchset from Oracle and extract it.
       Portal 11.1.1.6:
             https://updates.oracle.com/download/13517084.html
       WebLogic 10.3.6:

http://www.oracle.com/technetwork/middleware/weblogic/downloads/wls-main-097127.html

4. Update WebLogic
$JAVA_HOME/bin/java -jar wls1036_upgrade_generic.jar

Prompted for Configuration manager, Bypassed…

5. Upgrade Portal
a. $STAGEDIR/Disk1/runInstaller.sh

Upgrade Portal Schema using the Patch Set Assistant

1. Recompile any INVALID objects:
@?/rdbms/admin/utlrp.sql

2. Verify aq_tm_proccesses > 0

show parameter aq_tm_processes;
alter system set aq_tm_processes=1 scope=both;

3. Run the PSA utility
$ORACLE_HOME/oracle_common/bin/psa

After you put in the database credentials click on the Connect button to select the Portal schema, enter the password and click on Next:

4.	Recompile any INVALID objects following same steps as above.

5	. Verify upgrade was successful

		SQL> SELECT OWNER, VERSION, STATUS, UPGRADED FROM SCHEMA_VERSION_REGISTRY;

		OWNER	VERSION	STATUS	U
		DEV_PORTAL	11.1.1.6.0	VALID	Y

Post-Patching Tasks If Your Starting Point is Release 11.1.1.3.0

. Upgrading System Components

Startup the Weblogic Admin server for the Portal Domain

Check to see if Config is locked

ORACLE_HOME=/u01/app/oracle/product/fmw11g/PORTALas_1

Start OPMN Instance

$FUSION_HOME/PORTALasinst_1/bin/opmnctl start

Execute upgradenoj2eeapp.sh from $ORACLE_HOME/opmn/bin

[oracle@myserver bin]$ ./upgradenonj2eeapp.sh -oracleInstance /u01/app/oracle/product/fmw11g/PORTALasinst_1 -adminHost myserver -adminPort 7001 -adminUsername weblogic

Unregister instance...

Command requires login to weblogic admin server (myserver):

Username: weblogic

Password:

Unregistering instance

Command succeeded.

redeploy NonJ2EEManagement.ear...

Command requires login to weblogic admin server (myserver):

Username: weblogic

Password:

Redeploying NonJ2EEManagement Application...weblogic.Deployer invoked with options: -adminurl myserver:7001 -username weblogic -name NonJ2EEManagement -source /u01/app/oracle/product/fmw11g/PORTALas_1/opmn/applications/NonJ2EEManagement.ear -redeploy -upload -noexit

Task 0 initiated: [Deployer:149026]deploy application NonJ2EEManagement [Version=11.1.1] on AdminServer.

Task 0 completed: [Deployer:149026]deploy application NonJ2EEManagement [Version=11.1.1] on AdminServer.

Target state: redeploy completed on Server AdminServer

Done

Command succeeded.

re-register instance...

Command requires login to weblogic admin server (myserver):

Username: weblogic

Password:

Registering instance

Command succeeded.

Successfully upgraded NonJ2EEManagement.ear and the registeration.

Resolving JDBC Errors in Oracle Reports and Oracle Portal

You must perform the following in order to avoid these JDBC error messages:

Go to the bin directory inside your DOMAIN_HOME.

Edit the setDomainEnv.sh (on UNIX operating systems) or setDomainEnv.cmd (on Windows operating systems) file and search for EXT_PRE_CLASSPATH.

Change the definition to match the following:

On UNIX operating systems:

EXT_PRE_CLASSPATH=$ORACLE_HOME/jdbc/lib/ojdbc6.jar

export EXT_PRE_CLASSPATH

For example:

if [ "${EXT_PRE_CLASSPATH}" != "" ] ; then

if [ "${PRE_CLASSPATH}" != "" ] ; then

EXT_PRE_CLASSPATH=$ORACLE_HOME/jdbc/lib/ojdbc6.jar // add this line

export EXT_PRE_CLASSPATH // add this line

PRE_CLASSPATH="${EXT_PRE_CLASSPATH}${CLASSPATHSEP}${PRE_CLASSPATH}"

export PRE_CLASSPATH

else

EXT_PRE_CLASSPATH=$ORACLE_HOME/jdbc/lib/ojdbc6.jar // add this line

export EXT_PRE_CLASSPATH // add this line

PRE_CLASSPATH="${EXT_PRE_CLASSPATH}"

export PRE_CLASSPATH

Updating Fusion Middleware Shared Libraries

Run the upgradeJRF() command on the node or system where the Administration Server is located for each domain you want to update. Your domain location is passed as a parameter:

FUSION_HOME/oracle_common/common/bin/wlst.sh

wls:/offline> upgradeJRF('/u01/app/oracle/product/fmw11g/user_projects/domains/PortalDomain')

. Updating Configurations and Stores

wls:/offline> upgradeOpss(jpsConfig="/u01/app/oracle/product/fmw11g/user_projects/domains/PortalDomain/config/fmwconfig/jps-config.xml", jaznData="/u01/app/oracle/product/fmw11g/oracle_common/modules/oracle.jps_11.1.1/domain_config/system-jazn-data.xml")

{auditStore=null, password=null, user=null, url=null, jdbcDriver=null, jaznData=/u01/app/oracle/product/fmw11g/oracle_common/modules/oracle.jps_11.1.1/domain_config/system-jazn-data.xml, jpsConfig=/u01/app/oracle/product/fmw11g/user_projects/domains/PortalDomain/config/fmwconfig/jps-config.xml}

Starting upgrade of opss configuration and security stores.

Aug 23, 2012 11:55:38 AM oracle.security.jps.internal.audit.dynamic.AuditStoreFactory getInstance

WARNING: file /u01/app/oracle/product/fmw11g/user_projects/domains/PortalDomain/config/fmwconfig/audit-store.xml was not found

WLS ManagedService is not up running. Fall back to use system properties for configuration.

Upgrade of opss configuration and security stores is done.

NOTE: Ignore Warning above. See Note: 1435333.1

. Optimizing Performance for WebLogic Server Domains on 64-Bit Platforms

Modify $DOMAIN_HOME/bin/setDomainEnv.sh and add:

before line:

# Had to have a separate test here BECAUSE of immediate variable expansion on windows

if [ "${JAVA_USE_64BIT}" = "true" ] ; then

if [ "${JAVA_VENDOR}" = "Oracle" ] ; then

EXTRA_JAVA_PROPERTIES="-XXcompressedRefs:enable=TRUE

${EXTRA_JAVA_PROPERTIES}"

export EXTRA_JAVA_PROPERTIES

if [ "${JAVA_VENDOR}" = "Oracle" ] ; then

EXTRA_JAVA_PROPERTIES="-XX:+UseLargePagesForHeap

${EXTRA_JAVA_PROPERTIES}"

export EXTRA_JAVA_PROPERTIES

EXTRA_JAVA_PROPERTIES="-Djps.auth.debug=false ${EXTRA_JAVA_PROPERTIES}"

export EXTRA_JAVA_PROPERTIES

Saturday

ORA-01722: Invalid Number - Weird Problem

We have an ADF application, which hooks into Portal and SOA running on Weblogic 10.3.6 in a Fusion Middleware environment.

A few weeks ago after pushing a new release of the code to production, users encountered an ORA-01722 error in some screens of the application. We captured the SQL and could run it via SQL*Plus and Sql Developer with no errors.

We opened an SR with Oracle and they offered some suggestions and in the end recommended a patch to fix a shared cursor bug. We haven't applied that patch yet because we are on the latest PSU for 11.1.0.7 and that patch isn't available for it. So we've issued a patch request with Oracle.

The weird problem is, that when my co-worker starts the WebLogic Managed Server, the users do not encounter the ORA-01722 error. The only difference is that I am ssh'ing from a Mac and he is on windows. We ssh into the same generic account and execute the same steps. We've also confirmed that I can restart the server a dozen times with the error and my co-worker has never had the problem. We've verified this a few times.

Today we had some more maintenance and I tried restarting the server again. Users encountered the same ORA-01722 error. I restarted the server again but this time from my windows computer using SecureCRT. Error disappears.

The following is a diff between a mac and windows ssh session. There isn't a huge difference.

< TERM=xterm

---

> TERM=xterm-256color

5c5

< SSH_CLIENT=X.X.X.X 57952 22

---

> SSH_CLIENT=X.X.X.X 65516 22

7c7

< SSH_TTY=/dev/pts/1

---

> SSH_TTY=/dev/pts/4

10c10

< LS_COLORS=no=00:fi=00:di=00;34:ln=00;36:pi=40;33:so=00;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:ex=00;32:*.cmd=00;32:*.exe=00;32:*.com=00;32:*.btm=00;32:*.bat=00;32:*.sh=00;32:*.csh=00;32:*.tar=00;31:*.tgz=00;31:*.arj=00;31:*.taz=00;31:*.lzh=00;31:*.zip=00;31:*.z=00;31:*.Z=00;31:*.gz=00;31:*.bz2=00;31:*.bz=00;31:*.tz=00;31:*.rpm=00;31:*.cpio=00;31:*.jpg=00;35:*.gif=00;35:*.bmp=00;35:*.xbm=00;35:*.xpm=00;35:*.png=00;35:*.tif=00;35:

---

> LS_COLORS=

16c16

< LANG=en_US.UTF-8

---

> LANG=en_CA.UTF-8

23c23

< SSH_CONNECTION=X.X.X.X 57952 X.X.X.X 22

---

> SSH_CONNECTION=X.X.X.X 65516 X.X.X.X 22

The only one that catches my eye is the LANG env variable but I honestly don't see that as having any affect on the application. If there was an environmental difference I would expect to see other types of issues. ORA-01722 is pretty specific.

Also, we can only reproduce this on our production server. If I restart DEV/TEST from my Mac there are no issues. During a future maintenance window I may try changing the different environment settings after I ssh in from my Mac just to see if the error re-occurs.

Regardless, we'll still proceed with the patch request just in case it happens again. Weird Problem!