SOA Upgrade 11.1.1.3 -> 11.1.1.6

We upgraded our SOA install from 11.1.1.3 to 11.1.1.6 and hit a few issues.   One of the main issues was that after upgrading DEV our inflight processes disappeared.    We talked back and forth with Oracle and it was supposed to be supported but for some reason it wasn't working for us.    I know that from SOA 10g to SOA 11g in-flight processes are not supported as part of the upgrade.

One of the ways we use BPEL is workflow for a custom application.  It could take months for a workflow to complete and there are always in-flight processes.    So there is never a quiet time, or a point at which all workflows are complete to do the upgrade.

We put a plan in place to manually re-place all items back in the proper stage.    It seems like it was going to be a fair amount of effort so for DEV we only re-placed a subset of items to make sure we had the process correct.

However, once we upgrade TEST the in-flight processes somehow magically survived.    We aren't sure why, there could be some small inconsistencies between DEV and TEST which caused it.  Another reason could be that in DEV we upgraded to 11.1.1.6 first, then at a later point upgraded to 11.1.1.6.6.   In TEST we went directly to 11.1.1.6.6.

The SOA upgrade itself isn't a very resilient process.   If the PSA (Patch Set Assistant) gets interrupted for some reason its not smart enough to recover.  In DEV it wasn't a big issue, I had an export of the MDS and SOAINFRA schemas and I just dropped and recreated.    For PROD tho, the amount of data, thus downtime, would prevent us from doing this.    We only hit an issue once in DEV and TEST was smooth,  so for PROD we all had our fingers cross.

The PROD install was going smoothly until I had to run the PSA.     Shortly after I started the schema upgrade it failed and a dreaded error appeared.  In the $FMW_HOME/oracle_common/upgrade/logs directory I found:


[2013-08-21T19:28:09.239-04:00] [SOA] [ERROR] [] [upgrade.SOA.SOA1] [tid: 13] [ecid: 0000K2ZD5BJ9lZWpPws1yd1I5Klw000004,0] [[
oracle.sysman.assistants.common.dbutil.SQLFatalErrorException: java.sql.SQLException: ORA-00054: resource busy and acquire with NOWAIT specified or timeout expired

The problem was a database session holding a lock in the SOAINFRA schema.   We hit this problem in DEV and the solution was to shutdown applications that access SOAINFRA tables and change the schema password.   However, I didn't notice a hung database session by one of our custom applications.

If you try to run the PSA again it says the SOAINFRA schema isn't valid and won't let you continue with the upgrade.   So I tried manually updating the registry to state the schema is valid.

update schema_version_registry set status='VALID' where mr_name='SOAINFRA';

Great!  The installer started up again but it quickly failed.  Checking the logs:

2013-08-21 19:36:19.378 rcu:Extracted SQL Statement: [ALTER TABLE BPM_CUBE_PROCESS ADD (SubType VARCHAR2(200), DeploymentInfo BLOB)]

2013-08-21 19:36:19.378 rcu:Statement Type: 'DDL Statement'

JDBC SQLException - ErrorCode: 1430SQLState:72000 Message: ORA-01430: column being added already exists in table

Since this was production, I opened a P1 SR with Oracle right away.   In these situations I try to get Oracle involved as quickly as possible.   I'll still continue to do research on my end.  Sometimes I find the solution quicker, sometimes Oracle Support does.   

This time I found the following note:

Patch Set Assistant Failed for SOA 11.1.1.6.0 when Patch 13606871 is Applied on SOA 11.1.1.5.0 (Doc ID 1517404.1)

It provided a set of SQL statements to help rollback the schema upgrade.  Only problem was my upgrade was from 11.1.1.3, not 11.1.1.5.    Knowing that Oracle now supported manually rolling back the upgrade, then I looked at the script that was failing:

/u01/app/oracle/product/fmw11g_SOA/SOAHome_1/rcu/integration/soainfra//sql/upgrade_soainfra_111130_111140_oracle.tsql

After a few trial and errors I managed to write a script to undo all the changes.     Unfortunately if you missed one, which was easy to do as the script is a few thousand lines, you had to start over.   Finally I managed to find all updated objects and continued with the upgrade.

Another upgrade issue I encountered showed itself in the startup logs for soa_server1:

java.sql.SQLException: ORA-25226: dequeue failed, queue UPG_SOAINFRA.EDN_OAOO_QUEUE is not enabled for dequeue

The solution to this problem was to manually restart the queues:

SQL> show user
USER is "UPG_SOAINFRA"
SQL> exec dbms_aqadm.start_queue('IP_OUT_QUEUE',true,true);
PL/SQL procedure successfully completed.
SQL>  exec dbms_aqadm.start_queue('EDN_OAOO_QUEUE',true,true);
PL/SQL procedure successfully completed.

After the startup we waited anxiously to hear from the superusers to let us know the status of in-flight processes.    To much celebration (more so from them than us since they had alot of potential work to do) in-flight processes did not disappear.

runInstaller java.lang.reflect.InvocationTargetException

Well this one stumped me for a little bit today:











Not much information on Metalink or Google.    In the past one issue that would crop up every now and then was /tmp mounted with noexec.   I checked for that but it was ok... On a whim I decided to set my tmp dir to another location and it worked:

TMP=/u01/tmp; export TMP


Update: I recall why changing the TMP directory worked.    The problem was with noexec as one of the mount options of /tmp.   I had changed it in fstab but couldn't arrange downtime so I used environment variables to repoint tmp.   The server still hasn't been rebooted, so the change hasn't come into affect yet.  

error on line 1 at column 1: Document is empty

I'm not sure why this happens but every now and then the invalidator password gets corrupted.  When that happens, the following error will be seen when you try to access portal:

This page contains the following errors:
error on line 1 at column 1: Document is empty
Below is a rendering of the page up to the first error.
FIX:
Re-enter the invalidator password for Webcache and Portal.
Portal WebCache

OAM and EBS Breaks after Cloning

We are in the process of integrating E-Business Suite with Oracle Access Manager (OAM). Its setup in DEV and TEST but not in PROD yet.    There was a requirement to reclone DEV and as a result the integration with OAM broke.

We ran all the configuration steps on the EBS side, such as:

• $FND_TOP/bin/txkrun.pl -script=SetSSOReg -registerinstance=yes
• Verifying ASADMIN was active and the correct password.
• Profile options to configure authentation:
• Applications SSO Type
• Applications SSO Login types
• etc...
• Business Events to make sure account creation and modifications synchronizes correctly.
• We skipped all of the tasks to be performed on the OAM side since they were already done.

However, when restarting Access Gate we would see the following error in the logs:

weblogic.application.ModuleException:

        at weblogic.jdbc.module.JDBCModule.prepare(JDBCModule.java:302)

        at weblogic.application.internal.flow.ModuleListenerInvoker.prepare(ModuleListenerInvoker.java:199)

        at weblogic.application.internal.flow.DeploymentCallbackFlow$1.next(DeploymentCallbackFlow.java:517)

        at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:52)

        at weblogic.application.internal.flow.DeploymentCallbackFlow.prepare(DeploymentCallbackFlow.java:159)

        Truncated. see log file for complete stacktrace

Caused By: weblogic.common.resourcepool.ResourceSystemException:

Could not connect to 'oracle.apps.fnd.ext.jdbc.datasource.AppsDataSource'.

The returned message is: ORA-01017: invalid username/password; logon denied

We triple checked the datasource credentials defined in the Access Gate server but still nothing worked.  It turns one of the steps to deploy Access Gate was to copy the dbc file from $FND_SECURE over to the $MW_HOME/appsutil/accessgate/dev directory.   Since DEV was recloned the APPL_SERVER_ID value had changed.    We re-copied this file, bounced the Access Gate server and everything started working again.

Since we don't have EBS integrated with OAM in PROD yet i'm not sure if we will need to do any additional steps after a clone.   I'm thinking all the steps above will apply except for maybe business events as they will already be configured properly in PROD.   I'll update this post if I have to do anything else.

Portal Upgrade 11.1.1.3 to 11.1.1.6

A few months ago we upgraded our Fusion Middleware Portal environment from 11.1.1.3 to 11.1.1.6.   The upgrade went pretty smoothly, especially compared to our 11.1.1.1 to 11.1.1.3 upgrade.   It seems that Oracle has replaced alot of the manual post-upgrade tasks with scripts.   It seemed much easier this time around.

Below is the upgrade.   I'll post a followup shortly with some of the issues I encountered and how they were resolved.

1.  Shutdown Portal
   a.  $FUSION_HOME/PORTALasinst_1/bin/opmnctl stopall;
 
   b.  $DOMAIN_HOME/bin/stopManagedWebLogic.sh WLS_PORTAL
        $DOMAIN_HOME/bin/stopWebLogic.sh

2.  Backup application and Database

3.  If you haven't already download the patchset from Oracle and extract it.
       Portal 11.1.1.6:  
             https://updates.oracle.com/download/13517084.html
       WebLogic 10.3.6:

            http://www.oracle.com/technetwork/middleware/weblogic/downloads/wls-main-097127.html

4.  Update WebLogic
    $JAVA_HOME/bin/java -jar wls1036_upgrade_generic.jar



Prompted for Configuration manager, Bypassed…


5.  Upgrade Portal
   a. $STAGEDIR/Disk1/runInstaller.sh



 Upgrade Portal Schema using the Patch Set Assistant

   1. Recompile any INVALID objects:
            @?/rdbms/admin/utlrp.sql

   2. Verify aq_tm_proccesses > 0

        show parameter aq_tm_processes;
        alter system set aq_tm_processes=1 scope=both;

  3.  Run the PSA utility
         $ORACLE_HOME/oracle_common/bin/psa


After you put in the database credentials click on the Connect button to select the Portal schema, enter the password and click on Next:

4.	Recompile any INVALID objects following same steps as above.
5	. Verify upgrade was successful
		SQL> SELECT OWNER, VERSION, STATUS, UPGRADED FROM SCHEMA_VERSION_REGISTRY;
		OWNER		VERSION		STATUS	U
		DEV_PORTAL		11.1.1.6.0		VALID	Y

Post-Patching Tasks If Your Starting Point is Release 11.1.1.3.0 1 . Upgrading System Components a.  Startup the Weblogic Admin server for the Portal Domain Check to see if Config is locked b.   ORACLE_HOME=/u01/app/oracle/product/fmw11g/PORTALas_1 c.  Start OPMN Instance $FUSION_HOME/PORTALasinst_1/bin/opmnctl start d.   Execute upgradenoj2eeapp.sh from $ORACLE_HOME/opmn/bin [oracle@myserver bin]$ ./upgradenonj2eeapp.sh -oracleInstance /u01/app/oracle/product/fmw11g/PORTALasinst_1 -adminHost myserver -adminPort 7001 -adminUsername weblogic Unregister instance... Command requires login to weblogic admin server (myserver):   Username: weblogic   Password:  Unregistering instance Command succeeded. redeploy NonJ2EEManagement.ear... Command requires login to weblogic admin server (myserver):   Username: weblogic   Password:  Redeploying NonJ2EEManagement Application...weblogic.Deployer invoked with options:  -adminurl myserver:7001 -username weblogic -name NonJ2EEManagement -source /u01/app/oracle/product/fmw11g/PORTALas_1/opmn/applications/NonJ2EEManagement.ear -redeploy -upload -noexit   Task 0 initiated: [Deployer:149026]deploy application NonJ2EEManagement [Version=11.1.1] on AdminServer. Task 0 completed: [Deployer:149026]deploy application NonJ2EEManagement [Version=11.1.1] on AdminServer. Target state: redeploy completed on Server AdminServer Done Command succeeded. re-register instance... Command requires login to weblogic admin server (myserver):   Username: weblogic   Password:  Registering instance Command succeeded. Successfully upgraded NonJ2EEManagement.ear and the registeration. e.   Resolving JDBC Errors in Oracle Reports and Oracle Portal You must perform the following in order to avoid these JDBC error messages: Go to the bin directory inside your DOMAIN_HOME. Edit the setDomainEnv.sh (on UNIX operating systems) or setDomainEnv.cmd (on Windows operating systems) file and search for EXT_PRE_CLASSPATH. Change the definition to match the following: On UNIX operating systems: EXT_PRE_CLASSPATH=$ORACLE_HOME/jdbc/lib/ojdbc6.jar export EXT_PRE_CLASSPATH For example: if [ "${EXT_PRE_CLASSPATH}" != "" ] ; then    if [ "${PRE_CLASSPATH}" != "" ] ; then       EXT_PRE_CLASSPATH=$ORACLE_HOME/jdbc/lib/ojdbc6.jar // add this line       export EXT_PRE_CLASSPATH                           // add this line       PRE_CLASSPATH="${EXT_PRE_CLASSPATH}${CLASSPATHSEP}${PRE_CLASSPATH}"       export PRE_CLASSPATH    else       EXT_PRE_CLASSPATH=$ORACLE_HOME/jdbc/lib/ojdbc6.jar  // add this line       export EXT_PRE_CLASSPATH                            // add this line       PRE_CLASSPATH="${EXT_PRE_CLASSPATH}"       export PRE_CLASSPATH    fi fi f.   Updating Fusion Middleware Shared Libraries Run the upgradeJRF() command on the node or system where the Administration Server is located for each domain you want to update. Your domain location is passed as a parameter: FUSION_HOME/oracle_common/common/bin/wlst.sh wls:/offline> upgradeJRF('/u01/app/oracle/product/fmw11g/user_projects/domains/PortalDomain') g . Updating Configurations and Stores wls:/offline> upgradeOpss(jpsConfig="/u01/app/oracle/product/fmw11g/user_projects/domains/PortalDomain/config/fmwconfig/jps-config.xml", jaznData="/u01/app/oracle/product/fmw11g/oracle_common/modules/oracle.jps_11.1.1/domain_config/system-jazn-data.xml")   {auditStore=null, password=null, user=null, url=null, jdbcDriver=null, jaznData=/u01/app/oracle/product/fmw11g/oracle_common/modules/oracle.jps_11.1.1/domain_config/system-jazn-data.xml, jpsConfig=/u01/app/oracle/product/fmw11g/user_projects/domains/PortalDomain/config/fmwconfig/jps-config.xml} Starting upgrade of opss configuration and security stores. Aug 23, 2012 11:55:38 AM oracle.security.jps.internal.audit.dynamic.AuditStoreFactory getInstance WARNING: file /u01/app/oracle/product/fmw11g/user_projects/domains/PortalDomain/config/fmwconfig/audit-store.xml was not found WLS ManagedService is not up running. Fall back to use system properties for configuration. Upgrade of opss configuration and security stores is done. NOTE: Ignore Warning above.  See Note:  1435333.1 h . Optimizing Performance for WebLogic Server Domains on 64-Bit Platforms Modify $DOMAIN_HOME/bin/setDomainEnv.sh and add: before line: # Had to have a separate test here BECAUSE of immediate variable expansion on windows if [ "${JAVA_USE_64BIT}" = "true" ] ; then         if [ "${JAVA_VENDOR}" = "Oracle" ] ; then                 EXTRA_JAVA_PROPERTIES="-XXcompressedRefs:enable=TRUE ${EXTRA_JAVA_PROPERTIES}"                 export EXTRA_JAVA_PROPERTIES         fi fi if [ "${JAVA_VENDOR}" = "Oracle" ] ; then         EXTRA_JAVA_PROPERTIES="-XX:+UseLargePagesForHeap ${EXTRA_JAVA_PROPERTIES}"         export EXTRA_JAVA_PROPERTIES fi EXTRA_JAVA_PROPERTIES="-Djps.auth.debug=false ${EXTRA_JAVA_PROPERTIES}" export EXTRA_JAVA_PROPERTIES

ORA-01722: Invalid Number - Weird Problem

We have an ADF application, which hooks into Portal and SOA running on Weblogic 10.3.6 in a Fusion Middleware environment.

A few weeks ago after pushing a new release of the code to production, users encountered an ORA-01722 error in some screens of the application.    We captured the SQL and could run it via SQL*Plus and Sql Developer with no errors.

We opened an SR with Oracle and they offered some suggestions and in the end recommended a patch to fix a shared cursor bug.   We haven't applied that patch yet because we are on the latest PSU for 11.1.0.7 and that patch isn't available for it.  So we've issued a patch request with Oracle.

The weird problem is, that when my co-worker starts the WebLogic Managed Server, the users do not encounter the ORA-01722 error.   The only difference is that I am ssh'ing from a Mac and he is on windows.   We ssh into the same generic account and execute the same steps.  We've also confirmed that I can restart the server a dozen times with the error and my co-worker has never had the problem. We've verified this a few times.

Today we had some more maintenance and I tried restarting the server again.    Users encountered the same ORA-01722 error.   I restarted the server again but this time from my windows computer using SecureCRT.   Error disappears.

The following is a diff between a mac and windows ssh session.   There isn't a huge difference.

< TERM=xterm

---

> TERM=xterm-256color

5c5

< SSH_CLIENT=X.X.X.X 57952 22

---

> SSH_CLIENT=X.X.X.X 65516 22

7c7

< SSH_TTY=/dev/pts/1

---

> SSH_TTY=/dev/pts/4

10c10

< LS_COLORS=no=00:fi=00:di=00;34:ln=00;36:pi=40;33:so=00;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:ex=00;32:*.cmd=00;32:*.exe=00;32:*.com=00;32:*.btm=00;32:*.bat=00;32:*.sh=00;32:*.csh=00;32:*.tar=00;31:*.tgz=00;31:*.arj=00;31:*.taz=00;31:*.lzh=00;31:*.zip=00;31:*.z=00;31:*.Z=00;31:*.gz=00;31:*.bz2=00;31:*.bz=00;31:*.tz=00;31:*.rpm=00;31:*.cpio=00;31:*.jpg=00;35:*.gif=00;35:*.bmp=00;35:*.xbm=00;35:*.xpm=00;35:*.png=00;35:*.tif=00;35:

---

> LS_COLORS=

16c16

< LANG=en_US.UTF-8

---

> LANG=en_CA.UTF-8

23c23

< SSH_CONNECTION=X.X.X.X 57952 X.X.X.X 22

---

> SSH_CONNECTION=X.X.X.X 65516 X.X.X.X 22

The only one that catches my eye is the LANG env variable but I honestly don't see that as having any affect on the application.   If there was an environmental difference I would expect to see other types of issues.  ORA-01722 is pretty specific.

Also, we can only reproduce this on our production server.   If I restart DEV/TEST from my Mac there are no issues.    During a future maintenance window I may try changing the different environment settings after I ssh in from my Mac just to see if the error re-occurs.

Regardless, we'll still proceed with the patch request just in case it happens again.  Weird Problem!

AUDIT_TRAIL = DB, Portal and Grid Control

If your not familiar with Oracle’s auditing features then a good place to start is with Oracle’s Documentation.   I personally think Oracle’s documentation, especially for the database products is great. 

http://docs.oracle.com/cd/B28359_01/network.111/b28531/auditing.htm#autoId1

The above documentation linked to above describes what auditing is, why you would need it, etc.   There’s no need for me to repeat it here.

Our environment is pretty small, with a limited number of people who have access to the databases, etc.   So normally I have the audit trail disabled, however for this particular database we did need to increase logging to investigate some issues.  After we were finished, we used NOAUDIT to disable the extra logging we enabled.   However we left AUDIT_TRAIL set to DB.

Move to a few months later and on a routine scan of database performance I noticed a query consuming a fair amount of resources. 

SQL Details: gh9pd08vhptgr
SELECT TO_CHAR(current_timestamp AT TIME ZONE 'GMT', 'YYYY-MM-DD HH24:MI:SS TZD') AS curr_timestamp, COUNT(username) AS failed_count
FROM sys.dba_audit_session
WHERE returncode != 0 AND timestamp >= current_timestamp - TO_DSINTERVAL('0 0:30:00')




By looking at the screenshot above it seems pretty nasty but in reality it was just a small spike on the Grid Control Top Activity chart.    As I mentioned, our environment is small and our servers have more than enough horsepower, so it didn’t have a significant impact on performance. 

From Metalink note: Slow Performance Of DBA_AUDIT_SESSION Query From EM [ID 829103.1]  “There is a known performance issue with DBA_AUDIT_SESSION table per non-published Bug 7633167”

However with AUDIT_TRAIL=DB, LOGON and LOGOFF’s were still be recorded.  Since this database hosts our Portal schema it mean upwards of 200k of entries going into the sys.aud$ daily, for a grand total of 32 million rows!

The note provides two options for the query above, the first is to disable the Failed Login Count Metric, the second is to purge the sys.aud$ table.  Note 73408.1 How to Truncate, Delete, or Purge Rows from the Audit Trail Table SYS.AUD$

Since at the moment we don’t require auditing to be enabled, then I am simply going to truncate the sys.aud$ table and disable auditing by setting the database initialization parameter AUDIT_TRAIL=none and restarting the database at the next maintenance window.

If you do require auditing then you should setup a purging strategy.  If you need to do this, some good blog articles to read are:

http://www.pythian.com/news/1106/oracle-11g-audit-enabled-by-default-but-what-about-purging/
http://damir-vadas.blogspot.com/2010/06/auditing-database.html

Receiving Clear alerts after a Blackout–Grid Control

Grid Control allows you to set blackouts for targets so that while your performing maintenance you won’t get notified    You can also use them to disable notifications while your working on an issue.  Nothing worse than being paged multiple times while your trying to fix an issue. 


One of our applications has a component which requires a quick nightly bounce schedule via cron.   So I setup a blackout in Grid Control to start 5 minutes before and extend to 5 minutes after the restart.   However, at 3am I received a lovely page letting me know that an alert has cleared:


Subject: EM Alert: Clear:MyApp PROD - Test MyApp Login Page is now up

Target Name=MyApp
Target type=Web Application
Host=
Occurred At=Mar 5, 2012 3:15:00 AM EST
Message=Test MyApp Login Page is now up: MyApp Login Page has status 6 since 03/05/12 03:15:00 till 03/05/12 03:15:00 in America/New_York. Beacon RCPSC Status: 1 from 02/28/12 09:54:31 till 03/05/12 03:17:13 in -05:00. No new severities found after the blackout Metric data found after blackout, using the latest severity Latest severity from the beacon is 15 at 02/29/12 21:13:04 Beacon votes up Beacon Tenzing Status: 1 from 02/28/12 10:20:19 till 03/05/12 03:16:48 in America/New_York. No new severities found after the blackout Metric data found after blackout, using the latest severity Latest severity from the beacon is 15 at 02/29/12 19:26:14 Beacon votes up The final status of the test is UP from 03/05/12 03:15:00 till 03/05/12 03:17:13
Metric=[Test Response] Status


Strange.    Initially I thought that there may have been a system time issue between the servers, or that the restart had taken longer than I expected.   Looking into it tho that was not the case. A search on metalink turns up that there is a bug:

Bug 10210193 WEB APPS SERVICE GENERATE ERROR WHEN BLACKOUT START the Notification rule used has the metric [Test Response] Status which is generating the notification.


The solution is to remove the remove the [Test Response] metric from the rule.    No sleep interruptions the following night!

CPU Patches and Minimum Baselines

As you may or may not be aware, your environment has to meet a minimum baseline before you can apply CPU patches.  Typically Oracle supplies patches for both the latest patch set of a product and the previous patch set for a certain grace period. For example, below is clip taken from the EBS ECS Policy document  (Note: 1195034.1):

In addition, a given release update pack for the Applications Technology
product family will be treated as the minimum baseline 18 months after its
release. Specifically:

The Applications Technology RUP from 12.1.2 (R12.ATG_PF.B.Delta.2, Note
ID 845809.1) becomes the minimum prerequisite baseline on July 1, 2011.
The Applications Technology RUP from 12.1.3 (R12.ATG_PF.B.Delta.3, Note
ID 1066312.1) becomes the minimum prerequisite baseline on February 1,
2012

So the grace period for the ATG product family is 18 months.   ATG_PF.B.Delta.2 is the current baseline, however, after Feb. 1st the new baseline is ATG_PF.B.Delta.3.    That means if your not already on Delta.3, your going to upgrade before or as part of the July CPU release.

For other products the grace period may be different.  For example, for the database it is a minimum of 3 months, maximum of 1 year.  For more details see:  Database, FMW, EM Grid Control, and OCS Software Error Correction Support Policy [ID 209768.1]


The January 2012 CPU release information can be found here:

http://www.oracle.com/technetwork/topics/security/cpujan2012-366304.html

For Fusion Middleware and Database customers the best way to find this information is to keep an eye on the “Final Patch History” section of the patch availability document.  

For E-Business Suite customers make sure you read the yellow highlighted sections of the patch availability document.

We use E-Business Suite, Weblogic, Grid Control, Database and Fusion Middleware products.  So some key dates for us are:

• January 2012:
• Oracle Fusion Middleware 11.1.1.3
• Oracle WebLogic Server 10.3.3.0
• E-Business Suite:  ATG_PF.B.Delta.2
• April 2012:
• Oracle Fusion Middleware 11.1.1.4
• Oracle WebLogic Server 10.3.4.0
• July 2012:
• Oracle Database 11.2.0.2

The reason I mention this is because it catches people off guard all the time.  Upgrading to new baselines can add a significant amount of time to applying CPU’s, and its nice to make sure management, business owners, etc have as much time as possible to prepare.

Grid Control and Fusion Middleware–Required patches

We have been using Grid Control 11.1.0.1 for almost a year now to monitor our Fusion Middleware environments.   I’ve recently had an issue where I couldn’t get a job to restart a weblogic domain.   It errors out with:

Could not execute query: java.sql.SQLException: ORA-01403: no data found
ORA-06512: at line 88


Apparently you have to be on PSU 3+ and MLR3 applied before Oracle will help.   A year ago when we put it in, the only restriction I could find was that you had to be on GC 11g.    So, if you are using GC to monitor your fusion middleware environments you should look at this note and make sure your patched properly, before you need to open an SR.

Oracle Required Patches for Discovery and Monitoring Fusion Middleware Targets (Doc ID 1354421.1)


This note was published on Nov 3rd.  I don’t see a revision history so I am assuming it was only published recently.

11.1.1.1 -> 11.1.1.3 Fusion Middleware Upgrade

I've been meaning to post this awhile.. How long, well I believe 11.1.1.5 is out now.   I took a quick look at the upgrade documentation tho and the overall process doesn't seem to have changed much.  (I did see some extra post-upgrade steps depending on which version you start from..)

The biggest hurdle seems to be upgrading from 11.1.1.1 to 11.1.1.2...   I spent a fair bit of time trying to understand the documentation and perform the upgrade.   The process as described in the documentation didn't seem to make any sense to me.   After talking to Oracle support they found it difficult to follow as well and said they would file a request to get it reviewed.    We decided at that time to do a fresh install of 11.1.1.2 and upgrade it to 11.1.1.3.

I haven't checked back on the documentation to see if the sections were clarified.  If you followed or have seen my 11.1.1.1 installation guide you have an idea of my setup.    If you haven't, a quick review:

- Two Tier Environment
- VM18 has the IDM stack and Database installed.
- VM17 has Portal, SOA and SSO installed.

As I said above, instead of upgrading from 11.1.1.1 to 11.1.1.2 and patching to 11.1.1.3 we did a fresh install of 11.1.1.2.   The installation process for 11.1.1.2 is almost identical to 11.1.1.1 so I used the install guide I had already created as a base for the install.    From there I followed the steps below to patch up to 11.1.1.3

Note: The steps below only cover upgrading the products to 11.1.1.3.   Since we performed a fresh install it doesn't cover topics such as migrating data (example, OID accounts), domains hosting 3rd party software, portal data (transport sets), configuration (load balancers, reverse proxies, ..), etc.

Patch Weblogic from 10.2.2 to 10.2.3

1. Download the Upgrade Installer from Oracle Support.
   1. see "Downloading an Upgrade Installer From My Oracle Support"

2. Launch the install:
   1. [oracle@vm18 STAGE]$ $JAVA_HOME/bin/java -jar wls1033_upgrade_generic.jar

Upgrading IDM

Portal Patchset Install

• Prompted to run oracleRoot.sh

SOA Patchset Install

Updating Schemas with Patch Set Assistant

• Startup the database, ensure aq_tm_processes parameter is > 0
• In each home for Portal, SOA and IDM there is a Patch Set Assistant utility which was installed as part of the 11.1.1.3 patchset.

IDM psa

[oracle@vm18 bin]$ ./psa -dbConnectString //vm18:1521/FMW.world -dbaUserName sys -schemaUserName ODS
Oracle Fusion Middleware Patch Set Assistant 11.1.1.3.0
Enter the database administrator password for "sys":
Enter the schema password for schema user "ODS":
Log file is located at: /u01/app/oracle/product/mid11g3/Oracle_IDM1/upgrade/logs/psa2010-08-13-15-00-47PM.log
Updating the Oracle Fusion Middleware Metadata Schema to release 11.1.1.3.0.
Updating schema for component Oracle Internet Directory.
The command completed successfully


Start IDM
[oracle@vm18 bin]$ ./opmnctl startall
opmnctl startall: starting opmn and all managed processes...
[oracle@vm18 bin]$ pwd
/u01/app/oracle/product/mid11g3/IDMasinst_1/bin

Portal PSA

[oracle@vm17 bin]$ ./psa -dbConnectString //vm18:1521/FMW.world -dbaUserName sys -schemaUserName UPG_PORTAL
Oracle Fusion Middleware Patch Set Assistant 11.1.1.3.0
Enter the database administrator password for "sys":
Enter the schema password for schema user "UPG_PORTAL":
Log file is located at: /u01/app/oracle/product/mid11g3/PORTALas_1/upgrade/logs/psa2010-08-13-15-06-02PM.log
Updating the Oracle Fusion Middleware Metadata Schema to release 11.1.1.3.0.
Updating schema for component Oracle Portal.
The command completed successfully

SOA PSA

[oracle@vm17 bin]$ ./psa -dbConnectString //vm18:1521/FMW.world -dbaUserName sys -schemaUserName UPG_SOAINFRA
Oracle Fusion Middleware Patch Set Assistant 11.1.1.3.0
Enter the database administrator password for "sys":
Enter the schema password for schema user "UPG_SOAINFRA":
Log file is located at: /u01/app/oracle/product/mid11g3/SOAHome_1/upgrade/logs/psa2010-08-13-15-29-49PM.log
Updating the Oracle Fusion Middleware Metadata Schema to release 11.1.1.3.0.
Updating schema for component Oracle SOA.
---Redirecting sql log to /u01/app/oracle/product/mid11g3/SOAHome_1/upgrade/logs/psa2010-08-13-15-29-49PM.out
Executed SOAINFRA upgrade script.
The command completed successfully

Verify the PSA

SQL> select owner, version, status from schema_version_registry;
OWNER VERSION STATUS
------------------------------ ------------------------------ -----------
ODS 11.1.1.3.0 VALID
UPG_PORTAL 11.1.1.3.0 VALID
UPG_SOAINFRA 11.1.1.3.0 VALID

Check for Invalid objects and recompile:

SQL> select count(*) from dba_objects where status = 'INVALID';
COUNT(*)
----------
12
SQL> @?/rdbms/admin/utlrp

Post-patching tasks for the System Components:

(You may have to perform the steps below for the other products as well (IDM, SOA), not just for IDM.   I made note of it a while back but since we did the upgrade almost a year ago, I have misplaced it.)

/u01/app/oracle/product/mid11g3/PORTALasinst_1/bin
[oracle@vm17 bin]$ ./opmnctl unregisterinstance
Command requires login to weblogic admin server (vm17.rcpsc.edu):
Username: weblogic
Password:
Unregistering instance
Command succeeded.

[oracle@vm17 bin]$ ./opmnctl redeploy
Command requires login to weblogic admin server (vm17.rcpsc.edu):
Username: weblogic
Password:
Redeploying NonJ2EEManagement Application...weblogic.Deployer invoked with options: -adminurl vm17.rcpsc.edu:7001 -username weblogic -name NonJ2EEManagement -source /u01/app/oracle/product/mid11g3/PORTALas_1/opmn/applications/NonJ2EEManagement.ear -redeploy -upload -noexit
<Aug 13, 2010 4:20:54 PM EDT> <Info> <J2EE Deployment SPI> <BEA-260121> <Initiating redeploy operation for application, NonJ2EEManagement [archive: /u01/app/oracle/product/mid11g3/PORTALas_1/opmn/applications/NonJ2EEManagement.ear], to configured targets.>
Task 0 initiated: [Deployer:149026]deploy application NonJ2EEManagement [Version=11.1.1] on AdminServer.
Task 0 completed: [Deployer:149026]deploy application NonJ2EEManagement [Version=11.1.1] on AdminServer.
Target state: redeploy completed on Server AdminServer
Done
Command succeeded.

[oracle@vm17 bin]$ ./opmnctl registerinstance;
Command requires login to weblogic admin server (vm17.rcpsc.edu):
Username: weblogic
Password:
Registering instance
Command succeeded.

Update Version Number String shown in OEM

/u01/app/oracle/product/mid11g3/IDMasinst_1/bin
[oracle@vm18 bin]$ ./opmnctl updatecomponentregistration -componentName oid1 -Sport 3131 -Port 3060
Command requires login to weblogic admin server (192.168.80.148):
Username: weblogic
Password: