As every Apps DBA knows, we apply alot of patches. For some reason, every now and then, the global inventory gets corrupted (for lack of a better term) in our cloned environments. For example:
oraora@myserver=> opatch lsinventory
Oracle Interim Patch Installer version 1.0.0.0.55
Copyright (c) 2006 Oracle Corporation. All Rights Reserved..
We recommend you refer to the OPatch documentation under
OPatch/docs for usage reference. We also recommend using
the latest OPatch version. For the latest OPatch version
and other support related issues, please refer to document
293369.1 which is viewable from metalink.oracle.com
Oracle Home = /oradata/app/oradb/920_64bit
Location of Oracle Universal Installer components = /oradata/app/oradb/920_64bit/oui
Location of OraInstaller.jar = "/oradata/app/oradb/920_64bit/oui/jlib"
Oracle Universal Installer shared library = /oradata/app/oradb/920_64bit/oui/lib/solaris/liboraInstaller.so
Location of Oracle Inventory Pointer = /var/opt/oracle/oraInst.loc
Location of Oracle Inventory = /oradata/app/oradb/920_64bit/inventory
Path to Java = /oradata/app/oradb/920_64bit/jre/1.4.2/bin/java
Log file = /oradata/app/oradb/920_64bit/.patch_storage//*.log
Creating log file "/oradata/app/oradb/920_64bit/.patch_storage/LsInventory__03-05-2007_16-43-00.log"
Result:
LsInventory: OPatch Exception while accessing O2O
OPATCH_JAVA_ERROR : An exception of type "OPatchException" has occurred:
OPatch Exception:
OUI found no such ORACLE_HOME set in the environment
Can not get details for given Oracle Home
An exception occurs
null
There is no Interim Patch
OPATCH_JAVA_ERROR : An exception of type "OPatchException" has occurred:
Can not get a list of inventory on this home.
ERROR: OPatch failed because of Inventory problem.
If you do a search on metalink you'll see a few potential resolutions which range from verifying your oraInst.loc file to manually editing the inventory.xml file within your global inventory.
For 11i environemnts, Oracle provides us with a method to recreate the global inventory. For complete details you can view Note:295185.1 but the key steps are:
1. Verify ora inventory_loc within oraInst.loc
2. Take a backup of your inventory location.
3. Make sure you are on OUI 2.2 or later.
If your iAS ORACLE_HOME is missing from the inventory.xml file:
4. Sync your apps environment file.
5. Change to $IAS_ORACLE_HOME/appsoui/setup and execute perl OUIsetup.pl
If your RDBMS ORACLE_HOME is missing from the inventory.xml file:
4. Sync your database environment file.
5. Change to $ORACLE_HOME/appsoui/setup and execute perl OUIsetup.pl
ex. Excuting on the database tier:
oraora@myserver=> . ./DBORA.env
oraora@myserver=> cd $ORACLE_HOME/appsoui/setup
oraora@myserver=> perl OUIsetup.pl
ORACLE_HOME : /oradata/app/oradb/920_64bit (APPSDB_ORA)
OUI Location : /oradata/app/oradb/920_64bit/appsoui/oui
CLASSPATH : /oradata/app/oradb/920_64bit/appsoui/setup/OUISetup.jar:/oradata/app/oradb/920_64bit/appsoui/oui/lib/OraInstaller.jar:/oradata/app/oradb/920_64bit/appsoui/oui/lib/xmlparserv2.jar
JRE_INST : /oradata/app/oradb/920_64bit/appsoui/jre/1.1.8
Executing OUI setup now:
/oradata/app/oradb/920_64bit/appsoui/jre/1.3.1/bin/java -classpath /oradata/app/oradb/920_64bit/appsoui/setup/OUISetup.jar:/oradata/app/oradb/920_64bit/appsoui/oui/lib/OraInstaller.jar:/oradata/app/oradb/920_64bit/appsoui/oui/lib/xmlparserv2.jar -DDEBUG_MODE=false oracle.apps.ad.util.OUISetup /oradata/app/oradb/920_64bit /oradata/app/oradb/920_64bit/appsoui/oui /oradata/app/oradb/920_64bit/appsoui/jre/1.1.8 APPSDB_ORA
Log file : /oradata/app/oradb/920_64bit/appsoui/setup/OUIsetup.log
Checking inventory ...
Updating registration of ORACLE_HOME in global inventory...
Oracle home registration completed.
Removing OUI entries from the global inventory: /oradata/home/oraagf/oraInventory/ContentsXML/comps.xml
Removing OUI entries from the global inventory completed successfully
Registering the OUI component with local inventory...
OUI 2.2 registration completed.
OPatch lsinventory should now work fine.
Wednesday
Openworld 2006
This year I will be attending Openworld once again... I have to say tho, i'm not as excited as in previous years... Last year a coworker went with me and we met for lunches, suppers, etc. I find traveling alone pretty boring...
I finally finished my schedule tho and here are the sessions I plan on attending:
S283027 OAUG SysAdmin Special Interest Group Meeting
S283025 OAUG Upgrade Special Interest Group Meeting
S283047 Archive and Purge Special Interest Group Meeting
S283713 Is Your Production Support Team Working on the "Right Stuff"?
S282990 Top 10 Implementation Tips for Oracle Data Guard
S281377 Technology Directions for Oracle Applications
S281206 Database Worst Practices
S281398 Oracle and Pella: A Case Study in Reducing Oracle E-Business Suite Maintenance Downtime
S281651 Oracle E-Business Suite Customers: 10 Things You Can Do Now to Prepare for Oracle Fusion Applications
S280814 Oracle E-Business Suite Technology Updates
S281383 Deep Dive: Oracle E-Business Suite Release 12 New Technology Stack
S281393 Tuning Oracle E-Business Suite
S281396 E-Business Suite Lifecycle Management
S281397 Oracle E-Business Suite System Management: Release 12 New Features
S281394 How to Secure Your Oracle E-Business Suite Deployment
S281654 Partitioning and Purging Best Practices for Oracle E-Business Suite
S282383 Welcome to My Nightmare: The Common Performance Errors in Oracle Databases
S281384 Leveraging Oracle Database 10g Features with Oracle E-Business Suite
S281660 A Hidden Treasure: Oracle E-Business Suite Support Tools
As you can tell, they are mostly Oracle EBS related.
I finally finished my schedule tho and here are the sessions I plan on attending:
S283027 OAUG SysAdmin Special Interest Group Meeting
S283025 OAUG Upgrade Special Interest Group Meeting
S283047 Archive and Purge Special Interest Group Meeting
S283713 Is Your Production Support Team Working on the "Right Stuff"?
S282990 Top 10 Implementation Tips for Oracle Data Guard
S281377 Technology Directions for Oracle Applications
S281206 Database Worst Practices
S281398 Oracle and Pella: A Case Study in Reducing Oracle E-Business Suite Maintenance Downtime
S281651 Oracle E-Business Suite Customers: 10 Things You Can Do Now to Prepare for Oracle Fusion Applications
S280814 Oracle E-Business Suite Technology Updates
S281383 Deep Dive: Oracle E-Business Suite Release 12 New Technology Stack
S281393 Tuning Oracle E-Business Suite
S281396 E-Business Suite Lifecycle Management
S281397 Oracle E-Business Suite System Management: Release 12 New Features
S281394 How to Secure Your Oracle E-Business Suite Deployment
S281654 Partitioning and Purging Best Practices for Oracle E-Business Suite
S282383 Welcome to My Nightmare: The Common Performance Errors in Oracle Databases
S281384 Leveraging Oracle Database 10g Features with Oracle E-Business Suite
S281660 A Hidden Treasure: Oracle E-Business Suite Support Tools
As you can tell, they are mostly Oracle EBS related.
Thursday
11.5.9 Rebaseline
If your running Oracle EBS your probably aware by now that Oracle has rebaselined. I heard about this from Steven Chan's blog back in June I believe. (BTW, if you support Oracle Apps his blog is a must read IMHO).
So what does rebaselining mean? Well, Oracle regression tests their patches against a standard patch level... It would be impossible to do this for every conceivable combination. Since the July CPU has been certified against this baseline, then its recommended that you apply the patches as soon as possible.
What did that mean for us? Well, on top of the new baseline and security patches business requested an OTL (Oracle Time and Labour) upgrade as well. In total, that meant we had to apply around 70 patches. The first time you apply the patches it always takes a bit longer because you have to read through the patch notes, pre/post install steps and you usually hit problems which require some investigation. It took us about 50+ hours to put these in our first environment. That was approximately the LOE I provided. Tomorrow we start our stage environment and i'm hoping we can do it in 16-24 hours.
So far we have hit one post patch problem. If you are using autoconfig and applied patch 4489303 - TXK (FND) AUTOCONFIG TEMPLATE ROLLUP PATCH L (NOV 2005) (part of the rebaseline for 11.5.9) you may hit Bug 4891248. To resolve the issue you need a patch, Note:359703.1 has all the details.
Now we just wait for unit and user testing.. Since a number of modules have been touched i'm expecting problems.
So what does rebaselining mean? Well, Oracle regression tests their patches against a standard patch level... It would be impossible to do this for every conceivable combination. Since the July CPU has been certified against this baseline, then its recommended that you apply the patches as soon as possible.
What did that mean for us? Well, on top of the new baseline and security patches business requested an OTL (Oracle Time and Labour) upgrade as well. In total, that meant we had to apply around 70 patches. The first time you apply the patches it always takes a bit longer because you have to read through the patch notes, pre/post install steps and you usually hit problems which require some investigation. It took us about 50+ hours to put these in our first environment. That was approximately the LOE I provided. Tomorrow we start our stage environment and i'm hoping we can do it in 16-24 hours.
So far we have hit one post patch problem. If you are using autoconfig and applied patch 4489303 - TXK (FND) AUTOCONFIG TEMPLATE ROLLUP PATCH L (NOV 2005) (part of the rebaseline for 11.5.9) you may hit Bug 4891248. To resolve the issue you need a patch, Note:359703.1 has all the details.
Now we just wait for unit and user testing.. Since a number of modules have been touched i'm expecting problems.
Tuesday
OOUG DBA/Developer Day
Today I attended the OOUG, Ottawa Oracle User's group DBA/Developer day. If your interested in the presentations you can download them from here:
OOUG Website
Sort by date to find them quickly.
I attended:
Accountability for System Performance - Interesting discussion on 6 sigma, which is an objective way to measure system performance. It didn't dive very deep but was a good overview and points you in the right direction if you'd like to know more.
Profiling Oracle: How it works - An extension of the session above which dove deeper into how to profile your code and why you should do it. Also an interesting case study into why 'idle' events sometimes shouldn't be ignored.
SOA What? Introduction to Server Oriented Architecture - This session was just a brief overview of SOA and I attended just so I would get a bit more familiar with the topic. And finally I know what a Wiz-Dull (WDSL) is!
I didn't attend the last two sessions, had to leave for Toronto. This past weekend our company had a planned DRP (Disaster Recovery Planning) simulation so I had to travel to the recovery site.
OOUG Website
Sort by date to find them quickly.
I attended:
Accountability for System Performance - Interesting discussion on 6 sigma, which is an objective way to measure system performance. It didn't dive very deep but was a good overview and points you in the right direction if you'd like to know more.
Profiling Oracle: How it works - An extension of the session above which dove deeper into how to profile your code and why you should do it. Also an interesting case study into why 'idle' events sometimes shouldn't be ignored.
SOA What? Introduction to Server Oriented Architecture - This session was just a brief overview of SOA and I attended just so I would get a bit more familiar with the topic. And finally I know what a Wiz-Dull (WDSL) is!
I didn't attend the last two sessions, had to leave for Toronto. This past weekend our company had a planned DRP (Disaster Recovery Planning) simulation so I had to travel to the recovery site.
Don't hit CTRL-C while running adaddnode
Yesterday we were implementing a loadbalancer + 2 new application servers and an OS upgrade on the dbTier. After working for about 12 hours that day plus another 8 the night before (on top of a regular work day) I guess I was starting to get a bit tired... As well as feeling a little stressed (my sister-in-law was in a car accident the night before causing an unexpected trip to Toronto for my wife. So on top of performing the upgrade I had two restless kids on my hands...).
To save a fraction of a second, instead of typing out a complete path I decided to copy & paste it from the window I was running perl adaddnode.pl.... For some unknown reason, instead of hitting CTRL-INSERT (I was using SecureCRT), I hit CTRL-C.... Instantly I knew I made a boo boo.
No biggie I thought, I can just run the comand again. To my horror this is what I saw:
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Updating tables...
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
declare
*
ERROR at line 1:
ORA-00001: unique constraint (APPLSYS.AD_APPL_TOPS_U2) violated
ORA-06512: at line 16
I think its safe to say my heart skipped a few beats at that point. Just to give a bit of background.. I'm fairly new to the Apps DBA world, but i've been a DBA for about 8 years. Our primary apps DBA moved backed to India, and unable to find a replacement I took over the helm.
My first stop was metalink where I found note: 387679.1, When Adding A Node Through Rapid Clone We Are Getting Error. I followed the workaround but no luck. With no other leads I opened up a TAR. Meanwhile all our users arrived onsite to do sanity checking and the release coordinator was calling every 5 minutes, along with others who wanted updates. I stopped answering my phone at that point.
Oracle responded very quickly and recommended the same workaround (I mentioned in the TAR that I had tried it). In the end I had to truncate the ad_timestamps, ad_appl_tops tables and rerun the adaddnode script.
SUCCESS!
The rest went pretty smooth and by the end of the day we were about 3 hours behind schedule, a total of 27 instead of 24hrs. Not too bad given the other problems that we hit.
I have a few things left to migrate to the new servers such as OEM monitoring jobs but i'm happy to say business signed off on the upgrade saying they were impressed with the speed. So I guess it wasn't such a bad day after all.
To save a fraction of a second, instead of typing out a complete path I decided to copy & paste it from the window I was running perl adaddnode.pl.... For some unknown reason, instead of hitting CTRL-INSERT (I was using SecureCRT), I hit CTRL-C.... Instantly I knew I made a boo boo.
No biggie I thought, I can just run the comand again. To my horror this is what I saw:
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Updating tables...
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
declare
*
ERROR at line 1:
ORA-00001: unique constraint (APPLSYS.AD_APPL_TOPS_U2) violated
ORA-06512: at line 16
I think its safe to say my heart skipped a few beats at that point. Just to give a bit of background.. I'm fairly new to the Apps DBA world, but i've been a DBA for about 8 years. Our primary apps DBA moved backed to India, and unable to find a replacement I took over the helm.
My first stop was metalink where I found note: 387679.1, When Adding A Node Through Rapid Clone We Are Getting Error. I followed the workaround but no luck. With no other leads I opened up a TAR. Meanwhile all our users arrived onsite to do sanity checking and the release coordinator was calling every 5 minutes, along with others who wanted updates. I stopped answering my phone at that point.
Oracle responded very quickly and recommended the same workaround (I mentioned in the TAR that I had tried it). In the end I had to truncate the ad_timestamps, ad_appl_tops tables and rerun the adaddnode script.
SUCCESS!
The rest went pretty smooth and by the end of the day we were about 3 hours behind schedule, a total of 27 instead of 24hrs. Not too bad given the other problems that we hit.
I have a few things left to migrate to the new servers such as OEM monitoring jobs but i'm happy to say business signed off on the upgrade saying they were impressed with the speed. So I guess it wasn't such a bad day after all.
Subscribe to:
Posts (Atom)