Wednesday

Openworld 2006

This year I will be attending Openworld once again... I have to say tho, i'm not as excited as in previous years... Last year a coworker went with me and we met for lunches, suppers, etc. I find traveling alone pretty boring...

I finally finished my schedule tho and here are the sessions I plan on attending:

S283027 OAUG SysAdmin Special Interest Group Meeting
S283025 OAUG Upgrade Special Interest Group Meeting
S283047 Archive and Purge Special Interest Group Meeting
S283713 Is Your Production Support Team Working on the "Right Stuff"?
S282990 Top 10 Implementation Tips for Oracle Data Guard
S281377 Technology Directions for Oracle Applications
S281206 Database Worst Practices
S281398 Oracle and Pella: A Case Study in Reducing Oracle E-Business Suite Maintenance Downtime
S281651 Oracle E-Business Suite Customers: 10 Things You Can Do Now to Prepare for Oracle Fusion Applications
S280814 Oracle E-Business Suite Technology Updates
S281383 Deep Dive: Oracle E-Business Suite Release 12 New Technology Stack
S281393 Tuning Oracle E-Business Suite
S281396 E-Business Suite Lifecycle Management
S281397 Oracle E-Business Suite System Management: Release 12 New Features
S281394 How to Secure Your Oracle E-Business Suite Deployment
S281654 Partitioning and Purging Best Practices for Oracle E-Business Suite
S282383 Welcome to My Nightmare: The Common Performance Errors in Oracle Databases
S281384 Leveraging Oracle Database 10g Features with Oracle E-Business Suite
S281660 A Hidden Treasure: Oracle E-Business Suite Support Tools


As you can tell, they are mostly Oracle EBS related.

Thursday

11.5.9 Rebaseline

If your running Oracle EBS your probably aware by now that Oracle has rebaselined. I heard about this from Steven Chan's blog back in June I believe. (BTW, if you support Oracle Apps his blog is a must read IMHO).

So what does rebaselining mean? Well, Oracle regression tests their patches against a standard patch level... It would be impossible to do this for every conceivable combination. Since the July CPU has been certified against this baseline, then its recommended that you apply the patches as soon as possible.

What did that mean for us? Well, on top of the new baseline and security patches business requested an OTL (Oracle Time and Labour) upgrade as well. In total, that meant we had to apply around 70 patches. The first time you apply the patches it always takes a bit longer because you have to read through the patch notes, pre/post install steps and you usually hit problems which require some investigation. It took us about 50+ hours to put these in our first environment. That was approximately the LOE I provided. Tomorrow we start our stage environment and i'm hoping we can do it in 16-24 hours.

So far we have hit one post patch problem. If you are using autoconfig and applied patch 4489303 - TXK (FND) AUTOCONFIG TEMPLATE ROLLUP PATCH L (NOV 2005) (part of the rebaseline for 11.5.9) you may hit Bug 4891248. To resolve the issue you need a patch, Note:359703.1 has all the details.

Now we just wait for unit and user testing.. Since a number of modules have been touched i'm expecting problems.

Tuesday

OOUG DBA/Developer Day

Today I attended the OOUG, Ottawa Oracle User's group DBA/Developer day. If your interested in the presentations you can download them from here:

OOUG Website

Sort by date to find them quickly.

I attended:

Accountability for System Performance - Interesting discussion on 6 sigma, which is an objective way to measure system performance. It didn't dive very deep but was a good overview and points you in the right direction if you'd like to know more.

Profiling Oracle: How it works - An extension of the session above which dove deeper into how to profile your code and why you should do it. Also an interesting case study into why 'idle' events sometimes shouldn't be ignored.

SOA What? Introduction to Server Oriented Architecture - This session was just a brief overview of SOA and I attended just so I would get a bit more familiar with the topic. And finally I know what a Wiz-Dull (WDSL) is!

I didn't attend the last two sessions, had to leave for Toronto. This past weekend our company had a planned DRP (Disaster Recovery Planning) simulation so I had to travel to the recovery site.

Don't hit CTRL-C while running adaddnode

Yesterday we were implementing a loadbalancer + 2 new application servers and an OS upgrade on the dbTier. After working for about 12 hours that day plus another 8 the night before (on top of a regular work day) I guess I was starting to get a bit tired... As well as feeling a little stressed (my sister-in-law was in a car accident the night before causing an unexpected trip to Toronto for my wife. So on top of performing the upgrade I had two restless kids on my hands...).

To save a fraction of a second, instead of typing out a complete path I decided to copy & paste it from the window I was running perl adaddnode.pl.... For some unknown reason, instead of hitting CTRL-INSERT (I was using SecureCRT), I hit CTRL-C.... Instantly I knew I made a boo boo.

No biggie I thought, I can just run the comand again. To my horror this is what I saw:


++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Updating tables...
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

declare
*
ERROR at line 1:
ORA-00001: unique constraint (APPLSYS.AD_APPL_TOPS_U2) violated
ORA-06512: at line 16



I think its safe to say my heart skipped a few beats at that point. Just to give a bit of background.. I'm fairly new to the Apps DBA world, but i've been a DBA for about 8 years. Our primary apps DBA moved backed to India, and unable to find a replacement I took over the helm.

My first stop was metalink where I found note: 387679.1, When Adding A Node Through Rapid Clone We Are Getting Error. I followed the workaround but no luck. With no other leads I opened up a TAR. Meanwhile all our users arrived onsite to do sanity checking and the release coordinator was calling every 5 minutes, along with others who wanted updates. I stopped answering my phone at that point.

Oracle responded very quickly and recommended the same workaround (I mentioned in the TAR that I had tried it). In the end I had to truncate the ad_timestamps, ad_appl_tops tables and rerun the adaddnode script.

SUCCESS!

The rest went pretty smooth and by the end of the day we were about 3 hours behind schedule, a total of 27 instead of 24hrs. Not too bad given the other problems that we hit.

I have a few things left to migrate to the new servers such as OEM monitoring jobs but i'm happy to say business signed off on the upgrade saying they were impressed with the speed. So I guess it wasn't such a bad day after all.