Tuesday

OOUG DBA/Developer Day

Today I attended the OOUG, Ottawa Oracle User's group DBA/Developer day. If your interested in the presentations you can download them from here:

OOUG Website

Sort by date to find them quickly.

I attended:

Accountability for System Performance - Interesting discussion on 6 sigma, which is an objective way to measure system performance. It didn't dive very deep but was a good overview and points you in the right direction if you'd like to know more.

Profiling Oracle: How it works - An extension of the session above which dove deeper into how to profile your code and why you should do it. Also an interesting case study into why 'idle' events sometimes shouldn't be ignored.

SOA What? Introduction to Server Oriented Architecture - This session was just a brief overview of SOA and I attended just so I would get a bit more familiar with the topic. And finally I know what a Wiz-Dull (WDSL) is!

I didn't attend the last two sessions, had to leave for Toronto. This past weekend our company had a planned DRP (Disaster Recovery Planning) simulation so I had to travel to the recovery site.

Don't hit CTRL-C while running adaddnode

Yesterday we were implementing a loadbalancer + 2 new application servers and an OS upgrade on the dbTier. After working for about 12 hours that day plus another 8 the night before (on top of a regular work day) I guess I was starting to get a bit tired... As well as feeling a little stressed (my sister-in-law was in a car accident the night before causing an unexpected trip to Toronto for my wife. So on top of performing the upgrade I had two restless kids on my hands...).

To save a fraction of a second, instead of typing out a complete path I decided to copy & paste it from the window I was running perl adaddnode.pl.... For some unknown reason, instead of hitting CTRL-INSERT (I was using SecureCRT), I hit CTRL-C.... Instantly I knew I made a boo boo.

No biggie I thought, I can just run the comand again. To my horror this is what I saw:


++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Updating tables...
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

declare
*
ERROR at line 1:
ORA-00001: unique constraint (APPLSYS.AD_APPL_TOPS_U2) violated
ORA-06512: at line 16



I think its safe to say my heart skipped a few beats at that point. Just to give a bit of background.. I'm fairly new to the Apps DBA world, but i've been a DBA for about 8 years. Our primary apps DBA moved backed to India, and unable to find a replacement I took over the helm.

My first stop was metalink where I found note: 387679.1, When Adding A Node Through Rapid Clone We Are Getting Error. I followed the workaround but no luck. With no other leads I opened up a TAR. Meanwhile all our users arrived onsite to do sanity checking and the release coordinator was calling every 5 minutes, along with others who wanted updates. I stopped answering my phone at that point.

Oracle responded very quickly and recommended the same workaround (I mentioned in the TAR that I had tried it). In the end I had to truncate the ad_timestamps, ad_appl_tops tables and rerun the adaddnode script.

SUCCESS!

The rest went pretty smooth and by the end of the day we were about 3 hours behind schedule, a total of 27 instead of 24hrs. Not too bad given the other problems that we hit.

I have a few things left to migrate to the new servers such as OEM monitoring jobs but i'm happy to say business signed off on the upgrade saying they were impressed with the speed. So I guess it wasn't such a bad day after all.