Don't hit CTRL-C while running adaddnode

Yesterday we were implementing a loadbalancer + 2 new application servers and an OS upgrade on the dbTier. After working for about 12 hours that day plus another 8 the night before (on top of a regular work day) I guess I was starting to get a bit tired... As well as feeling a little stressed (my sister-in-law was in a car accident the night before causing an unexpected trip to Toronto for my wife. So on top of performing the upgrade I had two restless kids on my hands...).

To save a fraction of a second, instead of typing out a complete path I decided to copy & paste it from the window I was running perl adaddnode.pl.... For some unknown reason, instead of hitting CTRL-INSERT (I was using SecureCRT), I hit CTRL-C.... Instantly I knew I made a boo boo.

No biggie I thought, I can just run the comand again. To my horror this is what I saw:


++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Updating tables...
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

declare
*
ERROR at line 1:
ORA-00001: unique constraint (APPLSYS.AD_APPL_TOPS_U2) violated
ORA-06512: at line 16


I think its safe to say my heart skipped a few beats at that point. Just to give a bit of background.. I'm fairly new to the Apps DBA world, but i've been a DBA for about 8 years. Our primary apps DBA moved backed to India, and unable to find a replacement I took over the helm.

My first stop was metalink where I found note: 387679.1, When Adding A Node Through Rapid Clone We Are Getting Error. I followed the workaround but no luck. With no other leads I opened up a TAR. Meanwhile all our users arrived onsite to do sanity checking and the release coordinator was calling every 5 minutes, along with others who wanted updates. I stopped answering my phone at that point.

Oracle responded very quickly and recommended the same workaround (I mentioned in the TAR that I had tried it). In the end I had to truncate the ad_timestamps, ad_appl_tops tables and rerun the adaddnode script.

SUCCESS!

The rest went pretty smooth and by the end of the day we were about 3 hours behind schedule, a total of 27 instead of 24hrs. Not too bad given the other problems that we hit.

I have a few things left to migrate to the new servers such as OEM monitoring jobs but i'm happy to say business signed off on the upgrade saying they were impressed with the speed. So I guess it wasn't such a bad day after all.