Monday

SOA Upgrade 11.1.1.3 -> 11.1.1.6

We upgraded our SOA install from 11.1.1.3 to 11.1.1.6 and hit a few issues.   One of the main issues was that after upgrading DEV our inflight processes disappeared.    We talked back and forth with Oracle and it was supposed to be supported but for some reason it wasn't working for us.    I know that from SOA 10g to SOA 11g in-flight processes are not supported as part of the upgrade.

One of the ways we use BPEL is workflow for a custom application.  It could take months for a workflow to complete and there are always in-flight processes.    So there is never a quiet time, or a point at which all workflows are complete to do the upgrade.

We put a plan in place to manually re-place all items back in the proper stage.    It seems like it was going to be a fair amount of effort so for DEV we only re-placed a subset of items to make sure we had the process correct.

However, once we upgrade TEST the in-flight processes somehow magically survived.    We aren't sure why, there could be some small inconsistencies between DEV and TEST which caused it.  Another reason could be that in DEV we upgraded to 11.1.1.6 first, then at a later point upgraded to 11.1.1.6.6.   In TEST we went directly to 11.1.1.6.6.

The SOA upgrade itself isn't a very resilient process.   If the PSA (Patch Set Assistant) gets interrupted for some reason its not smart enough to recover.  In DEV it wasn't a big issue, I had an export of the MDS and SOAINFRA schemas and I just dropped and recreated.    For PROD tho, the amount of data, thus downtime, would prevent us from doing this.    We only hit an issue once in DEV and TEST was smooth,  so for PROD we all had our fingers cross.

The PROD install was going smoothly until I had to run the PSA.     Shortly after I started the schema upgrade it failed and a dreaded error appeared.  In the $FMW_HOME/oracle_common/upgrade/logs directory I found:


[2013-08-21T19:28:09.239-04:00] [SOA] [ERROR] [] [upgrade.SOA.SOA1] [tid: 13] [ecid: 0000K2ZD5BJ9lZWpPws1yd1I5Klw000004,0] [[
oracle.sysman.assistants.common.dbutil.SQLFatalErrorException: java.sql.SQLException: ORA-00054: resource busy and acquire with NOWAIT specified or timeout expired

The problem was a database session holding a lock in the SOAINFRA schema.   We hit this problem in DEV and the solution was to shutdown applications that access SOAINFRA tables and change the schema password.   However, I didn't notice a hung database session by one of our custom applications.

If you try to run the PSA again it says the SOAINFRA schema isn't valid and won't let you continue with the upgrade.   So I tried manually updating the registry to state the schema is valid.

update schema_version_registry set status='VALID' where mr_name='SOAINFRA';

Great!  The installer started up again but it quickly failed.  Checking the logs:

2013-08-21 19:36:19.378 rcu:Extracted SQL Statement: [ALTER TABLE BPM_CUBE_PROCESS ADD (SubType VARCHAR2(200), DeploymentInfo BLOB)]
2013-08-21 19:36:19.378 rcu:Statement Type: 'DDL Statement'
JDBC SQLException - ErrorCode: 1430SQLState:72000 Message: ORA-01430: column being added already exists in table

Since this was production, I opened a P1 SR with Oracle right away.   In these situations I try to get Oracle involved as quickly as possible.   I'll still continue to do research on my end.  Sometimes I find the solution quicker, sometimes Oracle Support does.   

This time I found the following note:

Patch Set Assistant Failed for SOA 11.1.1.6.0 when Patch 13606871 is Applied on SOA 11.1.1.5.0 (Doc ID 1517404.1)

It provided a set of SQL statements to help rollback the schema upgrade.  Only problem was my upgrade was from 11.1.1.3, not 11.1.1.5.    Knowing that Oracle now supported manually rolling back the upgrade, then I looked at the script that was failing:

/u01/app/oracle/product/fmw11g_SOA/SOAHome_1/rcu/integration/soainfra//sql/upgrade_soainfra_111130_111140_oracle.tsql

After a few trial and errors I managed to write a script to undo all the changes.     Unfortunately if you missed one, which was easy to do as the script is a few thousand lines, you had to start over.   Finally I managed to find all updated objects and continued with the upgrade.

Another upgrade issue I encountered showed itself in the startup logs for soa_server1:

java.sql.SQLException: ORA-25226: dequeue failed, queue UPG_SOAINFRA.EDN_OAOO_QUEUE is not enabled for dequeue
The solution to this problem was to manually restart the queues:

SQL> show user
USER is "UPG_SOAINFRA"
SQL> exec dbms_aqadm.start_queue('IP_OUT_QUEUE',true,true);
PL/SQL procedure successfully completed.
SQL>  exec dbms_aqadm.start_queue('EDN_OAOO_QUEUE',true,true);
PL/SQL procedure successfully completed.
After the startup we waited anxiously to hear from the superusers to let us know the status of in-flight processes.    To much celebration (more so from them than us since they had alot of potential work to do) in-flight processes did not disappear.

54 comments:

Unknown said...

Hi,It is difficult for company formation in Qatar to terminate a registered agency; in addition compensation is payable upon the termination of the agency, including upon the expiry of a fixed term agency.Thanks...

Path Infotech said...

Thanks for sharing the information

For more info : Oracle 10g Certification

Unknown said...

Nice Blog, Thank you for sharing this nice information about nice topic on your blog, it is very informatics info thank for this blog
Big Height

Unknown said...

Thanks for Information Our Online-Training-Informatica proven expert in all Hyperion Modules like Hyperion Financial Management, Hyperion Financial Data Quality, Hyperion Financial Reporting, Hyperion Essbase, Hyperion Planning, Smart view and Data Relationship management.Hyperion Essbase Online Training

Sekhar Byna said...

Very nice Article
you can also visit
Oracle apps blogs

Nikunj Sanghvi said...

Thanks for the informative article. Please visit our Resources website with more than 300 SQL queries, case studies, whitepapers etc:

https://resource.neocortex.com/

Please have a look at it and see if it would benefit your blog readers. We can also cross-promote your blog and content with us if you’re interested.

Lakshmi said...

Wonderful article, very useful and well explanation. Your post is extremely
incredible. I will refer this to my candidates...
hadoop Training in Chennai

Anonymous said...

Good blog post which provided a interesting information about.keep updating...
SEO Services in India

DataLoader said...

Thanks for the article. It really helped me.

gracylayla said...

Good Post! Thank you so much for sharing this pretty post, it was so good to read and useful to improve my knowledge as updated one, keep blogging…
Oracle EBS Training

liza said...


You need to kill time, you need entertainment. Refer to our website. hope you get the most comfort.
Thanks you for sharing!

Deer Hunting Tips Camping Trips Guide DEER HUNTING TIPS travel touring tips

liza said...


Thanks Still Hunting Method
Hunting psych tips Survival Tips Travel Touring Tips
for sharing this Post, Keep Updating such topics.

liza said...


I enjoyed over read your blog post. Your blog have nice information, I got good ideas from this amazing blog. I am always searching like this type blog post. I hope I will see again.

travel trekking tips
see the link Tent Camping 101 Exploring Smithriver

write my paper for me said...

Many thanks. For me it was useful.

https://mypaperedit.com/resume-editing said...

I am usually to blogging and i really admire your content. The article has really peaks my interest. I’m going to bookmark your web site and hold checking for brand spanking new information.

reflective essay thesis said...

Thank you for these posts and links, I hope this will come in handy.

kirankumar said...

Great blog article informative I liked it

We are the best piping design course in Hyderabad, India. Sanjary academy Offers Piping Design Course and Best Piping Design Training Institute in Hyderabad. Piping Design Institute in India Piping Design Engineering.
best Piping Design Course
piping design course with placement
pipe stress analysis course in hyderabad
piping engineering course in hyderabad
piping stress analysis course
best Piping Design institute
best institute of Piping Design Course in India
Piping Design Course
Piping Design Course in India­
Piping Design Course in Hyderabad

minecraft lan connection timed out said...

You need to take part in a contest for one of the finest websites on the net. I will highly recommend this website!
discord mic not working
m7703-1003
nvidia control panel missing
0x8007000d
steam content file locked

Tripu Design said...

Thank you for sharing your wonderful information. for professional graphics,
work contacts me. I am a freelance designer in gurgaon.
Freelance Graphic Designing:
Freelance Catalogue Designing in delhi
Freelance Catalogue Designing in gurgaon
Freelance Brochure Designing
Freelance Label Designing
Freelance Banner Designer
Freelance Poster Designer
graphic design services in delhi
graphic design services in gurgaon
Freelance Catalogue Designing in delhi
Freelance Catalogue Designing in gurgaon
Freelance Brochure Designing
Freelance Label Designing
Freelance Banner Designer
Freelance Poster Designer
graphic design services in delhi
graphic design services in gurgaon
Freelance Catalogue Designing in delhi
Freelance Catalogue Designing in gurgaon
Freelance Brochure Designing
Freelance Label Designing
Freelance Banner Designer
Freelance Poster Designer
graphic design services in delhi
graphic design services in gurgaon
Freelance Catalogue Designing in delhi
Freelance Catalogue Designing in gurgaon
Freelance Brochure Designing
Freelance Label Designing
Freelance Banner Designer
Freelance Poster Designer
graphic design services in delhi
graphic design services in gurgaon

sara jenifer said...

I like this topic.This site has lots of advantage.I found many interesting things from this site. It helps me in many ways.Thanks for posting this again.
RDVV BCOM TimeTable 2020
DAVV BCOM TimeTable 2020

Raj Maan said...

i am browsing this website dailly , and get nice facts from here all the time . 123movies

sammer mark said...

Thanks for sharing the informative post. In the event that you've refreshed your PC framework from a past rendition to the most current windows 10, you may be encountering this issue significantly more. Along these lines, don't stress more, our printer specialists are actually experienced for settling this specialized glitch in a flash. Our 24/7online assistance is open nonstop to give you moment answers for any sort of specialized inconveniences.
Setup HP DeskJet 3755 Printer on Windows 10
Fix HP Fax Error Code 344

Ruhi Sukhla said...

Terrific article! This is the type of information that should be shared across the web.

Vikram Ujjain University BA 2nd Year Result

Kailey Robinson said...

Download Video Player | Windows Media Player | Best Media Player

Unknown said...

3 Sevens marketing company is your premier digital marketing agency in Mississippi, which caters to digital solutions – marketing and development, to help any business large or small. Our professional team of experts helps you to channel your brand image to connect with your audience.

Unknown said...

3 Sevens marketing company is your premier digital marketing agency in Mississippi, which caters to digital solutions – marketing and development, to help any business large or small. Our professional team of experts helps you to channel your brand image to connect with your audience.

Unknown said...

Browse our men's gold chain collection, a variety of styles are available, everything from Cuban link to a rope. Shop now!

Unknown said...

The Gratitude List is a destination where you can find a collection of Gratitude journals for you. Gift yourself or your loved one, a moment of joy to transform their habits and life. We have the best gratitude journal LA. You can also check out our blog page that consists of information talking about the latest trends, new product launches, experiences of customers, or any tips about journals.

QuickBooks Error said...

Nice & Informative Blog !
Our team at QuickBooks Customer Service Number offers satisfied service to every QuickBooks client in a state of upheaval and uncertainty.

Unknown said...

The Gratitude List has a strong commitment with the unstoppable force for good, to create a change within oneself first, then for the people, community, ultimately for the world and planet. With sustainability in mind, we make sure that our manufacturing, packaging, and shipping are sustainable with a light carbon footprint. Buy a gratitude journall that is creative, beautiful, and treat yourself with positive things in your life

Hitesh Teli said...

Amazing website.click here to make free android apps without coding.

Jyotish Kumar said...

This article was so good that I did not skip a single word in this article, I could not know when time had passed while reading this article. I read your article daily and I also share it if I like it. I have written an article on online app kaise banaye after learning from you. Sir/Mam, you always help us by writing articles and it is also our duty to share your articles as much as possible. Sir/Mam, now allow me to go.

Unknown said...

You can buy the best gratitude journal in Florida from the gratitude list. We have different designs of gratitude diaries for you as well as your friends and families. So if you are looking to buy the best gratitude journals in Florida  or anywhere else in the world visit our website now

Fred said...

https://seoservicesagencyinbangalore.blogspot.com/

Jyotish said...

Hello,

Wow, "What a Fantastic Information You have given here".

Keep it Up and Thank you very much for your kind help through this article. You have done your work well as a blogger.

Regards,
Jyotish - Highest Paying Jobs in India 2021

Jyotish Sahni said...

Hello,

"What a Fantastic Article is This"

Really, I found this article very helpful because there is a lot of beneficial and helpful information in this great article. I shared this article with my friends because it is very compulsory for reading as because of your truthful and honest content. Thank you for helping me find the information I needed.

Ok Bye, Have a Wonderful Day.

Regards,
Jyotish

How to Make a Better Comment on the Blog
What is the another name for Veda

Vaibhav Raut said...

Love the information! devops online training

Candyce Herr said...


That was an emotional moment for everyone. This was entertaining. This requires eminent passion on my part to do it with Keto Advanced Fat Burner in order to push ahead with that. I feel like a kid in a candy store. There is so much news on Keto Advanced Fat Burner available. Keto Advanced Fat Burner is something that confirms the character and taste of the individual


Keto Advanced Fat Burner.

JBL Tempo Traveller Rentals said...

Its really good blog and very information, we are tempo traveller rental company in Gurgaon providing tempo traveller in gurgaon for local and outstation, we have luxury tempo traveller in gurgaon you can rent 12 seater, 17 seater , 20 seater , 26 seater Tempo traveller gurgaon to outstation tours ~ JBL Tempo Traveller Rentals Delhi NCR ~ 9560075009



Our location :
Tempo traveller in Gurgaon



tempo traveller in noida
tempo traveller in Ghaziabad
tempo traveller in faridabad
tempo traveller in faridabad
tempo traveller in delhi
************************

wrecking ball said...



Hey friend, it is very well written article, thank you for the valuable and useful information you provide in this post. Keep up the good work! FYI, please check these depression, stress and anxiety related articles:
Essay On Journey Of Guru Tegh Bahadur Ji , A radical awakening book review

Kylo said...

Read your blog post recently as I do not know how to execute
onn roku tv setup steps. You have explained the setup steps clearly. I’m happy that I could complete onn roku tv setup quickly. Kindly post a blog explaining the guidelines to fix setup issues that popup
Keep up the good work
My rating for your blog post would be 100 stars

Paula’s Cleaning Service said...

Paula's Cleaning Service provides honest and reliable commercial cleaning services in Northville. Our professional cleaners handle cleaning and disinfection services for offices and businesses in a range of industries

ba time table said...

I wish to say that this 3rd year exam date post is amazing.

Kylo said...

Read your blog post recently as I do not know how to execute
Roku tv setup steps. You have explained the setup steps clearly. I’m happy that I could complete Roku tv setup quickly. Kindly post a blog explaining the guidelines to fix setup issues that popup
Keep up the good work
My rating for your blog post would be 100 stars

Kylo said...

If you are interested to learn how to activate Roku using
Roku.com/link , let me suggest the blog post titled, How to activate Roku. Read the post a few days back. I could find clear guidelines to activate Roku. Spend your free time reading the post to learn Roku.com/link activation guidelines

Kylo said...

Excellent post with the title, How to activate Roku using the portal Roku.com/link activation code I’m Impressed after reading and I have no other words to comment
I could learn Roku.com/link activation code activation procedure quickly after reading your post. Kindly post similar blogs explaining the guidelines to add and activate the entertaining channels on Roku
Let me mark the 100-star rating for your blog post
Keep up the good work
Awaiting more informative blogs from now on

zarkazijar said...

I really enjoy reading your well-written articles. It looks like you spend a lot of effort and time on your blog. I have bookmarked it and I am looking forward to reading new articles. Keep up the good work. nocen cut off mark for all courses

Admit Card for B.Com Part 1 Exam said...

Great article. Your blogs are unique and simple that is understood by anyone.

Alina jonas said...
This comment has been removed by the author.
raju said...

wordpress design services agency Need professional WordPress Web Design Services? We're experts in developing attractive mobile-friendly WordPress websites for businesses. Contact us today!

optiplex said...

Thanks for sharing this informative post.

B3 Leather Bomber Jackets

Pilot Jackets

Leather shirts

Sruthi Karan said...

Thank you so much for your excellent blog! I really enjoy to visit your very interesting post, Well done!
Virginia Family Law Attorneys
Local Family Lawyers

apsu ba 1st year result said...

First You got a great blog. I will be interested in more similar topics. I see you got really very useful topics, I will be always checking your blog thanks.

apsu ba 2nd year result, apsu ba 3rd year result.

Kate Herun said...

https://bid4papers.com/

Bid4Papers is an online marketplace that connects students with professional academic writers. The platform enables students to get help with their assignments from experienced writers and editors. Bid4Papers also provides a bidding system that allows students to pick the writer with the most suitable offer for their assignment.