Grid Control and Fusion Middleware–Required patches

We have been using Grid Control 11.1.0.1 for almost a year now to monitor our Fusion Middleware environments.   I’ve recently had an issue where I couldn’t get a job to restart a weblogic domain.   It errors out with:

Could not execute query: java.sql.SQLException: ORA-01403: no data found
ORA-06512: at line 88


Apparently you have to be on PSU 3+ and MLR3 applied before Oracle will help.   A year ago when we put it in, the only restriction I could find was that you had to be on GC 11g.    So, if you are using GC to monitor your fusion middleware environments you should look at this note and make sure your patched properly, before you need to open an SR.

Oracle Required Patches for Discovery and Monitoring Fusion Middleware Targets (Doc ID 1354421.1)


This note was published on Nov 3rd.  I don’t see a revision history so I am assuming it was only published recently.

Openworld 2011– Day 3 Summary

I can’t believe Openworld has come and gone.  The last two days were a blur!  With the appreciation event and traveling home I didn’t get a chance to post a summary.   And of course, once I made it back home I had a ton of family activities, work to catch up on, etc.   So I’m only getting around to going over my notes now.    Since the presentations are available online, I thought I would still talk about some of the sessions I’d recommend downloading.

On Wednesday I attended 3 presentations, I skipped one so I could like up like a fanboi for Larry’s keynote.   On Sunday night I managed to pick absolutely the worst seat in the hall.    (The camera people were directly in front of me, which blocked my view of the stage and the pillars on each side blocked the screen showing the speakers!)  So I wanted to make sure I sat in a good section.

The first presentation I attended was Oracle E-Business Suite Technology Vision, Roadmap, and Customer Successes by Lisa Parekh.  Unfortunately tho, I hadn’t intended on going there.   I was in a rush, they scanned my badge, it showed green and I went on in.     It was a good session tho and provided a lot of information on their priorities, such as ease of use, security and performance.   There is a lot of information in this presentation and its available for download here: https://oracleus.wingateweb.com/published/oracleus2011/sessions/17248/S17248_1609600.pdf

My next session was Best Practices for Maintaining and Troubleshooting Oracle Fusion Applications.  I quickly discovered tho that Fusion Applications != Fusion Middleware.   Luckily tho, Lisa (from the last presentation) mentioned an EBS Patching session.  So I left and went over to that one.

Oracle E-Business Suite Directions: Slashing Downtimes with Online Patching, presented by Kevin Hudson was very informative.  This was a very detailed session getting down to how patching works underneath.  In a nutshell patching will take place while the environment remains online and a quick bounce after for the cutover.

Behind the scenes there will now be two APPL_TOP’s.   One will be the called the Run Edition and the other the Patch edition.   As the name says, the patch edition will be used for patching and the run edition is currently used by the application.  After applying a patch you cutover to the patch edition, which now becomes the run edition.

One thing I liked was that after applying a patch you don’t have to do the cutover right away.  You can wait until scheduled downtime.   So, you could apply the patch during business hours (if it doesn’t affect performance much) and do a quick bounce off hours. 

Some points raised by questions from attendees :
- In the future it may be possible to roll back patches.
- At any point until you cutover you can abort the patch.
- Shared APPL_TOP still supported but now you’ll be sharing a dual filesystem.
- You can still merge patches.

If you’re an EBS DBA then you should read this presentation as it really digs down into how online patching works behind the scenes:
https://oracleus.wingateweb.com/published/oracleus2011/sessions/17255/17255_Cho139369.pdf

Next was Larry’s keynote.  It was probably the most entertaining one he had in quite a few years.  The last time I remember it being like this was many years ago bashing Microsoft.   This time Salesforce.com was the unlucky recipient.  My favorite quote:  “Sticker than a roach motel!” I left towards the end of the keynote because it was starting to overlap the last session of the day.




The last session I attended on Thursday was Oracle Enterprise Manager Deployment Best Practices.  Best practices such as having multiple OMSs so that you don’t have a single point of failure either due to an outage or change.  A common mistake is to distribute OMSs in geographically separate locations under the assumption that it will improve performance.  Its recommended to keep your OMSs and repository close since performance is better and agents can tolerate the latency.

If you are in the planning phase of a new Grid Control implementation plan on an incremental rollout, “big bang” deployments are harder to deliver.  Monitoring the EM environment is important such as the status of DBMS_SCHEDULER, housekeeping tasks, incoming loader backlog notifications, etc.

One feature talked about, which has been around for awhile,  was out of band notifications.  When setup, this feature allows an agent to monitor the OMS and send an email notification if it goes down.   The metalink note on how to set it up can be found here: 
Setting Up Out of Bound (OOB) Notification for 'OMS and Repository' Target in Grid Console [ID 429257.1]


The session also covered some new features of 12c.

Before heading out to the appreciation event concert I stopped by the hotel for a jacket and a quick video chat with the family.   I thought a jacket on top of the sweater I had on all day was enough. You think I would have known since this wasn’t my first trip to Treasure Island!   Anyways, thankfully they had patio heaters setup everywhere so after watching Sting perform I met up with some friends for some food and watch them make fools of themselves trying to win some stuff toys.

Openworld 2011 - Day 2

Wow, the conference is flying past fast.  Its already the morning of day 3.  The appreciation event is tonight, hopefully it won't rain!   Tomorrow is the last day, yikes!

I didn't sleep well Sunday night (remember my neighbours?), so I decided to sleep in Tuesday morning if I could.   So unfortunately I missed the keynote in the morning.  I will definitely catch the replay later.

My first session Extreme Database Administration: New Features for Expert DBAs talked about some of the new features coming up on Oracle Enterprise Manager 12c.   It covered some of the same material as a session I attended Monday but if you don't diversify your schedule your bound to get some overlap.

The session talked about Real-Time ADM which allows for real-time analysis of hung or slow database systems.   It connects to the environment via a lightweight connection that doesn't acquire any additional latches or locks and by passes the SQL layer.  It detects resource constraints/limits, hangs, memory issues, deadlocks, etc...

Comparative performance analysis is another feature which will allow you to run a full ADDM analysis over two AWR snapshots.    Used to help answer the question, performance was good yesterday but terrible today.

Ash Analytics is an advanced ASH report which looks similar to the top activity screen but you are not limited to selecting certain time frames but can expand the analysis by dragging the graph to any point in time.

Next up the presenter talked about change management features including difficulties in obtaining accurate test data.   Application Data Model (ADM) is a new feature to help address those issues.  It scans application schemas to model the relationships between tables and columns.  Enables data masking and subsetting operations to be application aware.  Data subsetting can be done by percentages of tables or by columns (regions, year, etc) at runtime.   One example showed subsetting 100GB of data to 20GB took 12 minutes using the export method. Secure Database Testing is another new feature which integrates real time application testing with data masking.

Database Upgrade Automation looks like an interesting feature as well.  Grid Control will detect that a new version is available and suggest the best upgrade path.  It will analyze the database for potential upgrade issues such as space, version, invalid objects, etc.   You can mass deploy binaries to all targets and create out of place copies, then switch the instances to new installations.    You can switch back easily if issues are encountered.

Exadata Management allows for comprehensive tools to manage all of your Exa servers.   I didn't take many notes here since I won't be fortunate enough to use this hardware anytime soon.   Lastly Database as a Cloud Service was discussed.     This seems like a really good feature for large DBA shops who provision and support alot of database environments.    Easy provisioning and deployment of prepackaged, preconfigured databases with metering and charge back capabilities.

I missed the next session due to a call from my manager, so explored the exhibition halls.   Maybe i'm getting older but this is the first year I have not sat in on any presentations.    I find you sit there for 5-15 minutes to listen about a product i'll never use to get a trinket that my kids will never play with. ;)   I end up walking around with a bunch of extra stuff weighing me down all day and then risk having my luggage overweight on the return flight.  Its happened many times before!   Don't worry tho, my kids will still get a gift, one they will hopefully actually use tho. ;)  

Afterwards I attended Steven Chan's Oracle E-Business Suite Technology Certification Primer and Roadmap presentation.   This one is a must for me each year.  As the abstract says, "Is your Oracle E-Business Suite technology stack up-to-date? Are you taking advantage of all the latest options and capabilities? This Oracle Development session summarizes the latest certifications and roadmap for the Oracle E-Business Suite technology stack...." This is a centralized place to make sure you on supported versions, the desupport timelines, certifications with other products and features.    If you didn't attend be sure to keep an eye on Steven Chan's blog as i'm pretty sure he will post it there at some point.

I did a last minute switch up and went to Oracle Recovery Manager (Oracle RMAN) and Oracle Data Guard: Seven Cool Tips from Oracle . If we take advantage of any features coming up it may be data guard. So I thought this would be a good presentation to take in.  The "Tips" were ok, 3 on RMAN, 3 on Dataguard and one from the Customer Testimonial.    They talked about using Guaranteed Restore Points with no IO impact and the tip was that you didn't need to explicitly enable flashback database.

"Super" quick restore by pointing the system to an image copy of a datafile in the FRA in the event of a failure.  I'm not sure how practical that one is.. I don't know anyone that uses image backups.  Most people take advantage of RMAN because of the way it skips un-used blocks, compression, etc.

One of the data guard tips for 11.2 is flushing redo.  Automatically shipping redo logs over to the standby. Previously this had to be done manually because there was a potential for dataloss even in maximum availability unless it was done manually.

The customer testimonial was pretty good.   Enterprise is a 16 billion dollar car rental company and I was surprised to learn that they don't use RAC.   Lisa Reinheimer  described their environment and how they are using dataguard.  I always like listening to how other companies architect their environments.  It gives me ideas and lets me know if i'm on the right track (if i'm already using the features).   If your interested in seeing their setup you should download the presentation.

Finally I attended Best Practices for Oracle E-Business Suite Performance Tuning . I can't even begin to tell you how much material was covered in this presentation.  This presentation was conducted by members of the EBS performance team.    You really need to download this presentation if you support an EBS environment.   There was too much content to even dream of capturing it all to post a summary.   Its basically a centralized presentation listing out tons of tuning tips, metalink notes discussing best practices and performance patches.   Enhancements to new FND_TRACE in 12.1.2 , Note: 1362660.1 for example.   Recommended performance patches for EBS can be found in note 2440401.   Great presentation.   I'll be referring back to this one quite a bit.

That was my day.   Afterwards I went to supper with some old colleagues to an Italian Restaurant in North Beach.    A nice walk down to the pier to burn off supper and a trolly ride back to the hotel.  

Openworld 2011 - Day 1

The day started off with a keynote featuring Joe Tucci from EMC, Mark Hurd and Thomas Kurian.  I won't say much about the keynote, i'm sure more than enough analysis has been already done.    All I will say tho, similar to last evening, the focus was too heavy on hardware and the "Exa's".     Oracle has sold a thousand of these servers and plan to sell 3000 over the next year.   While impressive, it still means that the majority of customers currently don't have Exa servers in house.   So, in my humble opinion, the keynotes should try to appeal to a wider audience.

First up I had a hand's on lab, Using Oracle Application Management Suite for Oracle E-Business Suite.  We just purchased this pack and plan on implementing it over the next few months.   The lab had a few issues, some my own fault, some with the environment.   It was only an hour long and given the amount of material in the guide i'm not sure if the goal was for us to complete it all.   Really to cover the amount of material they had listed you would need at least half a day.  One good thing tho is that they offered to email us the documentation (or copy it if you had a USB key) so we could continue/read up on it on our own time.

Next I attended Who Ate My DB Time? Advanced Techniques for Database Performance Analysis.  It was a good presentation.  It covered some new features of OEM R12 and how they work behind the scenes.  It finished with a customer testimonial from Boeing.

Automating Patching with Oracle Application Management Suite for Oracle E-Business Suite, covered features from both the application management pack and change management pack.   Monitoring, lifecycle management, configuration management, etc.    Being able to compare two instances from profile options to context values.  AMP for 12c is still under development, so if you use this pack you won't be upgrading to the new OEM version right away.   The customer testimonial by Prithwi Barman at the end was very interesting.   I know that before we can even use AMP we need to apply the latest ATG patches plus various others.   But after hearing him talk it looks like its going to take a fair bit more effort than I thought.   I didn't add it up, but it took them quite a few man months to implement it from DEV to PROD and test features such custom patches.  

Prithwi also talked about the process they follow migrating patches and displayed a chart summarizing the process flow.   Alot of time goes into building these flows so its always nice to see how other companies do this.  It helps give you a kickstart designing your own and/or justifing some of the choices you make.

For example, in the attached slide you can see that the developers package the custom patches, pass them along through an approval layer before reaching the DBA's.   This makes perfect sense to me but I know will make a difficult sell.   Being able to say X company is doing it this way for Y and Z reasons is going to help alot.

The next session I attended was Maximize Efficiency of Oracle E-Business Suite 11i and 12 Through Diagnostics.   Not something that really falls under my responsibility other than to make sure its setup.   So my main goal was just to get some more information on how its used by developers, some issues they may encounter etc.    Quite frequently when they open an SR oracle asks them to run various diagnostics reports and upload them.    

It was very beneficial, the presenter Karen Brownfield, talked about some security issues such as the CRM HTML administration role having super user access to diagnostics.   How tests in 11i timeout after 5 minutes, RDA's timeout in R12.   Features such as save sets and groups.  If you have a set of tests you plan to execute at the same time frequently you can put them in a save set.   Groups are similar but available to all users instead of just the one who created it.

Last but not least was Oracle E-Business Suite Directions: Deployment and System Administration.    Features in the next version of R12 (12.2) include a dual file system which allows online patching to occur.   Basically two appltops are installed, the application runs out of one of them (runtime appltop) while patches are applied to the other (patch appltop).   Once patching is complete a brief downtime occurs during the switch over.  Downtime is predictable, minutes not hours/days.

The benefits of switching the application tier over to Weblogic were also explained.   WLS has strong support for high availability and failover.   Consolidated administration, configuration and deployment.   A very indepth troubleshooting framework.   Comprehensive, low-overhead application and diagnostics.

A brief stop by the OTN party and then supper with an old co-worker.  We haven't seen each other in years even tho we live in the same city.   We were actually on the same flight and ran into each other after deplaning in San Francisco.

Openworld 2011 - Day 0

It didn't seem like long ago that I registered for #oow11 but the first day (or is it day 0?) has come and gone. I guess because of the timezone difference I was up about an hour before my alarm was set to go off, which is pretty strange for me. After a quick shower I was out the door by 7:30am and on my way to register. I expected long lines but I had my kit by 8am.

Day 0, like previous years, was a day for Oracle user groups to hold meetings. My day started with Thomas Canty's presentation titled Sysadmin 101. Thomas talked about RAID, the pros/cons of each level, NFS performance, direct NFS and various command line tips. This is a great presentation for someone new to supporting Oracle on Linux that would like to learn a few linux tips.

I take a fair bit of time leading up to the conference to go through Oracle's schedule builder and arrange my day. However, its still easy to miss good sessions. So I always keep my eye out for long lines outside a session and the TV screens showing the titles. As I was making my way to a session I happened to notice Mark Rittman's session on OBIEE 11g Architecture and Internals.

We are currently using OBIEE 10g and there has been talk of upgrading to 11g. So I went in and grabbed a seat. We currently use Oracle Fusion Middleware so alot of the concepts such as weblogic domains, system/non-system components, etc. So the session was a perfect fit to get me up to speed on OBIEE11g. I'm really glad I spotted that one as I walked by.

Next was the E-Business (EBS) Applications Technology SIG . I was a little late and walked in on an overivew of WebLogic. Afterwards Steven Chan gave an abbreviated overview of his Certification Primer and Roadmap session that he's presenting on Tuesday. I actually just realized it was abbreviated and not the full presentation while looking at his website (http://blogs.oracle.com/stevenChan/). So I just modified my schedule to include the full session which is at 1:15 on Tuesday.

I have used RMAN since 9i and its one of those features which I feel you always have to keep up on. Whats more important than being able to recover your database in the event of a failure? So I attended Kamran Agayev's session on RMAN 11g new features. One thing I liked about his presentation was the demo's and real examples/benchmarks. Alot of presentations either have slides of commands and their output or they try and do the demo live which almost always runs into some sort of issue. Kamran recorded his demo's and played them back. I found that for me personally it helped understand the topics better.

Kamran talked about new features such as compression types (ZLIB), Virtual Private Catalogs, changes to commands such as VALIDATE, etc. If you use RMAN and need to get up to speed on 11g new features then i'd check out this presentation.

Finally I attended the Sysadmin SIG. The session was dominated by a discussion on Oracle's Critical patch updates. Its nice to know that i'm not alone in finding that the documentation around this is very confusing. I'm not sure when it changed, maybe it was with release 12 but its definitely not as straight forward. I think part of the reason is that they are trying to reuse documentation. So you'll click on one note and it will reference another and you are left to fit all the pieces together. The Fusion Middleware charts list various products such as infrastructure home, middle tier home, OC4J, forms, jdeveloper, etc, The first time I tried to apply a CPU patch for R12 I had to open an SR as I had no idea which patches to apply. The state of the documentation has left me with an uneasy feeling that i'm not capturing all the patches I need to be applying.

Maybe my memory is playing tricks on me, but i'm pretty sure when I supported an 11i environment all the patches were self contained in one note and it was very easy to find the patches you needed to apply.

On top of the confusing documentation other issues such as determining if your environment is affected by the vulnerabilities, risk analysis and testing requirements also came up.

One of the benefits of coming to Openworld, in my opinion, is being able to provide feed back to the right people at Oracle and having an open discussion with them. Hopefully they will go back with our comments and improve the CPU process.

Lastly the day concluded with the opening keynote. I have to say that over the past few years I haven't looked forward to them as much as I used to. Ever since Oracle acquired Sun, Larry's keynote seems to be dominated by hardware. Not that its a bad thing because I think its a good direction for Oracle. But honestly, my organization is too small to take advantage or be able to afford any of the Exa's (Exadata, Exalogic, Exalytics). Oracle mentioned they sold 1000 in the past year and in line to sell 3000 over the next. Good numbers but that also means the vast majority of Oracle's customers aren't using it or plan to in the near future. So, to me, it doesn't make sense for it to dominate the entire keynote.

After the keynote I stopped by for a quick bite to eat and went to a few stores to take a few pictures of purses for my wife so she could do some virtual shopping.

Well, it seems that the headboard of my hotel neighbours room has stopped hitting off our shared wall, so its time to catch some sleep. I have to get up again in 3.5 hours! Please ignore any typo's or grammatical errors as my eye's aren't quite focusing. ;)

Openworld 2011

UPDATE: Opps, I just updated the title, no idea why I said 2012!

-=-=-=-

Just completed registration for my Blogger pass. I am really looking forward to the conference this year as the sessions look really good.

If your in San Francisco for the conference and would like to meet up drop me an email. My address is on the right side of this page.

11.1.1.1 -> 11.1.1.3 Fusion Middleware Upgrade

I've been meaning to post this awhile.. How long, well I believe 11.1.1.5 is out now.   I took a quick look at the upgrade documentation tho and the overall process doesn't seem to have changed much.  (I did see some extra post-upgrade steps depending on which version you start from..)

The biggest hurdle seems to be upgrading from 11.1.1.1 to 11.1.1.2...   I spent a fair bit of time trying to understand the documentation and perform the upgrade.   The process as described in the documentation didn't seem to make any sense to me.   After talking to Oracle support they found it difficult to follow as well and said they would file a request to get it reviewed.    We decided at that time to do a fresh install of 11.1.1.2 and upgrade it to 11.1.1.3.

I haven't checked back on the documentation to see if the sections were clarified.  If you followed or have seen my 11.1.1.1 installation guide you have an idea of my setup.    If you haven't, a quick review:

- Two Tier Environment
- VM18 has the IDM stack and Database installed.
- VM17 has Portal, SOA and SSO installed.

As I said above, instead of upgrading from 11.1.1.1 to 11.1.1.2 and patching to 11.1.1.3 we did a fresh install of 11.1.1.2.   The installation process for 11.1.1.2 is almost identical to 11.1.1.1 so I used the install guide I had already created as a base for the install.    From there I followed the steps below to patch up to 11.1.1.3

Note: The steps below only cover upgrading the products to 11.1.1.3.   Since we performed a fresh install it doesn't cover topics such as migrating data (example, OID accounts), domains hosting 3rd party software, portal data (transport sets), configuration (load balancers, reverse proxies, ..), etc.

Patch Weblogic from 10.2.2 to 10.2.3

1. Download the Upgrade Installer from Oracle Support.
   1. see "Downloading an Upgrade Installer From My Oracle Support"

2. Launch the install:
   1. [oracle@vm18 STAGE]$ $JAVA_HOME/bin/java -jar wls1033_upgrade_generic.jar

Upgrading IDM

Portal Patchset Install

• Prompted to run oracleRoot.sh

SOA Patchset Install

Updating Schemas with Patch Set Assistant

• Startup the database, ensure aq_tm_processes parameter is > 0
• In each home for Portal, SOA and IDM there is a Patch Set Assistant utility which was installed as part of the 11.1.1.3 patchset.

IDM psa

[oracle@vm18 bin]$ ./psa -dbConnectString //vm18:1521/FMW.world -dbaUserName sys -schemaUserName ODS
Oracle Fusion Middleware Patch Set Assistant 11.1.1.3.0
Enter the database administrator password for "sys":
Enter the schema password for schema user "ODS":
Log file is located at: /u01/app/oracle/product/mid11g3/Oracle_IDM1/upgrade/logs/psa2010-08-13-15-00-47PM.log
Updating the Oracle Fusion Middleware Metadata Schema to release 11.1.1.3.0.
Updating schema for component Oracle Internet Directory.
The command completed successfully


Start IDM
[oracle@vm18 bin]$ ./opmnctl startall
opmnctl startall: starting opmn and all managed processes...
[oracle@vm18 bin]$ pwd
/u01/app/oracle/product/mid11g3/IDMasinst_1/bin

Portal PSA

[oracle@vm17 bin]$ ./psa -dbConnectString //vm18:1521/FMW.world -dbaUserName sys -schemaUserName UPG_PORTAL
Oracle Fusion Middleware Patch Set Assistant 11.1.1.3.0
Enter the database administrator password for "sys":
Enter the schema password for schema user "UPG_PORTAL":
Log file is located at: /u01/app/oracle/product/mid11g3/PORTALas_1/upgrade/logs/psa2010-08-13-15-06-02PM.log
Updating the Oracle Fusion Middleware Metadata Schema to release 11.1.1.3.0.
Updating schema for component Oracle Portal.
The command completed successfully

SOA PSA

[oracle@vm17 bin]$ ./psa -dbConnectString //vm18:1521/FMW.world -dbaUserName sys -schemaUserName UPG_SOAINFRA
Oracle Fusion Middleware Patch Set Assistant 11.1.1.3.0
Enter the database administrator password for "sys":
Enter the schema password for schema user "UPG_SOAINFRA":
Log file is located at: /u01/app/oracle/product/mid11g3/SOAHome_1/upgrade/logs/psa2010-08-13-15-29-49PM.log
Updating the Oracle Fusion Middleware Metadata Schema to release 11.1.1.3.0.
Updating schema for component Oracle SOA.
---Redirecting sql log to /u01/app/oracle/product/mid11g3/SOAHome_1/upgrade/logs/psa2010-08-13-15-29-49PM.out
Executed SOAINFRA upgrade script.
The command completed successfully

Verify the PSA

SQL> select owner, version, status from schema_version_registry;
OWNER VERSION STATUS
------------------------------ ------------------------------ -----------
ODS 11.1.1.3.0 VALID
UPG_PORTAL 11.1.1.3.0 VALID
UPG_SOAINFRA 11.1.1.3.0 VALID

Check for Invalid objects and recompile:

SQL> select count(*) from dba_objects where status = 'INVALID';
COUNT(*)
----------
12
SQL> @?/rdbms/admin/utlrp

Post-patching tasks for the System Components:

(You may have to perform the steps below for the other products as well (IDM, SOA), not just for IDM.   I made note of it a while back but since we did the upgrade almost a year ago, I have misplaced it.)

/u01/app/oracle/product/mid11g3/PORTALasinst_1/bin
[oracle@vm17 bin]$ ./opmnctl unregisterinstance
Command requires login to weblogic admin server (vm17.rcpsc.edu):
Username: weblogic
Password:
Unregistering instance
Command succeeded.

[oracle@vm17 bin]$ ./opmnctl redeploy
Command requires login to weblogic admin server (vm17.rcpsc.edu):
Username: weblogic
Password:
Redeploying NonJ2EEManagement Application...weblogic.Deployer invoked with options: -adminurl vm17.rcpsc.edu:7001 -username weblogic -name NonJ2EEManagement -source /u01/app/oracle/product/mid11g3/PORTALas_1/opmn/applications/NonJ2EEManagement.ear -redeploy -upload -noexit
<Aug 13, 2010 4:20:54 PM EDT> <Info> <J2EE Deployment SPI> <BEA-260121> <Initiating redeploy operation for application, NonJ2EEManagement [archive: /u01/app/oracle/product/mid11g3/PORTALas_1/opmn/applications/NonJ2EEManagement.ear], to configured targets.>
Task 0 initiated: [Deployer:149026]deploy application NonJ2EEManagement [Version=11.1.1] on AdminServer.
Task 0 completed: [Deployer:149026]deploy application NonJ2EEManagement [Version=11.1.1] on AdminServer.
Target state: redeploy completed on Server AdminServer
Done
Command succeeded.

[oracle@vm17 bin]$ ./opmnctl registerinstance;
Command requires login to weblogic admin server (vm17.rcpsc.edu):
Username: weblogic
Password:
Registering instance
Command succeeded.

Update Version Number String shown in OEM

/u01/app/oracle/product/mid11g3/IDMasinst_1/bin
[oracle@vm18 bin]$ ./opmnctl updatecomponentregistration -componentName oid1 -Sport 3131 -Port 3060
Command requires login to weblogic admin server (192.168.80.148):
Username: weblogic
Password:

Linux: Comparing packages on two servers and installing the difference.

I know this isn’t Oracle related but if you have a position in a small company like myself, chances are you are performing more than DBA duties.   We just recently hired a linux admin but he’s on vacation this week and there’s a new server to stage for a Portal environment.

In the past I have just run system-install-packages on an existing server and the new one and compare the packages.   Time consuming, error prone, etc.  Also, any patches not installed from the base system packages won’t be captured.

Given those reasons I decided to use the command line.   If you have done something similar in the past and have a better way of doing it feel free to leave a comment.

Step 1:  Get a list of RPM packages from both servers:

rpm -qa --qf "%{NAME}\n” | sort > rpms_newserver.txt

Do the same on the second server.    Since the packages may not have been installed in the same order, I pipe it to sort, otherwise I believe it would cause problems with the next step.

The rpms_newserver.txt document will have a listing of files, ex:

a2ps
acl
acpid
alacarte
.
.
.
zsh

Step 2: Put both package lists in the same directory on the new server and run the linux command diff to compare the files.

diff rpms_oldserver.txt rpms_newserver.txt | grep "<" > package_list.txt

The diff command compares two files and displays the differences.   To get a list of the lines missing from the rpms_newserver.txt file we grep for the less than arrow “<” and pipe it to a new file. 

Step 3: The package_list.txt document is in the same format as the files in step 1.   Next I used vi to insert the yum command into the file so it can be executed like a script.

Open the package_list.txt file with vi and type:

:1,$s/< /yum install -y /g

Save the document and now it looks like:

yum install -y a2ps
yum install -y alacarte
yum install -y alchemist


Step 4:

Execute the package_list.txt document:

sh package_list.txt

That’s it, once it finishes all the packages should be installed.

WebLogic - Smart Update patch download capability decomissioned?!

Well, this is rather unfortunate.  I am in the process of rolling out weblogic patches and found out the hard way that downloading and applying the patches via the Smart Update utility (bsu.sh) has been decomissioned!   Just a few weeks ago I posted how seamless and easy the process was to patch WebLogic.   


IMHO, this is a step backwards.  However, end of the day its not tragic.  You just have to download the patches manually via Metalink like other Oracle products and upload them to your cache directory.   Once they are in the cache directory just launch Smart Update and they will show up in the download patches section towards the bottom of the screen.  


By default the cache directory is $WL_HOME/utils/bsu/cache_dir

WebLogic Net Installers and Downloading of Patches Via Smart Update To Be Decommissioned [ID 1294294.1]

To better coordinate the distribution of patches to Oracle customers, we are planning to consolidate on My Oracle Support as the single source for patch delivery.  After the Smart Update changes are complete, you will be able to access patches and patch sets whenever you need them from the My Oracle Support Patches & Updates site.

To this end, the WebLogic Net Installers and Smart Update patch and patch set download capability will be decommissioned on June 10, 2011.

Free SQL E-Books

Noticed an article on ReadWriteWeb a few weeks ago listing free SQL E-Books .   Not all Oracle related but a couple seem interesting.

http://www.readwriteweb.com/hack/2011/06/free-e-books-on-sql.php

Grid Control – Refreshing Weblogic Domains

WebLogic domains, especially SOA, tend to be very dynamic. Over time new applications may be deployed or others upgraded to the latest release, SOA partitions created/deleted, BPEL processes new/updated/deleted, etc. By default, Grid control is not aware of these changes. This can be a problem if you use grid control to centrally manage/monitor your environment. There are two options available to you:

1. Manually refresh the weblogic domain.
2. Enable Automatic Refresh

Manually refreshing a Weblogic Domain


One of the places this can be done is from the WebLogic Domain drop down menu, click on Refresh WebLogic Domain.




The following screen will appear, click on Continue.



After you click on continue target discovery will take place and a confirmation window will show how many targets were found:



After you click on close you’ll be brought to the assign agents screen. I’m not aware of any reason why you would want to reassign to a different agent, so I just continued by clicking on Add Targets.



Once you have made any changes, if necessary, above click on add targets to save the changes.



Automatically refreshing a Weblogic Domain


If your environment changes quite frequently you may want to enable an automatic refresh. To do this login to Grid Control, click on the Targets Table, Middleware section and select the domain you wish to make the change for.



Notice above that my domain was last refreshed Apr. 1^st. Click on the timestamp next to WebLogic Domain Refreshed and the following window will appear:



After you click ok, a confirmation message will be displayed saying that it was successfully scheduled:



If you click on the timestamp above next to Weblogic Domain Refreshed again:



You can click on the job name and it will bring you to the Job Activity section of Grid Control. From here you can change the schedule of the refresh, etc.





Thats it, pretty simple.  I prefer to use the manual method because it provides a summary of the new or modified targets,  which you don't get by scheduling it automatically.    I haven't checked the Grid Control job logs tho to see if that information is included there.

Grid Control 11g–Blocked Agents

Today I noticed that one of my Grid Control agents was down. From the Grid Control homepage for the target:



I thought it had just crashed so I checked the status from the server:

[oracle@myserver bin]$ ./emctl status agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Agent Version : 11.1.0.1.0
OMS Version : 11.1.0.1.0
Protocol Version : 11.1.0.0.0
Agent Home : /u01/app/oracle/product/grid11g/agent11g
Agent binaries : /u01/app/oracle/product/grid11g/agent11g
Agent Process ID : 29687
Parent Process ID : 29661
Agent URL : https://myserver:3872/emd/main/
Repository URL : https://myserver:4900/em/upload
Started at : 2011-06-23 02:06:56
Started by user : oracle
Last Reload : 2011-06-23 02:06:56
Last successful upload : (none)
Last attempted upload : (none)
Total Megabytes of XML files uploaded so far : 0.00
Number of XML files pending upload : 432
Size of XML files pending upload(MB) : 52.54
Available disk space on upload filesystem : 2.74%
Collection Status : Disabled by Upload Manager
Last attempted heartbeat to OMS : 2011-06-24 10:37:34
Last successful heartbeat to OMS : unknown
---------------------------------------------------------------
Agent is Running and Ready


Notice the “Last successful heartbeat to OMS” is unknown. If I try and force an upload it errors:

[oracle@myserver bin]$ ./emctl upload
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
EMD upload error: uploadXMLFiles skipped :: OMS version not checked yet. If this issue persists check trace files for ping to OMS related errors.


Checking the agent logs ($GRID_HOME/agent11g/sysman/log/emagent.trc):

2011-06-24 10:40:03,074 Thread-1091160384 ERROR pingManager: Did not receive valid response to ping "ERROR-Agent is blocked. Blocked reason is: Agent is out-of-sync with repository. This most likely means that the agent was reinstalled or recovered. Please contact an EM administrator to unblock the agent by performing an agent resync from the console. Please contact EM adminstrator to unblock the agent"


So the agent is out of sync with with the repository and is now blocked. I remember back in the 9i days to clean up an agent was a bit of a pain but in 11g its pretty simple. Log back into Grid Control, click on the Setup link located in the top right hand corner. Next click on the Agents link in the blue bar, followed by the Misconfiguration Alerts and Blocked Agents tab.



If you click on the agent itself (instead of selecting it and clicking unblock) you will be brought to the agents page shown below:



To resynch it click on the Agent Resynchronization button.



Make sure the Unblock agent on successful completion… is selected and hit continue. Once completed you may notice that the agent is still showing as unreachable. If you check via the server again you’ll notice that its working fine.  Notice the Last successful heartbeat and last successful upload rows.

[oracle@myserver bin]$ ./emctl status agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Agent Version : 11.1.0.1.0
OMS Version : 11.1.0.1.0
Protocol Version : 11.1.0.0.0
.
.
Last successful upload : 2011-06-24 10:44:36
Total Megabytes of XML files uploaded so far : 19.37
Number of XML files pending upload : 0
Size of XML files pending upload(MB) : 0.00
Available disk space on upload filesystem : 2.76%
Last successful heartbeat to OMS : 2011-06-24 10:44:04
---------------------------------------------------------------
Agent is Running and Ready


After a short period of time it will show as up in the Grid Control Console as well.