Openworld 2011 - Day 2

Wow, the conference is flying past fast.  Its already the morning of day 3.  The appreciation event is tonight, hopefully it won't rain!   Tomorrow is the last day, yikes!

I didn't sleep well Sunday night (remember my neighbours?), so I decided to sleep in Tuesday morning if I could.   So unfortunately I missed the keynote in the morning.  I will definitely catch the replay later.

My first session Extreme Database Administration: New Features for Expert DBAs talked about some of the new features coming up on Oracle Enterprise Manager 12c.   It covered some of the same material as a session I attended Monday but if you don't diversify your schedule your bound to get some overlap.

The session talked about Real-Time ADM which allows for real-time analysis of hung or slow database systems.   It connects to the environment via a lightweight connection that doesn't acquire any additional latches or locks and by passes the SQL layer.  It detects resource constraints/limits, hangs, memory issues, deadlocks, etc...

Comparative performance analysis is another feature which will allow you to run a full ADDM analysis over two AWR snapshots.    Used to help answer the question, performance was good yesterday but terrible today.

Ash Analytics is an advanced ASH report which looks similar to the top activity screen but you are not limited to selecting certain time frames but can expand the analysis by dragging the graph to any point in time.

Next up the presenter talked about change management features including difficulties in obtaining accurate test data.   Application Data Model (ADM) is a new feature to help address those issues.  It scans application schemas to model the relationships between tables and columns.  Enables data masking and subsetting operations to be application aware.  Data subsetting can be done by percentages of tables or by columns (regions, year, etc) at runtime.   One example showed subsetting 100GB of data to 20GB took 12 minutes using the export method. Secure Database Testing is another new feature which integrates real time application testing with data masking.

Database Upgrade Automation looks like an interesting feature as well.  Grid Control will detect that a new version is available and suggest the best upgrade path.  It will analyze the database for potential upgrade issues such as space, version, invalid objects, etc.   You can mass deploy binaries to all targets and create out of place copies, then switch the instances to new installations.    You can switch back easily if issues are encountered.

Exadata Management allows for comprehensive tools to manage all of your Exa servers.   I didn't take many notes here since I won't be fortunate enough to use this hardware anytime soon.   Lastly Database as a Cloud Service was discussed.     This seems like a really good feature for large DBA shops who provision and support alot of database environments.    Easy provisioning and deployment of prepackaged, preconfigured databases with metering and charge back capabilities.

I missed the next session due to a call from my manager, so explored the exhibition halls.   Maybe i'm getting older but this is the first year I have not sat in on any presentations.    I find you sit there for 5-15 minutes to listen about a product i'll never use to get a trinket that my kids will never play with. ;)   I end up walking around with a bunch of extra stuff weighing me down all day and then risk having my luggage overweight on the return flight.  Its happened many times before!   Don't worry tho, my kids will still get a gift, one they will hopefully actually use tho. ;)  

Afterwards I attended Steven Chan's Oracle E-Business Suite Technology Certification Primer and Roadmap presentation.   This one is a must for me each year.  As the abstract says, "Is your Oracle E-Business Suite technology stack up-to-date? Are you taking advantage of all the latest options and capabilities? This Oracle Development session summarizes the latest certifications and roadmap for the Oracle E-Business Suite technology stack...." This is a centralized place to make sure you on supported versions, the desupport timelines, certifications with other products and features.    If you didn't attend be sure to keep an eye on Steven Chan's blog as i'm pretty sure he will post it there at some point.

I did a last minute switch up and went to Oracle Recovery Manager (Oracle RMAN) and Oracle Data Guard: Seven Cool Tips from Oracle . If we take advantage of any features coming up it may be data guard. So I thought this would be a good presentation to take in.  The "Tips" were ok, 3 on RMAN, 3 on Dataguard and one from the Customer Testimonial.    They talked about using Guaranteed Restore Points with no IO impact and the tip was that you didn't need to explicitly enable flashback database.

"Super" quick restore by pointing the system to an image copy of a datafile in the FRA in the event of a failure.  I'm not sure how practical that one is.. I don't know anyone that uses image backups.  Most people take advantage of RMAN because of the way it skips un-used blocks, compression, etc.

One of the data guard tips for 11.2 is flushing redo.  Automatically shipping redo logs over to the standby. Previously this had to be done manually because there was a potential for dataloss even in maximum availability unless it was done manually.

The customer testimonial was pretty good.   Enterprise is a 16 billion dollar car rental company and I was surprised to learn that they don't use RAC.   Lisa Reinheimer  described their environment and how they are using dataguard.  I always like listening to how other companies architect their environments.  It gives me ideas and lets me know if i'm on the right track (if i'm already using the features).   If your interested in seeing their setup you should download the presentation.

Finally I attended Best Practices for Oracle E-Business Suite Performance Tuning . I can't even begin to tell you how much material was covered in this presentation.  This presentation was conducted by members of the EBS performance team.    You really need to download this presentation if you support an EBS environment.   There was too much content to even dream of capturing it all to post a summary.   Its basically a centralized presentation listing out tons of tuning tips, metalink notes discussing best practices and performance patches.   Enhancements to new FND_TRACE in 12.1.2 , Note: 1362660.1 for example.   Recommended performance patches for EBS can be found in note 2440401.   Great presentation.   I'll be referring back to this one quite a bit.

That was my day.   Afterwards I went to supper with some old colleagues to an Italian Restaurant in North Beach.    A nice walk down to the pier to burn off supper and a trolly ride back to the hotel.  

Openworld 2011 - Day 1

The day started off with a keynote featuring Joe Tucci from EMC, Mark Hurd and Thomas Kurian.  I won't say much about the keynote, i'm sure more than enough analysis has been already done.    All I will say tho, similar to last evening, the focus was too heavy on hardware and the "Exa's".     Oracle has sold a thousand of these servers and plan to sell 3000 over the next year.   While impressive, it still means that the majority of customers currently don't have Exa servers in house.   So, in my humble opinion, the keynotes should try to appeal to a wider audience.

First up I had a hand's on lab, Using Oracle Application Management Suite for Oracle E-Business Suite.  We just purchased this pack and plan on implementing it over the next few months.   The lab had a few issues, some my own fault, some with the environment.   It was only an hour long and given the amount of material in the guide i'm not sure if the goal was for us to complete it all.   Really to cover the amount of material they had listed you would need at least half a day.  One good thing tho is that they offered to email us the documentation (or copy it if you had a USB key) so we could continue/read up on it on our own time.

Next I attended Who Ate My DB Time? Advanced Techniques for Database Performance Analysis.  It was a good presentation.  It covered some new features of OEM R12 and how they work behind the scenes.  It finished with a customer testimonial from Boeing.

Automating Patching with Oracle Application Management Suite for Oracle E-Business Suite, covered features from both the application management pack and change management pack.   Monitoring, lifecycle management, configuration management, etc.    Being able to compare two instances from profile options to context values.  AMP for 12c is still under development, so if you use this pack you won't be upgrading to the new OEM version right away.   The customer testimonial by Prithwi Barman at the end was very interesting.   I know that before we can even use AMP we need to apply the latest ATG patches plus various others.   But after hearing him talk it looks like its going to take a fair bit more effort than I thought.   I didn't add it up, but it took them quite a few man months to implement it from DEV to PROD and test features such custom patches.  

Prithwi also talked about the process they follow migrating patches and displayed a chart summarizing the process flow.   Alot of time goes into building these flows so its always nice to see how other companies do this.  It helps give you a kickstart designing your own and/or justifing some of the choices you make.

For example, in the attached slide you can see that the developers package the custom patches, pass them along through an approval layer before reaching the DBA's.   This makes perfect sense to me but I know will make a difficult sell.   Being able to say X company is doing it this way for Y and Z reasons is going to help alot.

The next session I attended was Maximize Efficiency of Oracle E-Business Suite 11i and 12 Through Diagnostics.   Not something that really falls under my responsibility other than to make sure its setup.   So my main goal was just to get some more information on how its used by developers, some issues they may encounter etc.    Quite frequently when they open an SR oracle asks them to run various diagnostics reports and upload them.    

It was very beneficial, the presenter Karen Brownfield, talked about some security issues such as the CRM HTML administration role having super user access to diagnostics.   How tests in 11i timeout after 5 minutes, RDA's timeout in R12.   Features such as save sets and groups.  If you have a set of tests you plan to execute at the same time frequently you can put them in a save set.   Groups are similar but available to all users instead of just the one who created it.

Last but not least was Oracle E-Business Suite Directions: Deployment and System Administration.    Features in the next version of R12 (12.2) include a dual file system which allows online patching to occur.   Basically two appltops are installed, the application runs out of one of them (runtime appltop) while patches are applied to the other (patch appltop).   Once patching is complete a brief downtime occurs during the switch over.  Downtime is predictable, minutes not hours/days.

The benefits of switching the application tier over to Weblogic were also explained.   WLS has strong support for high availability and failover.   Consolidated administration, configuration and deployment.   A very indepth troubleshooting framework.   Comprehensive, low-overhead application and diagnostics.

A brief stop by the OTN party and then supper with an old co-worker.  We haven't seen each other in years even tho we live in the same city.   We were actually on the same flight and ran into each other after deplaning in San Francisco.

Openworld 2011 - Day 0

It didn't seem like long ago that I registered for #oow11 but the first day (or is it day 0?) has come and gone. I guess because of the timezone difference I was up about an hour before my alarm was set to go off, which is pretty strange for me. After a quick shower I was out the door by 7:30am and on my way to register. I expected long lines but I had my kit by 8am.

Day 0, like previous years, was a day for Oracle user groups to hold meetings. My day started with Thomas Canty's presentation titled Sysadmin 101. Thomas talked about RAID, the pros/cons of each level, NFS performance, direct NFS and various command line tips. This is a great presentation for someone new to supporting Oracle on Linux that would like to learn a few linux tips.

I take a fair bit of time leading up to the conference to go through Oracle's schedule builder and arrange my day. However, its still easy to miss good sessions. So I always keep my eye out for long lines outside a session and the TV screens showing the titles. As I was making my way to a session I happened to notice Mark Rittman's session on OBIEE 11g Architecture and Internals.

We are currently using OBIEE 10g and there has been talk of upgrading to 11g. So I went in and grabbed a seat. We currently use Oracle Fusion Middleware so alot of the concepts such as weblogic domains, system/non-system components, etc. So the session was a perfect fit to get me up to speed on OBIEE11g. I'm really glad I spotted that one as I walked by.

Next was the E-Business (EBS) Applications Technology SIG . I was a little late and walked in on an overivew of WebLogic. Afterwards Steven Chan gave an abbreviated overview of his Certification Primer and Roadmap session that he's presenting on Tuesday. I actually just realized it was abbreviated and not the full presentation while looking at his website (http://blogs.oracle.com/stevenChan/). So I just modified my schedule to include the full session which is at 1:15 on Tuesday.

I have used RMAN since 9i and its one of those features which I feel you always have to keep up on. Whats more important than being able to recover your database in the event of a failure? So I attended Kamran Agayev's session on RMAN 11g new features. One thing I liked about his presentation was the demo's and real examples/benchmarks. Alot of presentations either have slides of commands and their output or they try and do the demo live which almost always runs into some sort of issue. Kamran recorded his demo's and played them back. I found that for me personally it helped understand the topics better.

Kamran talked about new features such as compression types (ZLIB), Virtual Private Catalogs, changes to commands such as VALIDATE, etc. If you use RMAN and need to get up to speed on 11g new features then i'd check out this presentation.

Finally I attended the Sysadmin SIG. The session was dominated by a discussion on Oracle's Critical patch updates. Its nice to know that i'm not alone in finding that the documentation around this is very confusing. I'm not sure when it changed, maybe it was with release 12 but its definitely not as straight forward. I think part of the reason is that they are trying to reuse documentation. So you'll click on one note and it will reference another and you are left to fit all the pieces together. The Fusion Middleware charts list various products such as infrastructure home, middle tier home, OC4J, forms, jdeveloper, etc, The first time I tried to apply a CPU patch for R12 I had to open an SR as I had no idea which patches to apply. The state of the documentation has left me with an uneasy feeling that i'm not capturing all the patches I need to be applying.

Maybe my memory is playing tricks on me, but i'm pretty sure when I supported an 11i environment all the patches were self contained in one note and it was very easy to find the patches you needed to apply.

On top of the confusing documentation other issues such as determining if your environment is affected by the vulnerabilities, risk analysis and testing requirements also came up.

One of the benefits of coming to Openworld, in my opinion, is being able to provide feed back to the right people at Oracle and having an open discussion with them. Hopefully they will go back with our comments and improve the CPU process.

Lastly the day concluded with the opening keynote. I have to say that over the past few years I haven't looked forward to them as much as I used to. Ever since Oracle acquired Sun, Larry's keynote seems to be dominated by hardware. Not that its a bad thing because I think its a good direction for Oracle. But honestly, my organization is too small to take advantage or be able to afford any of the Exa's (Exadata, Exalogic, Exalytics). Oracle mentioned they sold 1000 in the past year and in line to sell 3000 over the next. Good numbers but that also means the vast majority of Oracle's customers aren't using it or plan to in the near future. So, to me, it doesn't make sense for it to dominate the entire keynote.

After the keynote I stopped by for a quick bite to eat and went to a few stores to take a few pictures of purses for my wife so she could do some virtual shopping.

Well, it seems that the headboard of my hotel neighbours room has stopped hitting off our shared wall, so its time to catch some sleep. I have to get up again in 3.5 hours! Please ignore any typo's or grammatical errors as my eye's aren't quite focusing. ;)

Openworld 2011

UPDATE: Opps, I just updated the title, no idea why I said 2012!

-=-=-=-

Just completed registration for my Blogger pass. I am really looking forward to the conference this year as the sessions look really good.

If your in San Francisco for the conference and would like to meet up drop me an email. My address is on the right side of this page.

11.1.1.1 -> 11.1.1.3 Fusion Middleware Upgrade

I've been meaning to post this awhile.. How long, well I believe 11.1.1.5 is out now.   I took a quick look at the upgrade documentation tho and the overall process doesn't seem to have changed much.  (I did see some extra post-upgrade steps depending on which version you start from..)

The biggest hurdle seems to be upgrading from 11.1.1.1 to 11.1.1.2...   I spent a fair bit of time trying to understand the documentation and perform the upgrade.   The process as described in the documentation didn't seem to make any sense to me.   After talking to Oracle support they found it difficult to follow as well and said they would file a request to get it reviewed.    We decided at that time to do a fresh install of 11.1.1.2 and upgrade it to 11.1.1.3.

I haven't checked back on the documentation to see if the sections were clarified.  If you followed or have seen my 11.1.1.1 installation guide you have an idea of my setup.    If you haven't, a quick review:

- Two Tier Environment
- VM18 has the IDM stack and Database installed.
- VM17 has Portal, SOA and SSO installed.

As I said above, instead of upgrading from 11.1.1.1 to 11.1.1.2 and patching to 11.1.1.3 we did a fresh install of 11.1.1.2.   The installation process for 11.1.1.2 is almost identical to 11.1.1.1 so I used the install guide I had already created as a base for the install.    From there I followed the steps below to patch up to 11.1.1.3

Note: The steps below only cover upgrading the products to 11.1.1.3.   Since we performed a fresh install it doesn't cover topics such as migrating data (example, OID accounts), domains hosting 3rd party software, portal data (transport sets), configuration (load balancers, reverse proxies, ..), etc.

Patch Weblogic from 10.2.2 to 10.2.3

1. Download the Upgrade Installer from Oracle Support.
   1. see "Downloading an Upgrade Installer From My Oracle Support"

2. Launch the install:
   1. [oracle@vm18 STAGE]$ $JAVA_HOME/bin/java -jar wls1033_upgrade_generic.jar

Upgrading IDM

Portal Patchset Install

• Prompted to run oracleRoot.sh

SOA Patchset Install

Updating Schemas with Patch Set Assistant

• Startup the database, ensure aq_tm_processes parameter is > 0
• In each home for Portal, SOA and IDM there is a Patch Set Assistant utility which was installed as part of the 11.1.1.3 patchset.

IDM psa

[oracle@vm18 bin]$ ./psa -dbConnectString //vm18:1521/FMW.world -dbaUserName sys -schemaUserName ODS
Oracle Fusion Middleware Patch Set Assistant 11.1.1.3.0
Enter the database administrator password for "sys":
Enter the schema password for schema user "ODS":
Log file is located at: /u01/app/oracle/product/mid11g3/Oracle_IDM1/upgrade/logs/psa2010-08-13-15-00-47PM.log
Updating the Oracle Fusion Middleware Metadata Schema to release 11.1.1.3.0.
Updating schema for component Oracle Internet Directory.
The command completed successfully


Start IDM
[oracle@vm18 bin]$ ./opmnctl startall
opmnctl startall: starting opmn and all managed processes...
[oracle@vm18 bin]$ pwd
/u01/app/oracle/product/mid11g3/IDMasinst_1/bin

Portal PSA

[oracle@vm17 bin]$ ./psa -dbConnectString //vm18:1521/FMW.world -dbaUserName sys -schemaUserName UPG_PORTAL
Oracle Fusion Middleware Patch Set Assistant 11.1.1.3.0
Enter the database administrator password for "sys":
Enter the schema password for schema user "UPG_PORTAL":
Log file is located at: /u01/app/oracle/product/mid11g3/PORTALas_1/upgrade/logs/psa2010-08-13-15-06-02PM.log
Updating the Oracle Fusion Middleware Metadata Schema to release 11.1.1.3.0.
Updating schema for component Oracle Portal.
The command completed successfully

SOA PSA

[oracle@vm17 bin]$ ./psa -dbConnectString //vm18:1521/FMW.world -dbaUserName sys -schemaUserName UPG_SOAINFRA
Oracle Fusion Middleware Patch Set Assistant 11.1.1.3.0
Enter the database administrator password for "sys":
Enter the schema password for schema user "UPG_SOAINFRA":
Log file is located at: /u01/app/oracle/product/mid11g3/SOAHome_1/upgrade/logs/psa2010-08-13-15-29-49PM.log
Updating the Oracle Fusion Middleware Metadata Schema to release 11.1.1.3.0.
Updating schema for component Oracle SOA.
---Redirecting sql log to /u01/app/oracle/product/mid11g3/SOAHome_1/upgrade/logs/psa2010-08-13-15-29-49PM.out
Executed SOAINFRA upgrade script.
The command completed successfully

Verify the PSA

SQL> select owner, version, status from schema_version_registry;
OWNER VERSION STATUS
------------------------------ ------------------------------ -----------
ODS 11.1.1.3.0 VALID
UPG_PORTAL 11.1.1.3.0 VALID
UPG_SOAINFRA 11.1.1.3.0 VALID

Check for Invalid objects and recompile:

SQL> select count(*) from dba_objects where status = 'INVALID';
COUNT(*)
----------
12
SQL> @?/rdbms/admin/utlrp

Post-patching tasks for the System Components:

(You may have to perform the steps below for the other products as well (IDM, SOA), not just for IDM.   I made note of it a while back but since we did the upgrade almost a year ago, I have misplaced it.)

/u01/app/oracle/product/mid11g3/PORTALasinst_1/bin
[oracle@vm17 bin]$ ./opmnctl unregisterinstance
Command requires login to weblogic admin server (vm17.rcpsc.edu):
Username: weblogic
Password:
Unregistering instance
Command succeeded.

[oracle@vm17 bin]$ ./opmnctl redeploy
Command requires login to weblogic admin server (vm17.rcpsc.edu):
Username: weblogic
Password:
Redeploying NonJ2EEManagement Application...weblogic.Deployer invoked with options: -adminurl vm17.rcpsc.edu:7001 -username weblogic -name NonJ2EEManagement -source /u01/app/oracle/product/mid11g3/PORTALas_1/opmn/applications/NonJ2EEManagement.ear -redeploy -upload -noexit
<Aug 13, 2010 4:20:54 PM EDT> <Info> <J2EE Deployment SPI> <BEA-260121> <Initiating redeploy operation for application, NonJ2EEManagement [archive: /u01/app/oracle/product/mid11g3/PORTALas_1/opmn/applications/NonJ2EEManagement.ear], to configured targets.>
Task 0 initiated: [Deployer:149026]deploy application NonJ2EEManagement [Version=11.1.1] on AdminServer.
Task 0 completed: [Deployer:149026]deploy application NonJ2EEManagement [Version=11.1.1] on AdminServer.
Target state: redeploy completed on Server AdminServer
Done
Command succeeded.

[oracle@vm17 bin]$ ./opmnctl registerinstance;
Command requires login to weblogic admin server (vm17.rcpsc.edu):
Username: weblogic
Password:
Registering instance
Command succeeded.

Update Version Number String shown in OEM

/u01/app/oracle/product/mid11g3/IDMasinst_1/bin
[oracle@vm18 bin]$ ./opmnctl updatecomponentregistration -componentName oid1 -Sport 3131 -Port 3060
Command requires login to weblogic admin server (192.168.80.148):
Username: weblogic
Password: