X Server and EBS

According to Note: 231137.1 Troubleshooting the Self Service Framework with Oracle Applications, you need access to an X server for dynamic image generation. These images are used for buttons and menu bars that you see on self service pages. Before I get to far, I should probably add that if your on windows you don't need an X server.

Metalink note: 181244.1 lists the different options you have available and describes how to configure each. In summary the 3 options you have are:

1. Use the X display of your server.
2. If your server doesn't have a display you can use 3rd party software Xvfb or VNC.
3. A Desktop X display server - ie, use 3rd party software such as Exceed on a PC

If your environment is autoconfig enabled you can look at you context file to see where you display is currently pointed.

ora@myserver=> grep display *.xml ORA_myserver.xml: myserver:0.0

You can also see it in your $IAS_ORACLE_HOME/Apache/Jserv/etc/jserv.properties file

wrapper.env=DISPLAY=myserver:0.0


To test whether or not the application is able to access the X Server you can use the AOL test UI:

http://hostname:port/OA_HTML/jsp/fnd/aoljtest.jsp

The first thing you will be prompted for is your connection details:



After you click on the TEST button scroll to the bottom of the screen and click on "Enter AOL/J Setup Test". On the left hand side of the following screen you see a summary of the environment you connected to and a list of tests. Scroll down on the left hand side until you see "X Server Accessibility". Once you click on that link the right hand frame will show you whether or not the X Server is accessible.




Note 231137.1 also describes some of the common errors you may see if the X server is not accessible. However, sometimes you will get rather cryptic errors if the application can't talk to an X server.

For the record, we use the X display of our server (option 1 above). In order to use this option the unix administrators have to type xhost + on the console to grant access to the X server. A caveat is that the unix administrators have to lock their terminal. If they logout, then access is revoked. This past weekend we had some maintenance in which the servers needed to be rebooted. After which the unix admins logged out of the terminal. Sunday morning users reported the following error when in iProcurement:



As you can see, not alot of information to go on. Since this is self-service web application the first thing to check is the apache and jserv log files. In the jserv.log file was the following error:

[24/06/2007 14:38:08:378 EDT] OAButtonBean, localName='button': oracle.cabo.style: Could not initialize the graphical environment
. Please make sure that the DISPLAY environment variable is set correctly. Proceeding with image generation disabled...

Using the X server test via AOL Test UI we could verify that the application couldn't talk to the X server. After a call to the unix team they granted us access, "locked" the screen and once apache was bounced the error was resolved. So in a nutshell, keep that link above handy. We have hit numerous weird errors in the past and sometimes it involves a lot of investigation to find out in the end it was just the X Server.

I believe in R12 the need for an X server is gone but we are a long ways from R12..... For now we may investigate using option 2 or 3 above so that we can manage this piece ourselves.

10.2.0.2 Rman duplicate issue

Each night we use rman's duplicate database feature to refresh a test environment to aid in troubleshooting production issues. This past weekend we upgraded our production database from 9i to 10gR2. We also applied alot of patches and the simplest way to update our test environment was to reclone it.

Last night we used our refresh scripts (rman duplicate) to rebuild the test database. Early this morning users reported the following error:




Since this problem wasn't reproducible in production we knew it had to be something with the refresh. The first thing we did was to look for differences between production and test databases. Some objects were invalid in test but recompiling them didn't resolve the issue. Checking through the log files for the refresh I noticed that autoconfig didn't complete successfully.


Uploading Context file /apps/appsora/oraappl/admin/ora_myserver.xml
Context File upload for /apps/appsora/oraappl/admin/ora_myserver.xml failed.
Check log file created by FNDCPUCF program.
. . .
[AutoConfig Error Report]
The following report lists errors AutoConfig encountered during each
phase of its execution. Errors are grouped by directory and phase.
The report format is:


[PROFILE PHASE]
AutoConfig could not successfully execute the following scripts:
Directory: /apps/appsora/oracomn/admin/install/ora_myserver
afcpctx.sh INSTE8_PRF 1


AutoConfig is exiting with status 1


afcpctx.sh uploads you context file to the database by calling FNDCPUCF. I assuming FNDCPUCF stands for FND Concurrent Process Upload Context File but I can't seem to find much information on it. Since it fires off a concurrent request the next step was to check the log files generated for errors. The following error was in one of them:

Uploading context file...
Oracle error -1187: ORA-01187: cannot read from file 702 because it failed verification tests
ORA-01110: data file 702: '/oradata/dbf/oradata/temp02.dbf' has been detected in FND_GSM_UTIL.APPEND_CTX_FRAGMENT.

I haven't encountered that error before so but a quick search on Metalink turned up the following note:


Subject: RMAN After 10.2.0.2 Duplicate Database Gets Ora-1187 And Ora-1110 on TempFiles
Doc ID: Note:374934.1

In 9i tempfile information is removed from the database during the duplication process and you need to re-add them manually. In 10g, tempfiles are now included. However, if the files already exist on the OS then the tempfiles are not re-created and you will see errors in the alert log. So, in order to resolve this issue we dropped the tempfiles and re-added them (as per the metalink note).

To drop tempfiles:

alter database tempfile '/u01/oracle/mydb/temp01.dbf' drop;

To add a new one (assuming your temporary tablespace is named temp):

alter tempspace temp add tempfile '/u01/oracle/mydb/temp01.dbf' size 500M;

Once this was done, autoconfig completed without error and we restarted the application.

10g Upgrade issues

This weekend we are upgrading our E-Business Suite database from 9i to 10g (10.2.0.2 to be specific). Today while performing some final checks I noticed that an initialization parameter was not set properly in our test environment.

I had used the script bde_chk_cbo.sql from Metalink note: 174605.1 which builds a report consisting of the current, required and default values of your initialization parameters. The parameter in question was optimizer_secure_view_merging and according to the report it was undefined. Which means its using the default value of TRUE. According to Note 216205.1, Database Initialization Parameters for Oracle Applications 11i this parameter should be set to FALSE.

I checked my logs of the test upgrade and I did indeed set this parameter to FALSE. So I performed a search on metalink and found Note 404646.1, How To Set Parameter optimizer_secure_view_merging = FALSE Using Autoconfig. In summary, autoconfig isn't updating your pararameter file properly unless your on version 115.7 of the template file $ORACLE_HOME/appsutil/template/afinit_db102.ora. We have version 115.3 installed. This parameter is needed to avoid some performance issues with dictionary queries. If this applies to your environment, you should review the metalink note for the workaround.

Over the past month or so we have upgraded many environments to 10g, reverted some back to 9i, etc. It started to get confusing as to which environment was running 10g and which were still on 9i. (We have about 14 dev/test/project environments right now, with more planned.) One of my coworkers started up a 9i database with the 10g Oracle Home and we noticed something interesting:

ALERT: Compatibility of the database is changed from pre-10i to 10.2.0.0.0.
Increased the record size of controlfile section 2 to 256 bytes
The number of logical blocks in section 2 remains the same
Increased the record size of controlfile section 4 to 428 bytes
Control file expanded from 1614 blocks to 1644 blocks
Increased the record size of controlfile section 9 to 56 bytes
Control file expanded from 1644 blocks to 1672 blocks
Increased the record size of controlfile section 10 to 200 bytes
Control file expanded from 1672 blocks to 1694 blocks
kcv_lh_or_upgrade: 10.2 upgrading 1 incarnations
Control file expanded from 1694 to 1990 blocks for upgrade.
Setting recovery target incarnation to 1
Wed Jun 20 00:11:34 2007
Successful mount of redo thread 1, with mount id 764203184
Wed Jun 20 00:11:34 2007
Database mounted in Exclusive Mode
Completed: ALTER DATABASE MOUNT
Wed Jun 20 00:11:34 2007
ALTER DATABASE OPEN


As you can probably guess by the messages above, the control file was modified. When the database was shutdown and started with the 9i software the following happened:


ORA-00201: controlfile version 9.2.0.0.0 incompatible with ORACLE version
9.2.0.0.0


Yikes! On Monday I'm going to see if I can recreate the control files and startup the database but right now i'm off to upgrade our production EBS database to 10g.

Patching and EBS

A couple of people [ Steven Chan , Niall Litchfield ] have blogged over the past week about patching an Oracle EBS environment.

Steve says, "I'm compelled to make an admittedly-quixotic attempt to persuade you that the benefits of keeping your E-Business Suite environment up-to-date far outweigh the costs."

In general I agree with him based on my past experiences. A few times, due to a number of reasons, I haven't been able to do as much patching as I would have liked. That has come back and bitten us a few times by hitting issues that we wouldn't have encountered otherwise. There can be a substantial cost to troubleshoot issues. It usually requires time from DBA's, developers, QC and end users. That can add up pretty quickly.

For us tho there are three reasons why we are not as current as I would like. For large efforts, such as upgrades, its very difficult to put a dollar value on being reactive instead of proactive. There is also no guarantee that being upto date will reduce the number of issues or if they will resolve outstanding issues. That makes it difficult to make a case with upper management.

The other reason is because of testing requirements and/or testing resources. We have some custom modules and code so there is a fairly rigorous testing cycle for any change that goes into our environment. We have 2 teams dedicated for testing and at times we have to bring in end users. A testament to the testers tho is that we rarely encounter issues after changes go live.

Lastly, since business has a backlog of requests, our work often takes a back seat. Its a continuous fight to be able to put in some of the things we would like to do. I'm happy to say tho that we have done a bit of work over the past year to upgrade our hardware, consolidate environments and implement features such as Shared Application Filesystem and merging patches to reduce downtime. We recently put in multiple application tiers and without a shared filesystem maintenance would be a nightmare.

Currently we have some very aggressive projects going on at the moment. We are upgrading to 10gR2, implementing the Sales Compensation (OIC) and Projects modules. And if that wasn't enough, upgrading to 11.5.10.2. I think its safe to say that the next 6-8 months are going to be very busy.