Forget to configure your application server as an Administration Instance?

I’ve installed Oracle Application Server quite a few times and today I had to perform another one.   After I installed the application server I tried to login to the console but was greeted by a lovely message:

Oops! This link appears to be broken.

The apache error logs showed:

File does not exist: /u01/app/oracle/product/apps1013/Apache/Apache/htdocs/em/console


The access log:

"GET /em/console HTTP/1.1" 404 342


It turns out, while installing the application server I forgot to select a check box which designated the server as an administrator instance. 



In a single instance environment, such as the one I am setting up, you need to select this option to be able to manage the instance.   If this was a node in a cluster, you should only designate one instance as the administration instance.

So what do you do if you forgot to select it like I did?  Nothing to worry about, you just need to edit two files:

1. Open $ORACLE_HOME/j2ee/home/config/server.xml

Change start=”true” for the ascontrol line.  It should look like this:

<application name="ascontrol" path="../../home/applications/ascontrol.ear" parent="system" start="true" />


2. Edit $ORACLE_HOME/j2ee/home/config/default-web-site.xml

Add ohs-routing=”true” to the ascontrol line.  It should ook like this:

<web-app application="ascontrol" name="ascontrol" load-on-startup="true" root="/em" ohs-routing="true" />

Now you just need to restart the application server:

$ORACLE_HOME/opmn/bin/opmnctl stopall;
$ORACLE_HOME/opmn/bin/opmnctl startall;

and if you goto http://servername:/em/console the application server control should now work.

11g Fusion Install issues, don’t interrupt the install…!

Here are a couple of tips I found out the hard way with while installing fusion 11g.

1. SSO - At one particular stage I was in the process of installing single sign-on 10g following these instructions. After following the step listed at Section 10.2 step 5 I needed to perform some maintenance on the server. So I shutdown the environment, performed the work and restarted the servers.

Before I could continue my install I needed to start up the services such as the database, weblogic server, OID, etc. However, when I went to install OID it would not start properly. Within the OID logs I found:

[host: 0002] [pid: 8145] [tid: 0] Guardian: WARNING: Connected to incorrect OID base schema version, (version=10.1.4.0.1).

If I had been paying more attention to the installation steps I would have noticed that step 5 changed the directory version number in the repository to 10.1.4.0.1. After you install OID 10g, you re-run the inspre11.pl script with the –op3 flag to reset the value back to 11g.

Unfortunately OID needs to be running before you execute inspre11.pl, so I had to update the repository manually with the following SQL:

update ods.ds_attrstore set attrval = 'OID 11.1.1.1.0' where entryid =1 and attrname = 'orcldirectoryversion';
commit;

After that OID 11g started up fine and I could rerun the instpre11.pl script with the –op2 flag. Then I continued with the SSO install.

Another tip, review the following note before installing OID 10g:

Subject: Oracle Identity Management 10g (10.1.4.0.1) Release Notes Addendum

Doc ID:
465847.1

It lists some pre-reqs which are not in the install guide and some known issues. The main one for me is that libdb.so.2 was missing. The note describes how to resolve that issue.

2. Portal – The Portal install itself went without any issues, however, at the configuration wizard stage it would hang creating the weblogic domain. Sometimes I would see java.lang.OutOfMemoryError: PermGen space messages in the install log as well. I found the following notes:

Subject: FMW 11g “IDM” or “Portal/Forms/Reports/Discoverer” Configuration Wizards on 64bit Plantforms Hang at 0% ‘ Creating Domain’

Doc ID: 865462.1

Subject: FMW 11g ‘ Portal/Forms/Reports/Discoverer’ Config Wizard Fails – “Error creating ASInstance”.. Unable to Connec tot ‘Admin Server’ ..PermGen

I was using the 32bit version of Weblogic which ships with the Sun and JRockit JDK’s on top of OEL 5.3 64bit. The notes say that you must use 64bit FMW 11g with a 64bit JDK on a 64 bit OS. I downloaded the latest JRockit because it apparently doesn’t suffer from permgen issues but I still hit the same problem(althought no permgen messages in the log file..). It wasn’t until I downloaded the latest Sun JDK 1.6.0_16 that the configuration wizard was able to create the domain.

3. SOA – After installing Portal, SOA seemed to complete without a hiccup… Until I tried to login to SOA. At one point I noticed errors when trying to start the SOA domain with the startWebLogic.sh script:

javax.security.auth.login.FailedLoginException: [Security:090304]Authentication Failed: User weblogic javax.security.auth.login.LoginException: [Security:090301]Password Not Supplied

However, I wasn’t being prompted for a password. I could hardcode the password within the startWebLogic.sh script but it would still fail, however with another error:

<Sep 1, 2009 10:08:33 AM EDT> <Critical> <WebLogicServer> <BEA-000386> <Server subsystem failed. Reason: weblogic.security.SecurityInitializationException: Authentication for user weblogic denied
weblogic.security.SecurityInitializationException: Authentication for user weblogic denied

Weblogic checks for the existence of a boot.properties file upon startup. If it exists, it reads the username and password from this file and doesn’t prompt you. I deleted this file and after that I was prompted for the password.. I was hoping that the stored password was incorrect, causing my problems but no such luck.

I talked to Oracle support and they believed that something was corrupted and the next step would be to recreate the admin account. Since I am using the latest version, they haven’t tested the steps yet and said they’d get back to me in a couple of hours. I searched google and found a few hits with the same problem. A few people tried re-installing and somehow everything worked fine after that. I took a backup of my domain configuration and decided to try and recreate it.

This time during the install, I selected the optional configuration options for the Administration server and Managed servers. (http://download.oracle.com/docs/cd/E12839_01/doc.1111/e13925/config_screens.htm#CJAIIADH) I noticed the ports being used have already been taken by the Portal install. I chose new ports, finished the install and everything started fine.

I’m guessing my authentication errors are because my Portal domain was up and SOA was trying to authenticate against it. When installing Fusion you let the configuration wizards create the weblogic domains, you don’t create them with the weblogic assistants. I guess they aren’t smart enough to detect that ports are already in use, which is kind of surprising. This is my first FMW 11g install, so it is possible I messed something up the first time I tried to install SOA.

4. Repository Creation Utility – Another problem I hit was trying to run the RCU utility for SSO on a 64bit OEL 5 environment. I couldn’t get it to work. Since this doesn’t need to run very often I put it on a 32bit Redhat 4.7 environment and ran it from there without issues.

So after 4 SR’s and a fair bit of reading I finally have a Fusion Middleware 11g environment. I was surprised to find its much easier to install e-Business Suite than FMW!

Alloy/Deny Directives Ignored

Just a quick note to you let you know about a potential security vulnerability when you are using Oracles Webcache and HTTP server on the same server. Note 263943.1,
contains all of the details but in a nutshell instead of the client ip address being passed to the http server, webcache passes its ip address. Thus certain Allow/Deny rules that you may have setup in your httpd.conf file may not work as expected.

For example from the application server httpd.conf file:

<location>
  SetHandler server-status
  Order deny,allow
  Deny from all
  Allow from localhost myserver.mydomain myserver
</location>

The server status page contains some useful information to aid in troubleshooting performance problems with the application server (see note 280165.1). It contains information that you would not want potential crackers to see such as the servers version and requests currently in progress.

As you can see from the Deny, Allow entries above we only want requests from the localhost or myserver (myserver==localhost) to gain access. However, since webcache is on the same server it is passing the server IP addresses instead of the client. Thus the HTTP server believes every request is from a local client and grants access. Obviously, this is not the intended effect.

This is just one example and if you are relying on Allow/Deny rules to restrict access to certain parts of your applications, then you may be exposing more than you think.

The solution is to change the parameter UseWebCacheIp to ON in the Oracle HTTP Sever httpd.conf file and restart it. Please see note 263943.1 for versions affected and patchlevels required before making this change.

I am not sure why this parameter isn't enabled by default and I can't think of a reason as to why not. However, I am new to supporting a standalone application server and there must be a reason otherwise Oracle would have it enabled. Do you know of one? If so, please leave a comment.

How to Change an Application Server User Password

The solution depends on if you are using Oracle Internet Directory (OID) or not. If you are using OID you can login to the OID Console or the oidadmin GUI to reset the password easily.

But what if your application server is stand alone? If your trying to reset the oc4jadmin superuser account then the only solution is manually modify the $ORACLE_HOME/j2ee/home/config/system-jazn-data.xml file.

Search the file for your the username and place the new password, prefixed with an exclamation point "!" between the <credentials> </credentials> delimiters. I am assuming that you have to prefix it with the ! so that Oracle knows how to handle the password. ie, that its not encrypted.

Restart the application server and verify that you can login.

How about regular user accounts tho? Surprisingly you have to follow the same process. I must have poked around in the Application Server Console for half an hour in disbelief that I couldn't reset a user password without knowing the old password.

If I missed something or if you know of another way to reset user accounts, please post a comment.

New Job, New responsibilities

Recently I started a new job with the added benefit that I have some new stuff to learn. Previously I managed EBS environments with a few standalone databases and I will be doing that here as well. Additionally they also use Oracle Portal, Oracle Internet Directory and standalone application server.

I haven't used these before so I thought I would setup a vmware environment, install it and start plugging away. For the life of me, I couldn't find the install guide for the application server. After I wasted enough time looking I decided to uncompress the software and move forward anyways. It was then that I noticed the docs subfolder under Disk1, which containg an install guide.

The first step of the install is to execute runInstaller and check for pre-reqs:

prompt> mount_point/1012disk1/runInstaller -executeSysPrereqs

The OS, temp and swap space checks passed but failed checking the display:

Checking monitor: must be configured to display at least 256 colors. Failed <<<<
>>>> could not execute auto check for display colors using command
/usr/X11R6/bin/xdpyinfo. Check if the DISPLAY variable is set.

I was local to the box and my display was working properly.. I could also execute xdpyinfo manually and it would work ok. However, a which xdpyinfo turned up that the program resided in a different directory than the one runInstaller was trying to use.

[oracle@localhost Disk1]$ which xdpyinfo
/usr/bin/xdpyinfo
[oracle@localhost bin]$ ln -s /usr/bin/xdpyinfo /usr/X11R6/bin/xdpyinfo

So I created a symbolic link, as you can see above and tried executing runInstaller again. This time everything passed.

Not much of a tip a day... Just wanted to mention that I had started a new job and hopefully the dry spell of posts is over. I was very idle in my previous position (one of the reasons why I left), which made it more difficult to think of stuff to blog. I'll be using some new pieces of software now and we are in the process of implementing a new R12 environment. So I should have alot to talk about soon.