Fusion Middleware – PermGen memory issues

The other day a developer called me to say that the Fusion Middleware console was spinning.   A quick look at the the environment showed a java process consuming 100% of a cpu and in the AdminServer weblogic log there was the following error:

java.lang.OutOfMemoryError: PermGen space

To get the environment going again I had to kill the java process and restart the AdminServer.    I increased the amount of PermGen memory and so far it hasn’t happened again. 

To change the PermGen memory login to the WebLogic Administration Console:

After you login click on Lock & Edit, so you can make changes. Followed by Environment -> Servers under the Domain Structures widget. The right hand side of the screen will update. Click on the AdminServer(admin) link in the table:
Verify that the Configuration tab is selected and click on Server Start:
Scroll down until you see an Arguments text box and update the PermSize and MaxPermSize to the new values. In my example below I have already modified them to be twice their original value:
Once you have made the changes scroll back up to the top of the page and click on the Save button:
Click on the Activate Changes button:
As you can see below, the changes were saved successfully and it says no restarts are necessary. Since we updated the startup parameters tho, they will not come into affect until you perform a restart.
If you want to check out your changes on the server side, login to the server hosting your Portal domain. Change directory to your $PORTAL_DOMAIN/config directory and view the config.xml file. A backup of the config.xml file can be found at $PORTAL_DOMAIN/servers/domain_bak/config_prev/config.xml

From within the WebLogic console, the monitoring capabilities seem to be rather limited.

For example:
From the Fusion Middleware control console I see:
And from the WebLogic Administration Console (Server-> Monitoring -> Performance tab) I can see:
Now, I am very new to WebLogic, so there may be other areas to see more detailed information or a way to start recording it. I have it on my list to investigate further.

If you have Grid Control and the WebLogic management pack there are lots of performance metrics you can view and drill down into.

Lets say you don’t have Oracle Grid Control and would like to confirm the PermGen changes we made above or take a closer took at performance. There are a number of tools available and one of them, VisualVM is packaged with the Sun JDK.

To launch VisualVM go to your JDK_HOME/bin directory and launch jvisualvm. The first time you launch this tool there is a calibration that needs to occur.

When it finishes you’ll be presented with some statistics. After you click OK and the application starts you will see a screen similar to:
The tool automatically discovers the java processes. If you don’t see any here, or not as many as you expect there could be a number of reasons. From the VisualVM documentation:

The circumstances in which VisualVM will not automatically discover JMX agents, and thus the Java applications they expose, are the following:
  • The target application is running on the J2SE 5.0 platform and the* properties have not  been specified.
  • The target application is running on the same host as VisualVM but was started by a different user than the one who started VisualVM. VisualVM discovers running applications using the jps tool, which can only discover Java applications started by  the same user as the one who starts the VisualVM tool.
  • The target application is running on a remote host where jstatd is not running or is running but was started by a different user. The jstatd daemon provides an interface that allows remote monitoring applications to connect to Java applications on the host where it is running.
As you can see in my screenshot above, it automatically discovered quite a few WebLogic servers. I ran VisualVM on my application tier which contains Portal and SOA. As you can see next to each entry there is a process id. You can either double click on each one until you find the one your looking for, or say your looking for the Portal AdminServer you can grab it from your server. This screenshot is going to be a bit cluttered but basically i’m executing ps –ef, grepping for AdminServer and looking for a couple of key things in the output. The process id, that the process is for an AdminServer and the domain.

If I go back to VisualVM, find process id 32508 in the list and double click on it I see:
Take a look at the red circle. Wait?! It still says that the PermSize is 128 and MaxPermSize is 256m.

It turns out that the procedure I used above to change the PermSize settings, which modifies the config.xml file will only work if you are using a Node Manager to start the servers.

If you are using the and scripts to start up the AdminServer and managed servers respectively, then you will need to modify those scripts.

Since I am modifying the Java parameters for the AdminServer the script we need to modify is However, calls a script called This script initializes a number of variables, but the one we are looking for is EXTRA_JAVA_PROPERTIES.
This parameter will be modified multiple times, so it can be confusing as to which one to modify. Here is a clip:

EXTRA_JAVA_PROPERTIES=" -Xms512m -Xmx1024m -XX:PermSize=128m -XX:MaxPermSize=256m -Doracle.home=/u01/app/oracle/product/mid11g/PORTALas_1 -Ddomain.home=/u01/app/oracle/product/mid11g/user_projects/domains/PortalDomain ${EXTRA_JAVA_PROPERTIES}"

if [ "${SERVER_NAME}" = "WLS_REPORTS" ] ; then
EXTRA_JAVA_PROPERTIES="-Xms256m -Xmx512m -XX:PermSize=256m -XX:MaxPermSize=512m -Djava.library.path=/tmp/

The parameter we want to modify is the first one. The second one is actually Java parameters for the Reports manager server. Further down in the file you will see sections for the other managed servers in the Portal environment, such as Forms and Portal. Modify the first EXTRA_JAVA_PROPERTIES PermSize settings:

EXTRA_JAVA_PROPERTIES=" -Xms512m -Xmx1024m -XX:PermSize=256m -XX:MaxPermSize=512m -Doracle.home=/u01/app/oracle/product/mid11g/PORTALas_1 -Ddomain.home=/u01/app/oracle/product/mid11g/user_projects/domains/PortalDomain ${EXTRA_JAVA_PROPERTIES}"

After a restart of the AdminServer and viewing the java process in VisualVM we see:

We have now confirmed the changes have been made successfully. If you notice 9 lines above my circle there is another MaxPermSize entry. I believe that one is being captured by the variable:


I am not sure what happens in a case where a parameter is specified twice with different values. Does the second one override the first? Is it the one with the maximum value? I’ll tell you what I think in a second.
If you click on the Monitor tab->PermGen you will see the following:

This screen provides you with more information about the state of the java process such as the PermGen memory area. As you can see the size is around 256M and currently the WebLogic process is only using around 175M.

To test what happens when a parameter is specified twice, I put another –XXPermSize before the one in the JAVA_EXTRA_OPTIONS variable and tried a few test scenarios. I set the first to 128m and the second to 256m. When I looked at the PermGen memory in VisualVM the size was around 256. Then I swapped the parameters and restarted the server again. This time the PermGen size was 128m (the value of the second parameter). So it looks like if a parameter is specified twice the last occurrence is used.


11g Fusion Middleware Install Walkthrough – Part 3 Single Sign On

Part 3 describes the process to install and setup SSO on the application tier of my Fusion Middleware Environment. I had originally intended to talk about Portal and SOA here but I didn’t realize how long the SSO portion was.

Oracle Single Sign-On and Oracle Delegated Administration Services Release 10g are required components for Oracle Portal, Forms, Reports and Discoverer Release 10g and Release 11g.

Overview of the Installation Steps
  1. Executing
  2. SSO Repository Creation
  3. Install SSO and upgrade it to
Executing and SSO Repository Creation

More detail for the steps below can be found in Oracle’s documentation.

execute with –op1. The –op1 flag will enable anonymous bind, disable entry cache and change the OID version from to so the SSO repository creation assistant will be able to load the schema.

On the database tier set your ORACLE_HOME environment variable to your IDM installation, and ORACLE_INSTANCE to the instance home location. Execute with the –op1 parameter.
The syntax for is:

$OID11gR1_ORACLE_HOME/perl/bin/perl \
$OID11gR1_ORACLE_HOME/ldap/bin/ OID_HOST OID_PORT {-ssl | -nonssl} \
{-op1 | -op2 | -op3}

In part 2, I used the same password for all of the accounts, so ODS_PASSWORD and ORCLADMIN_PASSWORD is the same. That’s why you will see password specified twice in my example below:

[oracle@vm2 Oracle_IDM1]$ ORACLE_HOME=/u01/app/oracle/product/mid11g/Oracle_IDM1; export ORACLE_HOME

[oracle@vm2 Oracle_IDM1]$ ORACLE_INSTANCE=/u01/app/oracle/product/mid11g/IDMasinst_1; export ORACLE_INSTANCE

[oracle@vm2 Oracle_IDM1]$ $ORACLE_HOME/perl/bin/perl $ORACLE_HOME/ldap/bin/ vm2 3060 -nonssl oid1 vm2:1521:FMWD password password –op1

Use RepCA to load SSO and other schemas against DB before running -op2

SSO Repository Creation

In part 2 we used a repository creation utility to load the IDM, Portal and SOA 11g schemas. Since SSO is a previous version (10g), we need to use a different repository creation utility (RCU). The 11g RCU is much nicer because it doesn’t need to be installed before you can use it. The 10g SSO RCU needs to be installed like other Oracle products before it can be installed.

The 10g SSO RCU is 32bit, so if your trying to install this on a 64bit environment, you have to type linux32 bash before you launch the runInstaller. Even tho I ran it from a 32bit shell I still encountered issues. Since you don’t need the 10g SSO RCU after you load the SSO schema I decided to install it on an old 32bit server instead of trying to figure out what was wrong.

Unzip the 10g SSO RCU archive and launch the runInstaller:


Click on the Next button.


Verify the inventory directory and change it to suit your environment. Then click on the Next button.


Make sure you execute the script before clicking on the Continue button.
[root@appTier oraInventory]# ./
Creating the Oracle inventory pointer file (/etc/oraInst.loc)
Changing groupname of /home/oracle/oraInventory to oinstall.


Change if necessary and click on the Next button.


I hit this warning, I can’t remember which kernel parameter was at issue but after I reviewed the problem I determined it was ok to proceed. The next time I install it, I’ll take better notes. Click on the Yes button.


If you need other languages select them and click on the Next button.


Verify that Yes is selected so that the RCU assistant launches automatically after the installation is finished and click on the Next button.

Click on the Install button.

The RCU assistant will automatically start and show the screen below:


Click on the Next button:


Enter the path of the Oracle Home directory for the database installed in Part 2. As well, select a log file directory and click on the Next button

We need to load the schema and register it, so verify that the Load and Register option is selected and click on the Next button:


Enter the information for the database created in Part 2.


It appears that the SSO Repca utility ignores the SGA_TARGET parameter and looks explicitly for the various SGA parmeters to make sure your database is sized correctly and setup correctly. As long as your SGA_TARGET is set high enough you can quickly login to your database, execute the commands below and continue with the RCU assistant.

SQL> alter system set aq_tm_processes =1;
SQL> alter system set db_cache_size=200M;
SQL> alter system set shared_pool_size=200M;
SQL> alter system set java_pool_size=120M;


Click on Yes, this requirement is only for 10g databases and we are not using 10g Portal.


I am using an regular file system so I selected the first option and clicked on the Next button.


Since I am on a SAN I am using the same directory for all tablespaces but if you are using multiple directories you will need to manually enter the information for each tablespace. Click on the Next button when complete.


By default the system tablespace datafile should be in autoextend in 11g but verify.


Enter the hostname for the server in which you installed Identity Management on in Part 2. By default the SSL port is 3131 but if you are unsure you can look at the installation summary file you should have saved back in Part 2.


Enter the password for orcladmin which you specified while installing Oracle Identity Management in Part 2.


I choose the default and clicked on Next.


Once the SSO repository has been loaded into your database you will be presented with the following screen. Click OK and your finished.


The 10g SSO RCU changes the ODS password for some reason so you need to change it back to the one you used back in Part 2.

Login to your database via SQL*Plus and change the ODS password using alter user ods identified by PASSWORD, where PASSWORD represents the ODS schema password before running the 10g SSO RCU Assistant.

SQL> alter user ods identified by password;
User altered

Login to the server in which Oracle Identity Management was installed and set the ORACLE_HOME, ORACLE_INSTANCE and TNS_ADMIN environment variables to reflect the IDM install and create a wallet:

[oracle@vm2 ~]$ ORACLE_HOME=/u01/app/oracle/product/mid11g/Oracle_IDM1; export ORACLE_HOME
[oracle@vm2 ~]$ ORACLE_INSTANCE=/u01/app/oracle/product/mid11g/IDMasinst_1; export ORACLE_INSTANCE
[oracle@vm2 ~]$ TNS_ADMIN=$ORACLE_INSTANCE/config; export TNS_ADMIN

[oracle@vm2 ~]$ cat $TNS_ADMIN/tnsnames.ora

[oracle@vm2 ~]$ $ORACLE_HOME/ldap/bin/oidpasswd connect=OIDDB create_wallet=true
confirm password:
Replication password file exists
password set

Restart Oracle Internet Directory:

[oracle@vm2 bin]$ cd $ORACLE_INSTANCE/bin
[oracle@vm2 bin]$ pwd
[oracle@vm2 bin]$ ./opmnctl stopall;
opmnctl stopall: stopping opmn and all managed processes...
[oracle@vm2 bin]$ ./opmnctl startall;
opmnctl startall: starting opmn and all managed processes...

Execute the script with -op2, which resets the Oracle Internet Directory version and allows you to install Oracle Single Sign-On and Oracle Delegated Administration Services Release 10g ( The -op2 option will also verify the orcldirectoryversion attribute has a value of OID

[oracle@vm2 bin]$ $ORACLE_HOME/perl/bin/perl $ORACLE_HOME/ldap/bin/ vm2 3060 -nonssl oid1 OIDDB password password -op2

Install SSO/DAS against 11g OID before running -op3

Above when we executed with the –op1 flag we used vm2:1521:FMWD for the database information. For some reason, with the –op2 flag, you will get an error. See the following note for details. Since we set the TNS_ADMIN environment variable above, we can pass the tnsnames.ora entry, which is OIDDB.

Subject: Inspre11.Pl Fails With Error Message No Such File Or Directory At Line 310   Doc ID: 876623.1

Install Oracle Single Sign-On and Oracle Delegated Administration Services Release 10g ( and Upgrade to

On your application their unzip the SSO media and launch the runInstaller.


Click on the Next button:


Specify the directory for the SSO Oracle Home and click on the Next button.


Select Oracle Application Server Infrastructure 10g for the Single Sign-On (SSO) component and click on the Next button.


We already created the SSO repository, so select the second option Identity Management and click on the Next button.


If you haven’t already review Note 465847.1, it lists the required packages for OEL 5 and if you made the OS changes listed in Part 1 then you should be fine. So I ignored this error and continued.. I’m pretty sure it was looking for packages I have installed but of a lower release level. Click on Next to continue.


You’ll need to execute the script as part of this install. Click on the Next button to continue.


We are only using SSO and Delegated Administration Services, so I de-selected the other options and clicked on the Next button.

Click on the Next button.


Enter the hostname and port number for your IDM install from Part 2.


Enter the password for the orcladmin account which you specified during the IDM install in Part 2.


Create an Instance Name, I put SSO in front of the instance name to keep it logical. After you enter a password click on the Next button.


Click on the Install button.


During the install you may see this error. It can be safely ignored as per note: 465847.1


Before you click on the OK button above you should download and apply patch patch 5649850 before running run the script.

To apply this patch set your ORACLE_HOME environment variable to your SSO Oracle Home, unzip the patch, change into the patch directory and execute $ORACLE_HOME/OPatch/opatch apply.

If you get an error:

The Oracle Home /u01/app/oracle/product/mid11g/SSOHome_1 is not registered with the Central Inventory. OPatch was not able to get details of the home from the inventory.

Try upgrading Opatch to OPatch Version:

Once you have applied the patch click on the OK button.


Once the install has finished the configuration assistants will run. Assuming no issues are encountered you will be presented with the End of Installation screen below:


SSO has been installed but now we need to upgrade it to

Upgrade SSO to

Upgrade Oracle Single Sign-On and Oracle Delegated Administration Services to Release 10g ( by applying the Oracle Identity Management 10g ( Patch Set. You can get the Oracle Identity Management 10g ( Patch Set from My Oracle Support (formerly MetaLink) by searching for Bug or Patch Number 7215628.

Stop SSO:

If you don’t stop it beforehand I believe the installer will stop it for you.

echo "Stopping SSO Components. . ."
cd $SSO_HOME/bin
./emctl stop iasconsole
sleep 5
cd $SSO_HOME/opmn/bin
./opmnctl stopall

Download patch 7215628, unzip it and launch the runInstaller:


Click on Next.


Select the SSO Home from the drop down list and click on Next.


Enter your Metalink credentials if you would like configuration manager installed. I opted not to install it.

Enter the ias_admin password you specified during the SSO install above. Click on Next to continue.


Enter the orcladmin password from the IDM install performed in Part 2. Click on Next to continue.


Enter SYS password for the database created in Step 2.


I shutdown SSO before but if you didn’t, the installer will do that now. Click on OK to continue.


Click the on the Install button.


Once the install finishes you’ll be prompted to run the script. Perform that task and click on the OK button.


After the configuration assistants execute successfully the patch will be completed:


Click on Exit.

On the IDM server execute the script with -op3, which sets the Oracle Internet Directory version back to 11g Release 1 (11.1.1).

Set the environment variables setup the IDM environment:

oracle@vm2 ~]$ ORACLE_HOME=/u01/app/oracle/product/mid11g/Oracle_IDM1; export ORACLE_HOME
[oracle@vm2 ~]$ ORACLE_INSTANCE=/u01/app/oracle/product/mid11g/IDMasinst_1; export ORACLE_INSTANCE
[oracle@vm2 ~]$ TNS_ADMIN=$ORACLE_INSTANCE/config; export TNS_ADMIN

[oracle@vm2 ~]$ORACLE_HOME/perl/bin/perl $ORACLE_HOME/ldap/bin/ vm2 3060 -nonssl oid1 OIDDB password password –op3

Change orcldirectoryversion to and enable entrycache

Verify the install was successful by logging in to Oracle Delegated Administration Services. You should be redirected to Oracle Single Sign-On and if your able to login in successfully everything should be installed correctly.

The URL where vm7 is the server name for the application tier:


Part 4 will cover the Portal and SOA Installation.