Fusion Middleware – PermGen memory issues

The other day a developer called me to say that the Fusion Middleware console was spinning.   A quick look at the the environment showed a java process consuming 100% of a cpu and in the AdminServer weblogic log there was the following error:

java.lang.OutOfMemoryError: PermGen space

To get the environment going again I had to kill the java process and restart the AdminServer.    I increased the amount of PermGen memory and so far it hasn’t happened again. 

To change the PermGen memory login to the WebLogic Administration Console:


After you login click on Lock & Edit, so you can make changes. Followed by Environment -> Servers under the Domain Structures widget. The right hand side of the screen will update. Click on the AdminServer(admin) link in the table:

Verify that the Configuration tab is selected and click on Server Start:

Scroll down until you see an Arguments text box and update the PermSize and MaxPermSize to the new values. In my example below I have already modified them to be twice their original value:

Once you have made the changes scroll back up to the top of the page and click on the Save button:

Click on the Activate Changes button:

As you can see below, the changes were saved successfully and it says no restarts are necessary. Since we updated the startup parameters tho, they will not come into affect until you perform a restart.

If you want to check out your changes on the server side, login to the server hosting your Portal domain. Change directory to your $PORTAL_DOMAIN/config directory and view the config.xml file. A backup of the config.xml file can be found at $PORTAL_DOMAIN/servers/domain_bak/config_prev/config.xml

From within the WebLogic console, the monitoring capabilities seem to be rather limited.

For example:
From the Fusion Middleware control console I see:

And from the WebLogic Administration Console (Server-> Monitoring -> Performance tab) I can see:

Now, I am very new to WebLogic, so there may be other areas to see more detailed information or a way to start recording it. I have it on my list to investigate further.

If you have Grid Control and the WebLogic management pack there are lots of performance metrics you can view and drill down into.

Lets say you don’t have Oracle Grid Control and would like to confirm the PermGen changes we made above or take a closer took at performance. There are a number of tools available and one of them, VisualVM is packaged with the Sun JDK.

To launch VisualVM go to your JDK_HOME/bin directory and launch jvisualvm. The first time you launch this tool there is a calibration that needs to occur.


When it finishes you’ll be presented with some statistics. After you click OK and the application starts you will see a screen similar to:

The tool automatically discovers the java processes. If you don’t see any here, or not as many as you expect there could be a number of reasons. From the VisualVM documentation:

The circumstances in which VisualVM will not automatically discover JMX agents, and thus the Java applications they expose, are the following:

• The target application is running on the J2SE 5.0 platform and the  -Dcom.sun.management.jmxremote* properties have not  been specified.
• The target application is running on the same host as VisualVM but was started by a different user than the one who started VisualVM. VisualVM discovers running applications using the jps tool, which can only discover Java applications started by  the same user as the one who starts the VisualVM tool.
• The target application is running on a remote host where jstatd is not running or is running but was started by a different user. The jstatd daemon provides an interface that allows remote monitoring applications to connect to Java applications on the host where it is running.

As you can see in my screenshot above, it automatically discovered quite a few WebLogic servers. I ran VisualVM on my application tier which contains Portal and SOA. As you can see next to each entry there is a process id. You can either double click on each one until you find the one your looking for, or say your looking for the Portal AdminServer you can grab it from your server. This screenshot is going to be a bit cluttered but basically i’m executing ps –ef, grepping for AdminServer and looking for a couple of key things in the output. The process id, that the process is for an AdminServer and the domain.


If I go back to VisualVM, find process id 32508 in the list and double click on it I see:

Take a look at the red circle. Wait?! It still says that the PermSize is 128 and MaxPermSize is 256m.

It turns out that the procedure I used above to change the PermSize settings, which modifies the config.xml file will only work if you are using a Node Manager to start the servers.

If you are using the startWebLogic.sh and startManagedWebLogic.sh scripts to start up the AdminServer and managed servers respectively, then you will need to modify those scripts.

Since I am modifying the Java parameters for the AdminServer the script we need to modify is startWebLogic.sh. However, startWebLogic.sh calls a script called setDomainEnv.sh. This script initializes a number of variables, but the one we are looking for is EXTRA_JAVA_PROPERTIES.
This parameter will be modified multiple times, so it can be confusing as to which one to modify. Here is a clip:

----[clip]----
EXTRA_JAVA_PROPERTIES=" -Xms512m -Xmx1024m -XX:PermSize=128m -XX:MaxPermSize=256m -Doracle.home=/u01/app/oracle/product/mid11g/PORTALas_1 -Ddomain.home=/u01/app/oracle/product/mid11g/user_projects/domains/PortalDomain ${EXTRA_JAVA_PROPERTIES}"
export EXTRA_JAVA_PROPERTIES

if [ "${SERVER_NAME}" = "WLS_REPORTS" ] ; then
EXTRA_JAVA_PROPERTIES="-Xms256m -Xmx512m -XX:PermSize=256m -XX:MaxPermSize=512m -Djava.library.path=/tmp/
---[clip]----

The parameter we want to modify is the first one. The second one is actually Java parameters for the Reports manager server. Further down in the file you will see sections for the other managed servers in the Portal environment, such as Forms and Portal. Modify the first EXTRA_JAVA_PROPERTIES PermSize settings:

EXTRA_JAVA_PROPERTIES=" -Xms512m -Xmx1024m -XX:PermSize=256m -XX:MaxPermSize=512m -Doracle.home=/u01/app/oracle/product/mid11g/PORTALas_1 -Ddomain.home=/u01/app/oracle/product/mid11g/user_projects/domains/PortalDomain ${EXTRA_JAVA_PROPERTIES}"

After a restart of the AdminServer and viewing the java process in VisualVM we see:


We have now confirmed the changes have been made successfully. If you notice 9 lines above my circle there is another MaxPermSize entry. I believe that one is being captured by the variable:

MEM_MAX_PERM_SIZE_32BIT="-XX:MaxPermSize=512m"
export MEM_MAX_PERM_SIZE_32BIT

I am not sure what happens in a case where a parameter is specified twice with different values. Does the second one override the first? Is it the one with the maximum value? I’ll tell you what I think in a second.
If you click on the Monitor tab->PermGen you will see the following:


This screen provides you with more information about the state of the java process such as the PermGen memory area. As you can see the size is around 256M and currently the WebLogic process is only using around 175M.

To test what happens when a parameter is specified twice, I put another –XXPermSize before the one in the JAVA_EXTRA_OPTIONS variable and tried a few test scenarios. I set the first to 128m and the second to 256m. When I looked at the PermGen memory in VisualVM the size was around 256. Then I swapped the parameters and restarted the server again. This time the PermGen size was 128m (the value of the second parameter). So it looks like if a parameter is specified twice the last occurrence is used.