Sunday

Fusion Middleware – PermGen memory issues

The other day a developer called me to say that the Fusion Middleware console was spinning.   A quick look at the the environment showed a java process consuming 100% of a cpu and in the AdminServer weblogic log there was the following error:

java.lang.OutOfMemoryError: PermGen space

To get the environment going again I had to kill the java process and restart the AdminServer.    I increased the amount of PermGen memory and so far it hasn’t happened again. 

To change the PermGen memory login to the WebLogic Administration Console:

clip_image002
After you login click on Lock & Edit, so you can make changes. Followed by Environment -> Servers under the Domain Structures widget. The right hand side of the screen will update. Click on the AdminServer(admin) link in the table:
clip_image004
Verify that the Configuration tab is selected and click on Server Start:
clip_image006
Scroll down until you see an Arguments text box and update the PermSize and MaxPermSize to the new values. In my example below I have already modified them to be twice their original value:
clip_image008
Once you have made the changes scroll back up to the top of the page and click on the Save button:
clip_image010
Click on the Activate Changes button:
clip_image012
As you can see below, the changes were saved successfully and it says no restarts are necessary. Since we updated the startup parameters tho, they will not come into affect until you perform a restart.
clip_image014
If you want to check out your changes on the server side, login to the server hosting your Portal domain. Change directory to your $PORTAL_DOMAIN/config directory and view the config.xml file. A backup of the config.xml file can be found at $PORTAL_DOMAIN/servers/domain_bak/config_prev/config.xml

From within the WebLogic console, the monitoring capabilities seem to be rather limited.

For example:
From the Fusion Middleware control console I see:
clip_image016
And from the WebLogic Administration Console (Server-> Monitoring -> Performance tab) I can see:
clip_image018
Now, I am very new to WebLogic, so there may be other areas to see more detailed information or a way to start recording it. I have it on my list to investigate further.

If you have Grid Control and the WebLogic management pack there are lots of performance metrics you can view and drill down into.

Lets say you don’t have Oracle Grid Control and would like to confirm the PermGen changes we made above or take a closer took at performance. There are a number of tools available and one of them, VisualVM is packaged with the Sun JDK.

To launch VisualVM go to your JDK_HOME/bin directory and launch jvisualvm. The first time you launch this tool there is a calibration that needs to occur.

clip_image020
When it finishes you’ll be presented with some statistics. After you click OK and the application starts you will see a screen similar to:
clip_image022
The tool automatically discovers the java processes. If you don’t see any here, or not as many as you expect there could be a number of reasons. From the VisualVM documentation:

The circumstances in which VisualVM will not automatically discover JMX agents, and thus the Java applications they expose, are the following:
  • The target application is running on the J2SE 5.0 platform and the  -Dcom.sun.management.jmxremote* properties have not  been specified.
  • The target application is running on the same host as VisualVM but was started by a different user than the one who started VisualVM. VisualVM discovers running applications using the jps tool, which can only discover Java applications started by  the same user as the one who starts the VisualVM tool.
  • The target application is running on a remote host where jstatd is not running or is running but was started by a different user. The jstatd daemon provides an interface that allows remote monitoring applications to connect to Java applications on the host where it is running.
As you can see in my screenshot above, it automatically discovered quite a few WebLogic servers. I ran VisualVM on my application tier which contains Portal and SOA. As you can see next to each entry there is a process id. You can either double click on each one until you find the one your looking for, or say your looking for the Portal AdminServer you can grab it from your server. This screenshot is going to be a bit cluttered but basically i’m executing ps –ef, grepping for AdminServer and looking for a couple of key things in the output. The process id, that the process is for an AdminServer and the domain.

clip_image024
If I go back to VisualVM, find process id 32508 in the list and double click on it I see:
clip_image026
Take a look at the red circle. Wait?! It still says that the PermSize is 128 and MaxPermSize is 256m.

It turns out that the procedure I used above to change the PermSize settings, which modifies the config.xml file will only work if you are using a Node Manager to start the servers.

If you are using the startWebLogic.sh and startManagedWebLogic.sh scripts to start up the AdminServer and managed servers respectively, then you will need to modify those scripts.

Since I am modifying the Java parameters for the AdminServer the script we need to modify is startWebLogic.sh. However, startWebLogic.sh calls a script called setDomainEnv.sh. This script initializes a number of variables, but the one we are looking for is EXTRA_JAVA_PROPERTIES.
This parameter will be modified multiple times, so it can be confusing as to which one to modify. Here is a clip:

----[clip]----
EXTRA_JAVA_PROPERTIES=" -Xms512m -Xmx1024m -XX:PermSize=128m -XX:MaxPermSize=256m -Doracle.home=/u01/app/oracle/product/mid11g/PORTALas_1 -Ddomain.home=/u01/app/oracle/product/mid11g/user_projects/domains/PortalDomain ${EXTRA_JAVA_PROPERTIES}"
export EXTRA_JAVA_PROPERTIES

if [ "${SERVER_NAME}" = "WLS_REPORTS" ] ; then
EXTRA_JAVA_PROPERTIES="-Xms256m -Xmx512m -XX:PermSize=256m -XX:MaxPermSize=512m -Djava.library.path=/tmp/
---[clip]----


The parameter we want to modify is the first one. The second one is actually Java parameters for the Reports manager server. Further down in the file you will see sections for the other managed servers in the Portal environment, such as Forms and Portal. Modify the first EXTRA_JAVA_PROPERTIES PermSize settings:

EXTRA_JAVA_PROPERTIES=" -Xms512m -Xmx1024m -XX:PermSize=256m -XX:MaxPermSize=512m -Doracle.home=/u01/app/oracle/product/mid11g/PORTALas_1 -Ddomain.home=/u01/app/oracle/product/mid11g/user_projects/domains/PortalDomain ${EXTRA_JAVA_PROPERTIES}"

After a restart of the AdminServer and viewing the java process in VisualVM we see:

clip_image028
We have now confirmed the changes have been made successfully. If you notice 9 lines above my circle there is another MaxPermSize entry. I believe that one is being captured by the variable:

MEM_MAX_PERM_SIZE_32BIT="-XX:MaxPermSize=512m"
export MEM_MAX_PERM_SIZE_32BIT


I am not sure what happens in a case where a parameter is specified twice with different values. Does the second one override the first? Is it the one with the maximum value? I’ll tell you what I think in a second.
If you click on the Monitor tab->PermGen you will see the following:

clip_image030
This screen provides you with more information about the state of the java process such as the PermGen memory area. As you can see the size is around 256M and currently the WebLogic process is only using around 175M.

To test what happens when a parameter is specified twice, I put another –XXPermSize before the one in the JAVA_EXTRA_OPTIONS variable and tried a few test scenarios. I set the first to 128m and the second to 256m. When I looked at the PermGen memory in VisualVM the size was around 256. Then I swapped the parameters and restarted the server again. This time the PermGen size was 128m (the value of the second parameter). So it looks like if a parameter is specified twice the last occurrence is used.

Thursday

11g Fusion Middleware Install Walkthrough – Part 3 Single Sign On

Part 3 describes the process to install and setup SSO on the application tier of my Fusion Middleware Environment. I had originally intended to talk about Portal and SOA here but I didn’t realize how long the SSO portion was.

Oracle Single Sign-On and Oracle Delegated Administration Services Release 10g are required components for Oracle Portal, Forms, Reports and Discoverer Release 10g and Release 11g.

Overview of the Installation Steps
  1. Executing inspre11.pl
  2. SSO Repository Creation
  3. Install SSO 10.1.4.0.1 and upgrade it to 10.1.4.3
Executing inspre11.pl and SSO Repository Creation

More detail for the steps below can be found in Oracle’s documentation.

execute inspre11.pl with –op1. The –op1 flag will enable anonymous bind, disable entry cache and change the OID version from 11.1.1.1.0 to 10.1.4.0.1 so the SSO repository creation assistant will be able to load the schema.

On the database tier set your ORACLE_HOME environment variable to your IDM installation, and ORACLE_INSTANCE to the instance home location. Execute inspre11.pl with the –op1 parameter.
The syntax for inspre11.pl is:

$OID11gR1_ORACLE_HOME/perl/bin/perl \
$OID11gR1_ORACLE_HOME/ldap/bin/inspre11.pl OID_HOST OID_PORT {-ssl | -nonssl} \
OID_COMPONENT DB_CONNECT_STRING ODS_PASSWORD ORCLADMIN_PASSWORD \
{-op1 | -op2 | -op3}

In part 2, I used the same password for all of the accounts, so ODS_PASSWORD and ORCLADMIN_PASSWORD is the same. That’s why you will see password specified twice in my example below:

[oracle@vm2 Oracle_IDM1]$ ORACLE_HOME=/u01/app/oracle/product/mid11g/Oracle_IDM1; export ORACLE_HOME

[oracle@vm2 Oracle_IDM1]$ ORACLE_INSTANCE=/u01/app/oracle/product/mid11g/IDMasinst_1; export ORACLE_INSTANCE

[oracle@vm2 Oracle_IDM1]$ $ORACLE_HOME/perl/bin/perl $ORACLE_HOME/ldap/bin/inspre11.pl vm2 3060 -nonssl oid1 vm2:1521:FMWD password password –op1

Use RepCA to load SSO and other schemas against DB before running -op2



SSO Repository Creation

In part 2 we used a repository creation utility to load the IDM, Portal and SOA 11g schemas. Since SSO is a previous version (10g), we need to use a different repository creation utility (RCU). The 11g RCU is much nicer because it doesn’t need to be installed before you can use it. The 10g SSO RCU needs to be installed like other Oracle products before it can be installed.

The 10g SSO RCU is 32bit, so if your trying to install this on a 64bit environment, you have to type linux32 bash before you launch the runInstaller. Even tho I ran it from a 32bit shell I still encountered issues. Since you don’t need the 10g SSO RCU after you load the SSO schema I decided to install it on an old 32bit server instead of trying to figure out what was wrong.

Unzip the 10g SSO RCU archive and launch the runInstaller:

clip_image002

Click on the Next button.

clip_image004

Verify the inventory directory and change it to suit your environment. Then click on the Next button.

clip_image006

Make sure you execute the orainstRoot.sh script before clicking on the Continue button.
[root@appTier oraInventory]# ./orainstRoot.sh
Creating the Oracle inventory pointer file (/etc/oraInst.loc)
Changing groupname of /home/oracle/oraInventory to oinstall.


clip_image008

Change if necessary and click on the Next button.

clip_image010
clip_image012

I hit this warning, I can’t remember which kernel parameter was at issue but after I reviewed the problem I determined it was ok to proceed. The next time I install it, I’ll take better notes. Click on the Yes button.

clip_image014

If you need other languages select them and click on the Next button.

clip_image016

Verify that Yes is selected so that the RCU assistant launches automatically after the installation is finished and click on the Next button.

clip_image018
Click on the Install button.
clip_image020
clip_image022

The RCU assistant will automatically start and show the screen below:

clip_image024

Click on the Next button:

clip_image026

Enter the path of the Oracle Home directory for the database installed in Part 2. As well, select a log file directory and click on the Next button
.
clip_image028

We need to load the schema and register it, so verify that the Load and Register option is selected and click on the Next button:

clip_image030

Enter the information for the database created in Part 2.

clip_image032

It appears that the SSO Repca utility ignores the SGA_TARGET parameter and looks explicitly for the various SGA parmeters to make sure your database is sized correctly and setup correctly. As long as your SGA_TARGET is set high enough you can quickly login to your database, execute the commands below and continue with the RCU assistant.

SQL> alter system set aq_tm_processes =1;
SQL> alter system set db_cache_size=200M;
SQL> alter system set shared_pool_size=200M;
SQL> alter system set java_pool_size=120M;

clip_image034

Click on Yes, this requirement is only for 10g databases and we are not using 10g Portal.

clip_image036

I am using an regular file system so I selected the first option and clicked on the Next button.

clip_image038

Since I am on a SAN I am using the same directory for all tablespaces but if you are using multiple directories you will need to manually enter the information for each tablespace. Click on the Next button when complete.

clip_image040
clip_image042

By default the system tablespace datafile should be in autoextend in 11g but verify.

clip_image044
clip_image046

Enter the hostname for the server in which you installed Identity Management on in Part 2. By default the SSL port is 3131 but if you are unsure you can look at the installation summary file you should have saved back in Part 2.

clip_image048

Enter the password for orcladmin which you specified while installing Oracle Identity Management in Part 2.

clip_image050
clip_image052

I choose the default and clicked on Next.

clip_image054

Once the SSO repository has been loaded into your database you will be presented with the following screen. Click OK and your finished.

clip_image056

The 10g SSO RCU changes the ODS password for some reason so you need to change it back to the one you used back in Part 2.

Login to your database via SQL*Plus and change the ODS password using alter user ods identified by PASSWORD, where PASSWORD represents the ODS schema password before running the 10g SSO RCU Assistant.

SQL> alter user ods identified by password;
User altered


Login to the server in which Oracle Identity Management was installed and set the ORACLE_HOME, ORACLE_INSTANCE and TNS_ADMIN environment variables to reflect the IDM install and create a wallet:

[oracle@vm2 ~]$ ORACLE_HOME=/u01/app/oracle/product/mid11g/Oracle_IDM1; export ORACLE_HOME
[oracle@vm2 ~]$ ORACLE_INSTANCE=/u01/app/oracle/product/mid11g/IDMasinst_1; export ORACLE_INSTANCE
[oracle@vm2 ~]$ TNS_ADMIN=$ORACLE_INSTANCE/config; export TNS_ADMIN

[oracle@vm2 ~]$ cat $TNS_ADMIN/tnsnames.ora
OIDDB=(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=vm2)(PORT=1521))(CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=FMWD)))

[oracle@vm2 ~]$ $ORACLE_HOME/ldap/bin/oidpasswd connect=OIDDB create_wallet=true
password:
confirm password:
Replication password file exists
password set


Restart Oracle Internet Directory:

[oracle@vm2 bin]$ cd $ORACLE_INSTANCE/bin
[oracle@vm2 bin]$ pwd
/u01/app/oracle/product/mid11g/IDMasinst_1/bin
[oracle@vm2 bin]$ ./opmnctl stopall;
opmnctl stopall: stopping opmn and all managed processes...
[oracle@vm2 bin]$ ./opmnctl startall;
opmnctl startall: starting opmn and all managed processes...


Execute the inspre11.pl script with -op2, which resets the Oracle Internet Directory version and allows you to install Oracle Single Sign-On and Oracle Delegated Administration Services Release 10g (10.1.4.0.1). The -op2 option will also verify the orcldirectoryversion attribute has a value of OID 10.1.4.0.1.

[oracle@vm2 bin]$ $ORACLE_HOME/perl/bin/perl $ORACLE_HOME/ldap/bin/inspre11.pl vm2 3060 -nonssl oid1 OIDDB password password -op2


Install SSO/DAS against 11g OID before running -op3


Above when we executed inspre11.pl with the –op1 flag we used vm2:1521:FMWD for the database information. For some reason, with the –op2 flag, you will get an error. See the following note for details. Since we set the TNS_ADMIN environment variable above, we can pass the tnsnames.ora entry, which is OIDDB.

Subject: Inspre11.Pl Fails With Error Message No Such File Or Directory At Line 310   Doc ID: 876623.1

Install Oracle Single Sign-On and Oracle Delegated Administration Services Release 10g (10.1.4.0.1) and Upgrade to 10.1.4.3

On your application their unzip the SSO media and launch the runInstaller.

clip_image058

Click on the Next button:

clip_image060

Specify the directory for the SSO Oracle Home and click on the Next button.

clip_image062

Select Oracle Application Server Infrastructure 10g for the Single Sign-On (SSO) component and click on the Next button.

clip_image064

We already created the SSO repository, so select the second option Identity Management and click on the Next button.

clip_image066

If you haven’t already review Note 465847.1, it lists the required packages for OEL 5 and if you made the OS changes listed in Part 1 then you should be fine. So I ignored this error and continued.. I’m pretty sure it was looking for packages I have installed but of a lower release level. Click on Next to continue.

clip_image068

You’ll need to execute the root.sh script as part of this install. Click on the Next button to continue.

clip_image070

We are only using SSO and Delegated Administration Services, so I de-selected the other options and clicked on the Next button.
clip_image072

Click on the Next button.

clip_image074

Enter the hostname and port number for your IDM install from Part 2.

clip_image076

Enter the password for the orcladmin account which you specified during the IDM install in Part 2.

clip_image078

Create an Instance Name, I put SSO in front of the instance name to keep it logical. After you enter a password click on the Next button.

clip_image080

Click on the Install button.

clip_image082
clip_image084

During the install you may see this error. It can be safely ignored as per note: 465847.1


clip_image086

Before you click on the OK button above you should download and apply patch patch 5649850 before running run the root.sh script.

To apply this patch set your ORACLE_HOME environment variable to your SSO Oracle Home, unzip the patch, change into the patch directory and execute $ORACLE_HOME/OPatch/opatch apply.

If you get an error:

The Oracle Home /u01/app/oracle/product/mid11g/SSOHome_1 is not registered with the Central Inventory. OPatch was not able to get details of the home from the inventory.

Try upgrading Opatch to OPatch Version: 1.0.0.0.60

Once you have applied the patch click on the OK button.

clip_image088

Once the install has finished the configuration assistants will run. Assuming no issues are encountered you will be presented with the End of Installation screen below:

clip_image090

SSO has been installed but now we need to upgrade it to 10.1.4.3.0:

Upgrade SSO to 10.1.4.3.0

Upgrade Oracle Single Sign-On and Oracle Delegated Administration Services to Release 10g (10.1.4.3.0) by applying the Oracle Identity Management 10g (10.1.4.3.0) Patch Set. You can get the Oracle Identity Management 10g (10.1.4.3.0) Patch Set from My Oracle Support (formerly MetaLink) by searching for Bug or Patch Number 7215628.

Stop SSO:

If you don’t stop it beforehand I believe the installer will stop it for you.

echo "Stopping SSO Components. . ."
cd $SSO_HOME/bin
./emctl stop iasconsole
sleep 5
cd $SSO_HOME/opmn/bin
./opmnctl stopall

Download patch 7215628, unzip it and launch the runInstaller:

clip_image092

Click on Next.

clip_image094

Select the SSO Home from the drop down list and click on Next.

clip_image096
clip_image098

Enter your Metalink credentials if you would like configuration manager installed. I opted not to install it.

image
Enter the ias_admin password you specified during the SSO install above. Click on Next to continue.

clip_image102

Enter the orcladmin password from the IDM install performed in Part 2. Click on Next to continue.

clip_image104

Enter SYS password for the database created in Step 2.

clip_image106

I shutdown SSO before but if you didn’t, the installer will do that now. Click on OK to continue.

clip_image108

Click the on the Install button.

clip_image110

Once the install finishes you’ll be prompted to run the root.sh script. Perform that task and click on the OK button.

clip_image112
clip_image114

After the configuration assistants execute successfully the patch will be completed:

clip_image116

Click on Exit.

On the IDM server execute the inspre11.pl script with -op3, which sets the Oracle Internet Directory version back to 11g Release 1 (11.1.1).

Set the environment variables setup the IDM environment:

oracle@vm2 ~]$ ORACLE_HOME=/u01/app/oracle/product/mid11g/Oracle_IDM1; export ORACLE_HOME
[oracle@vm2 ~]$ ORACLE_INSTANCE=/u01/app/oracle/product/mid11g/IDMasinst_1; export ORACLE_INSTANCE
[oracle@vm2 ~]$ TNS_ADMIN=$ORACLE_INSTANCE/config; export TNS_ADMIN


[oracle@vm2 ~]$ORACLE_HOME/perl/bin/perl $ORACLE_HOME/ldap/bin/inspre11.pl vm2 3060 -nonssl oid1 OIDDB password password –op3

Change orcldirectoryversion to 11.1.1.1.0 and enable entrycache


Verify the install was successful by logging in to Oracle Delegated Administration Services. You should be redirected to Oracle Single Sign-On and if your able to login in successfully everything should be installed correctly.

The URL where vm7 is the server name for the application tier:

http://vm7:7777/oiddas/

Part 4 will cover the Portal and SOA Installation.