Tuesday

Agent Configuration Assistant Failed

Today I hit the following error while trying to perform a standalone Grid Control Agent install:

agent_fail


There could be a number of reasons as to why the agent configuration assistant would fail while you are trying to install the Grid Control Agent.   The commands which failed are listed in the $AGENT_HOME/cfgtoollogs/configToolsFailedCommands file.


[oracle@myserver agent10g] cat cfgtoollogs/configToolFailedCommands
# Copyright © 1999, 2009, Oracle. All rights reserved.
oracle.sysman.emcp.agent.AgentPlugIn


If you search metalink with “oracle.sysman.emcp.agent.AgentPlugIn”  you will get a number of hits but the 2 most useful ones I have looked at are:

Subject:
Troubleshooting the error oracle.sysman.emcp.agent.AgentPlugIn raised by the Agent Configuration Assistant (AgentCA)
Doc ID:
740628.1


Subject:
Troubleshooting the 'oracle.sysman.emcp.agent.AgentPlugIn has failed' error
Doc ID:
734981.1


In my particular case, the agent failed because I entered an incorrect agent password during the install.   Unfortunately there is no way to re-enter the password within OUI at this point so login to your server and  follow the steps below:
  1. cd $AGENT_HOME/bin
  2. ./emctl stop agent
  3. ./emctl unsecure agent
  4. ./emctl secure agent 
    • At this point you will be prompted for the agent registration password.  If you input the incorrect password it will fail.   Run this again until you enter the correct password.

    1. ./emctl start agent
    Now you should be able to login to your Grid Control console and see this target.

    Monday

    T-5 Days

    Until I depart for OpenWorld!   The last time I was there was 2006 and i’m looking forward to it again this year.   I guess it would be appropriate for me, given i’ll be posting from the conference, to state that I will be attending under the Blogger program.  I submitted an application over the summer and was pleasantly surprised to be accepted.   I don’t know what requirements are that your blog has to fulfill so I wasn’t sure how it would play out.

    This year I will be attending a couple of the Hands-on Labs.  On Monday I will be at the Oracle Real Application Clusters 11g Release 2, Part I/II labs and on Friday the Oracle Active Data Guard, Part I/II labs.   The rest of the sessions are a mix between E-Business Suite, Fusion Middleware and some database sessions.

    Of course on Sunday I will be attending the SIG meetings.  OAUG Database SIG, Fusion Middleware SIG and the E-Business Applications Technology SIG..  In the past I have found they have provided alot of useful information.

    I don’t normally attend many of the keynotes, especially if there are some interesting sessions at the same time.   I don’t spend a tremendous amount of time in the tradeshow area either but somehow I always manage to get a fair bit of swag.

    Outside of the conference I can’t wait to visit the Cheesecake Factory in Union Square.   Walking through China town and Fisherman’s Wharf is always a must.  I’ve been to Alcatraz before so I probably won’t go this time.  I’ll also try to check out a few of the places listed in Part 1 and Part 2  of things to do around the Moscone Center.  There is also a blogger meetup scheduled for Tuesday but i’m not 100% sure i’ll be able to attend yet.

    The rest of this week will be spent finishing some tasks, knowledge transfer to a colleague and finalizing my OpenWorld schedule.

    Wednesday

    Oracle VM Server – Sharing /OVS via NFS

    In order to enable High Availability for a server pool you have to have access to shared storage using either clustered OCFS2 on SAN, or ISCSI storage, or NFS on NAS. Lets say you don’t have access to that type of infrastructure or simply don’t need some of the HA features (live migration, automatic restart of failed VMs). There is a way to take advantage of some of the other cluster features without setting this up.

    Oracle VM Server comes packaged with a NFS server, so you can share out the /OVS directory and mount it on other servers. This will provide you with the ability to cold start VM’s on different servers in your server pool. This eases the headache of manually moving around VM’s if a server is maxed on resources or diskspace. 

    In our case, this works well for our DEV environment since we create/destroy/park a lot of VM’s. We have some older servers which don’t have a lot of storage and that which is does have is dreadfully slow.  We have a new server with plenty of blazingly fast disk and even over NFS its faster than local.

    Scenario: For the steps below I’m assuming you have at least two independent Oracle VM servers, meaning they both reside in their own server pools and have been added to VM Manager. The goal is to export the filesystem from server A, mount it on B and add B to the same server pool as A.

    1. Disable the Firewall. Disable the iptables firewall or add a rule. Since this is a dev environment I disabled iptables.
    [root@server_a ~]# service iptables stop
    Flushing firewall rules: [ OK ]
    Setting chains to policy ACCEPT: filter [ OK ]
    Unloading iptables modules: [ OK ]

    To disable iptables so it doesn’t start on the next reboot use chkconfig.  Chkconfig will show you which runlevels a service is enabled in and provide you with the ability to turn it off.

    [root@vmserver_a ~]# chkconfig --list  iptables
    iptables 0:off 1:off 2:on 3:on 4:on 5:on 6:off
    root@vmserver_a ~]# chkconfig iptables off
    [root@vmserver_a ~]# chkconfig --list iptables
    iptables 0:off 1:off 2:off 3:off 4:off 5:off 6:off

    2. Delete the server pool for server  B. Login to VM Manager and delete the server pool for B. Click on the Server Pools tab, select the Server Pool B and click on the Delete button. You will see a screen similar to the following:
    clip_image002

    Do not select either option and click on Yes.

    3. Export /OVS on Server A. Add the following to the /etc/exports file on Server A.

    /OVS vmserver_b.domain.name(rw,sync,no_root_squash)


    4. Start NFS on Server A. Execute the following on server A.

    [root@vmserver_a OVS]# service nfs start
    Starting NFS services: [ OK ]
    Starting NFS quotas: [ OK ]
    Starting NFS daemon: [ OK ]
    Starting NFS mountd: [ OK ]

    Set NFS to start automatically on reboot:

    [root@vmserver_a OVS]#  chkconfig nfs on

    5. Mount NFS export from Server A on Server B
    a. First stop the Oracle VM Server services on Server B.  Otherwise /OVS will be busy and you won’t be able to unmount it.

    [root@vmserver_b init.d]# service ovs-agent stop
    OVSAgentServer going down...
    OVSAgentServer stopped.

    [root@vmserver_b init.d]# service ovsrepositories stop
    Stopping OVS Storage Repository Mounter...
    OVS Storage Repository Mounter Shutdown: [ OK ]


    b. Unmount /OVS and remount it as /OVS_OLD 

    [root@vmserver_b init.d]# umount /OVS


    c. Edit /etc/fstab to mount /OVS from vmserver_a. I also mounted the local /OVS directory under /OVS_OLD in case there were vm’s I wanted to transfer.

    Fstab:
    /dev/sda1 /OVS_OLD ocfs2 defaults 1 0
    Vmserver_a:/OVS /OVS nfs rw,bg,hard,nointr,tcp,vers=3,timeo=300,rsize=32768,wsize=32768,actimeo=0
    [root@vmserver_b init.d]# mount /OVS_OLD
    [root@vmserver_b init.d]# mount /OVS


    d. Restart the OVS services on Server B

    [root@vmserver_b init.d]# service ovs-agent start
    OVSAgentServer starting...
    OVSLogServer starting...
    OVSAgentServer starting...
    OVSMonitorServer starting...
    OVSPolicyServer starting...
    OVSAgentServer started.

    [root@vmserver_b init.d]# service ovsrepositories start
    Starting OVS Storage Repository Mounter...
    OVS Storage Repository Mounter Startup: [ OK
    ]

    6. Add Server B to Server A’s server pool. Login in to Oracle VM Manager, select the Servers tab and click on the Add button. You will see a screen similar to the one below:
    clip_image004

    Enter the Server Host/IP for Server B. If its different than the name you want to give it, then enter Server Name as well. Type in the Server Agent Password for Server B and check Virtual Machine Server for Server Type. Click on the Add button when complete.

    On the Server Pools tab, in the column for Server Pool A you should see that it now 2 servers in its pool.

    Note: I am not an NFS expert so if you have any recommendations on better settings or how to improve performance, feel free to leave a comment.  I searched for recommended NFS settings and a few sites referenced Kevin Closson’s blog. Compared with the settings I used initially, these offered quite an improvement.   There are other ways to improve NFS performance such as using multiple NIC’s but I haven’t look into that yet.

    11g Fusion Install issues, don’t interrupt the install…!

    Here are a couple of tips I found out the hard way with while installing fusion 11g.

    1. SSO - At one particular stage I was in the process of installing single sign-on 10g following these instructions. After following the step listed at Section 10.2 step 5 I needed to perform some maintenance on the server. So I shutdown the environment, performed the work and restarted the servers.

    Before I could continue my install I needed to start up the services such as the database, weblogic server, OID, etc. However, when I went to install OID it would not start properly. Within the OID logs I found:

    [host: 0002] [pid: 8145] [tid: 0] Guardian: WARNING: Connected to incorrect OID base schema version, (version=10.1.4.0.1).

    If I had been paying more attention to the installation steps I would have noticed that step 5 changed the directory version number in the repository to 10.1.4.0.1. After you install OID 10g, you re-run the inspre11.pl script with the –op3 flag to reset the value back to 11g.

    Unfortunately OID needs to be running before you execute inspre11.pl, so I had to update the repository manually with the following SQL:

    update ods.ds_attrstore set attrval = 'OID 11.1.1.1.0' where entryid =1 and attrname = 'orcldirectoryversion';
    commit;

    After that OID 11g started up fine and I could rerun the instpre11.pl script with the –op2 flag. Then I continued with the SSO install.

    Another tip, review the following note before installing OID 10g:

    Subject: Oracle Identity Management 10g (10.1.4.0.1) Release Notes Addendum

    Doc ID:
    465847.1

    It lists some pre-reqs which are not in the install guide and some known issues. The main one for me is that libdb.so.2 was missing. The note describes how to resolve that issue.

    2. Portal – The Portal install itself went without any issues, however, at the configuration wizard stage it would hang creating the weblogic domain. Sometimes I would see java.lang.OutOfMemoryError: PermGen space messages in the install log as well. I found the following notes:

    Subject: FMW 11g “IDM” or “Portal/Forms/Reports/Discoverer” Configuration Wizards on 64bit Plantforms Hang at 0% ‘ Creating Domain’

    Doc ID: 865462.1

    Subject: FMW 11g ‘ Portal/Forms/Reports/Discoverer’ Config Wizard Fails – “Error creating ASInstance”.. Unable to Connec tot ‘Admin Server’ ..PermGen

    I was using the 32bit version of Weblogic which ships with the Sun and JRockit JDK’s on top of OEL 5.3 64bit. The notes say that you must use 64bit FMW 11g with a 64bit JDK on a 64 bit OS. I downloaded the latest JRockit because it apparently doesn’t suffer from permgen issues but I still hit the same problem(althought no permgen messages in the log file..). It wasn’t until I downloaded the latest Sun JDK 1.6.0_16 that the configuration wizard was able to create the domain.

    3. SOA – After installing Portal, SOA seemed to complete without a hiccup… Until I tried to login to SOA. At one point I noticed errors when trying to start the SOA domain with the startWebLogic.sh script:

    javax.security.auth.login.FailedLoginException: [Security:090304]Authentication Failed: User weblogic javax.security.auth.login.LoginException: [Security:090301]Password Not Supplied

    However, I wasn’t being prompted for a password. I could hardcode the password within the startWebLogic.sh script but it would still fail, however with another error:

    <Sep 1, 2009 10:08:33 AM EDT> <Critical> <WebLogicServer> <BEA-000386> <Server subsystem failed. Reason: weblogic.security.SecurityInitializationException: Authentication for user weblogic denied
    weblogic.security.SecurityInitializationException: Authentication for user weblogic denied

    Weblogic checks for the existence of a boot.properties file upon startup. If it exists, it reads the username and password from this file and doesn’t prompt you. I deleted this file and after that I was prompted for the password.. I was hoping that the stored password was incorrect, causing my problems but no such luck.

    I talked to Oracle support and they believed that something was corrupted and the next step would be to recreate the admin account. Since I am using the latest version, they haven’t tested the steps yet and said they’d get back to me in a couple of hours. I searched google and found a few hits with the same problem. A few people tried re-installing and somehow everything worked fine after that. I took a backup of my domain configuration and decided to try and recreate it.

    This time during the install, I selected the optional configuration options for the Administration server and Managed servers. (http://download.oracle.com/docs/cd/E12839_01/doc.1111/e13925/config_screens.htm#CJAIIADH) I noticed the ports being used have already been taken by the Portal install. I chose new ports, finished the install and everything started fine.

    I’m guessing my authentication errors are because my Portal domain was up and SOA was trying to authenticate against it. When installing Fusion you let the configuration wizards create the weblogic domains, you don’t create them with the weblogic assistants. I guess they aren’t smart enough to detect that ports are already in use, which is kind of surprising. This is my first FMW 11g install, so it is possible I messed something up the first time I tried to install SOA.

    4. Repository Creation Utility – Another problem I hit was trying to run the RCU utility for SSO on a 64bit OEL 5 environment. I couldn’t get it to work. Since this doesn’t need to run very often I put it on a 32bit Redhat 4.7 environment and ran it from there without issues.

    So after 4 SR’s and a fair bit of reading I finally have a Fusion Middleware 11g environment. I was surprised to find its much easier to install e-Business Suite than FMW!

    Friday

    Oracle VM: Did you accidently delete your VM?

    Lets say you deleted your VM (by accident or not) but now you want to restore it. Its pretty easy to do unless you selected the option “Remove image files from storage” when you deleted the VM.

    To delete a VM:

    From Oracle VM Manager, on the Virtual Machines tab, select the VM you would like to delete and from the “More Actions” drop down list and click on the “Go” button. The following confirmation screen will appear:

    image

    If you don’t want to clean up the image files from the VM Server manually, select “Remove image files from storage” and click on the delete button.

    To clean up the files manually ssh to your Oracle VM Server, go to the /OVS/running_pool and delete it:

    [root@ ~]# cd /OVS/running_pool/
    [root@ running_pool]# ls
    15_RC0001 22_RC0002 28_RC0003
    [root@ running_pool]# cd 28_RC0003/
    [root@ 28_RC0003]# ls -alrt
    total 1001481
    -rw-rw-rw- 1 root root 351 Aug 14 08:05 vm.cfg
    drwxrwxrwx 5 root root 4096 Aug 14 08:05 ..
    drwxrwxrwx 2 root root 4096 Aug 14 08:05 .
    -rw-rw-rw- 1 root root 1025507328 Aug 14 08:07 System.img
    [root@ 28_RC0003]# cd ..
    [root@ running_pool]# rm –rf 28_RC0003

    To restore a VM:

    For any number of reasons you decide that you want to restore a previously deleted VM. If you didn’t select the “Remove image files from storage” option and you didn’t clean up the files manually, your VM is still on the VM Server. If the VM is no longer on disk then you’ll need to restore from tape and place it back in the /OVS/running_pool directory. (Hopefully you backed it up!)

    Login to Oracle VM Manager, goto the Resources tab and select Virtual Machine Images.

    image

    Click on the Import button:

    image

    Select the second option, Select from Server Pool (Discover and register) and click on Next:

    image

    In the Virtual Machine Image Name drop down you should see you VM which is same name as the directory the VM is stored in in /OVS/running_pool. Select and fill in the other fields and click on the Next button. You will be shown a confirmation page, review the screen and click on the Confirm button. At which point you’ll be back to the Virtual Machine Images entry page:

    image

    Click on the Approve button. You will be shown another confirmation page, click on the Confirm button.

    image

    You should now see the VM in a powered off state. Click the Power On button and you should be good to go!