Error creating Database Control

For our production environments we use Oracle Enterprise Manager and Grid Control to monitor our databases and applications. Not all of our dev/test environments are hooked into this for various reasons.. There is one 10g feature i'd like to try out in a sandbox but instead of hooking the environment into Grid Control I thought I would just use Database Control.

Note:278100.1, How To Drop, Create And Recreate DB Control In A 10g Database, describes the process which seems pretty straight forward. However, while running the command to drop or recreate the repository the following error graced my screen:

oravis@myserver=> ./emca -deconfig dbcontrol db

STARTED EMCA at Jan 14, 2008 11:07:40 AM
EM Configuration Assistant, Version 10.2.0.1.0 Production
Copyright (c) 2003, 2005, Oracle.  All rights reserved.

Enter the following information:
Database SID: MYSID 

Do you wish to continue? [yes(Y)/no(N)]: Y
UnsatisfiedLinkError exception loading native library: njni10
Exception in thread "main" java.lang.UnsatisfiedLinkError: get
        at oracle.net.common.NetGetEnv.get(Native Method)
<SNIP>

I have never encountered this error before so the first step was a quick search of metalink, which didn't turn up any useful notes or discussions. A couple seemed to be related but the solutions didn't resolve my issue.

So the next step was to do some more investigation. The njni10 libraries existed under the $ORACLE_HOME/lib and $ORACLE_HOME/lib32 directories, so that wasn't the issue. Next I used the unix command ldd which lists the dependencies for executables or shared objects.

oravis@myserver=> ldd lib32/libnjni10.so
warning: ldd: lib32/libnjni10.so: is not executable
        libclntsh.so.10.1 =>     (file not found)
        libnnz10.so =>   /mnt/app/visdb/1020_64bit/lib32/libnnz10.so
        libkstat.so.1 =>         /usr/lib/libkstat.so.1
        libnsl.so.1 =>   /usr/lib/libnsl.so.1
        libsocket.so.1 =>        /usr/lib/libsocket.so.1
        libgen.so.1 =>   /usr/lib/libgen.so.1
        libdl.so.1 =>    /usr/lib/libdl.so.1
        libsched.so.1 =>         /usr/lib/libsched.so.1
        libc.so.1 =>     /usr/lib/libc.so.1
        libaio.so.1 =>   /usr/lib/libaio.so.1
        libm.so.1 =>     /usr/lib/libm.so.1
        libmp.so.2 =>    /usr/lib/libmp.so.2
        /usr/platform/SUNW,Sun-Fire-V440/lib/libc_psr.so.1

Can you see the problem? The problem is that library libclntsh.so.10.1 doesn't exist for some reason. Since this is a cloned EBS environment I checked the source system and indeed the file is there. To make sure it wasn't a problem specific to this environment I checked some other cloned instances and most of those didn't have the file either.

This library can be recreated by running the following commands:

oravis@myserver=> cd $ORACLE_HOME/rdbms/lib
oravis@myserver=> make -f ins_rdbms.mk client_sharedlib
/mnt/app/visdb/1020_64bit/bin/genclntsh
/mnt/app/visdb/1020_64bit/bin/genoccish
/mnt/app/visdb/1020_64bit/bin/genagtsh /mnt/app/visdb/1020_64bit/lib/libagtsh.so 1.0

If you run ldd against the library again you will see that the dependency issue has been resolved and I could proceed with the directions in the note above to create database control.

oravis@myserver=> ldd lib32/libnjni10.so
warning: ldd: lib32/libnjni10.so: is not executable
        libclntsh.so.10.1 =>     /mnt/app/visdb/1020_64bit/lib32/libclntsh.so.10.1
<SNIP>

Another quick comment. When you run the command to create the database control repository, don't cut and paste your password. It doesn't like it and causes emca to fail. As to why I cut and paste the password? Our passwords are in an encrypted file and it was quicker to cut and paste it than type it in.

The next step is to verify if this is a cloning issue and if so, how to resolve it. So the next time I have to clone an environment i'll be keeping a watchful eye.