Upgrade VM Manager 2.1.5

 
Just before Xmas one of our VM Servers came to a rather unfortunate end and now needs to be rebuilt.   However, I couldn’t delete the server from VM Manager since the agent was down.   There may be a backend update you can perform but apparently you can force a delete in later versions of VM Manger.
At first I downloaded VM Manager 3 but found this in the documentation:

Oracle VM Release 3.0 is not backwardly compatible with previous Oracle VM releases. If you are using Oracle VM Release 2.x you cannot upgrade to Release 3.0.3. You must reinstall your Oracle VM Servers and Oracle VM Manager. You can, however, reuse your virtual machine templates from Oracle VM Release 2.x by importing them into Oracle VM 3.0. http://docs.oracle.com/cd/E26996_01/e18548/BABEHCGC.html



Well, I didn’t really want to have to upgrade all the VM Servers at the same time.   However, version 2.2 is backwards compatible:

Oracle VM Manager is backwardly compatible with previous Oracle VM Server releases. To make sure you are not using a newer version of Oracle VM Server that cannot be managed Oracle VM Manager, you should upgrade Oracle VM Manager to Release 2.2 before you upgrade your Oracle VM Servers to Release 2.2.

http://docs.oracle.com/cd/E15458_01/doc.22/e16206/toc.htm#BEJFHFDB

I have never upgraded VM Manager before so I wasn’t sure what to expect.   The documentation said its easy and quick but deep inside a voice say saying, “Yeah Right!”
So, here are the steps I followed to upgrade VM Manager from version 2.1.5 to 2.2:

1. Backup existing VM Manager
   - Full backup of /opt directories oc4j and ovs-manager
   - Execute /opt/ovs-manager-2.1/bin/backup.sh
   - Took a manual export of OVS schema just to be safe.
2. VM Manager needs to be running.
3. Mount the VM Manager upgrade ISO on the server as root:
   [root@myserver patches]# mount -o loop,ro OracleVM-Manager-2.2.0.iso /mnt/OVM22
4. Execute runInstaller.sh to upgrade VM Manager

[root@myserver OVM22]# pwd/mnt/OVM22[root@myserver OVM22]# sh runInstaller.sh
Welcome to Oracle VM Manager 2.2
Please enter the choice: [1|2|3]1. Install Oracle VM Manager2. Uninstall Oracle VM Manager3. Upgrade Oracle VM Manager
Starting Oracle VM Manager 2.2 upgrade ...
Backup the database before upgrade is highly recommended, to backup the database now, choose 'N' and run:
/opt/ovs-manager-2.1/bin/backup.sh
Are you sure you want to upgrade Oracle VM Manager from version 2.1.5 to 2.2 ? [y|N] y
Please enter the password for database account 'OVS':
Now upgrading OC4J ...
Please enter the password for account 'oc4jadmin':
Starting OC4J ... Done.
Now upgrading the database schema ... Done.
Now upgrading the Oracle VM Manager application ...
Done.
Please enter the keystore password for the Web Service:
Confirm the password:
Setting keystore password for Web Service ... Done
Do you want to use HTTPS access for Oracle VM Manager (Y|n)?Y
Configuring OC4J to use HTTPS ... Done
Stopping OC4J ... Done
Starting OC4J ... Done
Upgrade Oracle VM Manager successfully.
Please check the log /var/log/ovm-manager/upgrade_2.1.5_2.2.log for more information.
To access the Oracle VM Manager 2.2 home page go to:
https://127.0.0.1:4443/OVS
To access the Oracle VM Manager web services WSDL page go to:
https://127.0.0.1:4443/OVSWS/LifecycleService.wsdl
https://127.0.0.1:4443/OVSWS/ResourceService.wsdl
https://127.0.0.1:4443/OVSWS/PluginService.wsdl
https://127.0.0.1:4443/OVSWS/ServerPoolService.wsdl
https://127.0.0.1:4443/OVSWS/VirtualMachineService.wsdl
https://127.0.0.1:4443/OVSWS/AdminService.wsdl
To access the Oracle VM Manager help page go to:
https://127.0.0.1:4443/help/help

That’s it, its as simple as they said it would be.   They also make the same claim about upgrading the VM Servers themselves.   I plan on taking a look at that next week.

ORA-01017: invalid username/password installing VM Manager 2.1.5 on 11gR2

While installing VM Manager 2.1.5 today an error occurred during the install.   I’m using an existing database instead of Oracle XE and its version 11.2.0.1.

Deploying Oracle VM Manager application to OC4J container.
Creating connection pool … Failed
Please see /var/log/ovm-manager/oc4j.log for more information.
From the oc4j.log:
2011-03-30 15:01:36.381 WARNING J2EE DS-00001 Exception occurred testing connection. Exception: java.sql.SQLException: ORA-01017: invalid username/password; logon denied



Nothing turned up on Metalink.  It didn’t make much sense to me until I remember that passwords by default in 11g are case sensitive.   I disabled the feature by issuing:

SQL> alter system set SEC_CASE_SENSITIVE_LOGON=false scope=spfile;

and restarted the database.   When I tried to install VM Manager again it was successful.   I plan to upgrade it to the latest version, 2.2.1 and will check to see if that version supports case insensitivity.

Hardware Partitioning with Oracle VM

I didn’t realize that Oracle VM could do hardware partitioning until the question was raised on Oracle-L and I looked into it.   I use Oracle VM for some Fusion Middleware and R12 environments but nothing overly complex.

I checked the Oracle VM FAQ document and one of the questions was, “How does partitioning relate to software licensing?”   The answer points to the following document which talks software vs hardware partitioning:

http://www.oracle.com/corporate/pricing/partitioning.pdf

That document refers to the following article which explains how to configure hardware partitioning:
http://www.oracle.com/technology/tech/virtualization/pdf/ovm-hardpart.pdf

The information can also be found on Oracle’s Wiki:
http://wiki.oracle.com/page/Hard+partitioning

Oracle VM Server – Sharing /OVS via NFS

In order to enable High Availability for a server pool you have to have access to shared storage using either clustered OCFS2 on SAN, or ISCSI storage, or NFS on NAS. Lets say you don’t have access to that type of infrastructure or simply don’t need some of the HA features (live migration, automatic restart of failed VMs). There is a way to take advantage of some of the other cluster features without setting this up.

Oracle VM Server comes packaged with a NFS server, so you can share out the /OVS directory and mount it on other servers. This will provide you with the ability to cold start VM’s on different servers in your server pool. This eases the headache of manually moving around VM’s if a server is maxed on resources or diskspace. 

In our case, this works well for our DEV environment since we create/destroy/park a lot of VM’s. We have some older servers which don’t have a lot of storage and that which is does have is dreadfully slow.  We have a new server with plenty of blazingly fast disk and even over NFS its faster than local.

Scenario: For the steps below I’m assuming you have at least two independent Oracle VM servers, meaning they both reside in their own server pools and have been added to VM Manager. The goal is to export the filesystem from server A, mount it on B and add B to the same server pool as A.

1. Disable the Firewall. Disable the iptables firewall or add a rule. Since this is a dev environment I disabled iptables.
[root@server_a ~]# service iptables stop
Flushing firewall rules: [ OK ]
Setting chains to policy ACCEPT: filter [ OK ]
Unloading iptables modules: [ OK ]

To disable iptables so it doesn’t start on the next reboot use chkconfig.  Chkconfig will show you which runlevels a service is enabled in and provide you with the ability to turn it off.

[root@vmserver_a ~]# chkconfig --list  iptables
iptables 0:off 1:off 2:on 3:on 4:on 5:on 6:off
root@vmserver_a ~]# chkconfig iptables off
[root@vmserver_a ~]# chkconfig --list iptables
iptables 0:off 1:off 2:off 3:off 4:off 5:off 6:off

2. Delete the server pool for server  B. Login to VM Manager and delete the server pool for B. Click on the Server Pools tab, select the Server Pool B and click on the Delete button. You will see a screen similar to the following:


Do not select either option and click on Yes.

3. Export /OVS on Server A. Add the following to the /etc/exports file on Server A.

/OVS vmserver_b.domain.name(rw,sync,no_root_squash)


4. Start NFS on Server A. Execute the following on server A.

[root@vmserver_a OVS]# service nfs start
Starting NFS services: [ OK ]
Starting NFS quotas: [ OK ]
Starting NFS daemon: [ OK ]
Starting NFS mountd: [ OK ]

Set NFS to start automatically on reboot:
[root@vmserver_a OVS]#  chkconfig nfs on

5. Mount NFS export from Server A on Server B
a. First stop the Oracle VM Server services on Server B.  Otherwise /OVS will be busy and you won’t be able to unmount it.

[root@vmserver_b init.d]# service ovs-agent stop
OVSAgentServer going down...
OVSAgentServer stopped.
[root@vmserver_b init.d]# service ovsrepositories stop
Stopping OVS Storage Repository Mounter...
OVS Storage Repository Mounter Shutdown: [ OK ]

b. Unmount /OVS and remount it as /OVS_OLD 

[root@vmserver_b init.d]# umount /OVS

c. Edit /etc/fstab to mount /OVS from vmserver_a. I also mounted the local /OVS directory under /OVS_OLD in case there were vm’s I wanted to transfer.

Fstab:
/dev/sda1 /OVS_OLD ocfs2 defaults 1 0
Vmserver_a:/OVS /OVS nfs rw,bg,hard,nointr,tcp,vers=3,timeo=300,rsize=32768,wsize=32768,actimeo=0
[root@vmserver_b init.d]# mount /OVS_OLD
[root@vmserver_b init.d]# mount /OVS

d. Restart the OVS services on Server B

[root@vmserver_b init.d]# service ovs-agent start
OVSAgentServer starting...
OVSLogServer starting...
OVSAgentServer starting...
OVSMonitorServer starting...
OVSPolicyServer starting...
OVSAgentServer started.
[root@vmserver_b init.d]# service ovsrepositories start
Starting OVS Storage Repository Mounter...
OVS Storage Repository Mounter Startup: [ OK ]

6. Add Server B to Server A’s server pool. Login in to Oracle VM Manager, select the Servers tab and click on the Add button. You will see a screen similar to the one below:


Enter the Server Host/IP for Server B. If its different than the name you want to give it, then enter Server Name as well. Type in the Server Agent Password for Server B and check Virtual Machine Server for Server Type. Click on the Add button when complete.

On the Server Pools tab, in the column for Server Pool A you should see that it now 2 servers in its pool.

Note: I am not an NFS expert so if you have any recommendations on better settings or how to improve performance, feel free to leave a comment.  I searched for recommended NFS settings and a few sites referenced Kevin Closson’s blog. Compared with the settings I used initially, these offered quite an improvement.   There are other ways to improve NFS performance such as using multiple NIC’s but I haven’t look into that yet.

Oracle VM: Did you accidently delete your VM?

Lets say you deleted your VM (by accident or not) but now you want to restore it. Its pretty easy to do unless you selected the option “Remove image files from storage” when you deleted the VM.

To delete a VM:

From Oracle VM Manager, on the Virtual Machines tab, select the VM you would like to delete and from the “More Actions” drop down list and click on the “Go” button. The following confirmation screen will appear:

If you don’t want to clean up the image files from the VM Server manually, select “Remove image files from storage” and click on the delete button.

To clean up the files manually ssh to your Oracle VM Server, go to the /OVS/running_pool and delete it:

[root@ ~]# cd /OVS/running_pool/
[root@ running_pool]# ls
15_RC0001 22_RC0002 28_RC0003
[root@ running_pool]# cd 28_RC0003/
[root@ 28_RC0003]# ls -alrt
total 1001481
-rw-rw-rw- 1 root root 351 Aug 14 08:05 vm.cfg
drwxrwxrwx 5 root root 4096 Aug 14 08:05 ..
drwxrwxrwx 2 root root 4096 Aug 14 08:05 .
-rw-rw-rw- 1 root root 1025507328 Aug 14 08:07 System.img
[root@ 28_RC0003]# cd ..
[root@ running_pool]# rm –rf 28_RC0003

To restore a VM:

For any number of reasons you decide that you want to restore a previously deleted VM. If you didn’t select the “Remove image files from storage” option and you didn’t clean up the files manually, your VM is still on the VM Server. If the VM is no longer on disk then you’ll need to restore from tape and place it back in the /OVS/running_pool directory. (Hopefully you backed it up!)

Login to Oracle VM Manager, goto the Resources tab and select Virtual Machine Images.

Click on the Import button:

Select the second option, Select from Server Pool (Discover and register) and click on Next:

In the Virtual Machine Image Name drop down you should see you VM which is same name as the directory the VM is stored in in /OVS/running_pool. Select and fill in the other fields and click on the Next button. You will be shown a confirmation page, review the screen and click on the Confirm button. At which point you’ll be back to the Virtual Machine Images entry page:

Click on the Approve button. You will be shown another confirmation page, click on the Confirm button.

You should now see the VM in a powered off state. Click the Power On button and you should be good to go!

Oracle VM: Adding a disk to a Guest OS

 

For my first VM, I used the Oracle Enterprise Linux template provided by Oracle.   Depending on which template you download you either have 4GB or 10GB of disk space configured.

From the README here are the details for two templates.

 

Template                   Kernel                    VCPU RAM Storage
------------------------   ------------------------- ---- --- -------

OVM_EL5U2_X86_64_PVM_10GB  2.6.18-92.1.17.0.2.el5xen    2 2GB 10GB
OVM_EL5U2_X86_64_PVM_4GB   2.6.18-92.1.17.0.2.el5xen    1 1GB 4GB

Depending on how you want to use the VM, this may not be enough space.  So the first thing your going to want to do is add another disk.   As you can see in the screenshot below, from the Virtual Machine tab select the VM you would like to add disk to and click on the configure button.

Once you click on the configure button you will be presented with the configuration screen.  Click on the storage link (tab) and you should see something similar to the following:

Click on the Create New Virtual Disk button and the following window will appear:

 

As you can see, I have already filled in the requested information. I usually use the mount point I plan on placing this disk under as the Virtual Disk Name .  Above I will be creating a ~10GB disk with the name u01.  Once you click on the Next button you will be provided with a summary screen:

After you click the Confirm button you will be brought back ot the Virtual Machine Configuration screen:

Notice that the disk I created, /u01 has a Disk Status of Creating.  If you login to the Oracle VM Machine your using to host this VM you will see that behind the scenes the dd command is being used to create a 10GB file named u01.img.   Below you can see I am ssh’d into the VM Server under the /OVS/running_pool directory where my VM is located.

Once the disk has finished being created it will be active:

Now that we have added the disk to the VM we need to format and mount it within the guest OS.   Notice the Front-end Device column in the screenshot above.  The value for u01 is xvdb and is the device name we will pass to fdisk to create a partition.   SSH or VNC into your VM as root.

[root@ ~]# fdisk -l
Disk /dev/xvdb: 10.4 GB, 10485760000 bytes
255 heads, 63 sectors/track, 1274 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/xvdb doesn't contain a valid partition table

 

Above is just a clip of the fdisk –l output showing the device /dev/xvdb and that it doesn’t contain any partitions.  Next create a partition:

[root@ ~]# fdisk /dev/xvdb

Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-1274, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-1274, default 1274):
Using default value 1274
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.

[root@ ~]# fdisk -l

Disk /dev/xvdb: 10.4 GB, 10485760000 bytes
255 heads, 63 sectors/track, 1274 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

    Device Boot      Start         End      Blocks   Id  System
/dev/xvdb1               1        1274    10233373+  83  Linux

Now that we have a valid partition, format it as an ext3 filesystem:

[root@ ~]# /sbin/mkfs -t ext3 /dev/xvdb1
mke2fs 1.39 (29-May-2006)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
1281696 inodes, 2558343 blocks
127917 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=2621440000
79 block groups
32768 blocks per group, 32768 fragments per group
16224 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
This filesystem will be automatically checked every 27 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.

Next label the partition so we don’t have to use the device name in /etc/fstab.  Finally add the partition details in /etc/fstab so that its automatically mounted upon startup and mount the partition:

[root@ ~]# /sbin/e2label /dev/xvdb1 /u01
[root@ ~]# vi /etc/fstab
[root@ ~]# cat /etc/fstab
/dev/VolGroup00/LogVol00 /                       ext3    defaults        1 1
LABEL=/boot             /boot                   ext3    defaults        1 2
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
/dev/VolGroup00/LogVol01 swap                    swap    defaults        0 0
LABEL=/u01              /u01                    ext3    defaults        1 2
[root@ ~]# mount /u01
[root@ ~]# df -k
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                      10125560   3011540   6591372  32% /
/dev/xvda1              101086     13229     82638  14% /boot
tmpfs                  1048576         0   1048576   0% /dev/shm
/dev/xvdb1            10072456    153696   9407092   2% /u01

 

Thats it!  To summarize, I used Oracle VM Manager to add a new disk to my VM.  Then I logged into the VM created a partition on the new disk, formatted it, added the details to /etc/fstab and finally mounted it.  Next step, 11G Fusion!

My first Oracle VM install.

We are starting to embark on a new phase of a project and for the DEV environment we are planning to use Oracle VM. The initial phase of this project will be evaluating different software solutions and we believe Oracle VM will be a good fit. Why do I believe it makes sense for us?

1. It will allow us to quickly build environments from templates provided by Oracle and GOLD images we create.
   
2. We will be building many environments to evaluate different software solutions. With Oracle VM we can keep the installations separate and when not in use they can be shut down.
   
3. We can place our DEV servers into a single server pool. This provides us with load balancing and the ability for the vm's to run on any server in the pool. In the event of planned hardware maintenance we can also migrate vm's to other servers in the the pool without downtime.
   
4. As a DBA it provides me with some extra freedoms. I can clone a VM to test patches, upgrades, etc and not disturb the developers.
   
5. I'm sure there are more benefits.
   

Number 2 and 3 will help maximize resources and maybe even be able to run more environments than I could on bare metal.

I'll be honest, I am a skeptic when it comes to virtualization. There have been plenty of discussions on the Oracle-L and OAUG mailing lists with pros/cons and success stories. I believe virtualization makes perfect sense for certain situations such as training. I've been using it myself for the past couple of years to practice/learn things I otherwise couldn't. (Such as RAC.) Time will tell whether or not we will see any benefit from this in our environment. I'll be sure to post a follow up in a couple of months.

Installing Oracle VM was pretty easy but I did have a couple of issues. The VM Manager installed within minutes. Although, my first few attempts failed. I created a CD from the ISO images provided by Oracle but instead of walking down 5 flights of stairs to the server room (bad ankle), I took the 'lazy' approach and copied the CD to local disk on the server. The install proceeded without any errors but the VM Manager wouldn't start properly. On my way out of the building I mounted the CD in the server, reinstalled and it worked flawlessly. Not sure why it caused me problems because I have staged other Oracle products the same way without issue.

Oracle VM Server was a bit more difficult but I believe that is due to my hardware configuration. I am not sure why but when installing any flavor of linux (including the VM Server), the SAN drive is placed ahead of local disk in the boot order. Apparently the BIOS is responsible for providing the details of the correct drive order but everything looked fine. The controller servicing local disk was listed before the controller connected to the SAN and on top of that, it wasn't even configured to be bootable. So by default, the boot loader was being installed on the MBR for the SAN. After a reboot I would receive GRUB errors.

With a full Linux distribution such as RedHat or Oracle Enterprise Linux you are provided with some advanced boot loader configuration options, one of them being to change the drive order. By manually placing the local disk ahead of the SAN, I am now provided with the option to install the boot loader on the MBR of the local disk. I can now install Linux and boot with no issues.

Oracle VM Server however, doesn't provide you with any advanced boot loader configuration options. So I had no choice but to place my SAN controller ahead of the local controller in the boot order and allow it to be bootable. However, when I rebooted I was greeted with "Error 21: Selected disk does not exist". I examined the grub config and noticed that it was trying to boot from "root (hd1,0)". hd1 should be corresponding to my local disk since it was placed second in the boot order and this is where I placed the /boot partition (along with the rest of the VM server install). I wanted to reserve the SAN drive for the /OVS directory, which is where all of the VM's are stored and also the bulk of my disk space.

For some reason tho, local disk is hd0. I edited the grub config to boot from "root (hd0,0)", restarted and it worked flawlessly. I'll admit, i'm not a server admin in a professional sense. I've been using linux since around ~93 (Slackware version 1), so I can figure things out but i've never supported it in a corporate environment with server class hardware before. I wouldn't be surprised if there is some hardware/BIOS config issues at play here.

Now that its installed, the fun begins!