Oracle FMW and more ...: OVM

Showing posts with label OVM. Show all posts

Tuesday, January 13, 2015

Upgrade OVM 3.2 to 3.3

Lately several people asked me how they should go about with upgrading their OVM 3.2.x environment to 3.3.x
They have the backend repository running on an Oracle Database, which is no longer supported. In release 3.3, only MySQL is supported as a backend repository database.
They were wondering how the migration process will take place.
Furthermore the documentation states that there is only one way to upgrade the OVM Server from 3.2 to 3.3.

See Chapter 5.1 of the Installation and Upgrade Guide for Release 3.3

If you are upgrading from Oracle VM Server Release 3.2 or earlier to Release 3.3:

You must set up two Yum repositories, one as a transitional repository containing the packages for Release 3.2, and a second containing the target packages for Release 3.3. When the Yum repositories are in place, you can run the UpgradeServers.py script on the Oracle VM Manager host to batch update servers as required

This would suggest that the ‘old fashion’ method of booting the OVM Server from CD no longer works.
Well I have tested the upgrade the ‘old fashion’ way and I can say that it will work with a few tweaks.
However, chapter 5.4.4 states that this method is not supported. So let’s see how the supported upgrade methods are carried out.

Test Environment

Let’s start with the setup of a small test environment based on VirtualBox VM’s.

Virtualbox Machines

Hostname	eth0 (LAN)	eth1 (Storage)
storage1.domain.local		10.10.10.170
ovm01.domain.local	192.168.100.171	10.10.10.171
ovmmanager.domain.local	192.168.100.170

ISCSI Storage on storage1.domain.local (simulated by Openfiler)

Volumes (LUN’S)	Size	Description
poolfs	20Gb	OVM Pool filesystem
repfs	50Gb	Storage Repository

OVM Server
OVM Server 3.2.8 installed on ovm01.domain.local
4 Gb memory
12 Gb disk
1 network interface for LAN connection
1 network interface for Storage connection

1 Virtual guest running OEL 6.5

OVM Manager
OVM Manager 3.2.8 installed on ovmmanager.domain.local
4 vCPU’s
8 Gb memory
25 Gb disk
1 network interface for LAN connection

Oracle 11.0.2.4 database as backend repository
Instance name: ovsdb

Upgrade OVM Manager

As with all releases, the upgrade starts with upgrading the OVM Manager.
The upgrade of OVM Manager is pretty straightforward.

It involves

Upgrading the OVM Manager software from 3.2.8 to 3.3.1
Migrating ovsdb from Oracle database to MySQL database

Make sure there is enough diskspace available for the upgrade. The runInstaller states that 7100MB should be available in the filesystem where the current OVM Manager is installed.

Start the installer from the mounted iso or from the cd.
In the example below I have mounted the iso on /media

[root@ovmmanager media]# ./runInstaller.sh

Oracle VM Manager Release 3.3.1 Installer

Oracle VM Manager Installer log file:
/var/log/ovmm/ovm-manager-3-install-2014-10-23-214811.log

Please select an installation type:
1: Install
2: Upgrade
3: Uninstall
4: Help

   Select Number (1-4): 2

Verifying upgrading prerequisites ...
Starting Upgrade ...

Reading database parameters from config ...

Choose option 2: Upgrade

Note. You can also start the installer with the Upgrade option directly on the commandline

./runInstaller.sh --installtype Upgrade

Provide the database password of the current ovs schema in the Oracle database.

Database Repository
==========================
Please enter the current Oracle VM Manager database password for user ovs: *******

Provide the current admin password, this is the password you use to logon to the OVM Console.

Oracle VM Manager application
=============================
Please enter the current Oracle VM Manager application password for user admin: ********

Now provide a password for the weblogic user for the new Weblogic 12c install. You can choose to make up a new password or just use the same as the old installation.

Oracle Weblogic Server 12c
==========================
Please enter a *new* Oracle WebLogic Server 12c user password for user weblogic: ********
Please enter a *new* Oracle WebLogic Server 12c user password for user weblogic (confirm): ********

Provide a FQDN name for which a SSL certificate will be created.

Please enter your fully qualified domain name, e.g. ovs123.us.oracle.com, (or IP address)
of your management server for SSL certification generation, more than one IP address are
detected: 192.168.100.170 10.0.3.15 [ovmmanager.domain.local]: ovmmanager.domain.local

The installer verifies the configuration and offers you the choice to continue with the upgrade or abort the installation.

Verifying configuration ...

Upgrading from version 3.2.8.733 to version 3.3.1.1065

Start upgrading Oracle VM Manager:
   1: Continue
   2: Abort

   Select Number (1-2): 1

The upgrade will now proceed.
The console output scrolls down on your screen. I will not bother you with the full output.
You can monitor the progress in another terminal session via:

tail -f /var/log/ovmm/ovm-manager-3-install-YYYY-MM-DD-HHMISS.log

At the end of the upgrade an installation summary is presented, showing the relevant informantion.

Installation Summary
--------------------
Database configuration:
  Database type               : MySQL
  Database host name          : localhost
  Database name               : ovs
  Database listener port      : 49500
  Database user               : ovs

Weblogic Server configuration:
  Administration username     : weblogic

Oracle VM Manager configuration:
  Username                    : admin
  Core management port        : 54321<
  UUID                        : 0004fb00000100002482670af80b9a35

Passwords:
There are no default passwords for any users. The passwords to use for Oracle VM Manager, Database, and Oracle WebLogic Server have been set by you during this installation. In the case of a default install, all passwords are the same.

Oracle VM Manager UI:
  https://ovmmanager.domain.local:7002/ovm/console
Log in with the user 'admin', and the password you set during the installation.

Note that you must install the latest ovmcore-console package for your Oracle Linux distribution to gain VNC and serial console access to your Virtual Machines (VMs).
Please refer to the documentation for more information about this package.

For more information about Oracle Virtualization, please visit:
  http://www.oracle.com/virtualization/

Oracle VM Manager upgrade complete.

Please remove configuration file /tmp/ovm_configX4zw9Q.

A few points to mention that answer some customer questions.

During the upgrade the content of the Oracle database is automatically exported.

Exporting core database, please be patient this may take a long time  ...

Further down the upgrade track, after MySQL database has been created, the export is imported into it.

Importing core database, please be patient this may take a long time ...

Don’t forget to shutdown and delete the Oracle database after the successful upgrade.

After the upgrade to 3.3 the OVM Manager is set to communicate with OVM Server using TLS. Since this was not the case with to OVM Server 3.2, we disable this option until all OVM Servers are upgraded.
To communicate with OVM Servers 3.2, disable TLS1 for SSL-based authentication in release 3.3.

vi /etc/sysconfig/ovmm
USETLS1=NO

After all OVM Servers have been upgraded, change it back to

USETLS1=YES

Remove any temporary files left by the installer.

rm -Rf /tmp/workdir_sel*
rm -Rf /tmp/ovm*

We can now proceed with the upgrade of the OVM Servers.

Upgrade OVM Server

Before you begin.

Make sure you have downloaded the latest installation media from edelivery.
According to an unpublished bug, there is a defect on initial releases of Oracle VM 3.3.1 installation media. This could make some servers hang on boot.

As mentioned earlier, the upgrade of OVM Server from release 3.2 or earlier to release 3.3 via booting from the CD is no longer supported.

We are going to create two Yum repositories.
A transitional repository containing 3.2 packages and a target repository containing 3.3 packages.
Both repositories will be created from the 3.3 installation media.

The repositories need to be accessible through http/https on any webserver.
If you don’t have a webserver available, you can use the Python SimpleHTTPServer module.
We will be using this module on the OVM Manager server as an example.

Creating the Yum repositories

Logon as root on the server.

Create a directory for the repository files.

mkdir -p /var/www/repos

Mount the iso on /media

mount -o loop OVS-3.3.1.iso /media

Copy the entire iso to the http root directory

cp -rp /media/* /var/www/repos/

Start the SimpleHTTPServer

cd /var/www
python –m SimpleHTTPServer 80
Serving HTTP on 0.0.0.0 port 80

Keep this session running until the upgrade has finished.

If you have any additional packages installed, see “5.4.1.2 Adding Packages to a Yum Repository” on how to add them to the repository.

Now configure the repositories in OVM Manager console.

Open the OVM Manager webconsole in a browser.

Go to : Tools and Resources -> Server Update Groups -> GlobalX86ServerUpdateConfiguration

Create the two repositories Transitional and Target
Click on the + to enter the “Create New Server Update Repository” screen.

Transitional repository

Name :	3.3_trans_repo
Repository name :	3.3_trans_repo
URL :	http://ovmmanager.domain.local/repos/Transition
Enabled :	Yes
Package Signature Type :	GPG
Package Signature Key :	http://ovmmanager.domain.local/repos/RPM-GPG-KEY-oracle-ol5

The Name must be exactly as above entered as the UpgradeServers.py will use this Name.

Target repository

Name :	3.3_ovs_repo
Repository name :	3.3_ovs_repo
URL :	http://ovmmanager.domain.local/repos/Server
Enabled :	Yes
Package Signature Type :	GPG
Package Signature Key :	http://ovmmanager.domain.local/repos/RPM-GPG-KEY-oracle

The Name must be exactly as above entered as the UpgradeServers.py will use this Name.

Upgrade the OVM Servers

After the repositories have been created in OVM Manager we can proceed with the upgrade of the OVM Servers.
The upgrade is performed by using the UpgradeServers.py script.

/u01/app/oracle/ovm-manager-3/ovm_tools/bin/UpgradeServers.py

For the upgrade from OVM Server 3.2 to 3.3 this script is the required method.
It is possible to upgrade between 3.3.x versions using the webconsole, however the preferred method is using the script.

With the UpgradeServers.py script you can upgrade one or multiple OVM Servers or complete Server Pools at once.
See 5.4.2 Upgrading Oracle VM Server using the UpgradeServers.py Script for the syntax.

In our example I will upgrade the one OVM Server in the test setup.

On the OVM Manager server

cd /u01/app/oracle/ovm-manager-3/ovm_tools/bin/
./UpgradeServers.py -u admin -v ovm01.domain.local

Enter your OVM Manager password: ******
Tue Jan 13 10:52:03 2015 INFO:
Tue Jan 13 10:52:03 2015 INFO:  UpgradeServers script starting...
Tue Jan 13 10:52:03 2015 INFO:  OVM Manager version: 3.3.1.1065
Tue Jan 13 10:52:03 2015 INFO:  Command line args: ['UpgradeServers.py', '-v', 'ovm01.domain.local']
Tue Jan 13 10:52:04 2015 INFO:  Server: ovm01.domain.local. Transition Server Update Repository: 3.3_trans_repo,        URL/path: http://ovmmanager.domain.local/repos/Transition
Tue Jan 13 10:52:04 2015 INFO:  Server: ovm01.domain.local. OVS Server Update Repository: 3.3_ovs_repo, URL/path:        http://ovmmanager.domain.local/repos/Server
Tue Jan 13 10:52:04 2015 INFO:  Getting update packages list in Server Update Repository: 3.3_ovs_repo, using old       est server: ovm01.domain.local, version: 3.2.8-736
Tue Jan 13 10:52:07 2015 INFO:  Checking servers for non-native packages (those installed after initial server in       stallation)
Tue Jan 13 10:52:08 2015 INFO:  Non-native package status:
Tue Jan 13 10:52:08 2015 INFO:  Non-native package              Status in 3.3_ovs_repo
Tue Jan 13 10:52:08 2015 INFO:  -----------------------------   ----------------------
Tue Jan 13 10:52:08 2015 INFO:  openscap                        OK: package exists
Tue Jan 13 10:52:08 2015 INFO:  openscap-python                 OK: package exists
Tue Jan 13 10:52:08 2015 INFO:  openscap-utils                  OK: package exists
Tue Jan 13 10:52:09 2015 INFO:  No non-generic plug-ins have been found that are in use on existing storage arrays
Type YES to continue with upgrade: YES

Tue Jan 13 10:54:00 2015 INFO:  Evaluating server: ovm01.domain.local, version: 3.2.8-736, for upgrading. [1 of 1 servers].
Tue Jan 13 10:54:00 2015 INFO:  Disabling Server Update Repository: 3.3_ovs_repo
Tue Jan 13 10:54:00 2015 INFO:  Enabling Server Update Repository: 3.3_trans_repo
Tue Jan 13 10:54:00 2015 INFO:  Waiting up to 25 seconds for updates of the Server Update Repositories to complete on server ovm01.domain.local.
Tue Jan 13 10:54:02 2015 INFO:  Finished updating Server Update Repositories.
Tue Jan 13 10:54:03 2015 INFO:  Starting upgrade of server: ovm01.domain.local, type: X86_64, version: 3.2.8-736, using Server Update Repository: 3.3_trans_repo
Tue Jan 13 10:54:04 2015 INFO:  No VMs are on server: ovm01.domain.local, starting server upgrade.
Tue Jan 13 10:55:14 2015 INFO:  Waiting for upgrade of server: ovm01.domain.local, to complete. The server is performing the upgrade. [STOPPING]
Tue Jan 13 10:55:44 2015 INFO:
……
Tue Jan 13 11:02:46 2015 INFO:  Waiting for upgrade of server: ovm01.domain.local, to complete. The server is performing the upgrade. [STOPPING]
Tue Jan 13 11:03:16 2015 INFO:  Waiting for upgrade of server: ovm01.domain.local, to complete. The server is rebooting after the upgrade. [STOPPED]
Tue Jan 13 11:03:46 2015 INFO:  Waiting for upgrade of server: ovm01.domain.local, to complete. The server is rebooting after the upgrade. [STOPPED]
Tue Jan 13 11:04:16 2015 INFO:  Server: ovm01.domain.local, upgraded successfully to version: 3.3.1-1065 (using Server Update Repository: 3.3_ovs_repo).
Tue Jan 13 11:04:16 2015 INFO:  Evaluating if any storage arrays need their plug-ins updated.
Tue Jan 13 11:04:16 2015 INFO:  No plug-ins were found that needed updating.
Tue Jan 13 11:04:16 2015 INFO:  Log file is available at /u01/app/oracle/ovm-manager-3/ovm_tools/bin/UpgradeServers.log
Tue Jan 13 11:04:16 2015 INFO:  UpgradeServers script stopping...

The OVM Server has been upgraded to 3.3.1

Tuesday, December 17, 2013

OVM Storage Repositories: Don’t overspend!

During the last few months, I have been asked a couple of times to assist in solving problems with crashing Virtual Machines in Oracle VM environments.

Failing disk checks on normal filesystems, read-only filesystems.

* Fsck could not correct all errors, manual repair needed [ !! ]
Give root password for maintenance
(or type Control-D to continue):

Kernel panic – not syncing: Attempted to kill init!

These are some of the errors I have seen during startup of VM’s.

In all of these cases the symptoms were the same. Virtual Machines (VM’s) were crashing and restarting them failed.
While watching the console windows during startup, they all seem to have problems with their filesystems. The errors occured on VM’s which, the engineers assured me, had enough disk space available.

After taking a closer look at the Storage Repository in which the VM’s were created, I found that the engineers had assigned more virtual disk space to the VM’s in Storage Repository than there was physical diskspace available. That is possible because the VM’s were created using Sparse Allocation.
Sparse Allocation creates a sparse disk, so the size of the disk is initially small and increases as it is used. Sparse allocation is faster than using Non-Sparse Allocation when creating a virtual machine. This is a great feature and has some other nice advantages, like faster migration, etc..

The caveat however, lies in the fact that you can overspend your physical diskspace.
When VM’s grow, they take up more and more space in your Storage Repository. Until ….. you run out of physical diskspace!! The Operating System (O/S) within the VM however, thinks that there is enough space. At that point the O/S within the VM’s start having i/o errors, choking up and finally crashing the VM.

To solve the problems start by creating some free space in the Storage Repository, either by extending the iSCSI LUN, NFS Share or by simply moving a VM to another Storage Repository. After the free space has been created, come some old fashion linux filesystem repair actions.

I found several different scenario’s after startup of VM’s and not all of them are recoverable!

At startup, the O/S forced checks and corrections of filesystems and was able to repair any inconsistencies automatically and the system boots up correct.
Problem solved

At startup, the O/S forced checks and corrections of filesystems and was unable to correct all errors and forced a manual repair.
After running fsck on the filesystem(s) in maintenance mode, the system boots up correct.
Problem solved

At startup, the O/S failed with a Kernel Panic. Outch, this can be very nasty!
This usually means that there is something wrong on a part of either the boot or root filesystem, needed by the kernel.
Change the VM to start from DVD/ISO* and boot into Linux Rescue mode. In rescue mode I was able to repair a filesystem only ones. In other cases the filesystems had sustained to much damage and the VM’s needed to be re-created/re-installed.

In one case the customer was very lucky. This VM had two virtual disks, one for O/S and one for Oracle software and database. The O/S disk was unrecoverable, but the other disk had no problems. The VM had just recently been migrated from another OVM Server. The ‘old’ VM was stil available and very little changes were made to the O/S. I was able to migrate the O/S disk from the old OVM Server and start the VM again.

Just to see what happens when you procede with filesystem repair on a filesystem with to much damage. I ran an fsck -y on this device. This ended up with a very large Lost&Found directory filled with #xxxxxx files and directories. Not quite the filesystem you can boot from. :-)

I wrote this article not to show you how to repair disk corruptions.
There ise enough information around to help you with that.
However, you can prevent this from happening!

Don’t overspend disk space in your Oracle VM environment!

(Or at least put some strict monitoring on physical diskspace usage)

* Since it is not possible to boot a PVM from DVD/ISO, change the VM to HVM.
see MOS Doc ID
884085.1: Oracle VM: How to configure a guest Virtual Machine to boot from CDROM/DVDROM

Monday, September 16, 2013

Quick & Easy migrate VM from Oracle VM 2.x to 3.x

I have come across customers running their Virtual Machines (VM) on Oracle VM 2.x wanting to migrate them to Oracle VM 3.x
Most of the time the engineers think the migration to 3.x is a lot of work. I have seen situations where they installed fresh new VM’s on Oracle VM 3.x and migrated the applications or they performed full system restores on them. A lot of work …, more than migrating!

The migration of Oracle VM 2.x to 3.x is, simply put, nothing more than importing the VM as a template into Oracle VM 3.x and create a new VM based on this template.

As of Oracle 3.x the only way to import a template, is to upload them using ftp of http(s).
A standard approach is to copy the img and cfg files to local storage on, for example, a ftp- of webserver and from there upload them to Oracle VM 3.x as a template. In al this can be a very time consuming job due to network connectivity. Especially when moving to and from DMZ’s through firewalls.

In this article I wil demonstrate a quick and easy way to migrate a VM from Oracle VM 2.x to 3.x
Without the hassle of first copying them to a local disk storage and the uploading them to Oracle VM 3.x.

Prerequisites:
Network connectivity, ftp or http(s), between the OVM Server 2.x and the new OVM Server 3.x

Steps to a quick and easy migration

Make the VM files available on the OVM Server 2.2

Shutdown the VM

[root@vm1 ~]# poweroff

Logon to the OVM Server
Check if the VM is really down.

[root@ovmserver1]# xm list
Name                            ID   Mem VCPUs      State   Time(s)
1548_vm1                         8 13312     4     -b---- 544793.2

Stop iptables (if running) for the duration of the migration.
Or create a rule that allows http access to the OVM Server is you prefer.

[root@ovmserver1]# service iptables stop
Flushing firewall rules:                                   [  OK  ]
Setting chains to policy ACCEPT: filter                    [  OK  ]
Unloading iptables modules:                                [  OK  ]

Change directory to the location of the img and vmcfg files of the VM.

cd /OVS/running_pool/1548_vm1
ls -l

-rw-r--r-- 1 root root          280 Nov 24  2011 README
-rw-r--r-- 1 root root  13316728320 Jun 10 17:11 System.img
-rw-rw-rw- 1 root root 209715200000 Sep  6 08:04 u01.img
-rw-r--r-- 1 root root          569 Sep  6 08:11 vm.cfg

Make a backup of the vm.cfg and remove the mac address and xenbr0 from it.
A new mac address and network will me provided by OVM 3.x later.

cp vm.cfg vm.cfg.org
vi vm.cfg

Before

vif = ['bridge=xenbr0,mac=00:16:3E:44:0D:7F,type=netfront']

After

vif = ['type=netfront']

Now to make the file available for direct upload we need a http of ftp server on OVM 2.x
I do not want to install additional software on the OVM Server.
Luckily Oracle (xen) uses a lot of python!!
Python has the SimpleHTTPServer module which provides a simple http Request handler.

Make sure you are in the directory where the img and vm.cfg files are located.

pwd
/OVS/running_pool/1548_vm1

Start a simple Http server

python -m SimpleHTTPServer 80
Serving HTTP on 0.0.0.0 port 80 ...

Leave this session open for the duration of the migration.

Import the 2.x VM as a template in 3.x using OVM Manager

Open the OVM Manager

Navigate to the Repository tab and go to the ‘VM Templates’.

Click on ‘Import VM Template’

Provide the url to the img and vm.cfg files
Put them all together in the the ‘VM Template URL’s’ box of the Import window.

http://10.x.x.10/System.img
http://10.x.x.10/u01.img
http://10.x.x.10/vm.cfg

Substitute the ip-adress with the ip-adress of your OVM Server 2.x

The files will now be uploaded (Or downloaded, depending on perspective)

Now we have a template called System.img
You can change this to a more descriptive name.

Now change the template to fit your needs.

Check/change name, netwerk interface, boot order, etc.

Create a new VM based on this template.

After creation has completed, change properties of the new VM as needed.
Note the MAC address(es) of the interface(s).
This might be needed later on during configuration of OS Guest.

Start the VM using Oracle VM Manager.

Open the console and logon as root.

It might be that you don’t see your network interface (eth0), only your Local Loopback address.

Most of the times you will find the old mac addess is still in ifcfg-eth0 (and -eth1,eth…. if used)
Also there may be a reference to it in /etc/udev/rules.d/60-net.rules or /etc/udev/rules.d/70-persistent-net.rules (depending on OS version)

Put the correct mac address in the ifcfg-eth0 files.

vi /etc/sysconfig/networking/devices/ifcfg-eth0

remove the udev file

rm -f /etc/udev/rules.d/60-net.rules

rm -f /etc/udev/rules.d/70-persistent-net.rules

Reboot

When the VM come back online, you should have the correct network interfaces back.

Finally.

Check /etc/sysctl.conf for ‘old’ Xen settings.
For example: xen.independent_wallclock
Adjust or remove them as needed.

Cleanup

Remove the VM from your old OVM 2.x server.

Or at least remove it from auto-startup.

Remove the template from your Storage Repository.

It takes up space and you risk someone re-using the template. This might cause ip conflicts, etc…

Enjoy migrations …