OVM Storage Repositories: Don’t overspend!

During the last few months, I have been asked a couple of times to assist in solving problems with crashing Virtual Machines in Oracle VM environments.

Failing disk checks on normal filesystems, read-only filesystems.

* Fsck could not correct all errors, manual repair needed [ !! ]
Give root password for maintenance
(or type Control-D to continue):

Kernel panic – not syncing: Attempted to kill init!

These are some of the errors I have seen during startup of VM’s.

In all of these cases the symptoms were the same. Virtual Machines (VM’s) were crashing and restarting them failed.
While watching the console windows during startup, they all seem to have problems with their filesystems. The errors occured on VM’s which, the engineers assured me, had enough disk space available.

After taking a closer look at the Storage Repository in which the VM’s were created, I found that the engineers had assigned more virtual disk space to the VM’s in Storage Repository than there was physical diskspace available. That is possible because the VM’s were created using Sparse Allocation.
Sparse Allocation creates a sparse disk, so the size of the disk is initially small and increases as it is used. Sparse allocation is faster than using Non-Sparse Allocation when creating a virtual machine. This is a great feature and has some other nice advantages, like faster migration, etc..

The caveat however, lies in the fact that you can overspend your physical diskspace.
When VM’s grow, they take up more and more space in your Storage Repository. Until ….. you run out of physical diskspace!! The Operating System (O/S) within the VM however, thinks that there is enough space. At that point the O/S within the VM’s start having i/o errors, choking up and finally crashing the VM.

To solve the problems start by creating some free space in the Storage Repository, either by extending the iSCSI LUN, NFS Share or by simply moving a VM to another Storage Repository. After the free space has been created, come some old fashion linux filesystem repair actions.

I found several different scenario’s after startup of VM’s and not all of them are recoverable!

• At startup, the O/S forced checks and corrections of filesystems and was able to repair any inconsistencies automatically and the system boots up correct.
  Problem solved

• At startup, the O/S forced checks and corrections of filesystems and was unable to correct all errors and forced a manual repair.
  After running fsck on the filesystem(s) in maintenance mode, the system boots up correct.
  Problem solved

• At startup, the O/S failed with a Kernel Panic. Outch, this can be very nasty!
  This usually means that there is something wrong on a part of either the boot or root filesystem, needed by the kernel.
  Change the VM to start from DVD/ISO* and boot into Linux Rescue mode. In rescue mode I was able to repair a filesystem only ones. In other cases the filesystems had sustained to much damage and the VM’s needed to be re-created/re-installed.

In one case the customer was very lucky. This VM had two virtual disks, one for O/S and one for Oracle software and database. The O/S disk was unrecoverable, but the other disk had no problems. The VM had just recently been migrated from another OVM Server. The ‘old’ VM was stil available and very little changes were made to the O/S. I was able to migrate the O/S disk from the old OVM Server and start the VM again.

Just to see what happens when you procede with filesystem repair on a filesystem with to much damage. I ran an fsck -y on this device. This ended up with a very large Lost&Found directory filled with #xxxxxx files and directories. Not quite the filesystem you can boot from. :-)

I wrote this article not to show you how to repair disk corruptions.
There ise enough information around to help you with that.
However, you can prevent this from happening!

Don’t overspend disk space in your Oracle VM environment!

(Or at least put some strict monitoring on physical diskspace usage)

* Since it is not possible to boot a PVM from DVD/ISO, change the VM to HVM.
see MOS Doc ID
884085.1: Oracle VM: How to configure a guest Virtual Machine to boot from CDROM/DVDROM

Quick & Easy migrate VM from Oracle VM 2.x to 3.x

I have come across customers running their Virtual Machines (VM) on Oracle VM 2.x wanting to migrate them to Oracle VM 3.x
Most of the time the engineers think the migration to 3.x is a lot of work. I have seen situations where they installed fresh new VM’s on Oracle VM 3.x  and migrated the applications or they performed full system restores on them. A lot of work …, more than migrating!

The migration of Oracle VM 2.x to 3.x is, simply put, nothing more than importing the VM as a template into Oracle VM 3.x and create a new VM based on this template.

As of Oracle 3.x the only way to import a template, is to upload them using ftp of http(s).
A standard approach is to copy the img and cfg files to local storage on, for example, a ftp- of webserver and from there upload them to Oracle VM 3.x as a template. In al this can be a very time consuming job due to network connectivity. Especially when moving to and from DMZ’s through firewalls.

In this article I wil demonstrate a quick and easy way to migrate a VM from Oracle VM 2.x to 3.x
Without the hassle of first copying them to a local disk storage and the uploading them to Oracle VM 3.x.

Prerequisites:
Network connectivity, ftp or http(s), between the OVM Server 2.x and the new OVM Server 3.x

Steps to a quick and easy migration

Make the VM files available on the OVM Server 2.2

Shutdown the VM

[root@vm1 ~]# poweroff

Logon to the OVM Server
Check if the VM is really down.

[root@ovmserver1]# xm list
Name                            ID   Mem VCPUs      State   Time(s)
1548_vm1                         8 13312     4     -b---- 544793.2

Stop iptables (if running) for the duration of the migration.
Or create a rule that allows http access to the OVM Server is you prefer.

[root@ovmserver1]# service iptables stop
Flushing firewall rules:                                   [  OK  ]
Setting chains to policy ACCEPT: filter                    [  OK  ]
Unloading iptables modules:                                [  OK  ]

Change directory to the location of the img and vmcfg files of the VM.

cd /OVS/running_pool/1548_vm1
ls -l

-rw-r--r-- 1 root root          280 Nov 24  2011 README
-rw-r--r-- 1 root root  13316728320 Jun 10 17:11 System.img
-rw-rw-rw- 1 root root 209715200000 Sep  6 08:04 u01.img
-rw-r--r-- 1 root root          569 Sep  6 08:11 vm.cfg

Make a backup of the vm.cfg and remove the mac address and xenbr0 from it.
A new mac address and network will me provided by OVM 3.x later.

cp vm.cfg vm.cfg.org
vi vm.cfg

Before

vif = ['bridge=xenbr0,mac=00:16:3E:44:0D:7F,type=netfront']

After

vif = ['type=netfront']

Now to make the file available for direct upload we need a http of ftp server on OVM 2.x
I do not want to install additional software on the OVM Server.
Luckily Oracle (xen) uses a lot of python!!
Python has the SimpleHTTPServer module which provides a simple http Request handler.

Make sure you are in the directory where the img and vm.cfg files are located.

pwd
/OVS/running_pool/1548_vm1

Start a simple Http server

python -m SimpleHTTPServer 80
Serving HTTP on 0.0.0.0 port 80 ...

Leave this session open for the duration of the migration.

Import the 2.x VM as a template in 3.x using OVM Manager

Open the OVM Manager

Navigate to the Repository tab and go to the ‘VM Templates’.

Click on ‘Import VM Template’

Provide the url to the img and vm.cfg files
Put them all together in the the ‘VM Template URL’s’ box of the Import window.

http://10.x.x.10/System.img
http://10.x.x.10/u01.img
http://10.x.x.10/vm.cfg

Substitute the ip-adress with the ip-adress of your OVM Server 2.x

The files will now be uploaded (Or downloaded, depending on perspective)

Now we have a template called System.img
You can change this to a more descriptive name.

Now change the template to fit your needs.

Check/change name, netwerk interface, boot order, etc.

Create a new VM based on this template.

After creation has completed, change properties of the new VM as needed.
Note the MAC address(es) of the interface(s).
This might be needed later on during configuration of OS Guest.

Start the VM using Oracle VM Manager.

Open the console and logon as root.

It might be that you don’t see your network interface (eth0), only your Local Loopback address.

Most of the times you will find the old mac addess is still in ifcfg-eth0 (and -eth1,eth…. if used)
Also there may be a reference to it in /etc/udev/rules.d/60-net.rules or /etc/udev/rules.d/70-persistent-net.rules (depending on OS version)

Put the correct mac address in the ifcfg-eth0 files.

vi /etc/sysconfig/networking/devices/ifcfg-eth0

remove the udev file

rm -f /etc/udev/rules.d/60-net.rules

or

rm -f /etc/udev/rules.d/70-persistent-net.rules

Reboot

When the VM come back online, you should have the correct network interfaces back.

Finally.

Check /etc/sysctl.conf for ‘old’ Xen settings.
For example: xen.independent_wallclock
Adjust or remove them as needed.

Cleanup

• Remove the VM from your old OVM 2.x server.

Or at least remove it from auto-startup.

• Remove the template from your Storage Repository.

It takes up space and you risk someone re-using the template. This might cause ip conflicts, etc…


Enjoy migrations …

Oracle 12c Revealed Demonstration

Last night my employer AMIS Services B.V. organized the seminar "Oracle Database 12c revealed". During this evening AMIS Oracle professionals offered the first opportunity to see the innovations in Oracle Database 12c in action in the Netherlands!
AMIS specialists took part in a year of beta testing. This was demonstrated, showing what's new and how we will use Oracle databases the next few years!

In one of the parallel session I demonstrated a few features of the new Oracle database 12c Multitenant option.

The full presentation was put on Slideshare.
Oracle 12c Revealed Demonstration