Thursday, June 4, 2015

Presenting at OGH DBA / Middleware day 2015

Deliver my presentation "Dynamically scale Weblogic in a private Cloud" on the OGH DBA / Middleware day 2015 today at 13:15.

Friday, April 10, 2015

Creating and scaling Dynamic Clusters using wlst

In my previous article, Creating and scaling Dynamic Clusters in Weblogic 12c, I described the creation and scaling of Dynamic Clusters. I used the Weblogic Console to create the Dynamic Clusters and change the number of servers.

Most of the time you will use some wlst scripting to create and manage your Weblogic environments.
In this article I will show you how to create Dynamic Clusters en how you can scale them.

The example scripts from the Oracle documentation where used as base for the following script.
It is just a simple create script to show you how easy it is to create a Dynamic Cluster via wlst. So no fancy functions and exception handling in there. Yet ...

createDynamicCluster.py
print '--- Set properties for dynamic Cluster creation'
clusterName='dyna-cluster'
serverTemplate='dyna-server-Template'
serverNamePrefix='dyna-server-'
listenAddress='192.168.100.4${id}'
listenPort=8000
listenPortSSL=9000
maxServerCount=2

print '--- Connect to the AdminServer'
try:
  connect('weblogic','Welcome01','t3://hostname.domain.local:7001')
except err:
  print "--- Can't connect to AdminServer, "+err
  sys.exit(2)

print '--- Start an edit session'
edit()
startEdit()

print '--- Creating the server template '+serverTemplate+' for the dynamic servers and set the attributes'
dynamicServerTemplate=cmo.createServerTemplate(serverTemplate)
dynamicServerTemplate.setListenAddress(listenAddress)
dynamicServerTemplate.setListenPort(listenPort)
dynamicServerTemplateSSL=dynamicServerTemplate.getSSL()
dynamicServerTemplateSSL.setListenPort(listenPortSSL)

print '--- Creating the dynamic cluster '+clusterName+', set the number of dynamic servers and designate the server template to it.'
dynamicCluster=cmo.createCluster(clusterName)
dynamicServers=dynamicCluster.getDynamicServers()
dynamicServers.setMaximumDynamicServerCount(maxServerCount)
dynamicServers.setServerTemplate(dynamicServerTemplate)

print '--- Designating the Cluster to the ServerTemplate'
dynamicServerTemplate.setCluster(dynamicCluster)

print '--- Set the servername prefix to '+serverNamePrefix
dynamicServers.setServerNamePrefix(serverNamePrefix)

print '--- Set Calculate Listen Port and Machinename based on server template'
dynamicServers.setCalculatedMachineNames(true)
dynamicServers.setCalculatedListenPorts(true)

print '--- Save and activate the changes'
save()
activate()
serverConfig()


Running the script with wlst will produce the following output and will create a Dynamic Cluster with two Dynamic Servers.

[oracle@wls01 ~]$ ${WL_HOME}/common/bin/wlst.sh createDynamicCluster.py
Initializing WebLogic Scripting Tool (WLST) ...

Welcome to WebLogic Server Administration Scripting Shell

Type help() for help on available commands

--- Set properties for dynamic Cluster creation
--- Connect to the AdminServer
Connecting to t3://wls01.domain.local:7001 with userid weblogic ...
Successfully connected to Admin Server "AdminServer" that belongs to domain "demo_domain".

Warning: An insecure protocol was used to connect to the
server. To ensure on-the-wire security, the SSL port or
Admin port should be used instead.

Start an edit session
Location changed to edit tree. This is a writable tree with
DomainMBean as the root. To make changes you will need to start
an edit session via startEdit().

For more help, use help('edit')

Starting an edit session ...
Started edit session, please be sure to save and activate your
changes once you are done.
--- Creating the server template dyna-server-Template for the dynamic servers and set the attributes
--- Creating the dynamic cluster dyna-cluster, set the number of dynamic servers and designate the server template to it.
--- Designating the Cluster to the ServerTemplate
--- Set the servername prefix to dyna-server-
--- Set Calculate Listen Port and Machinename based on server template
--- Save and activate the changes
Saving all your changes ...
Saved all your changes successfully.
Activating all your changes, this may take a while ...
The edit lock associated with this edit session is released
once the activation is completed.
Activation completed

As you might expect, it is way faster than clicking through the Weblogic Console.


Next step will be to scale the Dynamic Cluster up to four Dynamic Servers.

scaleDynamicCluster.py
print '--- Set properties for dynamic Cluster creation'
clusterName='dyna-cluster'
maxServerCount=4

print '--- Connect to the AdminServer'
try:
  connect('weblogic','Welcome01','t3://wls01.domain.local:7001')
except err:
  print "Can't connect to AdminServer, "+err
  sys.exit(2)

print '--- Start an edit session'
edit()
startEdit()

print '--- Change the maximum number of dynamic servers'
cd('/Clusters/%s' % clusterName )
dynamicServers=cmo.getDynamicServers()
dynamicServers.setMaximumDynamicServerCount(maxServerCount)

print '--- Save and activate the changes'
save()
activate()
serverConfig()

Running the script with wlst will produce the following output and will scale up to four Dynamic Servers.
[oracle@wls01 ~]$ ${WL_HOME}/common/bin/wlst.sh scaleDynamicCluster.py

Initializing WebLogic Scripting Tool (WLST) ...

Welcome to WebLogic Server Administration Scripting Shell

Type help() for help on available commands

--- Set properties for dynamic Cluster creation
--- Connect to the AdminServer
Connecting to t3://wls01.domain.local:7001 with userid weblogic ...
Successfully connected to Admin Server "AdminServer" that belongs to domain "demo_domain".

Warning: An insecure protocol was used to connect to the
server. To ensure on-the-wire security, the SSL port or
Admin port should be used instead.

--- Start an edit session
Location changed to edit tree. This is a writable tree with
DomainMBean as the root. To make changes you will need to start
an edit session via startEdit().

For more help, use help('edit')

Starting an edit session ...
Started edit session, please be sure to save and activate your
changes once you are done.
--- Change the maximum number of dynamic servers
--- Save and activate the changes
Saving all your changes ...
Saved all your changes successfully.
Activating all your changes, this may take a while ...
The edit lock associated with this edit session is released
once the activation is completed.
Activation completed

As mentioned before, the scripts are very limited and just show you how easy it is to create Dynamic Clusters using wlst. The scripts can be made as comprehensive as you need (want) them to be.
I will create some more examples and post them as I get them ready.

Imagine the possibilities when you create scripts you can connect to your monitoring system. Capacity on demand!

Tuesday, April 7, 2015

Creating and scaling Dynamic Clusters in Weblogic 12c

Introduced in Weblogic 12.1.2, dynamic clusters is a great feature to scale your private cloud.
Dynamic clusters provides you easy scaling of Weblogic clusters by adding and removing managed server instances on demand. They contain one or more dynamic servers. These dynamic servers are based on a single servertemplate that guarantees that every member of the cluster is exactly the same.

Creating Dynamic Clusters


Let's take a look at some of the possibilities as we create a dynamic cluster.

I have created a virtualbox environment.
This environment consists of four VM's with the following specs.

  • 2 vCPU's
  • 4 Gb memory
  • 50 Gb disk
  • Oracle Linux 6.6
  • Java 1.7.0_75
  • Weblogic 12.1.3.0.2

I created a simple domain called demo_domain with only an AdminServer and four machines.
After unpacking the domain to the four servers, the nodemanagers where started and are reachable by the AdminServer.


Now let go through the process of creating a dynamic cluster.

Open the Weblogic Console and navigate to Environment -> Clusters
Lock and Edit the domain in the Change Center
note. I make it a good practice always creating a domain in production mode, even in Development and Test.

Create a new dynamic cluster
New -> Dynamic Cluster

Provide the Clustername
Cluster name: dyna-cluster
Click Next

We will start of with a cluster containing two dynamic servers.
Number of Synamic Servers: 2
Server Name Prefix: dyna-server-
Click Next

For this demo all machines will take part.
Select 'Use any machine configured in this domain'
Click Next

Assign each dynamic server unique listen ports
Listen Port for First Server: 8000
SSL Listen Port for First Server: 9000
Click Next

Summary screen
Click Finish

With the creation of the Dynamic Cluster there is also a Server Template created for it.


Server templates

A single server template provides the basis for the creation of the dynamic servers. Using this single template provides the possibility of every member being created with exactly the same attributes. Where some of the server-specific attributes like Servername, listen-ports, machines, etc. can be calculated based upon tokens.
You can pre-create server templates and let Weblogic clone one when a Dynamic Cluster is created.
When none is available a server template is create with the Dynamic Cluster. The name and the listen ports are the only server template attributes that you provide during Dynamic Cluster creation.

Before we activate the changes to the domain, we are going to make a change to the server template.
As an example we are going to demonstrate the use of tokens for server-specific configuration. 

Navigate to Environment -> Clusters -> Server Templates

Click on the name: dyna-server-Template

We are going to use the ${ID} token in the Listen Address
Listen Address: 192.168.100.4${ID}
Click Save

The last digit of the listen address is used to make the listen address dynamic.

Activate changes in the Change Center of the Weblogic Console.
After activation the cluster and two managed servers are created.

We can now start the two servers.

In the previous steps we have added a dynamic cluster with two dynamic servers, based on a single server template, to the domain.

Scaling a Dynamic Cluster


When the capacity is insufficient and you need to scale-up, you can add dynamic servers on demand.
It requires only a few clicks.

Navigate to Environment -> Clusters
Click dyna-cluster

On the Configuration tab go to the Servers tab
Change the Maximum Number of DynamicServers to: 4
Click save

Activate changes in the Change Center of the Weblogic Console.
After activation two Dynamic Servers are added to the Dynamic Cluster.

Start the two new Dynamic Servers and you have doubled you capacity.



Scaling down works exactly the same.
Just lower the Maximum Number of DynamicServers and activate.

A few points to keep in mind when scaling up or down.

Up
  • New dynamic servers are not started upon creation
  • Think before you act with the use of Tokens.
    For example.
    In our demo, the number of Dynamic servers can't grow beyond nine servers, since we use the ${ID} as last digit of the listen address.

Down
  • Dynamic Servers above the new Maximum have to be shutdown before the change can be activated.
  • Dynamic Servers are removed in order, Last -> First
    (In our demo dyna-server-4 gets removed first, then dyna-server-3, etc..)
  • You cannot remove a Dynamic Server directly from the Environment -> Servers page



Tuesday, January 13, 2015

Upgrade OVM 3.2 to 3.3

Lately several people asked me how they should go about with upgrading their OVM 3.2.x environment to 3.3.x
They have the backend repository running on an Oracle Database, which is no longer supported. In release 3.3, only MySQL is supported as a backend repository database.
They were wondering how the migration process will take place.
Furthermore the documentation states that there is only one way to upgrade the OVM Server from 3.2 to 3.3.

See Chapter 5.1 of the Installation and Upgrade Guide for Release 3.3

If you are upgrading from Oracle VM Server Release 3.2 or earlier to Release 3.3:

You must set up two Yum repositories, one as a transitional repository containing the packages for Release 3.2, and a second containing the target packages for Release 3.3. When the Yum repositories are in place, you can run the UpgradeServers.py script on the Oracle VM Manager host to batch update servers as required

This would suggest that the ‘old fashion’ method of booting the OVM Server from CD no longer works.
Well I have tested the upgrade the ‘old fashion’ way and I can say that it will work with a few tweaks.
However, chapter 5.4.4 states that this method is not supported. So let’s see how the supported upgrade methods are carried out.

Test Environment

Let’s start with the setup of a small test environment based on VirtualBox VM’s.

Virtualbox Machines
Hostname eth0 (LAN) eth1 (Storage)
storage1.domain.local 10.10.10.170
ovm01.domain.local 192.168.100.17110.10.10.171
ovmmanager.domain.local192.168.100.170






ISCSI Storage on storage1.domain.local (simulated by Openfiler)
Volumes (LUN’S)SizeDescription
poolfs 20GbOVM Pool filesystem
repfs 50GbStorage Repository





OVM Server
OVM Server 3.2.8 installed on ovm01.domain.local
4 Gb memory
12 Gb disk
1 network interface for LAN connection
1 network interface for Storage connection

1 Virtual guest running OEL 6.5

OVM Manager
OVM Manager 3.2.8 installed on ovmmanager.domain.local
4 vCPU’s
8 Gb memory
25 Gb disk
1 network interface for LAN connection

Oracle 11.0.2.4 database as backend repository
Instance name: ovsdb

Upgrade OVM Manager


As with all releases, the upgrade starts with upgrading the OVM Manager.
The upgrade of OVM Manager is pretty straightforward.

It involves

  • Upgrading the OVM Manager software from 3.2.8 to 3.3.1
  • Migrating ovsdb from Oracle database to MySQL database

Make sure there is enough diskspace available for the upgrade. The runInstaller states that 7100MB should be available in the filesystem where the current OVM Manager is installed.

Start the installer from the mounted iso or from the cd.
In the example below I have mounted the iso on /media

[root@ovmmanager media]# ./runInstaller.sh

Oracle VM Manager Release 3.3.1 Installer

Oracle VM Manager Installer log file:
/var/log/ovmm/ovm-manager-3-install-2014-10-23-214811.log

Please select an installation type:
1: Install
2: Upgrade
3: Uninstall
4: Help

   Select Number (1-4): 2

Verifying upgrading prerequisites ...
Starting Upgrade ...

Reading database parameters from config ...

Choose option 2: Upgrade

Note. You can also start the installer with the Upgrade option directly on the commandline
./runInstaller.sh --installtype Upgrade

Provide the database password of the current ovs schema in the Oracle database.
Database Repository
==========================
Please enter the current Oracle VM Manager database password for user ovs: *******

Provide the current admin password, this is the password you use to logon to the OVM Console.
Oracle VM Manager application
=============================
Please enter the current Oracle VM Manager application password for user admin: ********

Now provide a password for the weblogic user for the new Weblogic 12c install. You can choose to make up a new password or just use the same as the old installation.
Oracle Weblogic Server 12c
==========================
Please enter a *new* Oracle WebLogic Server 12c user password for user weblogic: ********
Please enter a *new* Oracle WebLogic Server 12c user password for user weblogic (confirm): ********

Provide a FQDN name for which a SSL certificate will be created.
Please enter your fully qualified domain name, e.g. ovs123.us.oracle.com, (or IP address)
of your management server for SSL certification generation, more than one IP address are
detected: 192.168.100.170 10.0.3.15 [ovmmanager.domain.local]: ovmmanager.domain.local

The installer verifies the configuration and offers you the choice to continue with the upgrade or abort the installation.
Verifying configuration ...

Upgrading from version 3.2.8.733 to version 3.3.1.1065

Start upgrading Oracle VM Manager:
   1: Continue
   2: Abort

   Select Number (1-2): 1

The upgrade will now proceed.
The console output scrolls down on your screen. I will not bother you with the full output.
You can monitor the progress in another terminal session via:
tail -f /var/log/ovmm/ovm-manager-3-install-YYYY-MM-DD-HHMISS.log

At the end of the upgrade an installation summary is presented, showing the relevant informantion.
Installation Summary
--------------------
Database configuration:
  Database type               : MySQL
  Database host name          : localhost
  Database name               : ovs
  Database listener port      : 49500
  Database user               : ovs

Weblogic Server configuration:
  Administration username     : weblogic

Oracle VM Manager configuration:
  Username                    : admin
  Core management port        : 54321<
  UUID                        : 0004fb00000100002482670af80b9a35

Passwords:
There are no default passwords for any users. The passwords to use for Oracle VM Manager, Database, and Oracle WebLogic Server have been set by you during this installation. In the case of a default install, all passwords are the same.

Oracle VM Manager UI:
  https://ovmmanager.domain.local:7002/ovm/console
Log in with the user 'admin', and the password you set during the installation.

Note that you must install the latest ovmcore-console package for your Oracle Linux distribution to gain VNC and serial console access to your Virtual Machines (VMs).
Please refer to the documentation for more information about this package.

For more information about Oracle Virtualization, please visit:
  http://www.oracle.com/virtualization/

Oracle VM Manager upgrade complete.

Please remove configuration file /tmp/ovm_configX4zw9Q.

A few points to mention that answer some customer questions.

During the upgrade the content of the Oracle database is automatically exported.
Exporting core database, please be patient this may take a long time  ...

Further down the upgrade track, after MySQL database has been created, the export is imported into it.
Importing core database, please be patient this may take a long time ...

Don’t forget to shutdown and delete the Oracle database after the successful upgrade.

After the upgrade to 3.3 the OVM Manager is set to communicate with OVM Server using TLS. Since this was not the case with to OVM Server 3.2, we disable this option until all OVM Servers are upgraded.
To communicate with OVM Servers 3.2, disable TLS1 for SSL-based authentication in release 3.3.
vi /etc/sysconfig/ovmm
USETLS1=NO
After all OVM Servers have been upgraded, change it back to
USETLS1=YES
Remove any temporary files left by the installer.
rm -Rf /tmp/workdir_sel*
rm -Rf /tmp/ovm*

We can now proceed with the upgrade of the OVM Servers.

Upgrade OVM Server


Before you begin.

Make sure you have downloaded the latest installation media from edelivery.
According to an unpublished bug, there is a defect on initial releases of Oracle VM 3.3.1 installation media. This could make some servers hang on boot.

As mentioned earlier, the upgrade of OVM Server from release 3.2 or earlier to release 3.3 via booting from the CD is no longer supported.

We are going to create two Yum repositories.
A transitional repository containing 3.2 packages and a target repository containing 3.3 packages.
Both repositories will be created from the 3.3 installation media.

The repositories need to be accessible through http/https on any webserver.
If you don’t have a webserver available, you can use the Python SimpleHTTPServer module.
We will be using this module on the OVM Manager server as an example.

Creating the Yum repositories

Logon as root on the server.

Create a directory for the repository files.
mkdir -p /var/www/repos
Mount the iso on /media
mount -o loop OVS-3.3.1.iso /media
Copy the entire iso to the http root directory
cp -rp /media/* /var/www/repos/

Start the SimpleHTTPServer
cd /var/www
python –m SimpleHTTPServer 80
Serving HTTP on 0.0.0.0 port 80
Keep this session running until the upgrade has finished.

If you have any additional packages installed, see “5.4.1.2 Adding Packages to a Yum Repository” on how to add them to the repository.

Now configure the repositories in OVM Manager console.

Open the OVM Manager webconsole in a browser.

Go to : Tools and Resources -> Server Update Groups -> GlobalX86ServerUpdateConfiguration



Create the two repositories Transitional and Target
Click on the + to enter the “Create New Server Update Repository” screen.

Transitional repository

Name : 3.3_trans_repo
Repository name : 3.3_trans_repo
URL : http://ovmmanager.domain.local/repos/Transition
Enabled : Yes
Package Signature Type :GPG
Package Signature Key : http://ovmmanager.domain.local/repos/RPM-GPG-KEY-oracle-ol5









The Name must be exactly as above entered as the UpgradeServers.py will use this Name.

Target repository

Name : 3.3_ovs_repo
Repository name : 3.3_ovs_repo
URL : http://ovmmanager.domain.local/repos/Server
Enabled : Yes
Package Signature Type :GPG
Package Signature Key : http://ovmmanager.domain.local/repos/RPM-GPG-KEY-oracle









The Name must be exactly as above entered as the UpgradeServers.py will use this Name.

Upgrade the OVM Servers


After the repositories have been created in OVM Manager we can proceed with the upgrade of the OVM Servers.
The upgrade is performed by using the UpgradeServers.py script.
/u01/app/oracle/ovm-manager-3/ovm_tools/bin/UpgradeServers.py
For the upgrade from OVM Server 3.2 to 3.3 this script is the required method.
It is possible to upgrade between 3.3.x versions using the webconsole, however the preferred method is using the script.

With the UpgradeServers.py script you can upgrade one or multiple OVM Servers or complete Server Pools at once.
See 5.4.2 Upgrading Oracle VM Server using the UpgradeServers.py Script for the syntax.

In our example I will upgrade the one OVM Server in the test setup.

On the OVM Manager server

cd /u01/app/oracle/ovm-manager-3/ovm_tools/bin/
./UpgradeServers.py -u admin -v ovm01.domain.local

Enter your OVM Manager password: ******
Tue Jan 13 10:52:03 2015 INFO:
Tue Jan 13 10:52:03 2015 INFO:  UpgradeServers script starting...
Tue Jan 13 10:52:03 2015 INFO:  OVM Manager version: 3.3.1.1065
Tue Jan 13 10:52:03 2015 INFO:  Command line args: ['UpgradeServers.py', '-v', 'ovm01.domain.local']
Tue Jan 13 10:52:04 2015 INFO:  Server: ovm01.domain.local. Transition Server Update Repository: 3.3_trans_repo,        URL/path: http://ovmmanager.domain.local/repos/Transition
Tue Jan 13 10:52:04 2015 INFO:  Server: ovm01.domain.local. OVS Server Update Repository: 3.3_ovs_repo, URL/path:        http://ovmmanager.domain.local/repos/Server
Tue Jan 13 10:52:04 2015 INFO:  Getting update packages list in Server Update Repository: 3.3_ovs_repo, using old       est server: ovm01.domain.local, version: 3.2.8-736
Tue Jan 13 10:52:07 2015 INFO:  Checking servers for non-native packages (those installed after initial server in       stallation)
Tue Jan 13 10:52:08 2015 INFO:  Non-native package status:
Tue Jan 13 10:52:08 2015 INFO:  Non-native package              Status in 3.3_ovs_repo
Tue Jan 13 10:52:08 2015 INFO:  -----------------------------   ----------------------
Tue Jan 13 10:52:08 2015 INFO:  openscap                        OK: package exists
Tue Jan 13 10:52:08 2015 INFO:  openscap-python                 OK: package exists
Tue Jan 13 10:52:08 2015 INFO:  openscap-utils                  OK: package exists
Tue Jan 13 10:52:09 2015 INFO:  No non-generic plug-ins have been found that are in use on existing storage arrays
Type YES to continue with upgrade: YES

Tue Jan 13 10:54:00 2015 INFO:  Evaluating server: ovm01.domain.local, version: 3.2.8-736, for upgrading. [1 of 1 servers].
Tue Jan 13 10:54:00 2015 INFO:  Disabling Server Update Repository: 3.3_ovs_repo
Tue Jan 13 10:54:00 2015 INFO:  Enabling Server Update Repository: 3.3_trans_repo
Tue Jan 13 10:54:00 2015 INFO:  Waiting up to 25 seconds for updates of the Server Update Repositories to complete on server ovm01.domain.local.
Tue Jan 13 10:54:02 2015 INFO:  Finished updating Server Update Repositories.
Tue Jan 13 10:54:03 2015 INFO:  Starting upgrade of server: ovm01.domain.local, type: X86_64, version: 3.2.8-736, using Server Update Repository: 3.3_trans_repo
Tue Jan 13 10:54:04 2015 INFO:  No VMs are on server: ovm01.domain.local, starting server upgrade.
Tue Jan 13 10:55:14 2015 INFO:  Waiting for upgrade of server: ovm01.domain.local, to complete. The server is performing the upgrade. [STOPPING]
Tue Jan 13 10:55:44 2015 INFO:
……
Tue Jan 13 11:02:46 2015 INFO:  Waiting for upgrade of server: ovm01.domain.local, to complete. The server is performing the upgrade. [STOPPING]
Tue Jan 13 11:03:16 2015 INFO:  Waiting for upgrade of server: ovm01.domain.local, to complete. The server is rebooting after the upgrade. [STOPPED]
Tue Jan 13 11:03:46 2015 INFO:  Waiting for upgrade of server: ovm01.domain.local, to complete. The server is rebooting after the upgrade. [STOPPED]
Tue Jan 13 11:04:16 2015 INFO:  Server: ovm01.domain.local, upgraded successfully to version: 3.3.1-1065 (using Server Update Repository: 3.3_ovs_repo).
Tue Jan 13 11:04:16 2015 INFO:  Evaluating if any storage arrays need their plug-ins updated.
Tue Jan 13 11:04:16 2015 INFO:  No plug-ins were found that needed updating.
Tue Jan 13 11:04:16 2015 INFO:  Log file is available at /u01/app/oracle/ovm-manager-3/ovm_tools/bin/UpgradeServers.log
Tue Jan 13 11:04:16 2015 INFO:  UpgradeServers script stopping...
The OVM Server has been upgraded to 3.3.1



Related Oracle Documention:
Oracle VM Installation and Upgrade Guide for Release 3.3

Related Oracle support notes:
Alert: Verify Sufficient Space in /boot on Your Hypervisor/dom0 Before Upgrading to Oracle VM 3.3.x (Doc ID 1927958.1
During/After the Upgrade to Oracle VM 3.3.1 Using a Dell optiplex 755/780 Machine Does Not Boot Anymore (Doc ID 1932065.1)
Oracle VM Server 3.3.1 Installation Hangs (Doc ID 1918017.1)

Sunday, December 7, 2014

Securing OHS environments with latest SSL TLS protocols and SHA-2 certificates

Customer case

A while ago I was contacted by a customer about their old Oracle Application and Weblogic Server environment.
They were receiving complaints from users that they can't connect to the secure site any longer. Most of the complaints came from users that just recently updated their tablet or smartphone.
After a quick look in the logs of the OHS servers, I found out that the problem had to do with the SSL protocols being used.
The servers were providing connections through either SSLv3 or TLSv1.0, while the devices requested at least TLSv1.1.
The environment comprises of an Oracle HTTP server 10.1.x, for SSO, in front of their Application Server.
For the applications they are using OHS 11.1.1.x. in front of a mix of applications. Varying from oc4j 10.1.2 all the way up to 11.1.1, including Oracle Forms and Reports.
Unfortunately, due to this complexity of components, they were not able to upgrade the environment in time.

SSL Current Situation
 SSL Current Situation

Requirements

The customer asked to provide a solution with the following requirements.

  • Disable the old, insecure, SSLv3
  • Enable TLSv1.1 and TLSv1.2 for all sites
  • Current hostnames for the url’s must not change
  • Support SHA-2 SSL certificates for all sites

Circumstances I had to take into account

  • Oracle HTTP Server (OHS) 10.1.x and 11.1.1.x do not support TLS 1.1 and TLS 1.2.
    This is due to the Oracle NZ layer used by OHS 10g/11g for its SSL implementation which doesn't support TLS 1.1/1.2.
  • There is no support for SHA2 certificates (SHA256 or SHA512) or algorithms in Oracle Application Server 10g (10.1.2.X.X/10.1.3.X.X)
  • SHA2 is certified for Fusion Middleware 11g (11.1.1.X) with caveats
  • As part of their SHA-2 migration plan, Microsoft, Google, and Mozilla have announced that they will stop trusting SHA-1 certificates.
    Google will begin phasing out trust in SHA-1 certificates in November 2014.
  • Replacing the old 11.1.1.x OHS with FMW Webtier 12.1.3.0. is not an option.
    OSSO from the 10.1.x appserver is being used and in FMW Webtier 12.1.x the mod_osso module is no longer supported.
note. Oracle Traffic Director on Exalogic is also based on FMW 11.1.1.x !!


Solution

There are several options to meet the requirements set by the customer.
Unfortunately the best solution, upgrading the environment, cannot yet be implemented.

In this case the requirements were met by placing a reverse proxy in front of the entire environment.
The reverse proxy acts as an SSL terminator for client connections using the latest SHA-2 SSL Certificates.
To encrypt the connection, using TLSv1.0, between the reverse proxy and the backend OHS, I generated Self-Signed SHA-1 certificates compatible with the old servers .

As a reverse proxy I had the choice between using Oracle Fusion Middleware 12c 12.1.3 Webtier or the plain Apache HTTP Server.
I decided to go with Apache HTTP Server.

The reasons for this choice were.

(Security) Updates - (Security) updates are released more frequent for plain Apache than for Webtier
Easier to maintain - The server will be managed by Linux engineers, not the Oracle Engineers
Smaller footprint - I only need the reverse proxy functionality, not all the fancy stuff that comes with Oracle Webtier.

SSL Installed Solution

SSL Installed Solution

Pretty much all requirements were met by using the latest Apache with the correct SSL settings and new SSL Certificates.

For one requirement we needed to play a little trick:

Current hostnames for the url's must not change
After setup of the reverse proxy, all DNS entries for the url's hostnames where changed to the IP-addresses of the reverse proxy.
For the reverse proxy to be able to do its work, I placed the old IP-addresses in the local hosts file of the server running Apache HTTP Server.
So the clients browsers are accessing the url's via DNS resolving to the reverse proxy which on his turn resolves the hostsnames on the backend using /etc/hosts.

Final thoughts

It was not my intension to describe the complete setup of an Apache based reverse proxy here.
There are tons of how-to’s, blogs, etc. that describe all the setups and features.
The main purpose of this article is to make people aware of the fact that there are some changes in SSL security upcoming that can have a direct impact on your environment.

In the case described above, users were already experiencing problems with mobile devices and tablets. And as I finished the setup, their developers discovered that Java 1.8 uses TLSv1.2 by default.
So a problem, they did not yet relate to SSL protocols, was solved in the process.

As reminder

Oracle supports the use of TLSv1.1 and TLSv1.2 as of version FMW 12.1.x
Oracle supports the use of SHA-2 as of FMW 11.1.1.x (with caveats)

Related Oracle support notes:

Does Oracle HTTP Server (OHS) 10g Or Higher Support TLS 1.1 and TLS 1.2? (Doc ID 1503476.1)
Using OHS 12c With TLS 1.1 and 1.2 Protocols as an SSL Reverse-Proxy to OHS 11g (Doc ID 1920143.1)
Is SSLHonorCipherOrder and TLS 1.1/1.2 Supported for Oracle HTTP Server? (Doc ID 1485047.1)
How to Change SSL Protocols (to Disable SSL 3.0) in Oracle Fusion Middleware Products (Doc ID 1936300.1)

Tuesday, December 17, 2013

OVM Storage Repositories: Don’t overspend!

During the last few months, I have been asked a couple of times to assist in solving problems with crashing Virtual Machines in Oracle VM environments.

Failing disk checks on normal filesystems, read-only filesystems.

* Fsck could not correct all errors, manual repair needed [ !! ]
Give root password for maintenance
(or type Control-D to continue):

Kernel panic – not syncing: Attempted to kill init!

These are some of the errors I have seen during startup of VM’s.

In all of these cases the symptoms were the same. Virtual Machines (VM’s) were crashing and restarting them failed.
While watching the console windows during startup, they all seem to have problems with their filesystems. The errors occured on VM’s which, the engineers assured me, had enough disk space available.

After taking a closer look at the Storage Repository in which the VM’s were created, I found that the engineers had assigned more virtual disk space to the VM’s in Storage Repository than there was physical diskspace available. That is possible because the VM’s were created using Sparse Allocation.
Sparse Allocation creates a sparse disk, so the size of the disk is initially small and increases as it is used. Sparse allocation is faster than using Non-Sparse Allocation when creating a virtual machine. This is a great feature and has some other nice advantages, like faster migration, etc..

The caveat however, lies in the fact that you can overspend your physical diskspace.
When VM’s grow, they take up more and more space in your Storage Repository. Until ….. you run out of physical diskspace!! The Operating System (O/S) within the VM however, thinks that there is enough space. At that point the O/S within the VM’s start having i/o errors, choking up and finally crashing the VM.

To solve the problems start by creating some free space in the Storage Repository, either by extending the iSCSI LUN, NFS Share or by simply moving a VM to another Storage Repository. After the free space has been created, come some old fashion linux filesystem repair actions.

I found several different scenario’s after startup of VM’s and not all of them are recoverable!
  • At startup, the O/S forced checks and corrections of filesystems and was able to repair any inconsistencies automatically and the system boots up correct.
    Problem solved
  • At startup, the O/S forced checks and corrections of filesystems and was unable to correct all errors and forced a manual repair.
    After running fsck on the filesystem(s) in maintenance mode, the system boots up correct.
    Problem solved
  • At startup, the O/S failed with a Kernel Panic. Outch, this can be very nasty!
    This usually means that there is something wrong on a part of either the boot or root filesystem, needed by the kernel.
    Change the VM to start from DVD/ISO* and boot into Linux Rescue mode. In rescue mode I was able to repair a filesystem only ones. In other cases the filesystems had sustained to much damage and the VM’s needed to be re-created/re-installed.

In one case the customer was very lucky. This VM had two virtual disks, one for O/S and one for Oracle software and database. The O/S disk was unrecoverable, but the other disk had no problems. The VM had just recently been migrated from another OVM Server. The ‘old’ VM was stil available and very little changes were made to the O/S. I was able to migrate the O/S disk from the old OVM Server and start the VM again.

Just to see what happens when you procede with filesystem repair on a filesystem with to much damage. I ran an fsck -y on this device. This ended up with a very large Lost&Found directory filled with #xxxxxx files and directories. Not quite the filesystem you can boot from. :-)

I wrote this article not to show you how to repair disk corruptions.
There ise enough information around to help you with that.
However, you can prevent this from happening!

Don’t overspend disk space in your Oracle VM environment!

(Or at least put some strict monitoring on physical diskspace usage)

* Since it is not possible to boot a PVM from DVD/ISO, change the VM to HVM.
see MOS Doc ID
884085.1: Oracle VM: How to configure a guest Virtual Machine to boot from CDROM/DVDROM

Monday, September 16, 2013

Quick & Easy migrate VM from Oracle VM 2.x to 3.x

I have come across customers running their Virtual Machines (VM) on Oracle VM 2.x wanting to migrate them to Oracle VM 3.x
Most of the time the engineers think the migration to 3.x is a lot of work. I have seen situations where they installed fresh new VM’s on Oracle VM 3.x  and migrated the applications or they performed full system restores on them. A lot of work …, more than migrating!

The migration of Oracle VM 2.x to 3.x is, simply put, nothing more than importing the VM as a template into Oracle VM 3.x and create a new VM based on this template.

As of Oracle 3.x the only way to import a template, is to upload them using ftp of http(s).
A standard approach is to copy the img and cfg files to local storage on, for example, a ftp- of webserver and from there upload them to Oracle VM 3.x as a template. In al this can be a very time consuming job due to network connectivity. Especially when moving to and from DMZ’s through firewalls.

In this article I wil demonstrate a quick and easy way to migrate a VM from Oracle VM 2.x to 3.x
Without the hassle of first copying them to a local disk storage and the uploading them to Oracle VM 3.x.

Prerequisites:
Network connectivity, ftp or http(s), between the OVM Server 2.x and the new OVM Server 3.x

Steps to a quick and easy migration


Make the VM files available on the OVM Server 2.2


Shutdown the VM
[root@vm1 ~]# poweroff

Logon to the OVM Server
Check if the VM is really down.
[root@ovmserver1]# xm list
Name                            ID   Mem VCPUs      State   Time(s)
1548_vm1                         8 13312     4     -b---- 544793.2

Stop iptables (if running) for the duration of the migration.
Or create a rule that allows http access to the OVM Server is you prefer.
[root@ovmserver1]# service iptables stop
Flushing firewall rules:                                   [  OK  ]
Setting chains to policy ACCEPT: filter                    [  OK  ]
Unloading iptables modules:                                [  OK  ]

Change directory to the location of the img and vmcfg files of the VM.
cd /OVS/running_pool/1548_vm1
ls -l
-rw-r--r-- 1 root root          280 Nov 24  2011 README -rw-r--r-- 1 root root  13316728320 Jun 10 17:11 System.img -rw-rw-rw- 1 root root 209715200000 Sep  6 08:04 u01.img -rw-r--r-- 1 root root          569 Sep  6 08:11 vm.cfg

Make a backup of the vm.cfg and remove the mac address and xenbr0 from it.
A new mac address and network will me provided by OVM 3.x later.
cp vm.cfg vm.cfg.org
vi vm.cfg

Before
vif = ['bridge=xenbr0,mac=00:16:3E:44:0D:7F,type=netfront']

After
vif = ['type=netfront']

Now to make the file available for direct upload we need a http of ftp server on OVM 2.x
I do not want to install additional software on the OVM Server.
Luckily Oracle (xen) uses a lot of python!!
Python has the SimpleHTTPServer module which provides a simple http Request handler.

Make sure you are in the directory where the img and vm.cfg files are located.
pwd
/OVS/running_pool/1548_vm1

Start a simple Http server
python -m SimpleHTTPServer 80
Serving HTTP on 0.0.0.0 port 80 ...

Leave this session open for the duration of the migration.

Import the 2.x VM as a template in 3.x using OVM Manager


Open the OVM Manager

Navigate to the Repository tab and go to the ‘VM Templates’.


Click on ‘Import VM Template’

Provide the url to the img and vm.cfg files
Put them all together in the the ‘VM Template URL’s’ box of the Import window.

http://10.x.x.10/System.img
http://10.x.x.10/u01.img
http://10.x.x.10/vm.cfg

Substitute the ip-adress with the ip-adress of your OVM Server 2.x

The files will now be uploaded (Or downloaded, depending on perspective)



Now we have a template called System.img
You can change this to a more descriptive name.



Now change the template to fit your needs.

Check/change name, netwerk interface, boot order, etc.





Create a new VM based on this template.



After creation has completed, change properties of the new VM as needed.
Note the MAC address(es) of the interface(s).
This might be needed later on during configuration of OS Guest.

Start the VM using Oracle VM Manager.

Open the console and logon as root.

It might be that you don’t see your network interface (eth0), only your Local Loopback address.



Most of the times you will find the old mac addess is still in ifcfg-eth0 (and -eth1,eth…. if used)
Also there may be a reference to it in /etc/udev/rules.d/60-net.rules or /etc/udev/rules.d/70-persistent-net.rules (depending on OS version)

Put the correct mac address in the ifcfg-eth0 files.
vi /etc/sysconfig/networking/devices/ifcfg-eth0

remove the udev file
rm -f /etc/udev/rules.d/60-net.rules
or
rm -f /etc/udev/rules.d/70-persistent-net.rules

Reboot

When the VM come back online, you should have the correct network interfaces back.

Finally.

Check /etc/sysctl.conf for ‘old’ Xen settings.
For example: xen.independent_wallclock
Adjust or remove them as needed.

Cleanup


  • Remove the VM from your old OVM 2.x server.

Or at least remove it from auto-startup.

  • Remove the template from your Storage Repository.

It takes up space and you risk someone re-using the template. This might cause ip conflicts, etc…


Enjoy migrations …