Oracle FMW and more ...: June 2016

In my previous post Network considerations on an ODA X5-2 I described a solution where I created a private network to optimize network performance between the components running on an Oracle Database Appliance. I am talking about a Virtualized setup in this article, and rebuilding the second pair of public interfaces to become a private network bond, in order to get a faster connection between the virtual machines.
In this article I will describe the steps how to create a private network on an ODA X5-2 and configure an additional SCAN-listener for this private network.

Connect the nodes

Let's start by connecting the two nodes of the ODA directly to each other using Cat. 6 Ethernet cables. Make sure that the cables are long enough to route them through the Cable Management Arm. This way you don't have to disconnect the cables when you have to do maintenance on the nodes.
We will use the second pair of interfaces for the configuration of the private network. Connect interfaces net2 and net3 on node0 to net2 and net3 on node1. This will create a 10 Gb network between the two nodes.

Pict. 1 - Oracle Database Appliance X5-2 Back Panel with net2 and net3 on both nodes connected to each other

Now actually, you have a private network configured that you can assign to the VM's.

Configure the interface on the ODA_BASE

In Dom0 the pair of interfaces is known as net2 and is assigned to the ODA_BASE.
Within the ODA_BASE the interface is known as eth1.

In this example we will use subnet 192.168.10.0/24 for the private network.

Node0 : 192.168.10.10
Node1 : 192.168.10.11

There are several ways to assign an IP addresses to eth1 on the ODA_BASE on both nodes.

Using the Oracle Appliance Manager Configurator

If you are preparing for Oracle Database Appliance Installation and Deployment you configure the interface and ip-address during the deployment. The Oracle Appliance Manager Configurator will guide you through the process.

Using oakcli

If eth1 is unconfigured, you can use oakcli on the ODA_BASE of both nodes to configure it.

oakcli configure additionalnet

Provide Interface Name, DHCP, IP and Netmask

Manually change the IP address

Note. Don't change the primary public IP addresses this way!!!

If you need to change the IP Address of the primary public interface, you need to follow the procedure described in
ODA: How to Change Oracle Database Appliance IP Addresses Post Deployment (Doc ID 1504734.1) on Oracle support.

If eth1 is already configured and you want to give it another IP address we can edit the ifcfg-eth1 file on the ODA_BASE of both nodes.

On Node0

cd /etc/sysconfig/network-scripts
vi ifcfg-eth1

#auto generated by OAK
DEVICE=eth1
ONBOOT=yes
BOOTPROTO=none
TYPE=ETHERNET
IPADDR=192.168.10.10
NETMASK=255.255.255.0

Enable eth1 interface once.

ifup eth1

The next time the ODA_BASE is restarted, the interface will start automatically with the new address.

On Node1

cd /etc/sysconfig/network-scripts
vi ifcfg-eth1

#auto generated by OAK
DEVICE=eth1
ONBOOT=yes
BOOTPROTO=none
TYPE=ETHERNET
IPADDR=192.168.10.11
NETMASK=255.255.255.0

Enable eth1 interface once.

ifup eth1

The next time the ODA_BASE is restarted, the interface will start automatically with the new address.

Note that the eth1 interface on the ODA_BASE is also known as net2 on the Dom0 and in the GRID infrastructure. Continuing this article I will use the term net2 for the private network.

Configure an additional SCAN Listener

When an ODA is installed and deployed, the Oracle Appliance Manager Configurator configures the SCAN listener on the interface defined as firstnet. Usually this will be net1.

Now that we have a private network, we wat to be able to use it for connections to the database. So we are going to add a SCAN listener to net2.

As a guide to creating another SCAN listener I am using ODA (Oracle Database Appliance): HowTo Configure Multiple Public Network on GI (Grid Infrastructure) 12c (Doc ID 2101109.1)

Configure hosts file and DNS

Before you configure the SCAN listeners we are going to add some entries to the /etc/hosts file of the ODA_BASE on both nodes.
After initial deployment, the /etc/hosts file is filled with default values. For example.

# Following added by OneCommand
127.0.0.1    localhost.localdomain localhost
# PUBLIC HOSTNAMES

# PRIVATE HOSTNAMES
192.168.16.27    oda11-priv0.foo.bar oda11-priv0
192.168.16.28    oda12-priv0.foo.bar oda12-priv0

# VIP HOSTNAMES
10.10.1.13       oda11-vip.foo.bar oda11-vip
10.10.1.14       oda12-vip.foo.bar oda12-vip

# NET(0-3) HOSTNAMES
10.10.1.10       oda11.foo.bar oda11
10.10.1.11       oda12.foo.bar oda12

# Below are SCAN IP addresses for reference.
# SCAN_IPS=(10.10.1.7 10.10.1.8)
# ASR entry
141.146.1.169   transport.oracle.com

Where 10.10.1.0/24 is the subnet for net1 in this example.

Add the VIP hostnames of the newly created private network on net2.
To make things easy, used the same hostname as the public hostname and added -int to it.

# INTERNAL HOSTNAMES, ADDED MANUALLY
192.168.10.10    oda11-int.foo.bar oda11-int
192.168.10.11    oda12-int.foo.bar oda12-int

# INTERNAL VIP HOSTNAMES, ADDED MANUALLY
192.168.10.12    oda11-int-vip.foo.bar oda11-int-vip
192.168.10.11    oda12-int-vip.foo.bar oda12-int-vip

Also add the vip addess of the SCAN listener to DNS before you create the new SCAN listener. On an ODA by default there are two ip addresses registered for the SCAN listener, so for the new SCAN listener we will also register two addresses on the private interface.

192.168.10.7    oda1-int-scan
192.168.10.8    oda1-int-scan

Configure Listener in GI Infrastructure

Logon to the ODA_BASE of node0 as user grid

Set environment to GI

[grid@oda11 ~]$ . oraenv
ORACLE_SID = [grid] ? +ASM1
The Oracle base has been set to /u01/app/grid

Set interface type to pubic for eth1

[grid@oda11 ~]$ oifcfg setif -global eth1/192.168.10.0:public
[grid@odadbsandta01 ~]$ oifcfg getif
eth0  10.10.1.0  global  public
ibbond0  192.168.16.0  global  cluster_interconnect,asm
eth1  192.168.10.0  global  public

Logon to the ODA_BASE of node0 as user root

Set environment to GI

[root@odadbsandta01 ~]# . oraenv
ORACLE_SID = [root] ? +ASM1
The Oracle base has been set to /u01/app/grid

Add a network and vip addresses to GI

[root@oda11 grid]# srvctl add network -netnum 2 -S 192.168.10.0/255.255.255.0/eth1 -w static
[root@oda11 grid]# srvctl add vip -n oda11 -A oda11-int-vip.foo.bar /255.255.255.0 -netnum 2
[root@oda11 grid]# srvctl add vip -n oda12 -A oda12-int-vip.foo.bar/255.255.255.0 -netnum 2

Logon to the ODA_BASE of node0 as user grid

Set environment to GI

[grid@oda11 ~]$ . oraenv
ORACLE_SID = [grid] ? +ASM1
The Oracle base has been set to /u01/app/grid

Start the newly created vip's

[grid@oda11 ~]$ srvctl start vip -i oda11-int-vip.foo.bar
[grid@oda11 ~]$ srvctl start vip -i oda12-int-vip.foo.bar

Check if vip's are enabled and running on both nodes

[grid@oda11 ~]# srvctl status vip -n oda11
VIP oda11-vip.foo.bar is enabled
VIP oda11-vip.foo.bar is running on node: oda11
VIP oda11-int-vip.foo.bar is enabled
VIP oda11-int-vip.foo.bar is running on node: oda11

[grid@oda11 ~]# srvctl status vip -n oda12
VIP oda12-vip.foo.bar is enabled
VIP oda12-vip.foo.bar is running on node: oda12
VIP oda12-int-vip.foo.bar is enabled
VIP oda12-int-vip.foo.bar is running on node: oda12

Add a listener on the net2 interface and start it

[grid@oda11 ~]$ srvctl add listener -l INT_LISTENER -p 1521 -netnum 2 -s
[grid@oda11 ~]$ srvctl start listener -l INT_LISTENER
[grid@oda11 ~]$ srvctl status listener
Listener INT_LISTENER is enabled
Listener INT_LISTENER is running on node(s): oda12,oda11
Listener LISTENER is enabled
Listener LISTENER is running on node(s): oda12,oda11

Logon to the ODA_BASE of node0 as user root

Set environment to GI

[root@odadbsandta01 ~]# . oraenv
ORACLE_SID = [root] ? +ASM1
The Oracle base has been set to /u01/app/grid

Add SCAN and SCAN listener

[root@oda11 grid]# srvctl add scan -scanname oda1-int-scan -netnum 2
[root@oda11 grid]# srvctl start scan -netnum 2
[root@oda11 grid]# srvctl add scan_listener -netnum 2 -listener int -endpoints TCP:1521
[root@oda11 grid]# srvctl start scan_listener -netnum 2

[grid@oda11 ~]$ srvctl status scan
SCAN VIP scan1 is enabled
SCAN VIP scan1 is running on node oda12
SCAN VIP scan2 is enabled
SCAN VIP scan2 is running on node oda12

[grid@oda11 ~]$ srvctl status scan_listener
SCAN Listener LISTENER_SCAN1 is enabled
SCAN listener LISTENER_SCAN1 is running on node oda12
SCAN Listener LISTENER_SCAN2 is enabled
SCAN listener LISTENER_SCAN2 is running on node oda12

Register databases

In normal circumstances the databases should register themselves to the new listener. I have seen this working on an ODA X5-2 running version 12.1.2.4.0.
So normally you are done with the database part.

Unfortunately version 12.1.2.6.0 introduced a bug which prevents the registration of the database with the new listener.
The bug is related to bug 19884033 :
ORA-2097 AND ORA-119 ATTEMPTING TO SET LISTENER_NETWORKS PARAMETER BY THE AGENT
Unfortunately the fix for this bug didn't make it to the next release, nor the patchset of version 12.1.2.7.0.

DO NOT try to install the normal server patchset Patch 19884033!! I have tried this and almost ruined our ODA installation.
Please wait for a patch specifically for the ODA.

Waiting for the bugfix, Oracle support supplied the following workaround.

ALTER SYSTEM SET listener_networks='((NAME=net2)(LOCAL_LISTENER=(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=<net2_node1_vip>)(PORT=1521)))))','((NAME=net2)(REMOTE_LISTENER=net2_scan:1521))' SCOPE=BOTH SID='<instance1>'; 
ALTER SYSTEM SET listener_networks='((NAME=net2)(LOCAL_LISTENER=(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=<net2_node2_vip>)(PORT=1521)))))','((NAME=net2)(REMOTE_LISTENER=net2_scan:1521))' SCOPE=BOTH SID='<instance2>';

SQL> alter system register;

I am still working with Oracle support to get a one-off patch for this issue.

Using the private network

To use the private network, just add another network adapter to the vm or vmtemplate.

oakcli configure vm <vm name> -network "['type=netfront,bridge=net1','type=netfront,bridge=net2']"
oakcli configure vmtemplate <vm template name> -network "['type=netfront,bridge=net1','type=netfront,bridge=net2']"

In the VM the interfaces will be present as eth0 and eth1

Let's take a look at the pictures I used in the previous blog, which are part of the design as an example.

In this design I configured the Oracle Traffic Director (OTD) and Fusion Middleware (in this case SOA Suite and MFT) VM's with two interfaces.

The OTD is configured to listen on the public interface and routes traffic to the FMW components on the private interface.

The FMW components are configured to listen only on the private network. That way all traffic to the FMW components have to come from the OTD.
Also all cluster communication between the FMW components is routed through the private interface.

The FMW components route database traffic to and from the ODA_BASE through the private interface.

Note. The public interface on the FMW VM's is only used for SSH.

Summary

By creating a private network on an ODA, you are not only able to improve performance. You are also capable of separating traffic from public to private. This creates a real one-box solution and, if implemented correctly, increases security. You are able to hide the back-end systems from the clients.
At customer sites where I have implemented the private network, they are very pleased with the solution.

Experiences

One thing I experienced at customers is the network configuration they ordered for their ODA X5-2. Several times the ODA X5-2 was ordered off the shelve without consulting any of the network engineers on what kind of switches they have and what available bandwidth they support. Customers, at least the ones I visit, have more than enough 10Gb fiber interfaces available on their core switches, but 10Gb copper switches are not yet common to them. So they are faces with the fact that their new high performance machine is connected to 1Gb switch ports interfaces.

Since all traffic between components, with exception of interconnect traffic, runs through these interfaces. This could potentially cause a performance bottleneck.

Customer case

At a customer where I was asked to setup two ODA's last year this was also the case. They bought two ODA's with the standard configuration. Again the switches at the customer site supported 10Gb on fiber, but not on copper. Their copper switches supported 1Gb only. You can replace the InfiniBand card with a 10Gb fiber card, but this was not ordered and the ODA's where already re-imaged by an Oracle Engineer. Ordering the fiber cards and re-imaging the ODA's would delay the project too much, so I was asked to provide a solution with full bandwidth between the VM's (with clustered FMW products) and the Oracle databases on the ODA_BASE.

A quick solution was requested

I decided to take the second public interface and use it to create a private network within the ODA. This was done by directly connecting the interfaces of the two ODA nodes to each other. This created a 10Gb private network. All traffic between Oracle Traffic Director (OTD) and FMW components and between FMW components (including cluster communication) and the database now runs through that interface.

This way only the traffic from clients to OTD is going through the public interface.

Another solution would be to use the InfiniBand interface for this purpose. This would involve installing and configuring InfiniBand drivers, etc. in the VM's. (There is a whitepaper from Oracle about the configuration of InfiniBand on VM's.) This was considered as an option, but the use a normal Ethernet connection was favored because the impact on the VM's is minimal with that configuration.

The picture below is part of the design. Just to give you an idea about the flow of traffic. There are more FMW products running on this ODA, hence more traffic.

All traffic you see in this picture stays within the ODA.

A little over a month ago, I was faced with the same issue at a customer where I was asked to install an ODA X5-2 which will be used for JD Edwards.
At this customer I was, again, asked to implement the solution with the private network to optimize performance between the JD Edwards components.

And currently I am implementing a third ODA at the customer site I talked about earlier. They now have 10Gb copper switches at their new datacenter so the performance impact on the one interface would be less. However they still asked me to implement the private network configuration. The fact that traffic was not running through their LAN network and connections to the backend components (FMW and Database) are controlled by the OTD was considered an extra benefit.

Summary

Needless to say, it would be best to involve the network engineers as early as possible in the process. Just to make sure your fresh new ODA X5-2 gets the network connection it deserves.

Oracle FMW and more ...

Wednesday, June 29, 2016

How To create a private 10 Gb network on an ODA X5-2

Connect the nodes

Configure the interface on the ODA_BASE

Using the Oracle Appliance Manager Configurator

Using oakcli

Manually change the IP address

Note. Don't change the primary public IP addresses this way!!!

Configure an additional SCAN Listener

Configure hosts file and DNS

Configure Listener in GI Infrastructure

Register databases

Using the private network

Summary

Related Documents

Friday, June 24, 2016

Network considerations on an ODA X5-2

Experiences

Customer case

A quick solution was requested

Summary