Oracle FMW and more ...: How To create a private 10 Gb network on an ODA X5-2

In my previous post Network considerations on an ODA X5-2 I described a solution where I created a private network to optimize network performance between the components running on an Oracle Database Appliance. I am talking about a Virtualized setup in this article, and rebuilding the second pair of public interfaces to become a private network bond, in order to get a faster connection between the virtual machines.
In this article I will describe the steps how to create a private network on an ODA X5-2 and configure an additional SCAN-listener for this private network.

Connect the nodes

Let's start by connecting the two nodes of the ODA directly to each other using Cat. 6 Ethernet cables. Make sure that the cables are long enough to route them through the Cable Management Arm. This way you don't have to disconnect the cables when you have to do maintenance on the nodes.
We will use the second pair of interfaces for the configuration of the private network. Connect interfaces net2 and net3 on node0 to net2 and net3 on node1. This will create a 10 Gb network between the two nodes.

Pict. 1 - Oracle Database Appliance X5-2 Back Panel with net2 and net3 on both nodes connected to each other

Now actually, you have a private network configured that you can assign to the VM's.

Configure the interface on the ODA_BASE

In Dom0 the pair of interfaces is known as net2 and is assigned to the ODA_BASE.
Within the ODA_BASE the interface is known as eth1.

In this example we will use subnet 192.168.10.0/24 for the private network.

Node0 : 192.168.10.10
Node1 : 192.168.10.11

There are several ways to assign an IP addresses to eth1 on the ODA_BASE on both nodes.

Using the Oracle Appliance Manager Configurator

If you are preparing for Oracle Database Appliance Installation and Deployment you configure the interface and ip-address during the deployment. The Oracle Appliance Manager Configurator will guide you through the process.

Using oakcli

If eth1 is unconfigured, you can use oakcli on the ODA_BASE of both nodes to configure it.

oakcli configure additionalnet

Provide Interface Name, DHCP, IP and Netmask

Manually change the IP address

Note. Don't change the primary public IP addresses this way!!!

If you need to change the IP Address of the primary public interface, you need to follow the procedure described in
ODA: How to Change Oracle Database Appliance IP Addresses Post Deployment (Doc ID 1504734.1) on Oracle support.

If eth1 is already configured and you want to give it another IP address we can edit the ifcfg-eth1 file on the ODA_BASE of both nodes.

On Node0

cd /etc/sysconfig/network-scripts
vi ifcfg-eth1

#auto generated by OAK
DEVICE=eth1
ONBOOT=yes
BOOTPROTO=none
TYPE=ETHERNET
IPADDR=192.168.10.10
NETMASK=255.255.255.0

Enable eth1 interface once.

ifup eth1

The next time the ODA_BASE is restarted, the interface will start automatically with the new address.

On Node1

cd /etc/sysconfig/network-scripts
vi ifcfg-eth1

#auto generated by OAK
DEVICE=eth1
ONBOOT=yes
BOOTPROTO=none
TYPE=ETHERNET
IPADDR=192.168.10.11
NETMASK=255.255.255.0

Enable eth1 interface once.

ifup eth1

The next time the ODA_BASE is restarted, the interface will start automatically with the new address.

Note that the eth1 interface on the ODA_BASE is also known as net2 on the Dom0 and in the GRID infrastructure. Continuing this article I will use the term net2 for the private network.

Configure an additional SCAN Listener

When an ODA is installed and deployed, the Oracle Appliance Manager Configurator configures the SCAN listener on the interface defined as firstnet. Usually this will be net1.

Now that we have a private network, we wat to be able to use it for connections to the database. So we are going to add a SCAN listener to net2.

As a guide to creating another SCAN listener I am using ODA (Oracle Database Appliance): HowTo Configure Multiple Public Network on GI (Grid Infrastructure) 12c (Doc ID 2101109.1)

Configure hosts file and DNS

Before you configure the SCAN listeners we are going to add some entries to the /etc/hosts file of the ODA_BASE on both nodes.
After initial deployment, the /etc/hosts file is filled with default values. For example.

# Following added by OneCommand
127.0.0.1    localhost.localdomain localhost
# PUBLIC HOSTNAMES

# PRIVATE HOSTNAMES
192.168.16.27    oda11-priv0.foo.bar oda11-priv0
192.168.16.28    oda12-priv0.foo.bar oda12-priv0

# VIP HOSTNAMES
10.10.1.13       oda11-vip.foo.bar oda11-vip
10.10.1.14       oda12-vip.foo.bar oda12-vip

# NET(0-3) HOSTNAMES
10.10.1.10       oda11.foo.bar oda11
10.10.1.11       oda12.foo.bar oda12

# Below are SCAN IP addresses for reference.
# SCAN_IPS=(10.10.1.7 10.10.1.8)
# ASR entry
141.146.1.169   transport.oracle.com

Where 10.10.1.0/24 is the subnet for net1 in this example.

Add the VIP hostnames of the newly created private network on net2.
To make things easy, used the same hostname as the public hostname and added -int to it.

# INTERNAL HOSTNAMES, ADDED MANUALLY
192.168.10.10    oda11-int.foo.bar oda11-int
192.168.10.11    oda12-int.foo.bar oda12-int

# INTERNAL VIP HOSTNAMES, ADDED MANUALLY
192.168.10.12    oda11-int-vip.foo.bar oda11-int-vip
192.168.10.11    oda12-int-vip.foo.bar oda12-int-vip

Also add the vip addess of the SCAN listener to DNS before you create the new SCAN listener. On an ODA by default there are two ip addresses registered for the SCAN listener, so for the new SCAN listener we will also register two addresses on the private interface.

192.168.10.7    oda1-int-scan
192.168.10.8    oda1-int-scan

Configure Listener in GI Infrastructure

Logon to the ODA_BASE of node0 as user grid

Set environment to GI

[grid@oda11 ~]$ . oraenv
ORACLE_SID = [grid] ? +ASM1
The Oracle base has been set to /u01/app/grid

Set interface type to pubic for eth1

[grid@oda11 ~]$ oifcfg setif -global eth1/192.168.10.0:public
[grid@odadbsandta01 ~]$ oifcfg getif
eth0  10.10.1.0  global  public
ibbond0  192.168.16.0  global  cluster_interconnect,asm
eth1  192.168.10.0  global  public

Logon to the ODA_BASE of node0 as user root

Set environment to GI

[root@odadbsandta01 ~]# . oraenv
ORACLE_SID = [root] ? +ASM1
The Oracle base has been set to /u01/app/grid

Add a network and vip addresses to GI

[root@oda11 grid]# srvctl add network -netnum 2 -S 192.168.10.0/255.255.255.0/eth1 -w static
[root@oda11 grid]# srvctl add vip -n oda11 -A oda11-int-vip.foo.bar /255.255.255.0 -netnum 2
[root@oda11 grid]# srvctl add vip -n oda12 -A oda12-int-vip.foo.bar/255.255.255.0 -netnum 2

Logon to the ODA_BASE of node0 as user grid

Set environment to GI

[grid@oda11 ~]$ . oraenv
ORACLE_SID = [grid] ? +ASM1
The Oracle base has been set to /u01/app/grid

Start the newly created vip's

[grid@oda11 ~]$ srvctl start vip -i oda11-int-vip.foo.bar
[grid@oda11 ~]$ srvctl start vip -i oda12-int-vip.foo.bar

Check if vip's are enabled and running on both nodes

[grid@oda11 ~]# srvctl status vip -n oda11
VIP oda11-vip.foo.bar is enabled
VIP oda11-vip.foo.bar is running on node: oda11
VIP oda11-int-vip.foo.bar is enabled
VIP oda11-int-vip.foo.bar is running on node: oda11

[grid@oda11 ~]# srvctl status vip -n oda12
VIP oda12-vip.foo.bar is enabled
VIP oda12-vip.foo.bar is running on node: oda12
VIP oda12-int-vip.foo.bar is enabled
VIP oda12-int-vip.foo.bar is running on node: oda12

Add a listener on the net2 interface and start it

[grid@oda11 ~]$ srvctl add listener -l INT_LISTENER -p 1521 -netnum 2 -s
[grid@oda11 ~]$ srvctl start listener -l INT_LISTENER
[grid@oda11 ~]$ srvctl status listener
Listener INT_LISTENER is enabled
Listener INT_LISTENER is running on node(s): oda12,oda11
Listener LISTENER is enabled
Listener LISTENER is running on node(s): oda12,oda11

Logon to the ODA_BASE of node0 as user root

Set environment to GI

[root@odadbsandta01 ~]# . oraenv
ORACLE_SID = [root] ? +ASM1
The Oracle base has been set to /u01/app/grid

Add SCAN and SCAN listener

[root@oda11 grid]# srvctl add scan -scanname oda1-int-scan -netnum 2
[root@oda11 grid]# srvctl start scan -netnum 2
[root@oda11 grid]# srvctl add scan_listener -netnum 2 -listener int -endpoints TCP:1521
[root@oda11 grid]# srvctl start scan_listener -netnum 2

[grid@oda11 ~]$ srvctl status scan
SCAN VIP scan1 is enabled
SCAN VIP scan1 is running on node oda12
SCAN VIP scan2 is enabled
SCAN VIP scan2 is running on node oda12

[grid@oda11 ~]$ srvctl status scan_listener
SCAN Listener LISTENER_SCAN1 is enabled
SCAN listener LISTENER_SCAN1 is running on node oda12
SCAN Listener LISTENER_SCAN2 is enabled
SCAN listener LISTENER_SCAN2 is running on node oda12

Register databases

In normal circumstances the databases should register themselves to the new listener. I have seen this working on an ODA X5-2 running version 12.1.2.4.0.
So normally you are done with the database part.

Unfortunately version 12.1.2.6.0 introduced a bug which prevents the registration of the database with the new listener.
The bug is related to bug 19884033 :
ORA-2097 AND ORA-119 ATTEMPTING TO SET LISTENER_NETWORKS PARAMETER BY THE AGENT
Unfortunately the fix for this bug didn't make it to the next release, nor the patchset of version 12.1.2.7.0.

DO NOT try to install the normal server patchset Patch 19884033!! I have tried this and almost ruined our ODA installation.
Please wait for a patch specifically for the ODA.

Waiting for the bugfix, Oracle support supplied the following workaround.

ALTER SYSTEM SET listener_networks='((NAME=net2)(LOCAL_LISTENER=(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=<net2_node1_vip>)(PORT=1521)))))','((NAME=net2)(REMOTE_LISTENER=net2_scan:1521))' SCOPE=BOTH SID='<instance1>'; 
ALTER SYSTEM SET listener_networks='((NAME=net2)(LOCAL_LISTENER=(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=<net2_node2_vip>)(PORT=1521)))))','((NAME=net2)(REMOTE_LISTENER=net2_scan:1521))' SCOPE=BOTH SID='<instance2>';

SQL> alter system register;

I am still working with Oracle support to get a one-off patch for this issue.

Using the private network

To use the private network, just add another network adapter to the vm or vmtemplate.

oakcli configure vm <vm name> -network "['type=netfront,bridge=net1','type=netfront,bridge=net2']"
oakcli configure vmtemplate <vm template name> -network "['type=netfront,bridge=net1','type=netfront,bridge=net2']"

In the VM the interfaces will be present as eth0 and eth1

Let's take a look at the pictures I used in the previous blog, which are part of the design as an example.

In this design I configured the Oracle Traffic Director (OTD) and Fusion Middleware (in this case SOA Suite and MFT) VM's with two interfaces.

The OTD is configured to listen on the public interface and routes traffic to the FMW components on the private interface.

The FMW components are configured to listen only on the private network. That way all traffic to the FMW components have to come from the OTD.
Also all cluster communication between the FMW components is routed through the private interface.

The FMW components route database traffic to and from the ODA_BASE through the private interface.

Note. The public interface on the FMW VM's is only used for SSH.

Summary

By creating a private network on an ODA, you are not only able to improve performance. You are also capable of separating traffic from public to private. This creates a real one-box solution and, if implemented correctly, increases security. You are able to hide the back-end systems from the clients.
At customer sites where I have implemented the private network, they are very pleased with the solution.

Oracle FMW and more ...

Wednesday, June 29, 2016

How To create a private 10 Gb network on an ODA X5-2