Network considerations on an ODA X5-2

When you buy an Oracle Database Appliance (ODA) X5-2 off the shelve, you will get a machine with four times 10Gb copper Ethernet (bonded into two interfaces) for public communication and two 40Gb InfiniBand (bonded into one interface) for interconnect communication between the two ODA_BASE's. You would think that should be more than enough. Well in most cases it is. The interconnect running through InfiniBand is stunning for RAC and the public interface should be enough for most uses.

Oracle is doing a great job in selling the ODA X5-2 to their customers. It is a great machine with a lot of capabilities especially when you implement it with a Virtualized Platform Setup. In this configuration an Oracle VM hypervisor is put on top of the hardware and the ODA_BASE is placed in a special virtual machine (VM). The ODA_BASE is primarily used for running all databases and for managing, almost, all aspects of the entire ODA X5-2. Besides this special VM, you can create your own VM's. This gives you the opportunity to make optimal use of the hardware the ODA has to offer and keep track of your licenses. For most of our customers the virtualized ODA provides a one-box solution for their Fusion Middleware (FMW) implementations.

Experiences

One thing I experienced at customers is the network configuration they ordered for their ODA X5-2. Several times the ODA X5-2 was ordered off the shelve without consulting any of the network engineers on what kind of switches they have and what available bandwidth they support. Customers, at least the ones I visit, have more than enough 10Gb fiber interfaces available on their core switches, but 10Gb copper switches are not yet common to them. So they are faces with the fact that their new high performance machine is connected to 1Gb switch ports interfaces.

Since all traffic between components, with exception of interconnect traffic, runs through these interfaces. This could potentially cause a performance bottleneck.

Customer case

At a customer where I was asked to setup two ODA's last year this was also the case. They bought two ODA's with the standard configuration. Again the switches at the customer site supported 10Gb on fiber, but not on copper. Their copper switches supported 1Gb only. You can replace the InfiniBand card with a 10Gb fiber card, but this was not ordered and the ODA's where already re-imaged by an Oracle Engineer. Ordering the fiber cards and re-imaging the ODA's would delay the project too much, so I was asked to provide a solution with full bandwidth between the VM's (with clustered FMW products) and the Oracle databases on the ODA_BASE.

A quick solution was requested

I decided to take the second public interface and use it to create a private network within the ODA. This was done by directly connecting the interfaces of the two ODA nodes to each other. This created a 10Gb private network. All traffic between Oracle Traffic Director (OTD) and FMW components and between FMW components (including cluster communication) and the database now runs through that interface.

This way only the traffic from clients to OTD is going through the public interface.

Another solution would be to use the InfiniBand interface for this purpose. This would involve installing and configuring InfiniBand drivers, etc. in the VM's. (There is a whitepaper from Oracle about the configuration of InfiniBand on VM's.) This was considered as an option, but the use a normal Ethernet connection was favored because the impact on the VM's is minimal with that configuration.

The picture below is part of the design. Just to give you an idea about the flow of traffic. There are more FMW products running on this ODA, hence more traffic.

All traffic you see in this picture stays within the ODA.

A little over a month ago, I was faced with the same issue at a customer where I was asked to install an ODA X5-2 which will be used for JD Edwards.
At this customer I was, again, asked to implement the solution with the private network to optimize performance between the JD Edwards components.

And currently I am implementing a third ODA at the customer site I talked about earlier. They now have 10Gb copper switches at their new datacenter so the performance impact on the one interface would be less. However they still asked me to implement the private network configuration. The fact that traffic was not running through their LAN network and connections to the backend components (FMW and Database) are controlled by the OTD was considered an extra benefit.

Summary

Needless to say, it would be best to involve the network engineers as early as possible in the process. Just to make sure your fresh new ODA X5-2 gets the network connection it deserves.