IBM Integration Bus sizing and performance

It's not the first time that I am been in a situation where I have been asked to evaluate how many cores would be necessary to run a load on IBM Integration Bus (IIB).

In this post I will provide you a way to make this evaluation based on public IIB performance reports.

Required information  

Before starting your evaluation, you first need to identify a set of flows that corresponds to ESB patterns that you would like to achieve: transformation, aggregation, protocols, logging, ...

You would like as well to define on what operating system the run time will run.

For these flows you will need to define the following information

• Kind of messages: XML, non XML
• Average size of messages
• Load peak expressed in transaction/sec. I usually use the peak since this is the throughput that you would like to sustain. 
• Operating system where the IIB will run

Note that the CPU type has also it's importance. Normally you would have to add a corrector factor to compensate the fact that you have a more efficient processor.

Performance reports

Once this information is available, you can start your performance evaluation computation.
For this we will use performance report that is publicly available. For IBM Integration Bus, go to the link message throughput. (good link to bookmark is the performance topic). For older version, you can find performance reports at the support pack link.
On the message throughput link select your operating system.

These performance reports have been measure using a processor: IBM xSeries x3690 X5 with 2 x Deca-Core Intel(R) Xeon(R) E7-2860.
For those that would like to know the weight associated to this processor, please have a look at the PVU information page. You will find that this processor corresponds to a 70 PVU processor (2 sockets with 10 cores).

You will find different type of patterns that have been tested: aggregation, coordination request/reply, message routing, ...
For each of these tests there is a table providing the message rate, the CPU busy and the CPU ms/msg.

The information that we will keep are the 

• message rate
• the CPU ms/msg

Performance evaluation

How can we use these information?

Let's take an example!

Imagine that you would like to deploy a flow doing aggregation with an average non-persistent message size of 2 kb.
This flow will process 460000 messages a day but there is a peak! During one hour in the day the flow has to sustain 720000 messages.

We would like to size the bus such a way that he will sustain the peak. This corresponds to a throughput of 200 trs/sec.
Lets have a look to the tables available on the performance report page. There is one for the aggregation:

	Non Persistent			Full Persistent
Msg Size	Message Rate	% CPU Busy	CPU ms/msg	Message Rate	% CPU Busy	CPU ms/msg
256b	4020.1	99.3	4.941	476.9	12.4	5.215
2kB	3430.6	98.3	5.732	707.8	19.7	5.559

For 2kb non persistent message, the processing of one message takes 5.732 ms of CPU.

The processing of 200 transactions per seconds would take 200 (msg/sec) *5.732 (CPU ms/msg) = 1146.4 (msg/sec)*(CPU sec/1000 * 1/msg) = 1.1464 CPU. 

So you would need to at least 2 processors to process this load. The total load of the machine with 2 CPU would be 57.32%.

It is not recommended to have a processor loaded at more than 70 % but here the 2 CPU will do the deal.

Of course you could have more complex situation such MQ, JMS, database, routing....

It is possible to approach more complex situation by adding the CPU load for each scenarios.
If my integration needs to be exposed as web service using SOAP messages, performs routing and transforms messages, I would approximate my CPU load by adding the corresponding load factor: 2.633 (SOAP) + 0.488 (routing) + 0.896(transformation).

This is not ideal since each scenarios include the protocol processing like MQ for routing.
In the previous reports that are available in the support pack page, more tests have been done and it is possible to build a spreadsheets to isolate the processing of each mediations.
For example there is a test made for MQInput-MQOutput (x ms/msg), MQI-Transformation-MQO (y ms/msg) and JMSIn-JMSout (z ms/msg). For a JMSI-Transformation-JMSO integration flow, you could therefore approximate the CPU load using: y ms/msg - x ms/msg + z ms/msg.
The performance in IIB V9 is higher than for V8, so using the old performance reports would provide you a good estimation with a security factor.

References

message throughput information about IIB in function of the operating system.
performance information, tips and hints.
support pack old performance reports.
PVU information page to find the PVU for a defined CPU.

MQ Cluster demistified

I would like to provide through this post some highlighting on the MQ clustering: what is it, what does it brings and how to set it up.

MQ cluster is not about data replication or make data high available, it's about making object definitions known and available in a group of queue managers and providing a workload management capability.

 The "what is it"

Put it simply, a MQ Cluster is a collection of queue managers that have been defined to be part of a group called a cluster.
Within this group, queue managers know all objects that are shared within the cluster. Objects are like queues, topics.
Within a cluster all queue managers know how to send messages to the target clustered destination. The message transfer are handled automatically by the queue manager in the cluster.

The "what does it brings"

The cluster provides the following main advantages:

• Simplify administration for distribute queueing. When a message has to send to a destination located to another queue manager, the queue has to be simply shared in the cluster on the remote queue manager that's it. The transfer of the message from the queue manager where the application has put the message to the remote queue manage is carried out by the cluster: no remote queues, no transmission queues, no channels. 
• Workload balancing: when multiple instances of queues are defined, the cluster can balance the message distribution to all instances shared in the cluster. Priority and weight can be defined for specific queue managers. 
• Improve availability and simplify maintenance: when multiple instance of queues are defined and one queue manager that hosts this queue is not available, the messages are automatically routed to the remaining queues. This increases the availability since messages can still be processed but it also simplifies the maintenance as the queue manager is known to be stopped in the cluster. 

Some disadvantages that I have in mind:

• Queue manager in a cluster are directly connected with each others. It increases the number of channels which means consume more resources. 
• Less control on how the messages are sent from one queue manager to another 
• Less flexibility in the queue name resolution compared to the distributed queueing (alias, remote queues, ...). 

The "Principle" 

Let's first discuss about queue destinations.
When an application tries to put a message to a queue and this queue is not known by the local queue manager (the queue manager that the application is connected to), the queue manager queries the cluster (if it is part of one) to know if this queue is known.
If the queue is shared within the cluster, the information about its location is provided to the requestor queue manager. The local queue manager handles the message by putting it in the cluster transmission queue with a transmission header containing all the information to route it to the correct destination. The message is then sent by the local queue manager directly to the target queue manager.

Be aware that an application can get messages from local queue only, cluster defined or not. Local queues are queues defined on the queue manager that the application is connected to.
The story is a little bit different for topics. Indeed topic is used to define a destination where an application can publish or subscribe.
When a topic is shared within a cluster, it is known by all queue managers in this cluster.
Therefore if an application subscribes to a cluster topic it will receive all messages published to this topic even though the topic has been created in another queue manager.

The "how to make it" 

You will be puzzled how simple it is to create a cluster !!

Before going further there is one think to know about: the repositories. A repository is a store used by a queue manager to store cluster object definitions.

There are two types of repository: full and partial.
A full repository is a repository that contains cluster object definition from the whole cluster. When an object is shared in a cluster, the object definition is sent by this queue manager to the queue manager that holds the full repository.
A partial repository holds information about cluster object definitions that have been resolved from the queue manager that holds this repository. When an application puts a message for a remote queue shared in the cluster, the local queue manager requests the destination object information to the queue manager holding the full repository. This information is then stored in its local repository. Hence the name "partial repository" as it does not store all object definitions of the whole cluster.

In order to exchange these object definitions, queue managers use cluster channels:

• a cluster receiver channel to receive object definitions 
• a cluster sender channel to send object definitions to the queue manager(s) holding the full repository. 

Two rules about the repositories:

• A queue manager can hold either a full repository or a partial repository (not both) 
• Two (or one if not in production) full repositories is enough. All the other queue manager would hold a partial repository only. 

With this in mind, let's have a look how to make it !

The steps provided here after has to be performed in the same order.

For full repository

1. Set the queue manager property "REPOS" to the name of the cluster. This will tell to the queue manager that he will host a full repository. A runmqsc command would be

   ALTER QMGR REPOS('myCluster')

    
2. Create cluster receiver channel to define how to connect to this queue manager: 

   DEF CLUSRCVR(TO.MyQMgr) 
   CONNAME('ipaddressOfThisQMgr(portNumberOfThisQMgr)') 
   CLUSTER('myCluster')

3. Optionally cluster sender channels can be created to point to another full repository ONLY. 

 For partial repository

1. Create cluster receiver channel to define how to connect to this queue manager:

   DEF CLUSRCVR(TO.MyQMgr)
    CONNAME('ipaddressOfThisQMgr(portNumberOfThisQMgr)') 
   CLUSTER('myCluster')

2. Create a cluster sender channel to define how to reach a queue manager holding a full repository (not partial).

   DEF CLUSSDR(TO.FullRepoQMgr) 
   CONNAME('ipaddressOfFullRepoQMgr(port)') 
   CLUSTER('myCluster')

That's it !!
Now you can create a queue and share it to the cluster "myCuster" that you just created.

References 

Information about the number of repositories in a cluster:
https://www.ibm.com/developerworks/community/blogs/messaging/entry/wmq_clusters_why_only_two_full_repositories?lang=en
Best practices (old presentation but the principle are still valid):
http://www.academia.edu/5513555/WMQCluster_Best_Practices
And this ten things for having an healthy cluster
https://www.ibm.com/developerworks/community/blogs/aimsupport/entry/ten_quick_tips_for_healthy_mq_cluster?lang=en
Very useful information about cluster questions
https://www.ibm.com/developerworks/community/blogs/messaging/tags/clustering?lang=en
Very good article that provides information about MQ HA
http://www.ibm.com/developerworks/websphere/library/techarticles/0505_hiscock/0505_hiscock.html

WebSphere MQ & IIB on Ubuntu

I found that more and more people like Linux and this is a great things.

IBM Integration Bus for developers (free of charge !!!) is available for Linux distribution.

That's cool... however I find out that the installation may be always straightforward especially on the last Linux 64 bit distribution.

You will find in this small post what are the steps that I followed to install the package.

First I would recommend to install each component separately:

• WebSphere MQ
• IBM Integration Bus
• IIB Explorer

Download the IBM Integration Bus for developers package.

https://www14.software.ibm.com/webapp/iwm/web/pick.do?source=swg-wmbfd&S_TACT=109KA7GW&S_CMP=1586-0-120057

I have used the single package download.

WebSphere MQ

Installation

Installing software on Ubuntu usually entails using Synaptic or by using an apt-get command from the terminal. 

Unfortunately, there are still a number of packages out there that are only distributed in RPM format.

Ubuntu is Debian based and therefore uses.deb packages to install. If you want to install .rpm packages, you first should convert them into .deb packages with a conversion software such as alien. Then you can use gdebi or dpkg to install them.

Despite the large version number, alien is still (and will probably always be) rather experimental software. It has been used by many people for many years, but there are still many bugs and limitations

IBM distribute MQ as a set of RPMs and with what I just said, I recommend to use "rpm" and hence install the rpm package: ‘rpm’, ‘pax’ and ‘default-jre’ packages.

Detailed information can be found in the following technote:

http://www-01.ibm.com/support/docview.wss?uid=swg21656068 

In my installation I have made first 

1. Install the i386 libraries

sudo apt-get install libc6-i386
sudo apt-get install libgcc1:i386
sudo apt-get update 

2. Define the system configuration (provided here after)

3. Install the different rpm package as explaine at the previous link (for ubuntu 14.04 I had to use the --prefix /opt/mqm).

4. Define the WMQ configuration

System configuration

I would recommend to change the system configuration for Linux when using WebSphere MQ. I provide the information here after but detailed information can be found in the knowledge center ( Linux System requirements).

create /etc/sysctl.d/50-webspheremq.conf with the following:

kernel.shmmni = 4096
kernel.shmall = 2097152
kernel.shmmax = 268435456
kernel.sem = 500 256000 250 1024
fs.file-max = 524288

Make these changes live by running

sudo sysctl -p

These information can be found in the knowledge center
http://www-01.ibm.com/support/knowledgecenter/SSFKSJ_7.5.0/com.ibm.mq.ins.doc/q008550_.htm 

Check that this has been correctly updated

cat /proc/sys/kernel/shmmni 

WMQ Configuration

You can use the setmqinst command to change the installation description of an installation, or to set or unset an installation as the primary installation.

Before executing the command, you would have to create the following directory “/usr/lib64”. 

Create this directory with the following command:

sudo mkdir /usr/lib64

This command sets the installation with an installation path of /opt/mqm as the primary installation:

sudo /opt/mqm/bin/setmqinst -i -p /opt/mqm

Add the user to the group mqm in order to be administrator:

sudo addgroup $USER mqm

IIBExplorer installation

Before doing any installation, you should first install the prereqs librairies (see corresponding chapter).

Install IBExplorer. Go to the folder “/extractedfolder/messagebroker_ia_developer/IBExplorer”.

If the installer is not executable, you would have to change using the following command:

chmod +x install.bin

launch the installation:

sudo ./install.bin

And start the IBM Integration Toolkit with the following command:

sudo strmqcfg -clean

IBM Integration Toolkit

Before doing any installation, you should first install the prereqs librairies (see corresponding chapter).

The installation is then straightforward, just launch the installToolkit.

sudo ./installToolkit.sh

IBM Integration Bus

The runtime is installed in a very straightforward way.
Just go in the directory where the tar has been unzipped and launch the setup.

Intalling Eclipse prereqs

The installation manager, the IBM Integration Toolkit and IBM Integration Exporer requires to have 32bit libraries. I have found issues to run the installation without installing the "ia32-libs".
In order to install this library I have used the following command:
 

sudo -i
cd /etc/apt/sources.list.d
echo "deb http://old-releases.ubuntu.com/ubuntu/ raring main restricted universe multiverse" >ia32-libs-raring.list
apt-get update
apt-get install ia32-libs
rm /etc/apt/sources.list.d/ia32-libs-raring.list
apt-get update
exit
sudo apt-get install gcc-multilib

And the following libs:

apt-get install libc6-i386
apt-get install libgcc1:i386

A view of IBM Integration Bus runtime

If you are searching information on the components that IBM Integration Bus is composed of or just would like to know the terminolgy used (i.e. what is an Integration Node, an Integration Server, a message flow, ...) this post is for you !

I don't have the intention to explain every runtime components but rather provide enough information to understand the main components and the processes that are running.

First all, the IBM Integration Bus is a standalone application that is not running on a J2EE environment.
The runtime architecture provides a very strong resilience against failure and scalability capability.The core engine is very efficient and provides high performances with limited ressource usage. 
It can be easily deployed in a development workstation (development toolkit and runtime) without requiring a ferrari: I usually do demos (that includes IIB but DB2 as well) using a standard laptop having 4GB RAM.

The following figure describes the IBM Integration Bus runtime:

An integration node is composed by different processes and one queue manager. Note that in the former version of the Integration Bus, WebSphere Message Broker, the term IBM Integration Bus was called "Broker".
There are two kind of Integration node processes: those use to run the flows called Integration Servers  and those use for Integration Bus internal tasks.

The internal tasks processes are:

• TheIntegration Node wide listener, the "biphttplistener": this process is a http listener and it is started when the Integration node is configured to receive http request (http://www-01.ibm.com/support/knowledgecenter/SSMKHH_9.0.0/com.ibm.etools.mft.doc/bc43700_.htm?cp=SSMKHH_9.0.0%2F1-11-1-5-1-3-0-0&lang=en). It is started if the Integration node has been configured to have a Integration Node wide listener.
• The “watchdog” process, the "bipservice": it is a very small process that monitors (and restarts if necessary) the Admin Agent (bipbroker process).
• The Admin Agent (process name = bipbroker): this process is designed to monitor and administrate the Integration Servers. It performs such functions as handling deploy requests and restarting any Integration Servers that would stopped for unexpected reason. It does not process any message flows, but its size is governed by the size of the deploy requests that are being received.

The Integration servers are the processes where message flows are deployed and run. The name of the corresponding process is "dataflowengine". Message flows deployed in an Integration Server run as Threads.
Integration Servers are processes that are created by an administrator. 
IBM Integration Bus is proposed in different editions: express, standard and advanced. In the express and standard edition, only one Integration Server can be attached to an Integration Node.

All the integration servers within the same Integration Node use the same unique queue manager, the Integration Node queue manager. It is used as well to store eventual internal information as to get or put MQ messages processed by message flows.

The Integration Node queue manager is a standard WebSphere MQ Queue manager and can be administrated as any other queue manager. Even though it can be integrated into a WebSphere MQ network, using a MQ cluster or channels, and accessed by applications as a normal WMQ queue manager (using client connections for example), the usage of the Integration Node Queue Manager is restricted from a license point of view:it can process MQ messages as long as it is processed in one way or another by a message flow deployed on the Integration Node.

A message flow can be deployed on one or multiple Integration Servers. A number of instance of the message flow (corresponding to a number of threads) can be defined  on each Integration Server where it is deployed. Different message flows can be deployed on the same Integration Server.

Finally a message flow is composed by multiple message flow nodes (MQInput, MQOutput, SOAPInput, …) that are wired together.

IBM Integration Bus licensing principle on distributed explained

CPU Licensing principle for Distributed

Proper license has to be acquired in order to use IBM Integration Bus in a licensed environment.

The license agreement is related to the number of CPUs used by the IBM Integration Bus (IIB).

The number of CPUs used depends on the environment where the IBM Integration Bus is installed as the following figure shows:

If the IBM Integration Bus is installed on a non virtualized machine then the number of CPUs to take into consideration for license point of view is the total number of CPUs installed on that machine.

In a x86 virtualized environment, it is possible to allocate a variable number of CPUs to a virtual machine. In this environment the IIB is running in a virtual machine and the number of CPU to take into account for a license point of view is the number of virtual CPU allocated to this virtual machine with a maximum set to total number of physical CPU. This mean that if IIB is running in only one virtual machine having one virtual CPU, the number of CPU to take into account for the license will be 1 even though this virtual CPU is mapped to a physical CPU that is shared with other virtual machines. In the same way, if IIB is running in two virtual machines sharing the same physical CPU, each virtual machine having 1 virtual CPU mapped to the same physical CPU, the number of CPU to take into account for the license will be 2.

In a Power/ZSeries, it is possible as in the distributed environment to allocate a variable number of CPUs to a LPAR. However in this environment it is possible to share a CPU between LPARs with the restriction that a minimum of one CPU has to be taken into consideration for the license point of view. If we take the same example used above in the x86 environment then if IIB is running in only one LPAR having a micro partition of CPU, meaning that the LPAR is sharing the physical CPU with other LPARs, the number of CPU to take into account for the license will be 1. However if IIB is running on two LPARs sharing the same physical CPU, then the number of CPU to take into account for the license will still be 1.

IBM Integration Bus Editions Licensing

There are three mains edition available for IBM Integration Bus: express, standard and advanced.

The express edition targets customers that have a basic set of features and have only low capacity requirements. In this mode only a subset of message flow nodes can be used and only one Integration Server can be created by Integration Node. The available subset of nodes can be found at the following link (http://www-01.ibm.com/support/knowledgecenter/SSMKHH_9.0.0/com.ibm.etools.mft.doc/an67720_.htm?lang=en)

In the Standard Edition all the message flow nodes can be used (except the decision service node used to execute business rules and that requires specific entitlements). The limitation in this mode is that only one Integration Server can be created by Integration Node. The license doesn't restrict however the number of integration nodes.

In the Advanced Edition there are no limitation on the number of Integration Server per Integration Node. The restriction on the decision service remains.

The following figure shows this principle for the Standard and Advanced Edition:

Please note that this is my point of view and I am speaking for myself and not on behalf of IBM.

ActiveMQ & IIB with JMS

In this topic I will give you some hints on how to connect your IBM Integration Bus to a JMS provider ActiveMQ.
The following link is very useful:
http://www.ibm.com/developerworks/websphere/library/techarticles/1211_eswarachary/1211_eswarachary.html
There are however few information concerning the configuration.
I provided here after some extra informatio.

Configuration

Runtime configuration

In the MQ Explorer, select the "JMSProviders/ActiveMQ" configurable service  of the Integration node you want to use.




 The default_Path refers to the shared class directory of the IBM Integration Node.

 The jars of ActiveMQ have to be placed into a directory.
I usually place them in a dedicated directory under the homePath of the IIB Node which on windows is "C:\ProgramData\IBM\MQSI".
You can know the path using following command and look for "Local Work Path":

mqsiservice INodeName

Jar file

In order to be able to access activeMQ, the integration node needs the JMS provider jar file and place them in a location accessible to the node. I found convenient to copy the activemq-all.jar file only.

JNDI Bindings
In order to retrieve the connection factory, the Integration node needs to have a JNDI.

A default JNDI is provided with ActiveMQ: org.apache.activemq.jndi.ActiveMQInitialContextFactory

The jndi objects configuration can be found at the following link:
http://activemq.apache.org/jndi-support.html

You need to make this configuration file, the jndi.properties file, accessible to the IntegrationNode.
As explain at the link:
"
Once you have the jndi.properties edited and ready, it needs to be accessible to your application. The easiest way is to add jndi.properties to a jar file. When "new InitialContext()" is called, it will scan the resources and find the file.
 "
So place the file in a jar file - I just created a zip file, placed the property file in it and renamed the extension to "jar" - and place the jar file in the directory where you put the activemq jars.

In my configuration I have:




And in this directory I have the activemq.jar as well as another jar that contains my file "jndi.properties".

The content of my jndi.properties is:

java.naming.factory.initial=org.apache.activemq.jndi.ActiveMQInitialContextFactory
java.naming.provider.url=vm://localhost
connectionFactoryNames=connectionFactory, queueConnectionFactory, topicConnectionFactory
queue.jmsINQ=IN.Q
queue.jmsOUTQ=OUT.Q

There is a default connection factory "connectionfactory" and two queues.
The jndi name for the queue IN.Q defined on ActiveMQ is "queue.jmsINQ". The queue is accessed from the node using the alias name "jmsINQ" (without queue).

IIB Node configuration

Node configuration

The JMS Provider name on the JMSInput node is using a configurable service.
If the corresponding runtime configurable service property is not changed (let the defaulr), the property on the node will be used.
If you change the default property value on the conf. service then this property will be used.


Here is the configuration of my input node:

 

The connection factory "connectionfactory" is the one by default and defined in the jndi.properties.
And in the "basic" tab of the node, I put the alias name defined in the jndi properties file: jmsINQ for the input.



That's it !!

Testing

If you go on active MQ -> admin -> connections
you will see your connection.
Message can be send from ActiveMQ using the "send" tab.