Pages

Friday, June 30, 2017

Referenceable Variable in API Connect

This topic provides information about variables that can be defined and or accessed at runtime in API Connect.

These variables exists in the API Connect environment during the execution of the APIC assembly.
The assembly is the definition of the processing flow that is executed by the API Gateway at runtime when a request is received. This processing flow consists of policies and these can be "validate", "map", switch, javascript, xslt, invoke, ...

These variable can be referenced in APIC policies using the notation $(varName). For example in a invoke policy, you might want to evaluate dynamically the host name or the URL:



In this snapshot, the "backendurl" and the "payid" are variable that are referenced.

What are the variable that can be defined and referenced?


There are two main type of variables: context variable and API properties.
Context variables are variables that the value usually depends of the message request whereas API properties have their values defined when the API is published.

There are three main subcategories (my interpretation) of context variables:

  • user variables: these variables are defined in the assembly using javascript, xslt, or set variable
  • policies variables: these variables are instantiated by the policy that you set on the assembly. For example the validate policy creates context variable "decoded.claims" (validate jwt).
  • apic environment variables: these variables are set by the apic environment. For example the variable message (corresponds to the current data on the flow) and the request (corresponds to the input message and should be considered immutable).

How can these variables be accessed/defined ?

API Properties

API properties are defined by the "properties" in the API design UI (or the yaml). The value of these variables may differ depending on which catalog the API is published. 
This can be useful if for example within the same APIC cloud environment different catalogs are defined for different environment (f.e. DEV, TEST). You can for example define a "backendurl" that would be different if the API is deployed in the DEV catalog or in the TEST catalog.

These variable are accessed at runtime the same way as context variable using the notation $(myPropertyVar) or in a gatewayscript/xslt using the getvariable function (more later).

Context Variables

Unless set by the runtime, context variables are set programatically using either the setVariable, the gatewayscript (javascript) or the xslt policy.

In gatewayscript or xslt the variables are accessed using a provided getVariable or setVariable function.

Using gatewayscript:

apim.setvariable('payid',myvar,'set');
apim.getvariable('paid');

or using xslt:

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0"
  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
  xmlns:dp="http://www.datapower.com/extensions"
  xmlns:func="http://exslt.org/functions"
  xmlns:apim="http://www.ibm.com/apimanagement" extension-element-prefixes="dp func apim">
  <!-- Contains the APIM functions -->
  <xsl:import href="local:///isp/policy/apim.custom.xsl" />
  <xsl:template match="/">
    <xsl:call-template name="apim:setVariable">
      <xsl:with-param name="varName" select="'payid'" />
      <xsl:with-param name="value" select="'avalue'" />
    </xsl:call-template>
    <xsl:message>
      <xsl:text>Variable [</xsl:text>
      <xsl:value-of select="'payid'" />
      <xsl:text>] set to [</xsl:text>
      <xsl:value-of select="'avalue'" />
      <xsl:text>]</xsl:text>
    </xsl:message>
  </xsl:template>
</xsl:stylesheet>


More example can be found here for gatewayscript or for xslt.

Available runtime context variable


A list of runtime context variable that you can access are provided here at the knowledge center.

Notes that even when the API Gateway is DataPower, only the API runtime context variable can be reached. You can't for example access DataPower service variable.
If you would like to access them, you would need to create an APIC custom policy that would use either a gatewayscript action or xsl action to set an APIC environment variable using the apim set variable (using as example the snippet code provided before).

Monday, December 14, 2015

IBM Integration Bus Editions compared

IBM Integration Bus Edition/Mode comparison


This post will explain the difference between the different IBM Integration Bus edition.

IIB is available in four different modes described at the link http://goo.gl/oIVCLM.
Comparison of the different editions is also provided at the page IBM Integration Features.

These are (from the higher to the lower capacity): Advanced, Standard, Scale and Express.
Note that
  • There is a trade up path from any lower version to a higher one. 
  • The mode can be changed without requiring to reinstall the product, it's just a command line
In this post I will not linge on the scale mode as it was introduced to have a migration path for users running on WebSphere Enterprise Service Bus.
As this later will be end of support in 2018, users have the possibility to convert their WESB licenses to IBM Integration Bus in scale mode. This mode as some restrictions to match what WESB was offering.

The following schema provides an overview of the Advanced, Standard and Express mode:


















Express vs Standard

The Express and Standard mode have a common limitation: they can only have one Integration Server per node. This will be explained in further detailed later when comparing Advanced to Standard.
The difference between Standard and Express stands in features that are provided.
The features available for each mode are provided here at the knowledge center features per operation mode.
Express only provides a subset of features that are already very rich. You can implement flows using most of the nodes, performs transformations using graphical mapper, java, .NET.

Please find after the most important features that are not provided and would help you decide if this mode is appropriate for your usage:

  • Resequence nodes: to resequence the messages order using a built-in node
  • ERP nodes: SAP, SIEBEL, ...
  • CICS, IMS nodes
  • File Read node: to retrieve a file content in the middle of a flow
  • DataBase Input node: polling a database using a built-in node
  • Collector node: been able to collect messages from different sources using a correlation
  • Policy Enforcement Point node (PEP): been able to enforce the security in the middle of a flow
  • MQ Managed File Transfer nodes
  • ESQL code

Standard vs Advanced

In Standard all the features are allowed: in term of features, there is no difference between standard and advanced.
The difference stands in the number of Integration Server that can be defined per Integration Node.
Find at the following post the overview of the runtime if you are not familiar with the terms:  a-view-of-ibm-integration-bus-runtime.


This overview will help to understand the main differences and implication of the different editions:

  • Isolation: on advanced mode, you can have multiple integration server per Integration node. This means that you have a greater isolation when running on advanced mode as you can have flows deployed on separate integration server. You gain isolation in term of memory and address spaces.  If one flow makes the process to crash, it will not affect flows deployed on other Integration Server. 
  • Administration: you can administrate only one integration node at a time. You can administrate multiple Integration Servers of one Integration Node through the same web user interface. If you have multiple Integration Node, you would need to have multiple UI. Also if you need to apply a configuration change that requires the process to restart, all the flows deployed in the Integration Node will be impacted.
  • Queue Manager: in advanced mode all Integration Server of the Integration Node can access in binding mode the same queue manager associated with the Integration node (in V10, integration node doesn't required to have a queue manager - in some cases you would need to do so though). If you have multiple Integration Node it may be required to interconnect them.
  • Scalability: as each Integration Server is one process, it may be possible to better distribute the load across available processors when running in advanced mode. 

If you would like to know the limitation in term of license between these two modes, please find this information at the following post ibm-integration-bus-licensing-principle.

Thursday, July 30, 2015

Test SSL configuration with curl

In a previous article I explained how to configure IBM Integration Bus to use HTTPS with an InputHTTP node (httphttps-listener-behavior-with-iib.html).

I realized that it may not be evident to test the configuration.
I will provide here some hints on how to test a configuration where a server (for instance IBM Integration Bus) is configured to receive https connections with mutual authentication.

For these test I am using very useful tools: curl and openSSL.
useful information on curl can be found here.

Configuration

The keystore of the Integration Server node holds the personal certificate of the server. This certificate contains a public and private key.
The trustStore of the Integration Server node holds the certificate of the client that needs to be authenticated. This certificate contains only a public key. This key has been provided by the client.

In order to make a mutual authentication, the public key of the server has to be provided to the client and the client has to provide it's public key.

To extract the IBM Integration Server certificate from the JKS keystore, one can use the IBM Key Man tool. This tool is started from the menu (windows) or using strmqikm.
Select the folder Personal Certificate and click on extract certificate. If you select the type as "Base64-encoded ASCII data" the certificate will be in the PEM format (privacy-enhanced mail).
The tool provides as extension "arm". This format is equivalent to pem. The extension can be changed from PEM to ARM.

If you used the key man tool to create a self-signed certificate for the client, you would need to export the certificate in pkcs12 format (p12). This certificate would contains the public and private key.

Certificate format 

PEM  (privacy-enhanced mail) format
It's a "Base64-encoded ASCII data" certificate and this format is equivalent to arm. The extension can be changed from PEM to ARM.
PKSC12 certificate may have pfx or p12 as extension.
DER is a binary encoded certificate.
Get more information on SSLShopper

Testing

Curl requires PEM certificate.

In our example here, the client needs to have a personal certificate to be able to sign. The personal certificate for the client is in PKCS12 format, therefore you would need to convert it in PEM format.
This conversion can be done using openSSL (openssl commands) using the command:

openssl pkcs12 -in ClientPersonalCert.p12 -out ClientPersonalCert.pem -nodes

You can then use Curl to call the service.

curl --carcert serverCertificate.pem --cert clientPersonalCert.pem:<password> --cert-type PEM https://myserver:port/test


  • serverCertificate.pem is the certificate from the server that has been extracted from the keystore. It holds only a public key
  • clientPersonalCert.pem is the personal certificate of the client. This certificate has been exported from a keystore and has been converted into a PEM format.

If you need to perform an HTTP GET with query parameters, you may use the following curl command:

curl --carcert serverCertificate.pem --cert clientPersonalCert.pem:<password> --cert-type PEM -G -d "<myqueryParms>" https://myserver:port/test


    • G is to tells that you are issuing a GET
    • d is to provides the query parameters

Thursday, July 23, 2015

HTTP/HTTPS listener behavior with IIB HTTPInput nodes

HTTP/HTTPS listener behavior with IIB HTTPInput nodes

When a flow containing HTTPInput nodes are deployed on an Integration Server, the default behavior is to use the broker wide HTTP Listener.
This is different is you are deploying a flow using SOAP nodes. In this later case, the http listener used is the embedded HTTP listener of the Integration Server.

For your information, the broker wide listener is using MQ behind the scene. So it can be enabled on the version 10 if a default queue manager has not been configured.

In this blog I will explain how to configure the Integration Node to use the embedded listener of an Integration Server when using HTTP nodes. 
I will also explain how to configure the Integration Node to use SSL (HTTPS).

In the following text, I will assume that
* The integration node is called: IBMIBus
* The integration server is called: IServer1

Configuration for Embedded HTTP Listener

First check the configuration of the Integration Server using the following command:
mqsireportproperties IBMIBus -e IServer1 -o ExecutionGroup -a


This command will show the property "httpNodesUseEmbeddedListener". If this property is set to true, this means that when you will deploy a flow having a HTTPInput node, the embedded HTTP listener will be used.
To change this value use the following command:
mqsichangeproperties IBMIBus-e IServer1 -o ExecutionGroup -n httpNodesUseEmbeddedListener -v true
The port used by the embedded HTTP listener is defined dynamically when the first flow having HTTP nodes is deployed or when the Integration Server is started if it had such flow already deployed. If no flow having HTTP nodes has been deployed, the listener will not be activated.

To check the port used by the embedded HTTP listener, use the following command:
mqsireportproperties IBMIBus -e IServer1 -o HTTPConnector -a
The port can be specified if required (this will disable the automatic port number attribution). This is done using the following command:
mqsichangeproperties IBMIBus -e IServer1 -o HTTPConnector -n explicitlySetPortNumber -v 8085


Embedded listener configuration for SSL (HTTPS)

In this part, I will provide the commands to configure the embedded HTTP listener to use SSL.

Prerequisites
* The Integration Server has been configured to use embedded HTTP listener
* A key store has been created. It contains a certificate for the integration server (that can be used for the public and private key)
* A key store or trust store containing the client certificate if mutual authentication is required.
* The password used to access the keystore is "password".

The keystore and truststore configuration can be found at the following link:

Configuration

The Integration Server uses two objects to configure the SSL: the ComIbmJVMManager and the HTTPSConnector
The ComIBMJVMManager object is used for the entire Integration Server. It is used by input HTTP nodes as well as request HTTP nodes.
The HTTPSConnector is used only for the input HTTP nodes. 
If you need different keystore for the http request nodes and for the http input nodes then you may configure the ComIBMJVMManager for the HTTP request nodes and the HTTPSConnector for the input http node.
If there is no differences, you can configure only the ComIBMJVMManager object.

ComIBMJVMManager configuration


The following command is used to configure the object:
mqsichangeproperties IBMIBus -e IServer1 -o ComIbmJVMManager -n keystoreFile -v "c:\ks_IBMIBus.jks"
mqsichangeproperties IBMIBus -e IServer1 -o ComIbmJVMManager -n truststoreFile -v "c:\ks_IBMIBus.jks"
mqsichangeproperties IBMIBus -e IServer1 -o ComIbmJVMManager -n keystorePass -v <password>
mqsichangeproperties IBMIBus -e IServer1 -o ComIbmJVMManager -n truststorePass -v <password>
mqsichangeproperties IBMIBus -e IServer1 -o ComIbmJVMManager -n keystoreType -v JKS
mqsichangeproperties IBMIBus -e IServer1 -o ComIbmJVMManager -n truststoreType -v JKS
<password> the password to provide. You may provide the password directly in the command line or store the password in the secure integration node registry using the command mqsisetdbparms.
To use the secure registry, you have to provide the password in the command line with the form <MyIntegrationServer>Keystore::password. The command would then be:
mqsichangeproperties IBMIBus -e IServer1 -o ComIbmJVMManager -n keystorePass -v IServer1Keystore::password

Then store the password using the command line
mqissetdbparms IBMIBus -n IServer1Keystore::password -u ignore -p password
The user has no usage here, you may set whatever value you would like.

You need to restart the integration node if you change any of these properties.

If you need to configure the HTTPSConnector, follow the same approach.

Important Note: if you are using a browser tools like HttpRequestor from firefox, you would first need to accept the server certificate. This may be done by simply performing a GET of the service URL in firefox self. You would then be prompted to accept the certificate.

Specific server certificate to be used

You can specify the certificate to be used by the HTTPInput node for SSL. By default the first personal certificate found in the keystore is used. This certificate is used to authenticate the server to the client.
If you require to set a specific one set the property "keyAlias" of the object HTTPSConnector to the right alias.
mqsichangeproperties IBMIBus -e IServer1 -o HTTPSConnector -n keyAlias -v myAlias

Mutual authentication

To enable mutual authentication,  the property "clientAuth" of the object HTTPSConnector has to be set to true.
mqsichangeproperties IBMIBus -e IServer1 -o HTTPSConnector -n clientAuth -v true
By setting this value you would have using a browser:
Error code: ssl_error_handshake_failure_alert

Create a certificate and add the certificate containing the public/private key to the browser and the public certificate to the Integration Server Truststore (or keystore depending of your configuration). 

On firefox, this is done by going to option -> Advanced -> Certificates -> View Certificates -> Your Certificate -> import
You should have a pfx or p12 file ready.
You may create a self signed certificate for test, using the IBM key Management tool. 
Create a self signed certificate then export and select the "PKCS12" key file type.





Thursday, March 12, 2015

setup/script in IIB for Record-Replay


You will find in this post the commands that is necessary to configure the Integration Bus to record and replay the events generated by the flows.

Even though all the following information is available in the knowledge center, I found sometime difficult to find out all the necessary commands to be executed.

Parameters

Integration server

Integration Node name: <INName>
Integration Server name: <ISName>
IIB Queue Manager name: <IIBQMgrName>

Configurable Service

    Data capture store name. Configuration to define the database to be used: <DCStoreN>
DataCaptureSource name. Configuration to define the event source: <DCSourceN>
Data Destination Name: configuration used to define the queue where the message will be send when using the replay mechanism: <DDName>
Queue name used to send back the data: <ReplayQName>

Database configuration

ODBC Database DSN: <DSN>
Table Schema used to store the IIB events: <IIBSchema>
User/password for accessing the database under the schema <IIBSchema>: <DBUsr>/<BDPwd>

Script

Create configurable services for IIB

1. DataCaptureSource

mqsicreateconfigurableservice <INName> -c DataCaptureSource -o <DCSourceN> -n dataCaptureStore,topic -v <DCStoreN>,"$SYS/Broker/<INName>/Monitoring/#"
2. DataCaptureStore
mqsicreateconfigurableservice <INName> -c DataCaptureStore -o <DCStoreN> -n backoutQueue,commitCount,commitIntervalSecs,dataSourceName,egForRecord,egForView,queueName,schema,threadPoolSize,useCoordinatedTransaction  -v "SYSTEM.BROKER.DC.BACKOUT","10","5",<DSN>","<ISName>","<ISName>","SYSTEM.BROKER.DC.RECORD","<IIBSchema>","10","false"
3. DataDestination
mqsichangeproperties<INName> -c DataDestination -o <DDName> -n egForReplay,endpoint,endpointType  -v "<ISName>","wmq:/msg/queue/<ReplayQName>@<IIBQMgrName>","WMQDestination"

Database connection configuration

1. Create odbc connection in ODBC Data Source Administrator (Demo =  <DSN>)
2. Set security connection information
mqsisetdbparms <INName> -n <DSN> -u <DBUsr> -p <BDPwd>
3. Run script DataCaptureSchema.sql from db2 command line (non-administrator). This script are available under
<IIBInstallation>\ddl\db2

Tuesday, March 3, 2015

IBM Integration Explorer installation

I recently had some trouble with my IIB plugin (administration) after applying a MQ Explorer Fix:
I downloaded the MQX fix, and after the installation, the IIB nodes are not available anymore in my explorer.

It seems that the IIB is not installed.
I remembered that the plugin is configured by adding the links in the eclipse installation folder.
This seems to be ok as well.

I decided to uninstall the IBX and reinstall it but even that doesn't solve my issue.

After chatting with some Gurus, I finally discovered that eclipse does not notice that the plugin is no more linked or updated.

So what is the option?

The trick is to uninstall the IBX and reinstall it but under another directory.
From that moment, I usually set a version number on the folder name:
C:\IBM\IIBExplorerFP2

This is it.

Don't forget to start the IBX after using the -c option.

If you still have any trouble, let me know.

Monday, January 26, 2015

ESQL code to create mail with attachments using broker events

In this post I will provide an example on how to process events generated by a flow using the default IBM Integration Bus monitoring event capability.

The example will show how

  • to serialize a tree into BLOB using ESQL
  • how to send a mail with attachment
  • How to create/prepare the LocalEnvironment and Root for the EmailOutput node
  • The usage of the business event capabilities provided by IIB


The principle is simple:

  1. Configure a flow to generate an IIB event
  2. Create a subscription to this event with a WMQ Queue as endpoint
  3. Create a flow that consumes these events and send an email with attachments

The example in this post shows how to create mail with attachments using ESQL but this could be easily made using Java as well.

Configure a flow to generate an IIB event


The event generated as a well defined structure and the schema can be imported into a library using new model -> IBM predefined model.

Any nodes in a flow can be configured to generate events (Generating events in WebSphere Message Broker) that may contain context information and payload.

It is for example possible to configure a node to include the localEnvironment, ExceptionList and Message tree structure (under Root). These information will be placed into the IIB events under the folder "complexContent".
Note that the LocalEnvironment is reset when an exception occurs, so the data that would have been stored in this tree would be wiped when the message is propagated to the catch terminal of the input node (will be covered in a future post).

Finally it is also possible to include the full payload (as it was received) by selecting in the monitoring node properties "include payload as bitstream". The payload will then be included into the IIB events under "BistreamData".

Create a subscription

The IIB runtime is publishing the IIB events on the WMQ topic "$SYS/Broker/IBMIBus/Monitoring/#".
Using the WMQ Explorer you create a subscription to these events and select a destination queue:

 The flow that sends email

The flow is very simple: MQInput -> ComputeNode -> EmailOutput node
The compute node is used to create and configure the message that will be send using the emailoutput node. 
The node it self is configured the minimum properties: server:port, email to, from and security.
The rest will be provided by the code in ESQL (subject, body content and attachments).

In this example the complexContent included in the incoming business event is serialized into bitstream and will be send by mail as attachment.
The payload if present is also send as attachment.
The body of the mail is made of event origin data and using a DFDL to have a text document separated with CRLF.

The code is provided here after:




Friday, January 23, 2015

ESQL code Sample

In this post I will provide some example of ESQL code that could be useful.

The codes are made available using Gist.

TREE --> XML 

In the following gist, I provide an example on how to create a XML physical representation of a IIB in memory tree.
The principle is the following
  1. Create an element of type XMLNSC parser
  2. Copy or create a IIB tree
  3. Use the function ASBITSTREAM to serialize the tree in bitstream using the parser (here XML)


BLOB --> XML

In the following gist, I provide an example on how to create a XML tree from a BLOB.
In the example the BLOB is provided as test in a hexadecimal representation. The code parses it to an XML and append it in the current XML.


Friday, January 16, 2015

XPATH in IIB

In this post, I will provide some example of xpath expression allowing to perform complex transformation within a GDM (Graphical Data Mapper).

I will append this post with new example that I would found that could be of any usage.

For this first post, I will use a sample message having the following structure

<?xml version="1.0" encoding="UTF-8"?>
<Q1:INVOICE xmlns:Q1="http://www.acme.be/acme"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.acme.be/acmeInvoice.xsd ">
<CUSTOMERID>ID001</CUSTOMERID>
<NAME>X</NAME>
<INVOICE_ITEM>
<STOCK>OUT</STOCK>
<EXI>IMPORT</EXI>
<CONTAINER>abc</CONTAINER>
<ISOCODE>2210</ISOCODE>
<QUANTITY>1</QUANTITY>
</INVOICE_ITEM>
<INVOICE_ITEM>
<STOCK>OUT</STOCK>
<EXI>EMPTY</EXI>
<CONTAINER>abcd</CONTAINER>
<ISOCODE>2210</ISOCODE>
<QUANTITY>2</QUANTITY>
</INVOICE_ITEM>
<INVOICE_ITEM>
<STOCK>OUT</STOCK>
<EXI>EMPTY</EXI>
<CONTAINER>abcde</CONTAINER>
<ISOCODE>4532</ISOCODE>
<QUANTITY>4</QUANTITY>
</INVOICE_ITEM>
<INVOICE_ITEM>
<STOCK>OUT</STOCK>
<EXI>EMPTY</EXI>
<CONTAINER>abcdef</CONTAINER>
<ISOCODE>4532</ISOCODE>
<QUANTITY>2</QUANTITY>
</INVOICE_ITEM>
<INVOICE_ITEM>
<STOCK>IN</STOCK>
<EXI>IMPORT</EXI>
<CONTAINER>CONTAINER</CONTAINER>
<ISOCODE>ISOCODE</ISOCODE>
<QUANTITY>2</QUANTITY>
</INVOICE_ITEM>
</Q1:INVOICE>


Sum, Count

It is possible to compute the sum of elements under a repeating node in one expression.
For example if you would like to make the sum of the "Quantity" field of all the INVOICE_ITEM nodes, you could do this by performing the following map:

XPath expression is "fn:sum($INVOICE_ITEM/QUANTITY)"

Count can be used in the same way. Count will provide the number of elements.

Predicates

Predicates allows to select only a subset of nodes based on a criteria.
In the above example, it may be possible to compute the sum of QUANTITY for INVOICE_ITEM having its STOCK child element equal to "OUT".
This could be done by using predicates.
The above xpath expression would be changed with:

fn:sum($INVOICE_ITEM[STOCK='OUT']/QUANTITY)

The predicates here is "[STOCK='OUT']".

It is possible to use an input element within the predicates. For example, imagine that you have a list of Stock. And you would like to make the quantity sum only for the STOCK that are in the list.
This could be achieved by providing as input the stocklist to the xpath transform:
With the XPath expression
fn:sum($INVOICE_ITEM[STOCK=$STOCK1]/QUANTITY)

It is possible to have more than one "where" clause. For example is we would like the sum of  Quantity for STOCK='OUT' and EXI='IMPORT' then the following Xpath could be used:

fn:sum($INVOICE_ITEM[STOCK='OUT' and EXI='IMPORT']/QUANTITY)

be careful that the "and" is case sensitive.

Distinct Values

Last useful xpath expression for this post. Distinct values.
Distinct values could be used to retrieve only elements having distinct values.
For example let's say that the input message where STOCK can have the value "IN', "OUT'.
In the example above there are 4 Invoice Item with Stock equals to "OUT" and only 1 with "IN".
The XPATH expression used here is "fn:distinct-values($INVOICE_ITEM/STOCK)"
The input is the INVOICE_ITEM repeating node and the output is a repeating element STOCK under STOCKLIST.
The Stocklist will be populated by STOCK element having the value "IN" and "OUT":

<STOCKLIST>
        <STOCK>IN</STOCK>
        <STOCK>OUT</STOCK>
</STOCKLIST>

If you need to sort on multiple elements then the option that you may have is first create a concatenation of these fields in a previous map and then use the distinct values xpath expression.
So for example if you need to have the list of invoice items having distinct values for STOCK and EXI, then you may create a first map that concatenates STOCK and EXI using the xpath expression "fn:concat" and place this list in the localenvironment and then in the next map use the disctinct values xpath expression
.







HA Cache deployment with IBM Integration Bus

I have been involved in project where customer is looking to have a possibility to cache data  in a high available way within the integration layer.

In this post I will provide some points that have to be take into account when designing the IIB deployment architecture in order to have a high available cache.

Introduction

IBM Integration Bus provides an out of the box caching mechanism based on WebSphere eXtreme Scale.
WebSphere eXtreme Scale provides a scalable, in-memory data grid. The data grid dynamically caches, partitions, replicates, and manages data across multiple servers.

This cache can be used to store reference data that are regularly accessed or to hold a routing table.

The cache is not enabled by default and it is really easy to enable it: the default configuration is activated by setting a configuration parameters through the IBM Explorer administration tool.
To activate the cache across different Integration Node instances, a XML configuration file (templates are provided) has to be defined.
More information on the cache can be found here What's new in the Global Cache in IBM Integration Bus v9

Specialized skills in extremes scale is not necessary in order to use the cache. There is however two important cache components that may be good to know

  • Catalog servers: component that is embedded in an integration server and that controls placement of data and monitors the health of containers. You must have at least one catalog server in your global cache.
  • Container servers: component that is embedded in the integration server that holds a subset of the cache data. Between them, all container servers in the global cache host all of the cache data at least once. If more than one container exists, the default cache policy ensures that all data is replicated at least once. In this way, the global cache can cope with the loss of container servers without losing data.
More information about terminologies can be found here
Global cache terminologies

Principle

The catalog and container servers are embedded in Integration Servers.

To have a high available cache the following is required:

  • At least two catalog servers have to be online: without catalog server it is not possible to reach the data in memory
  • At least two container servers have to be online: this is necessary to replicate data on two different location

One more important point to know: it is not possible to configure an integration node to host a catalog server when it is configured as a multi-instance Integration node.

Possible deployment architecture

If the target is an active/active deployment, the following architecture is possible:
In this architecture, the catalog server is deployed in one Integration Server on both side. The other Integration servers are used to host containers. 
To improve the performance, the catalog server would be placed on a dedicated Integration Server (separated from the containers). This is not required though, a catalog server may reside on the same server as a container.
If the license doesn't permit to have multiple Integration Server per Integration Nodes (standard edition), then you could create a separate Integration Node on the same server to host the catalog server. 

If the target deployment consists of multi-instance queue managers, because for example the message residing on MQ has to be quickly recovered, the following architecture is possible:

Due to the fact that a multi-instance Integration Node can't host a catalog server (a configuration restriction), it is necessary to define an extra Integration Node to hold the catalog server (Integration Node - Catalog). This Integration Node doesn't need to be high available.
The multi-instance Integration Nodes are configured to host the container servers. Two active integration nodes are required to provide an high available cache (replication made on two different servers).

Additional information

The first time that the cache is configured using multiple catalog servers, the cache will become operational when at least two catalog servers are started. If the catalog servers are defined in two Integration Nodes, these two nodes would need to be started before been able to use the cache. Once the cache has been activated (this can be checked by looking at the logs or administration events) it is possible to lose (or stop) catalog servers (at least one catalog server should stay online) without impact to the cache access. This can be useful if maintenance on one instance has to be performed.

The default configuration is to have a maximum of 4 container servers per Integration Nodes. But more containers can be configured by configuring the Integration Server manually using the IBM Integration Explorer.

There are no limitation in term of container servers that can participate to the global cache. If you require more memory, you can just add a new container in the system. There is no need to restart the whole system !! Or you could also access an external extreme scale component like XC10 (Deciding between the embedded global cache and an external WebSphere eXtreme Scale grid).

Integration Server roles can be changed using the IBM Integration Explorer. It is possible to define a policy configuration file and assign it to the Integration Node using the IBM Integration Explorer. Once this has been done, you can start the Integration Node to take the configuration into account. If the global cache policy of the Integration Node is changed using the IBM Integration Explorer to "NONE" when the Integration Node is running, the current configuration will be held even if the Integration Node is restarted. It is therefore possible to change the Integration Server roles afterwards. More information on how to set the roles are provided here How to fix integration server roles in a Global Cache configuration in IBM Integration Bus and WebSphere Message Broker V8.

References

Information about global cache


Tuesday, October 28, 2014

IBM Integration Bus sizing and performance

It's not the first time that I am been in a situation where I have been asked to evaluate how many cores would be necessary to run a load on IBM Integration Bus (IIB).

In this post I will provide you a way to make this evaluation based on public IIB performance reports.

Required information  

Before starting your evaluation, you first need to identify a set of flows that corresponds to ESB patterns that you would like to achieve: transformation, aggregation, protocols, logging, ...

You would like as well to define on what operating system the run time will run.

For these flows you will need to define the following information
  • Kind of messages: XML, non XML
  • Average size of messages
  • Load peak expressed in transaction/sec. I usually use the peak since this is the throughput that you would like to sustain. 
  • Operating system where the IIB will run
Note that the CPU type has also it's importance. Normally you would have to add a corrector factor to compensate the fact that you have a more efficient processor.

Performance reports

Once this information is available, you can start your performance evaluation computation.
For this we will use performance report that is publicly available. For IBM Integration Bus, go to the link message throughput. (good link to bookmark is the performance topic). For older version, you can find performance reports at the support pack link.
On the message throughput link select your operating system.

These performance reports have been measure using a processor: IBM xSeries x3690 X5 with 2 x Deca-Core Intel(R) Xeon(R) E7-2860.
For those that would like to know the weight associated to this processor, please have a look at the PVU information page. You will find that this processor corresponds to a 70 PVU processor (2 sockets with 10 cores).

You will find different type of patterns that have been tested: aggregation, coordination request/reply, message routing, ...
For each of these tests there is a table providing the message rate, the CPU busy and the CPU ms/msg.

The information that we will keep are the 
  • message rate
  • the CPU ms/msg

Performance evaluation

How can we use these information?

Let's take an example!

Imagine that you would like to deploy a flow doing aggregation with an average non-persistent message size of 2 kb.
This flow will process 460000 messages a day but there is a peak! During one hour in the day the flow has to sustain 720000 messages.

We would like to size the bus such a way that he will sustain the peak. This corresponds to a throughput of 200 trs/sec.
Lets have a look to the tables available on the performance report page. There is one for the aggregation:

Non PersistentFull Persistent
Msg SizeMessage Rate% CPU BusyCPU ms/msgMessage Rate% CPU BusyCPU ms/msg
256b4020.199.34.941476.912.45.215
2kB3430.698.35.732707.819.75.559

For 2kb non persistent message, the processing of one message takes 5.732 ms of CPU.

The processing of 200 transactions per seconds would take 200 (msg/sec) *5.732 (CPU ms/msg) = 1146.4 (msg/sec)*(CPU sec/1000 * 1/msg) = 1.1464 CPU. 

So you would need to at least 2 processors to process this load. The total load of the machine with 2 CPU would be 57.32%.

It is not recommended to have a processor loaded at more than 70 % but here the 2 CPU will do the deal.

Of course you could have more complex situation such MQ, JMS, database, routing....

It is possible to approach more complex situation by adding the CPU load for each scenarios.
If my integration needs to be exposed as web service using SOAP messages, performs routing and transforms messages, I would approximate my CPU load by adding the corresponding load factor: 2.633 (SOAP) + 0.488 (routing) 0.896(transformation).

This is not ideal since each scenarios include the protocol processing like MQ for routing.
In the previous reports that are available in the support pack page, more tests have been done and it is possible to build a spreadsheets to isolate the processing of each mediations.
For example there is a test made for MQInput-MQOutput (x ms/msg), MQI-Transformation-MQO (y ms/msg) and JMSIn-JMSout (z ms/msg). For a JMSI-Transformation-JMSO integration flow, you could therefore approximate the CPU load using: y ms/msg - x ms/msg + z ms/msg.
The performance in IIB V9 is higher than for V8, so using the old performance reports would provide you a good estimation with a security factor.

References

message throughput information about IIB in function of the operating system.
performance information, tips and hints.
support pack old performance reports.
PVU information page to find the PVU for a defined CPU.

Tuesday, October 7, 2014

MQ Cluster demistified

I would like to provide through this post some highlighting on the MQ clustering: what is it, what does it brings and how to set it up.

MQ cluster is not about data replication or make data high available, it's about making object definitions known and available in a group of queue managers and providing a workload management capability.

 The "what is it"


Put it simply, a MQ Cluster is a collection of queue managers that have been defined to be part of a group called a cluster.
Within this group, queue managers know all objects that are shared within the cluster. Objects are like queues, topics.
Within a cluster all queue managers know how to send messages to the target clustered destination. The message transfer are handled automatically by the queue manager in the cluster.

The "what does it brings"

The cluster provides the following main advantages:
  • Simplify administration for distribute queueing. When a message has to send to a destination located to another queue manager, the queue has to be simply shared in the cluster on the remote queue manager that's it. The transfer of the message from the queue manager where the application has put the message to the remote queue manage is carried out by the cluster: no remote queues, no transmission queues, no channels. 
  • Workload balancing: when multiple instances of queues are defined, the cluster can balance the message distribution to all instances shared in the cluster. Priority and weight can be defined for specific queue managers. 
  • Improve availability and simplify maintenance: when multiple instance of queues are defined and one queue manager that hosts this queue is not available, the messages are automatically routed to the remaining queues. This increases the availability since messages can still be processed but it also simplifies the maintenance as the queue manager is known to be stopped in the cluster. 

Some disadvantages that I have in mind:
  • Queue manager in a cluster are directly connected with each others. It increases the number of channels which means consume more resources. 
  • Less control on how the messages are sent from one queue manager to another 
  • Less flexibility in the queue name resolution compared to the distributed queueing (alias, remote queues, ...). 

The "Principle" 

Let's first discuss about queue destinations.
When an application tries to put a message to a queue and this queue is not known by the local queue manager (the queue manager that the application is connected to), the queue manager queries the cluster (if it is part of one) to know if this queue is known.
If the queue is shared within the cluster, the information about its location is provided to the requestor queue manager. The local queue manager handles the message by putting it in the cluster transmission queue with a transmission header containing all the information to route it to the correct destination. The message is then sent by the local queue manager directly to the target queue manager.

Be aware that an application can get messages from local queue only, cluster defined or not. Local queues are queues defined on the queue manager that the application is connected to.
The story is a little bit different for topics. Indeed topic is used to define a destination where an application can publish or subscribe.
When a topic is shared within a cluster, it is known by all queue managers in this cluster.
Therefore if an application subscribes to a cluster topic it will receive all messages published to this topic even though the topic has been created in another queue manager.

The "how to make it" 

You will be puzzled how simple it is to create a cluster !!

Before going further there is one think to know about: the repositories. A repository is a store used by a queue manager to store cluster object definitions.

There are two types of repository: full and partial.
A full repository is a repository that contains cluster object definition from the whole cluster. When an object is shared in a cluster, the object definition is sent by this queue manager to the queue manager that holds the full repository.
A partial repository holds information about cluster object definitions that have been resolved from the queue manager that holds this repository. When an application puts a message for a remote queue shared in the cluster, the local queue manager requests the destination object information to the queue manager holding the full repository. This information is then stored in its local repository. Hence the name "partial repository" as it does not store all object definitions of the whole cluster.

In order to exchange these object definitions, queue managers use cluster channels:
  • a cluster receiver channel to receive object definitions 
  • a cluster sender channel to send object definitions to the queue manager(s) holding the full repository. 
Two rules about the repositories:
  • A queue manager can hold either a full repository or a partial repository (not both) 
  • Two (or one if not in production) full repositories is enough. All the other queue manager would hold a partial repository only. 
With this in mind, let's have a look how to make it !

The steps provided here after has to be performed in the same order.

For full repository
  1. Set the queue manager property "REPOS" to the name of the cluster. This will tell to the queue manager that he will host a full repository. A runmqsc command would be
    ALTER QMGR REPOS('myCluster')
     
  2. Create cluster receiver channel to define how to connect to this queue manager: 
    DEF CLUSRCVR(TO.MyQMgr) 
    CONNAME('ipaddressOfThisQMgr(portNumberOfThisQMgr)') 
    CLUSTER('myCluster')
  3. Optionally cluster sender channels can be created to point to another full repository ONLY. 
 For partial repository
  1. Create cluster receiver channel to define how to connect to this queue manager:
    DEF CLUSRCVR(TO.MyQMgr)
     CONNAME('ipaddressOfThisQMgr(portNumberOfThisQMgr)') 
    CLUSTER('myCluster')
  2. Create a cluster sender channel to define how to reach a queue manager holding a full repository (not partial).
    DEF CLUSSDR(TO.FullRepoQMgr) 
    CONNAME('ipaddressOfFullRepoQMgr(port)') 
    CLUSTER('myCluster')
That's it !!
Now you can create a queue and share it to the cluster "myCuster" that you just created.

References 

Information about the number of repositories in a cluster:
https://www.ibm.com/developerworks/community/blogs/messaging/entry/wmq_clusters_why_only_two_full_repositories?lang=en
Best practices (old presentation but the principle are still valid):
http://www.academia.edu/5513555/WMQCluster_Best_Practices
And this ten things for having an healthy cluster
https://www.ibm.com/developerworks/community/blogs/aimsupport/entry/ten_quick_tips_for_healthy_mq_cluster?lang=en
Very useful information about cluster questions
https://www.ibm.com/developerworks/community/blogs/messaging/tags/clustering?lang=en
Very good article that provides information about MQ HA
http://www.ibm.com/developerworks/websphere/library/techarticles/0505_hiscock/0505_hiscock.html

Wednesday, September 24, 2014

WebSphere MQ & IIB on Ubuntu

I found that more and more people like Linux and this is a great things.
IBM Integration Bus for developers (free of charge !!!) is available for Linux distribution.
That's cool... however I find out that the installation may be always straightforward especially on the last Linux 64 bit distribution.
You will find in this small post what are the steps that I followed to install the package.

First I would recommend to install each component separately:
  • WebSphere MQ
  • IBM Integration Bus
  • IIB Explorer

Download the IBM Integration Bus for developers package.

I have used the single package download.

WebSphere MQ


Installation

Installing software on Ubuntu usually entails using Synaptic or by using an apt-get command from the terminal. 
Unfortunately, there are still a number of packages out there that are only distributed in RPM format.
Ubuntu is Debian based and therefore uses.deb packages to install. If you want to install .rpm packages, you first should convert them into .deb packages with a conversion software such as alien. Then you can use gdebi or dpkg to install them.
Despite the large version number, alien is still (and will probably always be) rather experimental software. It has been used by many people for many years, but there are still many bugs and limitations

IBM distribute MQ as a set of RPMs and with what I just said, I recommend to use "rpm" and hence install the rpm package: ‘rpm’, ‘pax’ and ‘default-jre’ packages.
Detailed information can be found in the following technote:

In my installation I have made first 
1. Install the i386 libraries
sudo apt-get install libc6-i386
sudo apt-get install libgcc1:i386
sudo apt-get update 
2. Define the system configuration (provided here after)
3. Install the different rpm package as explaine at the previous link (for ubuntu 14.04 I had to use the --prefix /opt/mqm).
4. Define the WMQ configuration

System configuration

I would recommend to change the system configuration for Linux when using WebSphere MQ. I provide the information here after but detailed information can be found in the knowledge center ( Linux System requirements).

create /etc/sysctl.d/50-webspheremq.conf with the following:

kernel.shmmni = 4096
kernel.shmall = 2097152
kernel.shmmax = 268435456
kernel.sem = 500 256000 250 1024
fs.file-max = 524288
Make these changes live by running
sudo sysctl -p


Check that this has been correctly updated
cat /proc/sys/kernel/shmmni 

WMQ Configuration

You can use the setmqinst command to change the installation description of an installation, or to set or unset an installation as the primary installation.

Before executing the command, you would have to create the following directory “/usr/lib64”. 
Create this directory with the following command:
sudo mkdir /usr/lib64
This command sets the installation with an installation path of /opt/mqm as the primary installation:
sudo /opt/mqm/bin/setmqinst -i -p /opt/mqm

Add the user to the group mqm in order to be administrator:
sudo addgroup $USER mqm

IIBExplorer installation

Before doing any installation, you should first install the prereqs librairies (see corresponding chapter).

Install IBExplorer. Go to the folder “/extractedfolder/messagebroker_ia_developer/IBExplorer”.
If the installer is not executable, you would have to change using the following command:
chmod +x install.bin

launch the installation:
sudo ./install.bin

And start the IBM Integration Toolkit with the following command:
sudo strmqcfg -clean

IBM Integration Toolkit

Before doing any installation, you should first install the prereqs librairies (see corresponding chapter).

The installation is then straightforward, just launch the installToolkit.

sudo ./installToolkit.sh

IBM Integration Bus

The runtime is installed in a very straightforward way.
Just go in the directory where the tar has been unzipped and launch the setup.

Intalling Eclipse prereqs

The installation manager, the IBM Integration Toolkit and IBM Integration Exporer requires to have 32bit libraries. I have found issues to run the installation without installing the "ia32-libs".
In order to install this library I have used the following command:
 
sudo -i
cd /etc/apt/sources.list.d
echo "deb http://old-releases.ubuntu.com/ubuntu/ raring main restricted universe multiverse" >ia32-libs-raring.list
apt-get update
apt-get install ia32-libs
rm /etc/apt/sources.list.d/ia32-libs-raring.list
apt-get update
exit
sudo apt-get install gcc-multilib

And the following libs:
apt-get install libc6-i386
apt-get install libgcc1:i386

Friday, August 22, 2014

A view of IBM Integration Bus runtime

If you are searching information on the components that IBM Integration Bus is composed of or just would like to know the terminolgy used (i.e. what is an Integration Node, an Integration Server, a message flow, ...) this post is for you !

I don't have the intention to explain every runtime components but rather provide enough information to understand the main components and the processes that are running.

First all, the IBM Integration Bus is a standalone application that is not running on a J2EE environment.
The runtime architecture provides a very strong resilience against failure and scalability capability.The core engine is very efficient and provides high performances with limited ressource usage. 
It can be easily deployed in a development workstation (development toolkit and runtime) without requiring a ferrari: I usually do demos (that includes IIB but DB2 as well) using a standard laptop having 4GB RAM.

The following figure describes the IBM Integration Bus runtime:

An integration node is composed by different processes and one queue manager. Note that in the former version of the Integration Bus, WebSphere Message Broker, the term IBM Integration Bus was called "Broker".
There are two kind of Integration node processes: those use to run the flows called Integration Servers  and those use for Integration Bus internal tasks.
The internal tasks processes are:
  • TheIntegration Node wide listener, the "biphttplistener": this process is a http listener and it is started when the Integration node is configured to receive http request (http://www-01.ibm.com/support/knowledgecenter/SSMKHH_9.0.0/com.ibm.etools.mft.doc/bc43700_.htm?cp=SSMKHH_9.0.0%2F1-11-1-5-1-3-0-0&lang=en). It is started if the Integration node has been configured to have a Integration Node wide listener.
  • The “watchdog” process, the "bipservice": it is a very small process that monitors (and restarts if necessary) the Admin Agent (bipbroker process).
  • The Admin Agent (process name = bipbroker): this process is designed to monitor and administrate the Integration Servers. It performs such functions as handling deploy requests and restarting any Integration Servers that would stopped for unexpected reason. It does not process any message flows, but its size is governed by the size of the deploy requests that are being received.
The Integration servers are the processes where message flows are deployed and run. The name of the corresponding process is "dataflowengine". Message flows deployed in an Integration Server run as Threads.
Integration Servers are processes that are created by an administrator. 
IBM Integration Bus is proposed in different editions: express, standard and advanced. In the express and standard edition, only one Integration Server can be attached to an Integration Node.
All the integration servers within the same Integration Node use the same unique queue manager, the Integration Node queue manager. It is used as well to store eventual internal information as to get or put MQ messages processed by message flows.
The Integration Node queue manager is a standard WebSphere MQ Queue manager and can be administrated as any other queue manager. Even though it can be integrated into a WebSphere MQ network, using a MQ cluster or channels, and accessed by applications as a normal WMQ queue manager (using client connections for example), the usage of the Integration Node Queue Manager is restricted from a license point of view:it can process MQ messages as long as it is processed in one way or another by a message flow deployed on the Integration Node.
A message flow can be deployed on one or multiple Integration Servers. A number of instance of the message flow (corresponding to a number of threads) can be defined  on each Integration Server where it is deployed. Different message flows can be deployed on the same Integration Server.
Finally a message flow is composed by multiple message flow nodes (MQInput, MQOutput, SOAPInput, …) that are wired together.