Enterprise Integration – Part I: Concepts of Messaging Solutions And Enterprise Integration Patterns

In a loose series of Blogs I will write about the present state on messaging and enterprise integration. In this first part I will write about:

1. Today’s IT infrastructure
2. Messaging as a best practice
3. Messaging key concepts
4. Using Enterprise Integration Patterns for Messaging
5. Next Blog: Spring Integration

I try to give a deeper explanation based on my experience since 2000 of doing integration for several customers. I am happy for further input and discussions. Please feel free to comment this Blog or contact me.

Today’s IT infrastructure

Nowadays almost every enterprise is dealing with a heterogeneous system landscape. Its infrastructure is a conglomerate of applications:

• running on different platforms
• spread on different locations
• connected in a network

Within this landscape there are multiple applications exchanging data with another back and forth.
This integration of several applications are facing with the same challenges [EIP]:

• Networks are unreliable and slow
• Any application is different (by its programming language, operating platforms, and data format)
• Any application is changing on the long run

Messaging as a best practice

In an enterprise there are many applications integrating data from another. Ideally each integration scenario are implemented on the same centralized platform.
Integration where data are send to or received from another application can be seen as a communication of two or more applications.
There are 4 different integration styles of how communication can be achieved[EIP]:

1. File Transfer: an application exports data into a file which is then read by the target application(s).
2. Shared database: several applications are sharing the same database.
3. Remote Procedure Invocation: An application sends data directly to the target application(s).
4. Messaging: An application sends a message to a (queued) channel. Other applications listening to the channel can received the message. The communication is asynchronous since the message can be received at a later time.

Hohpe and Woolf points out why messaging is superior compared to the other approaches:
“The quick answer is that messaging is more immediate than File Transfer, better encapsulated than Shared Database, and more reliable than Remote Procedure Invocation.”, “Why Use Messaging?” on page xxxiii

Messaging (indirect communication) is the direct opposite of a remote procedure invocation where a sender and a receiver are connected through a queue. The responsibility of this message-oriented middleware is taken by a messaging system: it is responsible for receiving incoming messages from source applications, processing the messages through several channels and sending them to the target applications. Among this there are other responsibilities a messaging system is taking care of: system management, error logging and security.

Messaging key concepts

The messaging approach outlines two concepts where (1) applications are loosely coupled and (2) the messaging (communication) is asynchronously:

1. Loosely coupling: “Each [application] makes its requirements explicit and makes few assumptions about the other [application].”
2. Asynchronous communication: “Asynchronous communication is a mediated form of communication in which the sender and receiver are not concurrently engaged in communication.”

When talking about messaging I use the terms of Enterprise Integration Patterns as described below. At this point it is sufficient to know that messaging is an integration flow between several applications. Each integration flow is a process description using channels and endpoints. The first and latter endpoint are normally interfaces to an application.
Comparing the messaging approach with the other integration styles and in the context of a heterogeneous system landscape the advantages of loosely coupling and asynchronous communication are:
1. Loosely coupling

• There are legacy applications and their data format which cannot be directly influenced. In this case a messaging system can act as a mediator dealing with the given data and translates it into a common data format for further processing.
• On the long run applications in a system landscape are changing. This has an impact only on the application endpoint within the integration flow that needs to be changed in the messaging system. The remaining components of the integration flow keeps unchanged.
  A similar scenario is that an application might be replaced by or merged into another application. Here only the application endpoint needs to be changed or replaced.

2. Asynchronous communication

• Unlike a direct communication a messaging system follows a fire-and-forget principle: a sender can send a message and continue working without waiting for the processing of the receiver. The messaging system takes the responsibility for a reliable transmission. Even if the sender needs a reply (receiving another data) the messaging system can use a callback. Also in this case the sender does not have to wait for a response.
• In a system landscape applications might be for many reasons not available. A messaging system can queue messages and re-send them when an application is available.
• Another advantage of queuing messages is in case of high frequent data requests (like real-time scenarios) where the sender is flooded with lots of small message requests. A messaging system can control such an integration flow an aggregate all the messages into a single one.
• Since sender and receiver do not need to rely on each other their tasks on both sides are performed with a maximum. Just imagine both applications are running on servers with different CPUs. Each application can process on their best speed. There is no bottleneck to each other.

Using Enterprise Integration Patterns for Messaging

There are many possibilities to implement messaging for integration. When dealing with messaging there is no better way than using the Enterprise Integration Patterns (EIP). The book “Enterprise Integration Patterns” from Gregor Hohpe and Bobby Woolf is the de facto book about messaging solutions for enterprise integration.
EIP breaks down an integration scenario into into several parts. Each part is a pattern describing a specific solution. A pattern is a prescription of what needs to be done to solve a limited variety of problems. In that sense an integration scenario using the EIP pattern as a language is a combination of patterns guiding a message (of one or more applications) from the beginning to the end of the integration flow (to other applications).
A very good starting point is Hohpe’s Site www.enterpriseintegrationpatterns.com. For a detailed description I strongly recommend Hohpe’s and Woolf’s book.
EIP provides patterns for the following messaging concepts:

(Message) Endpoints	This is the interface for communication between an application and the messaging system. In Spring Integration for example every component in the messaging system connected to the channel is an endpoint (like Router, Splitter, Translator, Aggregator etc.).
Message Channels	Data are transported within the messaging system through channels. There are various kinds of channels like: Point-to-Point-Channel, Publish-Subscribe-Channel, Datatype Channel, Deadletter Channel, Channel Adapter, Message Bus etc.
Messages	A message consists of two parts containing a header and a body. The body is the actual data an endpoint is sending to or receiving from a channel. A body message carries meta data like message id, origin and destination. Messages within the integration flow can be split, aggregated, routed, changed or trashed by endpoints.
Pipes and Filters	This architecture describes a chain of processing steps (Filters) that are connected by channels (Pipes).
Message Routers	This special endpoint navigates a message to a certain channel. A simple example is a routing based on the message itself. Depending on whether it is a CSV or XML message it routes it either to a CSV or XML channel.
Message Translators	Within an integration flow a message can be converted or changed for further processing and delivering in an appropriate data format for a target application.

Next Blog: Spring Integration

In my next blog I will write my experiences I have made with Spring Integration (SI). It made quite a lot of changes from the milestone releases until the current version 1.0.
I will try to outline:

• how consequently the Enterprise Integration Patterns is implemented by using a message bus with various kinds of channels and endpoints.
• advantages of SI using the Spring Framework like AOP, Dependency Injection, Annotation- or XML-based configuration.

By using and evaluating other integration frameworks I enjoyed the ease of use which I didn’t know better since using Spring and Spring Integration. I can implement and use my existing integration components with no need of using or deriving from any Spring class. SI encapsulates all the message processing. POJOs can be used directly to send to any channels. No wrapping in a Message class is needed. But more about that will follow in my next Blog.

Tai
P.S.: Please share your opinion and feel free to leave a comment below (emails are not published) or contact me.