Delayed message delivery

Objective

This document describes the delivery delay handling in the communication over the message queue in a multi server context.

Updates in recent Hotfixes!

With the current hotfixes of Version 10.0 and 10.1 we were able to introduce a priority server cache invalidation which typically is executed in way under 50 milliseconds.
In earlier releases the synchronization delay could be multiple seconds. Still, there is a synchronization gap between the servers, even if it's very small now. So we still recommend to configure the Active MQ cluster like describen on this page. The default setting for the delay has been adjusted to 50 milliseconds which makes this delay now unnoticeable to the user.

General

The multi-server deployment of Product 360 is designed in a way to allow the highest form of independence of the servers in order to minimize any overhead which would be introduced through excessive locking mechanisms. Each server makes sure that concurrent data modification of the same object is synchronized. It's the nature of a highly scale able architecture like this that caches might be outdated for a short time as change events are delivered asynchronously. The application logic anticipates this delay and compensates by only persisting what is really modified - thus we do not update the database with old data and simultaneously keep the synchronization times to an absolute minimum.

If in that environment the client is connected to server 1 and the client does make a change to the persisted data values, and that change triggers a workflow instance that is communicating over the message queue with server 2, it might be that server 2 cannot see the persistence value changes done by server 1 in the case that the workflow execution and message transport is faster than the cache invalidation event.

To avoid that a workflow can run into this problem, there is a delivery delay introduced for outgoing queue messages to the workflow. The delivery delay ensures the Informatica BPM server will get the message only after the delivery delay has passed and not earlier. The default setting for the delivery delay is 50 milliseconds in a multi server environment. The delivery delay is handled by the message queue instance itself, which introduces the need, that the message queue server is configured to provide that functionality.

images/download/attachments/373863732/Out_of_sync_persistence_issue_in_a_workflow_case.png

It's important to note that workflows will be triggered with a minimum delay of 50 milliseconds with these settings.

Configuration

Configuration of Active MQ server

To enable the message delivery delay feature the activemq.xml file in the conf folder of the Active MQ server has to be adjusted in the way that the scheduler support is activated:

<broker [...] schedulerSupport="true">
...
</broker>

After the adjustment the Active MQ instance has to be restarted.

Product 360 configuration (server.properties)

It is possible to configure the delivery delay on a per queue basis. As a default a 50 milliseconds interval is used in the multi server environment (in a single server env it is 0 milliseconds).

To configure a single queue to have a different deliver delay [milliseconds] use the following syntax:

queue.[queue id].delivery.delay = 50

Per default this value is written into the Active MQ specific message header called AMQ_SCHEDULED_DELAY, but it is possible to overwrite this header key with the following syntax:

queue.[queue id].delivery.delay.header = AMQ_SCHEDULED_DELAY

Note: the value is always written as a numerical long as header type.