Product Update: Migrating to Concurrent Integrations

At Elastic Path we aim to constantly improve our products so that our customers can innovate at speed. Today I’ll be sharing a product update that seeks to do just that! As of June 16, 2022 all new integrations have been delivered concurrently, but existing integrations are still delivered serially. We intend to retire this old serial-delivery system in the coming months as it suffers from some technical issues at scale that can cause long delays in events being delivered resulting in poor shopper experience. This blog post explains what has changed, the rationale for the new approach, and provides guidance on migrating to the new concurrent system. This blog post is best suited for technical users, especially those Elastic Path customers that created integrations prior to June 16, 2022.

The Old Way: Serial Event Processing

Prior to the changes each event for a given store was processed in turn. This meant that the webhook calls occurred one at a time and events were generally processed in order.

The big problem with this approach is scalability. Even if a webhook (or delivery to a queuing system such as Amazon SQS) takes a relatively speedy 200ms to complete that limits the throughput of events for a given integration to only 18,000 events per hour which is quite possible to reach for larger or busier stores. In the past this has caused events for particularly busy stores to back up and not be delivered for several days. For many integrations this can negatively affect the shopper experience, for example if the event triggers an order confirmation email.

The New Way: Handling Events Concurrently

Events observed by all integrations created since 2022-06-16 are delivered concurrently. This means that instead of waiting for an event to be delivered before moving onto the next, multiple events may be delivered at the same time. This means that events will not back up and your store will scale better at busier times. However, this change does have two important implications for handling events:

Try Elastic Path today

Access a free Elastic Path trial today to see our leading composable commerce solution in action.

Sign up for a free trial

Many events may arrive at the same time

Any system receiving events from Elastic Path Commerce Cloud must be able to cope with processing multiple events at the same time. In order to manage this we recommend using a queuing technology such as Amazon SQS for later processing rather than handling the event directly with a synchronous webhook. This decouples receiving events and processing them, which copes better with spikes in event volume and allows the resources required to process the events to be smaller. If processing events directly as they are received there must be sufficient resources available to complete processing of each event.

Events arrive in any order

Since the inception of the Integrations API, we have only ever guaranteed that each event would be delivered at least once and not made any undertaking to deliver events in order. However the serial event processing mechanism we are retiring did usually deliver events in order and so it is probable that existing webhooks are not designed to cope with out-of-order events.

In order to cope with events that arrive out of order one can inspect the updated_at time of the event. This has millisecond resolution and should be enough to determine which of a number of events is the most recent. You should store the most recent version of an entity (at least its id and updated_at time) and then discard any events that have an older updated_at than that stored for the entity. Events are “fully-hydrated” meaning they carry all the up-to-date data about the entity they refer to, so with a record of the most recent event no information is lost.

Here is an example payload for a currency.updated event:

{
 "id": "c60245c7-7499-496a-b1b2-6b09f26d64a0",
 "store_id": "f952d632-88f2-4ab0-80c9-9fefa87c2eee",
 "org_id": "9ea0fcb3-1bc4-4040-854a-6884f6b7ad6e",
 "triggered_by": "currency.updated",
 "integration": {
 "id": "ff7b7520-f77f-4a4d-8aff-dcc509c53928",
 "integration_type": "webhook",
 "name": "demo",
 "description": "Demonstration integration"
 },
 "payload": {
 "data": {
 "id": "cee66481-0c6e-4e29-b83e-335bcbf5dd06",
 "type": "currency",
 "code": "GBP",
 "exchange_rate": 1,
 "format": "£{price}",
 "decimal_point": ".",
 "thousand_separator": ",",
 "decimal_places": 2,
 "default": false,
 "enabled": true,
 "links": {
 "self": "https://useast.api.elasticpath.com/currencies/cee66481-0c6e-4e29-b83e-335bcbf5dd06"
 },
 "meta": {
 "timestamps": {
 "created_at": "2023-06-01T09:21:27.204Z",
 "updated_at": "2023-06-01T09:40:34.943Z"
 },
 "owner": "store"
 }
 }
 },
 "configuration": {
 "url": "https://test.com/test/",
 "secret_key": ""
 }
}

You can see that the currency cee66481-0c6e-4e29-b83e-335bcbf5dd06 has its update time recorded in payload.meta.timestamps.updated_at.

Imagine a situation in which this currency was created, updated and then deleted all within a short period of time. If the events arrive in the expected order that they were created that would look like this:

If the updated event arrived before the created event, or if there are multiple updated events and they arrive in the wrong order then the updated_at time can be used to determine what to do.

Note that it is also possible, although unlikely, to receive a deleted event for an entity before its created event. If a deleted event is received all subsequent events for the entity should be ignored.

Migration

Switching an older integration to the concurrent method is as simple as setting the is_concurrent flag to be true. This is only achievable through the API, not Commerce Manager. For example, to switch integration c60245c7-7499-496a-b1b2-6b09f26d64a0 to be processed concurrently:

PUT https://useast.api.elasticpath.com/v2/integrations/c60245c7-7499-496a-b1b2-6b09f26d64a0
{
 "data": {
 "id": "c60245c7-7499-496a-b1b2-6b09f26d64a0",
 "type": "integration",
 "is_concurrent": true
 }
}

Note: For integrations created after June 16, 2022 it is not possible to revert back to the old serial behavior by setting is_concurrent to false but this can be done for the older integrations, allowing a failed attempt at migration to be reverted.

Sunsetting serial event processing

Elastic Path will sunset serial processing of events on January 31, 2024. We will communicate with any customers who have yet to migrate well in advance of that date to assist in the migration process. Any remaining integrations that have not been migrated will simply start being processed concurrently. If you have any questions, please contact your customer success manager.