The Outbox Pattern

Introduction

Sometimes, when processing a business operation, you need to communicate with an external component in the Fire-and-forget mode. That component can be, for example:
– external service
– message bus
– mail server
– same database but different database transaction
– another database

Examples of this type of integration with external components:
– sending an e-mail message after placing an order
– sending an event about new client registration to the messaging system
– processing another DDD Aggregate in different database transaction – for example after placing an order to decrease number of products in stock

The question that arises is whether we are able to guarantee the atomicity of our business operation from a technical point of view? Unfortunately, not always, or even if we can (using 2PC protocol), this is a limitation of our system from the point of latency, throughput, scalability and availability. For details about these limitations, I invite you to read the article titled It’s Time to Move on from Two Phase Commit.

The problem I am writing about is presented below:

After execution of line 24 transaction is committed. In line 28 we want to send an event to event bus, but unfortunately 2 bad things can happen:
– our system can crash just after transaction commit and before sending the event
– event bus can be unavailable at this moment so the event cannot be sent

Outbox pattern

If we cannot provide atomicity or we don’t want to do that for the reasons mentioned above, what could we do to increase the reliability of our system? We should implement the Outbox Pattern.

Outbox pattern

The Outbox Pattern is based on Guaranteed Delivery pattern and looks as follows:

Outbox pattern

When you save data as part of one transaction, you also save messages that you later want to process as part of the same transaction. The list of messages to be processed is called an Outbox, just like in e-mail clients.

The second element of the puzzle is a separate process that periodically checks the contents of the Outbox and processes the messages. After processing each message, the message should be marked as processed to avoid resending. However, it is possible that we will not be able to mark the message as processed due to communication error with Outbox:

Outbox messages processing

In this case when connection with Outbox is recovered, the same message will be sent again. What all this means to us? Outbox Pattern gives At-Least-Once delivery. We can be sure that message will be sent once, but can be sent multiple times too! That’s why another name for this approach is Once-Or-More delivery. We should remember this and try to design receivers of our messages as Idempotents, which means:

In Messaging this concepts translates into a message that has the same effect whether it is received once or multiple times. This means that a message can safely be resent without causing any problems even if the receiver receives duplicates of the same message.

Ok, Enough theory, let’s see how we can implement this pattern in .NET world.

Implementation

Outbox message

At the beginning, we need to define the structure of our OutboxMessage:

What is important, the OutboxMessage class is part of the Infrastructure and not the Domain Model! Try to talk with business about Outbox, they will think about the outlook application instead of the messaging pattern. πŸ™‚ I didn’t include ProcessedDate property because this class is only needed to save message as part of transaction so this property always will be NULL in this context.

Saving the message

For sure I do not want to program writing messages to the Outbox every time in each Command Handler, it is against DRY principle. For this reason, the Notification Object described in the post about publishing Domain Events can be used. Following solution is based on linked article with little modification – instead of processing the notifications immediately, it serializes them and writes them to the database.

As a reminder, all Domain Events resulting from an action are processed as part of the same transaction. If the Domain Event should be processed outside of the ongoing transaction, you should define a Notification Object for it. This is the object which should be written to the Outbox. The code looks like:

Example of Domain Event:

And Notification Object:

First thing to note is Json.NET library usage. Second thing to note are 2 constructors of CustomerRegisteredNotification class. First of them is for creating notification based on Domain Event. Second of them is to deserialize message from JSON string which is presented in following section regarding processing.

Processing the message

The processing of Outbox messages should take place in a separate process. However, instead of a separate process, we can also use the same process but another thread depending on the needs. Solution which is presented below can be used in both cases.

At the beginning, we need to use a scheduler that will periodically run Outbox processing. I do not want to create the scheduler myself (it is known and solved problem) so I will use one the mature solution in .NET – Quartz.NET. Configuration of Quartz scheduler is very simple:

Firstly, scheduler is created using factory. Then, new instance of IoC container for resolving dependencies is created. The last thing to do is to configure our job execution schedule. In case above it will be executed every 15 seconds but its configuration really depends on how many messages you will have in your system.

This is how ProcessOutboxJob looks like:

The most important parts are:
Line 1 – [DisallowConcurrentExecution] attribute means that scheduler will not start new instance of job if other instance of that job is running. This is important because we don’t want process Outbox concurrently.
Line 25 – Get all messages to process
Line 30 – Deserialize message to Notification Object
Line 32 – Processing the Notification Object (for example sending event to bus)
Line 38 – Set message as processed

As I wrote earlier, if there is an error between processing the message (line 32) and setting it as processed (line 38), job in the next iteration will want to process it again.

Notification handler template looks like this:

Finally, this is view of our Outbox:

Outbox view

Summary

In this post I described what are the problems with ensuring the atomicity of the transaction during business operation processing. I’ve raised the topic of 2PC protocol and motivation to not use it. I presented what the Outbox Pattern is and how we can implement it. Thanks to this, our system can be really more reliable.

Source code

If you would like to see full, working example – check my GitHub repository.

Additional Resources

Refactoring Towards Resilience: Evaluating Coupling – Jimmy Bogard
The Outbox: An EIP Pattern – John Heintz
Asynchronous message-based communication – Microsoft

Related posts

Domain Model Encapsulation and PI with Entity Framework 2.2
Simple CQRS implementation with raw SQL and DDD
How to publish and handle Domain Events

Domain Model Validation

Introduction

In previous post I described how requests input data can be validated on Application Services Layer. I showed FluentValidation library usage in combination with Pipeline Pattern and Problem Details standard. In this post I would like to focus on the second type of validation which sits in the Domain LayerDomain Model validation.

What is Domain Model validation

We can divide the validation of the Domain Model into two types based on scope – Aggregates scope and Bounded Context scope

Aggregates scope

Let’s remind what the Aggregate is by quoting a fragment of the text from Vaughn Vernon Domain-Driven Design Distilled book:

Each Aggregate forms a transactional consistency boundary. This means that within a single
Aggregate, all composed parts must be consistent, according to business rules, when the controlling
transaction is committed to the database.

The most important part of this quote in context of validation I underlined. It means that under no circumstances we can’t persist Aggregate to database which has invalid state or breaks business rules. These rules are often called “invariants” and are defined by Vaughn Vernon as follows:

… business invariants β€” the rules to which the software must always adhere β€” are guaranteed to be consistent following each business operation.

So in context of Aggregates scope, we need to protect these invariants by executing validation during our use case (business operation) processing.

Bounded Context scope

Unfortunately, validation of Aggregates invariants is not enough. Sometimes the business rule may apply to more than one Aggregate (they can be even aggregates of different types).

For example, assuming that we have Customer Entity as Aggregate Root, the business rule may be “Customer email address must be unique”. To check this rule we need to check all emails of Customers which are separated Aggregate Roots. It is outside of the scope of one Customer aggregate. Of course, supposedly, we could create new entity called CustomerCatalog as Aggregate Root and aggregate all of the Customers to it but this is not good idea for many reasons. The better solution is described later in this article.

Let’s see what options we have to solve both validation problems.

Three solutions

Return Validation Object

This solution is based on Notification Pattern. We are defining special class called Notification/ValidationResult/Result/etc which “collects together information about errors and other information in the domain layer and communicates it”.

What does it mean for us? Is means that for every entity method which mutates the state of Aggregate we should return this validation object. The keyword here is entity because we can have (and we likely will have) nested invocations of methods inside Aggregate. Recall the diagram from the post about Domain Model encapsulation:

Domain model encapsulation

The program flow will look like:

Validation Object Flow

and the code structure (simplified):

However, if we don’t like to return ValidationResult from every method which mutates the state we can apply different approach which I described in article about publishing Domain Events. In short, in this solution we need to add ValidationResult property for every Entity (as Domain Events collection) and after Aggregate processing we have to examine these properties and decide if the whole Aggregate is valid.

Deferred validation

Second solution how to implement validation is to execute checking after whole Aggregate’s method is processed. This approach is presented for example by Jeffrey Palermo in his article. The whole solution is pretty straightforward:

Deferred validation

Always Valid

Last but not least solution is called “Always Valid” and it’s just about throwing exceptions inside Aggregate methods. It means that we finish processing of the business operation with the first violation of the Aggregate invariant. In this way, we are assured that our Aggregate is always valid:

Always Valid program flow

Comparison of solutions

I have to admit that I don’t like Validation Object and Deferred Validation approach and I recommend Always Valid strategy. My reasoning is as follows.

Returning Validation Object approach pollutes our methods declarations, adds accidental complexity to our Entities and is against Fail-Fast principle. Moreover, Validation Object becomes part of our Domain Model and it is for sure not part of ubiquitous language. On the other hand Deferred Validation implies not encapsulated Aggregate, because the validator object must have access to aggregate internals to properly check invariants.

However, both approaches have one advantage – they do not require throwing exceptions which should be thrown only when something unexpected occurs. Business rule broken is not unexpected.

Nevertheless, I think this is one of the rare exception when we can break this rule. For me, throwing exceptions and having always valid Aggregate is the best solution. “The ends justify the means” I would like to say. I think of this solution like implementation of Publish-Subsribe Pattern. Domain Model is the Publisher of broken invariants messages and Application is the Subscriber to this messages. The main assumption is that after publishing message the publisher stops processing because this is how exceptions mechanism works.

Always Valid Implementation

Exception throwing is built into the C# language so practically we have everything. Only thing to do is create specific Exception class, I called it BusinessRuleValidationException:

Suppose we have a business rule defined that you cannot order more than 2 orders on the same day. So it looks implementation:

What we should do with the thrown exception? We can use approach from REST API Data Validation and return appropriate message to the client as Problem Details object standard. All we have to do is to add another ProblemDetails class and set up mapping in Startup:

The result returned to client:

Problem details validation domain model

For simpler validation like checking for nulls, empty lists etc you can create library of guards (see Guard Pattern) or you can use external library. See GuardClauses created by Steve Smith for example.

BC scope validation implementation

What about validation which spans multiple Aggregates (Bounded Context scope)? Let’s assume that we have a rule that there cannot be 2 Customers with the same email address. There are two approaches to solve this.

The first way is to get required aggregates in CommandHandler and then pass them to aggregate’s method/constructor as arguments:

However, this is not always a good solution because as you can see we need to load all Customer Aggregates to memory. This could be serious performance issue. If we can not afford it then we need to introduce second approach – create Domain Service which is defined as (source – DDD Reference):

When a significant process or transformation in the domain is not a natural responsibility of an entity or value object, add an operation to the model as a standalone interface declared as a service

So, for that case we need to create ICustomerUniquenessChecker service interface:

This is the implementation of that interface:

Finally, we can use it inside our Customer Aggregate:

The question here is whether pass Domain Service as an argument to aggregate’s constructor/method or execute validation in Command Handler itself. As you can see above I am fan of former approach because I like keep my command handlers very thin. Another argument for this option is that if I ever need to register Customer from a different use case I will not be able to bypass and forget about this uniqueness rule because I will have to pass this service.

Summary

A lot of was covered in this post in context of Domain Model Validation. Let’s summarize:
– We have two types of Domain Model validation – Aggregates scope and Bounded Context scope
– There are generally 3 methods of Domain Model validation – using Validation Object, Deferred Validation or Always Valid (throwing exceptions)
Always Valid approach is preferred
– For Bounded Context scope validation there are 2 methods of validations – passing all required data to aggregate’s method or constructor or create Domain Service (generally for performance reason).

Source code

If you would like to see full, working example – check my GitHub repository.

Additional Resources

Validation in Domain-Driven Design (DDD) – Lev Gorodinski
Validation in a DDD world– Jimmy Bogard

Related posts

REST API Data Validation
Domain Model Encapsulation and PI with Entity Framework 2.2
Simple CQRS implementation with raw SQL and DDD
How to publish and handle Domain Events