Modular Monolith: Integration Styles

This post is part of articles series about Modular Monolith architecture:

1. Modular Monolith: A Primer
2. Modular Monolith: Architectural Drivers
3. Modular Monolith: Architecture Enforcement
4. Modular Monolith: Integration Styles (this)

Introduction

No module or application in a larger system works in 100% isolation. In order to deliver business value, individual elements must somehow integrate with each other. Let me write here a quote from the book “Thinking in Systems: A Primer”, where Donella H. Meadows defines the system concept in general:

A system is an interconnected set of elements that is coherently organized in a way that achieves something. If you look at that definition closely for a minute, you can see that a system must consist of three kinds of things: elements, interconnections, and a function or purpose.

The concept of systems integration is defined as follows (wiki):

process of linking together different computing systems and software applications physically or functionally, to act as a coordinated whole.

As you can see from the definitions above, in order to provide a system that fulfills its purpose, we must integrate elements to form a whole. In previous articles in this series, we discussed the attributes of these elements which are, in our terminology, called modules.

In this post, I would just like to discuss the missing part – Integration Styles for modules in Modular Monolith architecture.

Enterprise Integration Patterns book

The title of this post is not accidental. It sounds exactly like Chapter 2 of Gregor Hohpe and Bobby Wolf great book Enterprise Integration Patterns. This book is considered a bible of information about systems integration and messaging. This article takes some knowledge from this chapter and relates it to the monolithic and modular architecture.

In any case, everyone interested in the topic of integration, I invite you to read the book or materials that are available online at https://www.enterpriseintegrationpatterns.com/ site.

Integration Styles

Integration Criteria

Like everything in nature, each Integration Style has its pros and cons. Therefore, we must define criteria on the basis of which we will compare all styles. Then, based on that criteria, we will decide on the method of integration in the future.

We can distinguish the following criteria: Coupling, Complexity, Data Timeliness.

1. Coupling

Coupling is a measure of the degree to which 2 modules are dependent on each other (wiki):

coupling is the degree of interdependence between software modules; a measure of how closely connected two routines or modules are; the strength of the relationships between modules.

If you’ve read the previous posts in the series, you already know that one of the most important attributes of modular design is independence. Therefore, it will be easy to guess that coupling is one of the more important criteria in terms of integration style.

2. Complexity

The second criterion for evaluating the Integration Style is its level of complexity. Some integration methods are simple – require little work, are easy to understand and use. However, others are more complicated, require more commitment, knowledge, and discipline.

3. Data Timeliness

The last criterion is the length of time between when one module decides to share some data and other modules have that data. This means how soon after a state change in a given module, the rest of the modules concerned will take this change into account. Of course, the shorter this time, the better.

Now that we know all the most important criteria, let’s move on to the ways of integrating our modules. Let’s discuss 4 Integration Styles: File Transfer, Shared Database Data, Direct Call and Messaging.

File Transfer

The first option is to integrate our modules using a regular file. Such a file must be exported from the source module and imported into the target module. This can happen in 3 ways:
– manual, where the user manually imports/exports
– automatic, where files are imported and exported automatically by systems
– hybrid, where the file is imported/exported automatically on one side and vice versa on the other

Modular Monolith Integration Styles - File Transfer
File Transfer

One of the main tasks of this type of integration is to determine the format of a given file. What is important, it is the only dependency that two modules integrating in this way have. You can visualize it as a really huge message that is carried over by the filesystem. For this reason, it can be assumed that the coupling is very low in this case.

As for the level of complexity of this approach, it can be evaluated as an average. On the one hand, generating a file in a specific format is not difficult in these times. On the other hand, uploading to a shared resource, managing files, handling duplicates, and so on is more complicated and time-consuming.

From a timeliness point of view, modules integration via files is slow (not to mention manual export/import). Most often it is performed in larger batches at some time intervals (so-called batches), often at night. For this reason, the delay can be a day, a week or more.

To be honest, I’ve seen file-sharing integration many, many times between systems, and probably never in a monolith – which is rather understandable. This Integration Style I have described for the sake of completeness of this topic. The most popular integration method for monoliths is the Shared Database Data.

Shared Database Data

In the EIP book, this method of integration is called Shared Database, but I believe that it is not quite the right name. Sharing the database does not always have to mean sharing data, because modules can store its data in separate tables (most often it is done through database schemas). Therefore, in my opinion, Shared Database Data is a better term.

Modular Monolith Integration Styles - Shared Data
Integration Style: Shared Data

In Shared Database Data modules share a certain set of data in the database. In this way, the data are always integrated and consistent with each other because, generally speaking, they are the same data. If module A writes data to table X, module B can read the data immediately after the database transaction is completed.

The level of complexity of such a solution is very small. Nowadays every application/module needs a database, so there is no need to add anything extra with this approach.

The solution looks perfect at first glance. However, its biggest disadvantage is a very high coupling. By sharing data, modules share their state which couples them together. The high coupling means no autonomy for the module. In addition, one little change to database structure or even data itself can break another module without notice. It implies that each change to the database must be consulted and coordinated. This way database becomes bottle-neck of changes. The whole solution is not evolutionary anymore.

The shared state has another significant disadvantage – it is very hard or even impossible to create one, unified data model which will ensure that the requirements of all modules are met. The attempt to unify most often ends with a very weak, ambiguous model that is difficult to understand, develop and maintain.

To reduce coupling while still maintaining the same level of data timeliness we can use Direct Call.

Direct Call

The third option is to directly call the method of the module we’re integrating with. In this case, we use the encapsulation mechanism. The module exposes only what is needed. The whole behavior is closed in a method. In this way, the state of our module is not exposed to the outside as it is in the case of the Shared Database Data approach. Thanks to this, the caller is not able to break anything from the outside.

Modular Monolith Integration Styles - Direct Call
Integration Styles – Direct Call

Not sharing the data implies that each module has its own data set. It can be the same database broken down by schemas or each module can have even a separate database created in different technology. In a scenario with one database, it is important to keep the data really in isolation. It means no constrains between tables from separate modules and no transactions between them.

Both the caller and the callee should treat each other as external. Both modules will use a different language and have different concepts modeled. Therefore, the Anti-Corruption Layer (ACL) should be applied. On the caller side, it could just be a gateway, on the callee side a facade. Thanks to this, modules encapsulation are kept.

In the case of a distributed system, the Direct Call is known as Remote Procedure Invocation/Call (RPI/RPC). Unfortunately, this technique is very often used in Microservice architecture and can lead to the so-called Distributed Monolith anti-pattern architecture. As the call is always synchronous, we are dealing with temporal coupling. Both caller and calle must be available in the same time. In the case of a monolith, it is not a problem because this is its nature, in the case of microservices it is much worse – it reduces architecture quality attributes like autonomous development and deployment. Read other article in this series about architectural drivers for more details.

The Direct Call Integration Style seems to be a very good choice when it comes to integrating our modules, but has some drawbacks too. First, the call is synchronous, so the caller has to wait for the result. Second, the calling module needs to know about the module it is calling, it has to have a direct dependency. Moreover, it has to know the intent of what it wants to do. Coupling is lower than in Shared Database Data, but still exists. If we want to avoid these drawbacks, we can use the last integration style: Messaging

Messaging

The File Transfer integration style has a great advantage – it does not create dependencies between modules. However, it has a big drawback – the data timeliness in most cases is unacceptable. Messaging does not have this disadvantage. The data timeliness is not so good as in the case of Direct Call because it is asynchronous communication, but it can be safely said that it is very good and acceptable in most cases.

Modular Monolith Integration Styles - Messaging - in memory
Integration Styles – Messaging (in memory)
Modular Monolith Integration Styles - Messaging - separate process
Integration Styles – Messaging (separate process)

Appropriate use of Messaging for the implementation of Event-Driven Architecture causes no dependency between modules. Modules integrate through events. However, these are not Domain Events because a domain event is local and should be encapsulated in a given Bounded Context. Integration Events contain only as much as needed to avoid the so-called Fat Events. Integration Events should be as small as possible, as they are part of the contract made available by the given module. As you know, the smaller the contract, the more stable it is -> the less frequently other modules have to change.

In addition, asynchronous processing, that causes Eventual Consistency, on the other hand supports performance advantages, scales better and it is more reliable.

What are the disadvantages of Messaging? First, due to the nature of asynchronicity, the state of our entire system can be eventually consistent as described above. That is why it is so important to have the boundaries of the modules well defined. This is almost as important as in Microservices architecture, but the advantage of the Modular Monolith architecture is that it is much easier to change these boundaries.

The second disadvantage of Messaging is that it is more complex. To provide asynchronous processing and Event-Driven Architecture, we’re going to need some sort of Event Bus. It can be an in-memory broker or a separate component (eg RabbitMQ). In addition, we will also need job processing mechanisms for internal processing – outbox, inbox, internal commands messages. There is need to write some of this infrastructure code. Fortunately, this is a generic problem – we do it only once and there are a lot of libraries and frameworks which support this.

Comparison

Below I present a comparison of all 3 Integration Styles taking into account 3 criteria – Coupling, Data Timeliness and Complexity.

Comparison - Coupling vs Complexity
Comparison – Coupling vs Complexity
Comparison - Coupling vs Data Timeliness
Comparison – Coupling vs Data Timeliness

What can we deduce from these diagrams? First of all, there is no one perfect style. Fortunately, we can see some heuristics:

1. If the system is very, very simple (essential complexity is low) and you do not care about modularity: choose simplicity and use Shared Database Data style.
2. If the system is complex (essential complexity is high) you must care about modularity. Options are:
– choose Messaging if you prefer highest level of autonomy and eventual consistency between modules is acceptable
– choose Messaging if you care a lot about performance, reliability, scalability
– choose Direct Call if you must have strong consistency, don’t need maximum level of modules autonomy or Data Timeliness is the primary factor.

Additionally, you can mix different styles. Most often, when we are talking about modular architecture, it is best to use Direct Call and Messaging together. Some modules can communicate synchronously and some asynchronously, depending on the need.

Summary

As I mentioned at the beginning, no module, component, or system lives in complete isolation. We must follow the “Divide and Conquer” principle so we divide our solution into smaller parts, but finally – we have to integrate these parts together to create a system.

Let’s summarize all 4 styles again:

File transfer – provides low coupling but has almost always unacceptable data timeliness so it is impractical in the monolith
Shared Database Data – the simplest, quick, but couples modules together
Direct Call – provides lower coupling than Shared Database Data, encapsulates modules, relatively simple
Messaging – ensures the lowest coupling, modularity, autonomy but at the cost of complexity

Which Integration Style to choose? Everything, as usual, “it depends”. However, I hope that I managed to at least to some extent explain what it depends on. No silver bullet, again.

Related Posts

1. Modular Monolith: A Primer
2. Modular Monolith: Architectural Drivers
3. Modular Monolith: Architecture Enforcement
4. How to publish and handle Domain Events
5. Handling Domain Events: Missing Part
6. Processing multiple aggregates – transactional vs eventual consistency

Handling concurrency – Aggregate Pattern and EF Core

Introduction

In this post I would like to discuss a frequently overlooked, though in some circumstances very important topic – concurrency handling in context of protection of the so-called Domain Invariants.

In other words, the question is the following: how to ensure that even in a multi-threaded environment we are able to always guarantee the immediate consistency for our business rules?

It’s best to describe this problem with an example…

Problem

Let’s assume that our domain has the concept of Order and Order Line. Domain expert said that:

One Order can have a maximum of 5 Order Lines.

In addition, he said that this rule must be met at all times (without exception). We can model it in the following way:

Conceptual Model
Conceptual Model

The Aggregate

One of the main goals of our system is to enforce business rules. One way to do this is to use the Domain-Driven Design tactical building block – an Aggregate. The Aggregate is a concept created to enforce business rules (invariants). Its implementation may vary depending on the paradigm we use, but In object-oriented programming, it is an object-oriented graph as Martin Fowler describes it:

A DDD aggregate is a cluster of domain objects that can be treated as a single unit.

And Eric Evans in DDD reference describes:

Use the same aggregate boundaries to govern transactions and distribution. Within an aggregate boundary, apply consistency rules synchronously

Back to the example. How can we ensure that our Order will never exceed 5 Order Lines? We must have an object that will take care of it. To do this, it must have information about the current number of Order Lines. So it must have a state and based on this state its responsibility is to decide whether invariant is broken or not.

In this case, the Order seems to be the perfect object for this. It will become root of our Aggregate, which will have Order Lines. It will be his responsibility to add the order line and check that the invariant has not been broken.

Order Aggregate
Order Aggregate

Let’s see how the implementation of such a construct may look like:

Everything’s fine right now, but this is only a static view of our model. Let’s see what the typical system flow is when invariant is not broken and when is broken:

Process of adding Order Line - one thread
Process of adding Order Line – success
Process of adding Order Line - rule broken
Process of adding Order Line – rule broken

Simplified implementation bellow:

Concurrency problem

Everything works nice and clean if we operate in a not heavy loaded environment. However, let’s see what can happen when 2 threads almost at the same time want to perform our operation:

Process of adding Order Line - business rule broken
Process of adding Order Line – business rule broken

As you can see in the diagram above, 2 threads load exactly the same aggregate at the same time. Let’s assume that the Order has 4 Order Lines. The aggregate with 4 Order Lines will be loaded in both the first and second threads. The exception will not be thrown, because of 4 < 5. Finally, depending on how we persist the aggregate, the following scenarios are possible:

a) If we have a relational database and a separate table for Order Lines then 2 Order Lines will be added giving a total of 6 Order Lines – the business rule is broken.

b) If we store aggregate in one atomic place (for example in document database as a JSON object), the second thread will override the first operation and we (and the User) won’t even know about this.

The reason for this behavior is that the second thread read data from the database (point 2.1) before the first one committed it (point 1.4.3).

Let’s see how we can solve this problem.

Solution

Pessimistic concurrency

The first way to ensure that our business rule will not be broken is to use Pessimistic concurrency. In that approach, we allow only one thread to process a given Aggregate. This leads to the fact that the processing thread must block the reading of other threads by creating a lock. Only when the lock is released, the next thread can get the object and process it.

Pessimistic concurrency
Pessimistic concurrency

The main difference from the previous approach is that the second thread waits for the previous one to finish (time between points 2.1 and 1.3). This approach causes us to lose performance because we can process transactions only one after the other. Moreover, it can lead to deadlocks.

How can we implement this behavior using EntityFramework Core and SQL server?

Unfortunately, EF Core does not support Pessimistic Concurrency. However, we can do it easily by ourselves using raw SQL and the query hint mechanism of the SQL Server engine.

Firsty, the database transaction must be set. Then the lock must be set. This can be done in two ways – either read data with query hint (XLOCK, PAGELOCK) or by updating the record at the beginning:

In this way, the transaction on the first thread will receive a Exclusive Lock (on write) and until it releases it (through the transaction commit) no other thread will be able to read this record. Of course, assuming that our queries operate in at least read committed transaction isolation level to avoid the so-called dirty reads.

Optimistic concurrency

An alternative, and most often preferred solution is to use Optimistic Concurrency. In this case, the whole process takes place without locking the data. Instead, the data in the database is versioned and during the update, it is checked – whether there was a change of version in the meantime.

Optimistic Concurrency
Optimistic Concurrency

What does the implementation of this solution look like? Most ORMs support Optimistic Concurrency out of the box. It is enough to indicate which fields should be checked when writing to the database and it will be added to the WHERE statement. If it turned out that our statement has updated 0 records, it means that the version of the record has changed and we need to do a rollback. Though often the current fields are not used to check the version and special column called “Version” or “Timestamp” is added.

Back to our example, does adding a column with a version and incrementing it every time the Order entity is changed solve the problem? Well no. The aggregate must be treated as a whole, as a boundary of transaction and consistency.

Therefore, the version incrementation must take place when we change anything in our aggregate. If we have added the Order Line and we keep it in a separate table, ORM support for optimistic concurrency will not help us because it works on updates and here there are inserts and deletes left.

How can we know that the state of our Aggregate has changed? If you follow this blog well, you already know the answer – by Domain Events. If a Domain Event was thrown from the Aggregate, it means that the state has changed and we need to increase Aggregate version.

The implementation can look like this.

First, we add a _version field to each Aggregate Root and the method to increment that version.

Secondly, we add a mapping for the version attribute and indicate that it is a EF Concurrency Token:

The last thing to do is incrementing the version if any Domain Events have been published:

With this setup, all updates will execute this statement:

For our example, for second thread no record will be updated ( @@ ROWOCOUNT = 0 ), so EntityFramework will throw the following message:

Microsoft.EntityFrameworkCore.DbUpdateConcurrencyException: Database operation expected to affect 1 row(s) but actually affected 0 row(s). Data may have been modified or deleted since entities were loaded. See http://go.microsoft.com/fwlink/?LinkId=527962 for information on understanding and handling optimistic concurrency exceptions.

and our Aggregate will be consistent – the 6th Order Line will not be added. The business rule is not broken, mission accomplished.

Summary

In summary, the most important issues here are:

  • The Aggregate’s main task is to protect invariants (business rules, the boundary of immediate consistency)
  • In a multi-threaded environment, when multiple threads are running simultaneously on the same Aggregate, a business rule may be broken
  • A way to solve concurrency conflicts is to use Pessimistic or Optimistic concurrency techniques
  • Pessimistic Concurrency involves the use of a database transaction and a locking mechanism. In this way, requests are processed one after the other, so basically concurrency is lost and it can lead to deadlocks.
  • Optimistic Concurrency technique is based on versioning database records and checking whether the previously loaded version has not been changed by another thread.
  • Entity Framework Core supports Optimistic Concurrency. Pessimistic Concurrency is not supported
  • The Aggregate must always be treated and versioned as a single unit
  • Domain events are an indicator, that state was changed so Aggregate version should be changed as well
  • GitHub sample repository

    Especially for the needs of this article, I created a repository that shows the implementation of 3 scenarios:

    – without concurrency handling
    – with Pessimistic Concurrency handling
    – with Optimistic Concurrency handling

    Link: https://github.com/kgrzybek/efcore-concurrency-handling

    Related Posts

    Handling Domain Events: Missing Part
    Domain Model Encapsulation and PI with Entity Framework 2.2
    Attributes of Clean Domain Model

Attributes of Clean Domain Model

Introduction

There is a lot of talk about clean code and architecture nowadays. There is more and more talk about how to achieve it. The rules described by Robert C. Martin are universal and in my opinion, we can use them in various other contexts.

In this post I would like to refer them to the context of the Domain Model implementation, which is often the heart of our system. We want to have a clean heart, aren’t we?

As we know from the Domain-Driven Design approach, we have two spaces – problem space and solution space. Vaughn Vernon defines these spaces in DDD Distilled book as follows:

Problem space is where you perform high-level strategic analysis and design steps within the constraints of a given project.

Solution space is where you actually implement the solution that your problem space discussions identify as your Core Domain.

So the implementation of the Domain Model is a solution to a problem. To talk about whether the Domain Model is clean, we must define what a clean solution means first.

What is Clean Solution

I think that everyone has a definition of what a clean solution means or at least feels if what he does is clean or not. In any case, let me describe what it means to me.

First of all – a given solution has to solve a defined problem. This is mandatory. Even most-elegant solution which doesn’t solve our problem is worth nothing (in context of problem).

Secondly, a clean solution should be optimal from the point of view of time. It should take no more or less time than necessary.

Thirdly, it should solve exactly one defined problem with simplicity in mind but with adaption to change. We should keep always the balance – no less (no design or under-design), no more (over-design, see YAGNI). This could be summarized by a quote from Antoine Airman’s Odyssey book:

Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away.

Note: Of course we should strive for perfection by being aware that we will never achieve it.

Fourthly, we should solve problems based on the experience of others and learn from their mistakes. This is why architectural and design patterns or principles are so useful. We can take someone’s solution and adapt it to our needs. Moreover, we can take tools created by others and use them. This is game-changer in the IT world.

Last, but not least attribute is the level of understanding of the solution. We should be able to understand our solution even after a lot of time. More importantly, others should be able to understand it. The code is much more read than written. Martin Fowler said:

Any fool can write code that a computer can understand. Good programmers write code that humans can understand.

This is one of my favorite IT quotes. I try to remember it constantly as I write the code.

To summarize, the clean solution should solve exactly our problem in optimal time with a good balance of simplicity vs complexity. It should be developed based on other experiences, common knowledge, using appropriate tools and should be understandable. Considering all these factors, how can we determine that our Domain Model is a clean solution?

Clean Domain Model

1. Uses Ubiquitous Language

Ubiquitous Language which is one of the fundamentals of DDD strategic patterns in Domain Model plays key role. I like to think about writing code in a Domain Model like writing a book for business. Business people should be able to understand all logic, terms, and concepts which are used – even this is a computer program and they don’t know programming language! If we use the same language, concepts and meaning everywhere in a defined context, understanding of our problems and solutions can only increase the probability of success of our project.

2. Is Rich in Behavior

Domain Model, as the name suggests, models the domain. We have behavior in the domain, so our model should also have it. If this behavior is absent, we have dealing with the so-called Anemic Domain Model anti-pattern. For me, this is really a Data Model, not model of domain. This kind of model is useful sometimes but not when we have complex behvior and logic. It makes our code procedural and all the benefits of object-oriented programming disappear. The real Domain Model is rich in behavior.

3. Is encapsulated

Encapsulation of Domain Model is very important. We want to expose only the minimum information about our model to the outside world. We should prevent leaking business logic to our application logic or even worse – GUI. Ideally, we should expose only public methods on our aggregates (only entry to Domain Model). More information how achieve good level of Domain Model encapsulation you can read in my eariler post Domain Model Encapsulation and PI with Entity Framework 2.2.

4. Is Persistence Ignorant

Persistence Ignorance (PI) is another well-known concept and desirable attribute of Domain Model. We don’t want to have high coupling between our model and the persistence store. These are different concepts, tasks and responsibilities. Change in one thing should have minimal impact on other and vice-versa. Low coupling in this area means greater adaption of our system to change. Again, you can read how to minimize this coupling in .NET world using EF Core in the post which is linked above.

5. Sticks to SOLID Principles

As the Domain Model is object-oriented, all SOLID principles should be followed. Examples:

  • SRP – one Entity represents one business concept. One method does one thing. One event represents one fact.
  • O/C – business logic implementation should be easy to extend without changing other places. For that case, Strategy/Policy Pattern is most often used.
  • LSP – because of sticking to the composition over inheritance rule, sticking to LSP rule should be easy. Inheritance is the strongest level of coupling between classes and this is what we want to avoid
  • ISP – interfaces of our Domain Services or policies should be small – ideally should have one method
  • DI – to decouple our Domain Model from the rest of the application and infrastructure we need to use the Dependency Injection and Dependency Inversion principle. This combination gives us a powerful weapon – we can keep the same level of abstraction in our model, use business language in the whole model and hide implementation details from him.

6. Uses Domain Primitives

Most of the codebases I see operate on primitive types – strings, ints, decimals, guids. In the Domain Model, this level of abstraction is definitely too low.

We should always try to express significant business concepts using classes (entities or value objects). In this way, our model is more readable, easier to maintain, rich in behavior. In addition, our code is more secure because we can add validation at the class level. More about Domain Primitives you can read here.

7. Is efficient

Even the best-written Domain Model that cannot handle a request in a satisfactory time is useless. That is why entity size and aggregate boundaries are so important.

When designing aggregates, you need to know how they will be processed – how often, how many users will try to use them at the same time, whether there will be periods of increased activity and so on. The smaller the aggregate, the shorter data needs to be loaded and saved and the transaction is shorter. On the other hand, the smaller the aggregate, the smaller the consistency boundary so the correct balance is needed.

8. Is expressive

Class structure in Object-Oriented languages is a powerful tool to model business concepts. Unfortunately, it is often used incorrectly. The most common mistake is modeling many concepts with one class. For this (often unconsciously) statuses and bit flags are used.

Most often, such a class can be divided into several classes, which supports SRP. A good heuristics here are looking at the business language and looking at our entity’s behavior.

9. Is testable and tested

Domain Models are used to solve complicated problems. A complicated problem means complicated business logic to solve it. If you have a complicated solution you have to be sure that it works correctly and you can’t be afraid of changing this logic.

This is where unit tests enter, which are an integral part of a clean Domain Model. Clean Domain Model is testable and should have a maximum test coverage factor. This is the heart of our application and we should always be one hundred percent sure that it works as we expected.

10. It should be written with ease

This point is a summary of all the above. If writing your Domain Model code is a problem every time – it’s hard to change it, understand it, test it, use it – something is wrong. What you should do?

Review all the points described by me above and see what your model does not meet? Maybe it depends on the infrastructure? Maybe he speaks a non-business language? Maybe it’s anemic, poorly encapsulated or untestable? If your model meets all attributes described in this post, then solving business problems should be easy and pleasant. I assure you.

So I have a question – today are you relaxed by writing your business logic or did you hit your head against the wall again? Is your Domain Model clean?

Related posts

Domain Model Encapsulation and PI with Entity Framework 2.2
Domain Model Validation
Handling Domain Events: Missing Part

Handling Domain Events: Missing Part

Introduction

Some time ago I wrote post about publishing and handling domain events. In addition, in one of the posts I described the Outbox Pattern, which provides us At-Least-Once delivery when integrating with external components / services without using the 2PC protocol.

This time I wanted to present a combination of both approaches to complete previous posts. I will present a complete solution that enables reliable data processing in the system in a structured manner taking into account the transaction boundary.

Depth of the system

At the beginning I would like to describe what is a Shallow System and what is a Deep System.

Shallow System

The system is considered to be shallow when, most often, after doing some action on it, there is not much going on.

A typical and most popular example of this type of system is that it has many CRUD operations. Most of the operations involve managing the data shown on the screen and there is little business logic underneath. Sometimes such systems can also be called a database browser. 😉

Another heuristic that can point to a Shallow System is the ability to specify the requirements for such a system practically through the GUI prototype. The prototype of the system (possibly with the addition of comments) shows us how this system should work and it is enough – if nothing underneath is happening then there is nothing to define and describe.

From the Domain-Driven Design point of view, it will most often look like this: the execution of the Command on the Aggregate publishes exactly one Domain Event and… nothing happens. No subscribers to this event, no processors / handlers, no workflows defined. There is no communication with other contexts or 3rd systems either. It looks like this:

Shalow system in context of DDD
Shallow system in context of DDD.

Execute action, process request, save data – end of story. Often, in such cases, we do not even need DDD. Transaction Script or Active Record will be enough.

Deep system

The Deep System is (as one could easily guess) the complete opposite of the Shallow System.

A Deep System is one that is designed to resolve some problems in a non-trivial and complicated domain. If the domain is complicated then the Domain Model will be complicated as well. Of course, the Domain Model should be simplified as it is possible and at the same time it should not lose aspects that are most important in a given context (in terms of DDD – Bounded Context). Nevertheless, it contains a lot of business logic that needs to be handled.

We do not specify a Deep System by the GUI prototype because too much is happening underneath. Saving or reading data is just one of the actions that our system does. Other activities are communication with other systems, complicated data processing or calling other parts of our system.

This time, much more is happening in the context of Domain-Driven Design implementation. Aggregates can publish multiple Domain Events , and for each Domain Event there can be many handlers responsible for different behavior. This behavior can be communication with an external system or executing a Command on another Aggregate, which will again publish its events to which another part of our system will subscribe. This scheme repeats itself and our Domain Model reacts in a reactive manner:

Deep system
Deep system in context of DDD.

Problem

In post about publishing and handling domain events was presented very simple case and the whole solution did not support the re-publishing (and handling) of events by another Aggregate, which processing resulted from the previous Domain Event. In other words, there was no support for complex flows and data processing in a reactive way. Only one Command -> Aggregate -> Domain Event -> handlers scenario was possible.

It will be best to consider this in a specific example. Let’s assume the requirements that after placing an Order by the Customer:
a) Confirmation email to the Customer about placed Order should be sent
b) New Payment should be created
c) Email about new Payment to the Customer should be sent

These requirements are illustrated in the following picture:

Let’s assume that in this particular case both Order placement and Payment creation should take place in the same transaction. If transaction is successful, we need to send 2 emails – about the Order and Payment. Let’s see how we can implement this type of scenario.

Solution

The most important thing we have to keep in mind is the boundary of transaction. To make our life easier, we must make the following assumptions:

1. Command Handler defines transaction boundary. Transaction is started when Command Handler is invoked and committed at the end.
2. Each Domain Event handler is invoked in context of the same transaction boundary.
3. If we want to process something outside the transaction, we need to create a public event based on the Domain Event. I call it Domain Event Notification, some people call it a public event, but the concept is the same.

The second most important thing is when to publish and process Domain Events? Events may be created after each action on the Aggregate, so we must publish them:
– after each Command handling (but BEFORE committing transaction)
– after each Domain Event handling (but WITHOUT committing transaction)

Last thing to consider is processing of Domain Event Notifications (public events). We need to find a way to process them outside transaction and here Outbox Pattern comes in to play.

The first thing that comes to mind is to publish events at the end of each Command handler and commit the transaction, and at the end of each Domain Event handler only publish events. We can, however, try a much more elegant solution here and use the Decorator Pattern. Decorator Pattern allows us to wrap up our handling logic in infrastructural code, similar like Aspect-oriented programming and .NET Core Middlewares work.

We need two decorators. The first one will be for command handlers:

As you can see, in line 16 the processing of a given Command takes place (real Command handler is invoked), in line 18 there is a Unit of Work commit. UoW commit publishes Domain Events and commits the existing transaction:

In accordance with the previously described assumptions, we also need a second decorator for the Domain Event handler, which will only publish Domain Events at the very end without committing database transaction:

Last thing to do is configuration our decorators in IoC container (Autofac example):

Add Domain Event Notifications to Outbox

The second thing we have to do is to save notifications about Domain Events that we want to process outside of the transaction. To do this, we use the implementation of the Outbox Pattern:

As a reminder – the data for our Outbox is saved in the same transaction, which is why At-Least-Once delivery is guaranteed.

Implementing flow steps

At this point, we can focus only on the application logic and does not need to worry about infrastructural concerns. Now, we only implementing the particular flow steps:

a) When the Order is placed then create Payment:

b) When the Order is placed then send an email:

c) When the Payment is created then send an email:

The following picture presents the whole flow:

Flow of processing
Flow of processing

Summary

In this post I described how it is possible to process Commands and Domain Events in a Deep System in a reactive way.

Summarizing, the following concepts has been used for this purpose:

– Decorator Pattern for events dispatching and transaction boundary management
– Outbox Pattern for processing events in separate transaction
– Unit of Work Pattern
– Domain Events Notifications (public events) saved to the Outbox
– Basic DDD Building Blocks – Aggregates and Domain Events
– Eventual Consistency

Source code

If you would like to see full, working example – check my GitHub repository.

Additional Resources

The Outbox: An EIP Pattern – John Heintz
Domain events: design and implementation – Microsoft

Related posts

How to publish and handle Domain Events
Simple CQRS implementation with raw SQL and DDD
The Outbox Pattern

The Outbox Pattern

Introduction

Sometimes, when processing a business operation, you need to communicate with an external component in the Fire-and-forget mode. That component can be, for example:
– external service
– message bus
– mail server
– same database but different database transaction
– another database

Examples of this type of integration with external components:
– sending an e-mail message after placing an order
– sending an event about new client registration to the messaging system
– processing another DDD Aggregate in different database transaction – for example after placing an order to decrease number of products in stock

The question that arises is whether we are able to guarantee the atomicity of our business operation from a technical point of view? Unfortunately, not always, or even if we can (using 2PC protocol), this is a limitation of our system from the point of latency, throughput, scalability and availability. For details about these limitations, I invite you to read the article titled It’s Time to Move on from Two Phase Commit.

The problem I am writing about is presented below:

After execution of line 24 transaction is committed. In line 28 we want to send an event to event bus, but unfortunately 2 bad things can happen:
– our system can crash just after transaction commit and before sending the event
– event bus can be unavailable at this moment so the event cannot be sent

Outbox pattern

If we cannot provide atomicity or we don’t want to do that for the reasons mentioned above, what could we do to increase the reliability of our system? We should implement the Outbox Pattern.

Outbox pattern

The Outbox Pattern is based on Guaranteed Delivery pattern and looks as follows:

Outbox pattern

When you save data as part of one transaction, you also save messages that you later want to process as part of the same transaction. The list of messages to be processed is called an Outbox, just like in e-mail clients.

The second element of the puzzle is a separate process that periodically checks the contents of the Outbox and processes the messages. After processing each message, the message should be marked as processed to avoid resending. However, it is possible that we will not be able to mark the message as processed due to communication error with Outbox:

Outbox messages processing

In this case when connection with Outbox is recovered, the same message will be sent again. What all this means to us? Outbox Pattern gives At-Least-Once delivery. We can be sure that message will be sent once, but can be sent multiple times too! That’s why another name for this approach is Once-Or-More delivery. We should remember this and try to design receivers of our messages as Idempotents, which means:

In Messaging this concepts translates into a message that has the same effect whether it is received once or multiple times. This means that a message can safely be resent without causing any problems even if the receiver receives duplicates of the same message.

Ok, Enough theory, let’s see how we can implement this pattern in .NET world.

Implementation

Outbox message

At the beginning, we need to define the structure of our OutboxMessage:

What is important, the OutboxMessage class is part of the Infrastructure and not the Domain Model! Try to talk with business about Outbox, they will think about the outlook application instead of the messaging pattern. 🙂 I didn’t include ProcessedDate property because this class is only needed to save message as part of transaction so this property always will be NULL in this context.

Saving the message

For sure I do not want to program writing messages to the Outbox every time in each Command Handler, it is against DRY principle. For this reason, the Notification Object described in the post about publishing Domain Events can be used. Following solution is based on linked article with little modification – instead of processing the notifications immediately, it serializes them and writes them to the database.

As a reminder, all Domain Events resulting from an action are processed as part of the same transaction. If the Domain Event should be processed outside of the ongoing transaction, you should define a Notification Object for it. This is the object which should be written to the Outbox. The code looks like:

Example of Domain Event:

And Notification Object:

First thing to note is Json.NET library usage. Second thing to note are 2 constructors of CustomerRegisteredNotification class. First of them is for creating notification based on Domain Event. Second of them is to deserialize message from JSON string which is presented in following section regarding processing.

Processing the message

The processing of Outbox messages should take place in a separate process. However, instead of a separate process, we can also use the same process but another thread depending on the needs. Solution which is presented below can be used in both cases.

At the beginning, we need to use a scheduler that will periodically run Outbox processing. I do not want to create the scheduler myself (it is known and solved problem) so I will use one the mature solution in .NET – Quartz.NET. Configuration of Quartz scheduler is very simple:

Firstly, scheduler is created using factory. Then, new instance of IoC container for resolving dependencies is created. The last thing to do is to configure our job execution schedule. In case above it will be executed every 15 seconds but its configuration really depends on how many messages you will have in your system.

This is how ProcessOutboxJob looks like:

The most important parts are:
Line 1 – [DisallowConcurrentExecution] attribute means that scheduler will not start new instance of job if other instance of that job is running. This is important because we don’t want process Outbox concurrently.
Line 25 – Get all messages to process
Line 30 – Deserialize message to Notification Object
Line 32 – Processing the Notification Object (for example sending event to bus)
Line 38 – Set message as processed

As I wrote earlier, if there is an error between processing the message (line 32) and setting it as processed (line 38), job in the next iteration will want to process it again.

Notification handler template looks like this:

Finally, this is view of our Outbox:

Outbox view

Summary

In this post I described what are the problems with ensuring the atomicity of the transaction during business operation processing. I’ve raised the topic of 2PC protocol and motivation to not use it. I presented what the Outbox Pattern is and how we can implement it. Thanks to this, our system can be really more reliable.

Source code

If you would like to see full, working example – check my GitHub repository.

Additional Resources

Refactoring Towards Resilience: Evaluating Coupling – Jimmy Bogard
Asynchronous message-based communication – Microsoft

Related posts

Domain Model Encapsulation and PI with Entity Framework 2.2
Simple CQRS implementation with raw SQL and DDD
How to publish and handle Domain Events

Domain Model Validation

Introduction

In previous post I described how requests input data can be validated on Application Services Layer. I showed FluentValidation library usage in combination with Pipeline Pattern and Problem Details standard. In this post I would like to focus on the second type of validation which sits in the Domain LayerDomain Model validation.

What is Domain Model validation

We can divide the validation of the Domain Model into two types based on scope – Aggregates scope and Bounded Context scope

Aggregates scope

Let’s remind what the Aggregate is by quoting a fragment of the text from Vaughn Vernon Domain-Driven Design Distilled book:

Each Aggregate forms a transactional consistency boundary. This means that within a single
Aggregate, all composed parts must be consistent, according to business rules, when the controlling
transaction is committed to the database.

The most important part of this quote in context of validation I underlined. It means that under no circumstances we can’t persist Aggregate to database which has invalid state or breaks business rules. These rules are often called “invariants” and are defined by Vaughn Vernon as follows:

… business invariants — the rules to which the software must always adhere — are guaranteed to be consistent following each business operation.

So in context of Aggregates scope, we need to protect these invariants by executing validation during our use case (business operation) processing.

Bounded Context scope

Unfortunately, validation of Aggregates invariants is not enough. Sometimes the business rule may apply to more than one Aggregate (they can be even aggregates of different types).

For example, assuming that we have Customer Entity as Aggregate Root, the business rule may be “Customer email address must be unique”. To check this rule we need to check all emails of Customers which are separated Aggregate Roots. It is outside of the scope of one Customer aggregate. Of course, supposedly, we could create new entity called CustomerCatalog as Aggregate Root and aggregate all of the Customers to it but this is not good idea for many reasons. The better solution is described later in this article.

Let’s see what options we have to solve both validation problems.

Three solutions

Return Validation Object

This solution is based on Notification Pattern. We are defining special class called Notification/ValidationResult/Result/etc which “collects together information about errors and other information in the domain layer and communicates it”.

What does it mean for us? Is means that for every entity method which mutates the state of Aggregate we should return this validation object. The keyword here is entity because we can have (and we likely will have) nested invocations of methods inside Aggregate. Recall the diagram from the post about Domain Model encapsulation:

Domain model encapsulation

The program flow will look like:

Validation Object Flow

and the code structure (simplified):

However, if we don’t like to return ValidationResult from every method which mutates the state we can apply different approach which I described in article about publishing Domain Events. In short, in this solution we need to add ValidationResult property for every Entity (as Domain Events collection) and after Aggregate processing we have to examine these properties and decide if the whole Aggregate is valid.

Deferred validation

Second solution how to implement validation is to execute checking after whole Aggregate’s method is processed. This approach is presented for example by Jeffrey Palermo in his article. The whole solution is pretty straightforward:

Deferred validation

Always Valid

Last but not least solution is called “Always Valid” and it’s just about throwing exceptions inside Aggregate methods. It means that we finish processing of the business operation with the first violation of the Aggregate invariant. In this way, we are assured that our Aggregate is always valid:

Always Valid program flow

Comparison of solutions

I have to admit that I don’t like Validation Object and Deferred Validation approach and I recommend Always Valid strategy. My reasoning is as follows.

Returning Validation Object approach pollutes our methods declarations, adds accidental complexity to our Entities and is against Fail-Fast principle. Moreover, Validation Object becomes part of our Domain Model and it is for sure not part of ubiquitous language. On the other hand Deferred Validation implies not encapsulated Aggregate, because the validator object must have access to aggregate internals to properly check invariants.

However, both approaches have one advantage – they do not require throwing exceptions which should be thrown only when something unexpected occurs. Business rule broken is not unexpected.

Nevertheless, I think this is one of the rare exception when we can break this rule. For me, throwing exceptions and having always valid Aggregate is the best solution. “The ends justify the means” I would like to say. I think of this solution like implementation of Publish-Subsribe Pattern. Domain Model is the Publisher of broken invariants messages and Application is the Subscriber to this messages. The main assumption is that after publishing message the publisher stops processing because this is how exceptions mechanism works.

Always Valid Implementation

Exception throwing is built into the C# language so practically we have everything. Only thing to do is create specific Exception class, I called it BusinessRuleValidationException:

Suppose we have a business rule defined that you cannot order more than 2 orders on the same day. So it looks implementation:

What we should do with the thrown exception? We can use approach from REST API Data Validation and return appropriate message to the client as Problem Details object standard. All we have to do is to add another ProblemDetails class and set up mapping in Startup:

The result returned to client:

Problem details validation domain model

For simpler validation like checking for nulls, empty lists etc you can create library of guards (see Guard Pattern) or you can use external library. See GuardClauses created by Steve Smith for example.

BC scope validation implementation

What about validation which spans multiple Aggregates (Bounded Context scope)? Let’s assume that we have a rule that there cannot be 2 Customers with the same email address. There are two approaches to solve this.

The first way is to get required aggregates in CommandHandler and then pass them to aggregate’s method/constructor as arguments:

However, this is not always a good solution because as you can see we need to load all Customer Aggregates to memory. This could be serious performance issue. If we can not afford it then we need to introduce second approach – create Domain Service which is defined as (source – DDD Reference):

When a significant process or transformation in the domain is not a natural responsibility of an entity or value object, add an operation to the model as a standalone interface declared as a service

So, for that case we need to create ICustomerUniquenessChecker service interface:

This is the implementation of that interface:

Finally, we can use it inside our Customer Aggregate:

The question here is whether pass Domain Service as an argument to aggregate’s constructor/method or execute validation in Command Handler itself. As you can see above I am fan of former approach because I like keep my command handlers very thin. Another argument for this option is that if I ever need to register Customer from a different use case I will not be able to bypass and forget about this uniqueness rule because I will have to pass this service.

Summary

A lot of was covered in this post in context of Domain Model Validation. Let’s summarize:
– We have two types of Domain Model validation – Aggregates scope and Bounded Context scope
– There are generally 3 methods of Domain Model validation – using Validation Object, Deferred Validation or Always Valid (throwing exceptions)
Always Valid approach is preferred
– For Bounded Context scope validation there are 2 methods of validations – passing all required data to aggregate’s method or constructor or create Domain Service (generally for performance reason).

Source code

If you would like to see full, working example – check my GitHub repository.

Additional Resources

Validation in Domain-Driven Design (DDD) – Lev Gorodinski
Validation in a DDD world– Jimmy Bogard

Related posts

REST API Data Validation
Domain Model Encapsulation and PI with Entity Framework 2.2
Simple CQRS implementation with raw SQL and DDD
How to publish and handle Domain Events

Domain Model Encapsulation and PI with Entity Framework 2.2

Introduction

In previous post I presented how to implement simple CQRS pattern using raw SQL (Read Model) and Domain Driven Design (Write Model). I would like to continue presented example focusing mainly on DDD implementation. In this post I will describe how to get most out of the newest version Entity Framework v 2.2 to support pure domain modeling as much as possible.

I decided that I will constantly develop my sample on GitHub. I will try to gradually add new functionalities and technical solutions. I will also try to extend domain so that the application will become similar to the real ones. It is difficult to explain some DDD aspects on trivial domains. Nevertheless, I highly encourage you to follow my codebase.

Goals

When we create our Domain Model we have to take many things into account. At this point I would like to focus on 2 of them: Encapsulation and Persistence Ignorance.

Encapsulation

Encapsulation has two major definitions (source – Wikipedia):

A language mechanism for restricting direct access to some of the object’s components

and

A language construct that facilitates the bundling of data with the methods (or other functions) operating on that data

What does it mean to DDD Aggregates? It just simply mean that we should hide all internals of our Aggregate from the outside world. Ideally, we should expose only public methods which are required to fulfill our business requirements. This assumption is presented below:

Persistence Ignorance

Persistence Ignorance (PI) principle says that the Domain Model should be ignorant of how its data is saved or retrieved. It is very good and important advice to follow. However, we should follow it with caution. I agree with opinion presented in the Microsoft documentation:

Even when it is important to follow the Persistence Ignorance principle for your Domain model, you should not ignore persistence concerns. It is still very important to understand the physical data model and how it maps to your entity object model. Otherwise you can create impossible designs.

As described, we can’t forget about persistence, unfortunately. Nevertheless, we should aim at decoupling Domain Model from rest parts of our system as much as possible.

Example Domain

For a better understanding of the created Domain Model I prepared the following diagram:

It is simple e-commerce domain. Customer can place one or more Orders. Order is a set of Products with information of quantity ( OrderProduct). Each Product has defined many prices ( ProductPrice) depending on the Currency.

Ok, we know the problem, now we can go to the solution…

Solution

1. Create supporting architecture

First and most important thing to do is create application architecture which supports both Encapsulation and Persistence Ignorance of our Domain Model. The most common examples are:
Clean Architecture
Onion Architecture
Ports And Adapters / Hexagonal Architecture

All of these architectures are good and and used in production systems. For me Clean Architecture and Onion Architecture are almost the same. Ports And Adapters / Hexagonal Architecture is a little bit different when it comes to naming, but general principles are the same. The most important thing in context of domain modeling is that each architecture Business Logic/Business Layer/Entities/Domain Layer 1) is in the center and 2) has no dependency to other components/layers/modules. It is the same in my example:

What this means in practice for our code in Domain Model?
1. No data access code.
2. No data annotations for our entities.
3. No inheritance from any framework classes, entities should be Plain Old CLR Object

2. Use Entity Framework in Infrastructure Layer only

Any interaction with database should be implemented in Infrastructure Layer. It means you have to add there entity framework context, entity mappings and implementation of repositories. Only interfaces of repositories can be kept in Domain Model.

3. Use Shadow Properties

Shadow Properties are great way to decouple our entities from database schema. They are properties which are defined only in Entity Framework Model. Using them we often don’t need to include foreign keys in our Domain Model and it is great thing.

Let’s see the Order Entity and its mapping which is defined in CustomerEntityTypeConfiguration mapping:

As you can see on line 15 we are defining property which doesn’t exist in Order entity. It is defined only for relationship configuration between Customer and Order. The same is for Order and ProductOrder relationship (see lines 23, 24).

4. Use Owned Entity Types

Using Owned Entity Types we can create better encapsulation because we can map directly to private or internal fields:

Owned types are great solution for creating our Value Objects too. This is how MoneyValue looks like:

5. Map to private fields

We can map to private fields not only using EF owned types, we can map to built-in types too. All we have to do is give the name of the field and column:

6. Use Value Conversions

Value Conversions are the “bridge” between entity attributes and table column values. If we have incompatibility between types, we should use them. Entity Framework has a lot of value converters implemented out of the box. Additionally, we can implement custom converter if we need to.

This converter simply converts “StatusId” column byte type to private field _status of type OrderStatus.

Summary

In this post I described shortly what Encapsulation and Persistence Ignorance is (in context of domain modeling) and how we can achieve these approaches by:
– creating supporting architecture
– putting all data access code outside our domain model implementation
– using Entity Framework Core features: Shadow Properties, Owned Entity Types, private fields mapping, Value Conversions

Related posts

Simple CQRS implementation with raw SQL and DDD
How to publish and handle Domain Events
REST API Data Validation

Simple CQRS implementation with raw SQL and DDD

Introduction

I often come across questions about the implementation of the CQRS pattern. Even more often I see discussions about access to the database in the context of what is better – ORM or plain SQL.

In this post I wanted to show you how you can quickly implement simple REST API application with CQRS using the .NET Core. I immediately point out that this is the CQRS in the simplest edition – the update through the Write Model immediately updates the Read Model, therefore we do not have here the eventual consistency. However, many applications do not need eventual consistency, while the logical division of writing and reading using two separate models is recommended and more effective in most solutions.

Especially for this article I prepared sample, fully working application, see full source on Github.

My goals

These are my goals that I wanted to achieve by creating this solution:
1. Clear separation and isolation of Write Model and Read Model.
2. Retrieving data using Read Model should be as fast as possible.
3. Write Model should be implemented with DDD approach. The level of DDD implementation should depend on level of domain complexity.
4. Application logic should be decoupled from GUI.
5. Selected libraries should be mature, well-known and supported.

Design

High level flow between components looks like:

As you can see the process for reads is pretty straightforward because we should query data as fast as possible. We don’t need here more layers of abstractions and sophisticated approaches. Get arguments from query object, execute raw SQL against database and return data – that’s all.

It is different in the case of write support. Writing often requires more advanced techniques because we need execute some logic, do some calculations or simply check some conditions (especially invariants). With ORM tool with change tracking and using Repository Pattern we can do it leaving our Domain Model intact (ok, almost).

Solution

Read model

Diagram below presents flow between components used to fulfill read request operation:

The GUI is responsible for creating Query object:

Then, query handler process query:

The first thing is to get open database connection and it is achieved using SqlConnectionFactory class. This class is resolved by IoC Container with HTTP request lifetime scope so we are sure, that we use only one database connection during request processing.

Second thing is to prepare and execute raw SQL against database. I try not to refer to tables directly and instead refer to database views. This is a nice way to create abstraction and decouple our application from database schema because I want to hide database internals as much as possible.

For SQL execution I use micro ORM Dapper library because is almost as fast as native ADO.NET and does not have boilerplate API. In short, it does what it has to do and it does it very well.

Write model

Diagram below presents flow for write request operation:

Write request processing starts similar to read but we create the Command object instead of the query object:

Then, CommandHandler is invoked:

Command handler looks different than query handler. Here, we use higher level of abstraction using DDD approach with Aggregates and Entities. We need it because in this case problems to solve are often more complex than usual reads. Command handler hydrates aggregate, invokes aggregate method and saves changes to database.

Customer aggregate can be defined as follows:

Architecture

Solution structure is designed based on well-known Onion Architecture as follows:

Only 3 projects are defined:
– API project with API endpoints and application logic (command and query handlers) using Feature Folders approach.
– Domain project with Domain Model
– Infrastructure project – integration with database.

Summary

In this post I tried to present the simplest way to implement CQRS pattern using raw sql scripts as Read Model side processing and DDD approach as Write Model side implementation. Doing so we are able to achieve much more separation of concerns without losing the speed of development. Cost of introducing this solution is very low and and it returns very quickly.

I didn’t describe DDD implementation in detail so I encourage you once again to check the repository of the example application – can be used as a kit starter for your app the same as for my applications.

Related posts

Domain Model Encapsulation and PI with Entity Framework 2.2
How to publish and handle Domain Events
REST API Data Validation

How to publish and handle Domain Events

2019-06-19 UPDATE: Please check Handling Domain Events: Missing Part post which is a continuation of this article

Introduction

Domain Event is one of the building blocks of Domain Driven Design. It is something that happened in particular domain and it captures memory of it. We create Domain Events to notify other parts of the same domain that something interesting happened and these other parts potentially can react to.

Domain Event is usually immutable data-container class named in the past tense. For example:

Three ways of publishing domain events

I have seen mainly three ways of publishing domain events.

1. Using static DomainEvents class

This approach was presented by Udi Dahan in his Domain Events Salvation post. In short, there is a static class named DomainEvents with method Raise and it is invoked immediately when something interesting during aggregate method processing occurred. Word immediately is worth emphasizing because all domain event handlers start processing immediately too (even aggregate method did not finish processing).

2. Raise event returned from aggregate method

This is approach when aggregate method returns Domain Event directly to ApplicationService. ApplicationService decides when and how to raise event. You can become familiar with this way of raising events reading Jan Kronquist Don’t publish Domain Events, return them! post.

3. Add event to Events Entity collection.

In this way on every entity, which creates domain events, exists Events collection. Every Domain Event instance is added to this collection during aggregate method execution. After execution, ApplicationService (or other component) reads all Eventscollections from all entities and publishes them. This approach is well described in Jimmy Bogard post A better domain events pattern.

Handling domain events

The way of handling of domain events depends indirectly on publishing method. If you use DomainEvents static class, you have to handle event immediately. In other two cases you control when events are published as well handlers execution – in or outside existing transaction.

In my opinion it is good approach to always handle domain events in existing transaction and treat aggregate method execution and handlers processing as atomic operation. This is good because if you have a lot of events and handlers you do not have to think about initializing connections, transactions and what should be treat in “all-or-nothing” way and what not.

Sometimes, however, it is necessary to communicate with 3rd party service (for example e-mail or web service) based on Domain Event. As we know, communication with 3rd party services is not usually transactional so we need some additional generic mechanism to handle these types of scenarios. So I created Domain Events Notifications.

Domain Events Notifications

There is no such thing as domain events notifications in DDD terms. I gave that name because I think it fits best – it is notification that domain event was published.

Mechanism is pretty simple. If I want to inform my application that domain event was published I create notification class for it and as many handlers for this notification as I want. I always publish my notifications after transaction is committed. The complete process looks like this:

1. Create database transaction.
2. Get aggregate(s).
3. Invoke aggregate method.
4. Add domain events to Events collections.
5. Publish domain events and handle them.
6. Save changes to DB and commit transaction.
7. Publish domain events notifications and handle them.

How do I know that particular domain event was published?

First of all, I have to define notification for domain event using generics:

All notifications are registered in IoC container:

In EventsPublisher we resolve defined notifications using IoC container and after our unit of work is completed, all notifications are published:

This is how whole process looks like presented on UML sequence diagram:

You can think that there is a lot of things to remember and you are right!:) But as you can see whole process is pretty straightforward and we can simplify this solution using IoC interceptors which I will try to describe in another post.

Summary

1. Domain event is information about something which happened in the past in modeled domain and it is important part of DDD approach.
2. There are many ways of publishing and handling domain events – by static class, returning them, exposing by collections.
2. Domain events should be handled within existing transaction (my recommendation).
3. For non-trasactional operations Domain Events Notifications were introduced.

Related posts

Handling Domain Events: Missing Part
The Outbox Pattern

Processing commands with Hangfire and MediatR

In previous post about processing multiple instance aggregates of the same type I suggested to consider using eventual consistency approach. In this post I would like to present one way to do this.

Setup

In the beginning let me introduce stack of technologies/patterns:
1. Command pattern – I am using commands but they do not look like theses described in GoF book. They just simple classes with data and they implement IRequest  marker interface of MediatR.

2. Mediator pattern. I am using this pattern because i want to decouple my clients classes (commands invokers) from commands handlers. Simple but great library created by Jimmy Bogard named MediatR implements this pattern very well. Here is simple usage.

and handler:

3. Hangfire. Great open-source library for processing and scheduling background jobs even with GUI monitoring interface. This is where my commands are scheduled, executed and retried if error occured.

Problem

For some of my uses cases, I would like to schedule processing my commands, execute them parallel with retry option and monitor them. Hangfire gives me all these kind of features but I have to have public method which I have to pass to Hangifre method (for example BackgroundJob.Enqueue). This is a problem – with mediator pattern I cannot (and I do not want) pass public method of handler because I have decoupled it from invoker. So I need special way to integrate MediatR with Hangfire without affecting basic assumptions.

Solution

My solution is to have three additional classes:
1. CommandsScheduler – serializes commands and sends them to Hangfire.

2. CommandsExecutor – responods to Hangfire jobs execution, deserializes commands and sends them to handlers using MediatR.

3. MediatorSerializedObject – wrapper class for serialized/deserialized commands with additional properties – command type and additional description.

Finally with this implementation we can change our client clasess to use CommandsScheduler:

and our commands are scheduled, invoked and monitored by Hangfire. I sketched sequence diagram which shows this interaction:

Processing commands with MediatR and Hanfire

Additionally, we can introduce interface for CommandsSchedulerICommandsScheduler. Second implementation will not use Hangfire at all and only will execute MediatR requests directly – for example in development process when we do not want start Hangfire Server.

Summary

I presented the way of processing commands asynchronously using MediatR and Hangfire. With this approach we have:
1. Decoupled invokers and handlers of commands.
2. Scheduling commands mechanism.
3. Invoker and handler of command may be other processes.
4. Commands execution monitoring.
5. Commands execution retries mechanism.

These benefits are very important during development using eventual consistency approach. We have more control over commands processing and we can react quickly if problem will appear.