Feature Folders

Introduction

Today I would like to suggest a less-common but in my opinion a much better way to organize our codebase. Meet the Feature Folders.

Problem

For ages we have been (at least in the .NET environment) used to thinking about our code structure taking into account the technical aspects. For example MVC application project templates assume the division our objects into separate directories – Controllers, Views, Scripts and so on. We can see the same in many tutorials. If we need add new feature, following this approach we should add all objects in different places.

This approach is found not only on the application layer but on others too. I have seen many times in the business logic layer the big folders called Aggregates, Entities, Domain Events with a lot of classes unrelated to each other. But what does it mean “unrelated”?

We can specify 2 types of relationships between objects – technical and business.

Technical relationship tells whether two objects have similar meaning from technical perspective. That is, whether they have the same usage. Two controllers are technically related, but controller and application service are not related – their purpose is different.

Business relationship tells whether two objects support the fulfillment of the same use case. These can be definitely different objects from a technical point of view – for example: a validator and a command handler.

Let’s see how technical folders can look like:
Technical folders

What is the problem with this design? As you can see, we have three modules: Commands, Handlers and Validators. Each of these modules has very low cohesion because as Wiki says:

cohesion refers to the degree to which the elements inside a module belong together

For example the new requirement appears and we need add new attribute which can be editable. We need change all of three objects associated with editing so in this particular design we need change all three modules as well. The same if we would like to move all functionality to a separate service. This is not good. We have to try achieve as high cohesion as possible.

Solution

The solution is to stop thinking about the technical aspects of our objects and focus instead on the business relationship. It will provide high cohesion with all its benefits (maintainability, reusability, less complexity and so on).

For every feature/use case we should create separate Feature Folder with all related from business perspective objects:

The same rule applies for business logic layer and I think it is even more important here. We should design our domain model per aggregates:

This is very simple approach but improves our design and especially in projects with many features makes work definitely easier.

There is an extra bonus. With this design we create templates for later requirements. For example if we need add implementation for adding a new product, we can copy paste whole feature folder, rename objects and we know exactly what we have to implement. Great!

Summary

In this post I showed what Feature Folders are. By good codebase organization we can achieve more better and elegant design. If you still use “technical folders” I encourage you to try this solution – you will not regret it. 🙂

Using Database Project and DbUp for database management

Introduction

In previous post I described two popular ways to manage database changes.

The first one was state versioning where you keep whole current design of your database in repository and when you need change something then you need only change this state. Later, when you want to deploy changes, your schema and target database is compared and the migration script is generated.

The second way is to versioning transitions to desired state, which means creating migration script for every change.

In this post I wanted to show implementation of these two approaches in .NET environment combined together – what I think is the best way to manage database changes.

Step one – Database Project

The first thing to do is create Database Project. This type of project is available only when you have SQL Server Data Tools installed. It can be installed together with Visual Studio 2017 or separately – see this page for more information.

When you have SQL Server Data Tools you can add new Database Project the standard way:

Now we can add database objects to our project in the form of SQL scripts. Each script should define one database object – table, view, procedure, function and so on. It is common to create root folders as schemes are named.

TIP: I do not recommend creating database objects in “dbo” schema. I advise to create good named schemes per module/purpose/functionality. Creating your own schemes also allow you to better manage your object namespaces.

The sample database project may look like this:

What is worth to notice is the Build Action setting of every script is set to Build. This is the setting after which Visual Studio recognizes database objects from ordinary scripts and build them together. If we for example remove script defining orders schema, VS will not be able to build our project:

This is great behavior because we have compile-time check and we can avoid more runtime errors.

When we finished database project, we can compare it to other project or database and create migration script. But as I described in previous post this is not optimal way to migrate databases. We will use DbUp library instead.

Step two – DbUp

DbUp is open source .NET library that provide you a way to deploy changes to database. Additionally, it tracks which SQL scripts have been run already, has many sql scripts providers available and other interesting features like scripts pre-processing.

You can ask a question why DbUp and not EF Migrations or Fluent Migrator? I have used all of them and I have to say that DbUp seems to me the most pure solution. I don’t like C# “wrapers” to generate SQL for me. DDL is easy language and I think we don’t need special tool for generating it.

DbUp is library so we can reference it to each application we want. What we need is simple console application which can be executed both on developer environment and CI build server. Firstly, we need reference DbUp NuGet package. Then we can add simple code to Main method:

This console application accepts two parameters: connection string to target database and file system path to scripts directory. It assumes following directory layout:
/PreDeployment
/Migrations
/PostDeployment

For “pre” and “post” deployment scripts we are defining NullJournal – in this way scripts will be run every time.

We should keep directory scripts in Database Project created earlier. DbUp executes scripts in alphabetical order. It can look like this:

Finally, we run migrations running our console application:

Executed scripts are listed in app.MigrationsJournal table:

And that’s all! We can develop and change our database in effective way now. 🙂

Summary

In this post I described how to implement both state and transitions versioning using Database Project na DbUp library. What has been achieved is:
– Compile-time checks (Database project)
– Ease of development (Both)
– History of definition of all objects (Database project)
– Quick access to schema definition (Database project)
– Ease of resolving conflicts (Database project)
– IDE support (Database project)
– Full control of defining transitions (DbUp)
– Pre and post deployment scripts execution (DbUp)
– Deployment automation (DbUp)
– The possibility of manual deployment (DbUp)
– History of applied transitions (DbUp).

Using this machinery the development of database should be definitely easier and less error-prone.

Database change management

Introduction

Database change management is not an easy task. It is even more difficult when we are at the beginning of a project where the data model is constantly changing. More people in the team is another difficulty because we have to coordinate our work and not interfere with each other. Of course, we can delegate database change management to one person, but this results in a bottleneck. In addition, we should take care of the deployment automation, because without automation we can not have truly continuous integration and delivery.

In this post I will describe:
– what we need to have good database change management mechanism
– two common approaches to solve this problem
– recommendation, which approach should be taken

The ideal solution

I think we should stick to the same rules as when managing code changes. Which means that we should implement the following practices:

Everything should be in the source control

We should follow what was the change, who did it and when. There is no better tool for this than source control. Every little change, every little script should be thrown into the repository.

Every developer should be able to run migrations

Every developer should have local database on his own environment and should be able to always upgrade his database after downloading the changes. Without it, developer will have new version of application and old version of database. This will cause unpredictable behavior. Shared development databases say no! What is also important, the deployment of changes should be very fast.

Easy conflict resolution

Conflicts on the same database objects should be easy to solve and should not require a lot of work.

Versioning

We always should know in what version our database is. In addition, we should know what migrations have already been deployed and which ones not.

Deployment automation

Our mechanism should provide us with the ability to automate the deployment of changes to tests and productions environments. We should use the same mechanism during development to make sure everything will be fine and will work as expected.

The possibility of manual deployment

Sometimes changes can only be deployed manually by human due to procedures or regulations in the company. For this reason, we should be able to generate a set of scripts for manual deployment.

The ability to run “pre” and “post” deployment scripts

It is common to execute some logic before or/and after deployment. Sometimes we need regenerate some data, sometimes we need to check something (constraints integrity for example). This type of feature is very useful.

IDE support

It would be perfect to have the full support of our IDE. Quick access to objects schema (without connection to database), prompts, compile-time checking – these are the things we need to be very productive during database development.

A lot of these requirements, let’s see what solutions we have.

Versioning the State

The first approach to database change management is state versioning. We hold whole database definition in our source control repository and when we need change something we change objects definitions. When the upgrade time comes, our tool compares our actual definitions with target database and generates migration script for us (and he can execute it right away). The process looks as follows:

As I described we only change objects definitions. For example we have orders table and in our repository we have:

When we need change something we just change definition:

I altered Name column and added Description column. That’s all – I changed state of my schema. The transition to this state is on the tool side.

Pros

– Compile-time checks (whole schema in one place)
– Ease of development
– History of definition of all objects
– Quick access to schema definition
– Ease of resolving conflicts
– IDE support

Cons

The biggest downside of this approach is that sometimes tool you are using can not be able to generate migration script based on differences. This is situation when you:
– try to add new not nullable column without default value.
– you have different order of columns in project vs target database. Then tool tries to change this order by creating temp tables, coping data and renaming – something that we do not always want because from a theoretical point of view order of columns doesn’t matter.
– try to rename object, column
– change type of column (how to convert?)

All of these problems (despite the many advantages) make it not a good approach to database change management.

Versioning transitions

Another approach is to versioning transitions instead of state. In this approach we create sequence of migration scripts which lead us to the desired database state.

For the example with orders table, instead of change definition we just add migration script as follows:

In this approach we should be aware of 2 things:
– order of executing scripts does matter
– is required to store which scripts were executed.

Pros

– Full control of defining transitions
– Executing scripts one by one in correct order guarantees success of deployment to other environments
– Possibility to implement undo/downgrade features

Cons

– Not so easy as change state definition
– Lack of objects history
– Lack of IDE support
– Hardly visible conflicts (2 developers add changes to the same table in 2 separate files)
– Need to keep order of scripts
– Need to keep which scripts were executed

So this is not perfect solution either, has some disadvantages but it is still better than versioning state because we have more control and possibilities. And now we are going to…

Best solution

The best solution is simple but not easy – versioning transitions AND versioning states. 🙂

If you look at the pros and cons of both solutions you have to admit that they are complementary. By using both approaches you can get rid of all cons leaving advantages.

Well, almost.. 🙁 Combining two approaches costs a little more time and work. You need always create migration script and change definition as well. In addition, you need to be careful about consistency and integrity with two models. Transitions should always lead to defined state. If you have mismatch, you do not know how final schema should look like.

Fortunately a little more work earlier saves you a lot of work later. I worked separately with both approaches in many projects earlier. Now when I combined them I truly feel certain about development and deployment of my databases. I wish I knew it before. 🙂

Summary

In this post I described:
– how ideal database change mechanism should look like
– state versioning approach
– transitions versioning approach
– the best solution that is a combination of both

In next post I will show how to setup database change management in .NET based on these assumptions.

How to store sensitive configuration data

Introduction

In this post I would like to discuss places where we can store configuration of our application and warn against keeping sensitive configuration data in the code repository. Additionally, I will show example of configuration based on User Secrets and Environment Variables in .NET Core.

Types of configuration data

There are plenty of configuration data types. I will try to name a few of them:
– store configurations (connection strings, providers, timeouts)
– external API configurations (urls, endpoints)
– caching
– hosting configuration (url, port, schema)
– file system paths (storing files, loading other resources)
– framework configuration
– libraries configuration
– business logic parameters
– …

Apart from the configuration type, we can add additional categorization:

Sensitive vs Insensitive

Sensitive data that must be protected from unauthorized access to safeguard the security of an our application or user’s data. Examples of sensitive data are private key for external API or username and password in database connection string. Example of insensitive configuration data could be database timeout length setting.

Local vs Global

Local configuration is dependent on environment. For example connection string to database is different for local development, test or production environments because they are using their own storage.

Global configuration is something independent from environment. For example it could be list of predefined file extensions which user can upload.

Required vs Optional

Configuration data for specific component could be required or optional. For database connection string server and database name are required parameters but for example max connection per pool is not.

Handling type

There are 3 types of handling configuration data:

1) Fetched once, used many time
2) Fetched every now and then, used many times
3) Fetched each time it is used

Configuration sources

How many configuration types exist there are so many sources of it. Diagram below presents possible variations:

Application configuration file

This is the most common option. Configuration data is included in application and can be changed by editing/replacing configuration file.

External configuration file

It is also configuration file but situated outside the application – on the same computer or somewhere on the network.

Environment variable

Configuration data is stored in operating system environment variables. This option has become popular by the trend on containerization.

Store

Sometimes there is a need to store configuration data in the database. The reason for this is externalization of configuration (when more than one process needs this data) or we need easy way to edit that configuration.

System / Service

When configuration is more complicated and dynamic it is good approach to get configuration from external system / service. An example of such a service is the Oculus which is service discovery mechanism in microservices environment.

Storing sensitive configuration data

If you would have to remember one thing from this entry, that would be it: Don’t store sensitive data in your code repositories. Never. Period.

Even if you have private repositories on Github or Bitbucket .
Even if you have private repositories hosted on your internal servers and whole infrastructure is hidden behind the NAT.
Even if you created only local repository which is not published to remote server.

Don’t do it.

How many times you thought in this way:

Hmm…Ok, I have to add credentials configuration for database/api/whatever now. For now I will simply add them to configuration file and when I get ready to deploy new feature to test/prod I will change it. Or I will do it before committing my changes. Yes, I will only check if this connection works..

…and the data suddenly appears in the repository. Because we forgot. Because we took care of something else. Because…

Too much “becauses”. The right solutions are simple and I will show two of them: storing configuration in external configuration file (User Secrets) and Environment variables.

User Secrets

User Secrets is nothing else than JSON configuration file stored in a system-protected user profile folder on the local machine in path %APPDATA%\Microsoft\UserSecrets\<user_secrets_id>\secrets.json. User secrets id is GUID generated by Secret Manager tool.

Process of adding User Secrets is pretty simple. Firstly, we need right click on the project and select Manage User Secrets option.

After this step Visual Studio will generate secrets.json file and add entry to .csproj file:

Then we can add needed configuration to generated secrets.json file, for example:

Last thing to do is reference Microsoft.Extensions.Configuration.UserSecrets provider configuration package and register it in application startup:

Environment variables

User Secrets are good in development scenarios. For other scenarios when we need deploy our application to other environments (test, stage, prod) we should use Environment variables instead. First step is to define variable for hosting operating system / container. For Windows it looks like:

setx SampleKey SampleValue /M

Second and last step is to configure provider referenced from Microsoft.Extensions.Configuration.EnvironmentVariables:

Extension method AddEnvironmentVariables() has overload with prefix parameter. It can be useful for variables filtering or when we have multiple environments on single host. This is example configuration:

How to implement GetEnvironment() method depends on your preferences. It can be determined based on project configuration or other factor like external file.

Summary

In this post I described the diversity of configuration data as well as their possible sources. In addition, I showed how to keep sensitive configuration data securely on the .NET Core runtime environment using User Secrets and Environment variables. The security of applications and our data is often overlooked in the initial phase of the project, which is a big mistake and can be tragic in consequences. It’s better to think about it from the very beginning.

Cache-Aside Pattern in .NET Core

Introduction

Often the time comes when we need to focus on optimizing the performance of our application. There are many ways to do this and one way is cache some data. In this post I will describe briefly the Cache-Aside Pattern and its simple implementation in .NET Core.

Cache-Aside Pattern

This pattern is very simple and straightforward. When we need specific data, we first try to get it from the cache. If the data is not in the cache, we get it from the source, add it to the cache and return it. Thanks to this, in the next query the data will be get from the cache. When adding to the cache, we need to determine how long the data should be stored in the cache. Below is an algorithm diagram:

First implementation

Implementation of this pattern in .NET Core is just as easy as its theoretical part. Firstly, we need register IMemoryCache interface:

Afterwards, we need add Microsoft.Extensions.Caching.Memory NuGet package.

And that’s all. Assuming that we want to cache basic information about our users, the implementation of the pattern looks as follows:

First of all, we are injecting .NET Core framework IMemoryCache interface implementation. Then in line 18 we check whether the data is in the cache. If it is not in the cache, we get it from the source (i.e. database), add to cache and return.

Code smells

This way of implementation you can find on MSDN site. I could finish this post at this point, but I must admit that there are a few things that I do not like about this code.

First of all, I think the interface IMemoryCache is not abstract enough. It suggests that the data is kept in application memory but the client code should not care where it is stored. Moreover, if we want to keep the cache in the database in the future, the name of this interface will not be correct.

Secondly, client code should not be responsible for logic of the naming cache key. It is Single Responsibility Principle violation. It should only provide data to create this key name.

Lastly, client code should not care about cache expiration. It should be configured in other place – application configuration.

In next section I will show how we can eliminate these 3 code smells.

Improved implementation

The first and most important step is to define a new, more abstract interface: ICacheStore

Then we need to define interface for our cache key classes:

This interface has CacheKey string property which is used during resolving cache key in our MemoryCacheStore implementation:

Finally, we need to configure IoC container to resolve MemoryCacheStore instance as ICacheStore together with expiration configuration taken from application configuration:

This is how new implementation looks like:

After this set up we can finally use this implementation in our client code. For each new object that we want to store in cache we need:

1) Add expiration configuration

2) Class that defines the cache key

Finally, the new client code looks like this:

In the code above we use more abstract ICacheStore interface, don’t care about creation of cache key and expiration configuration. It is more elegant solution and less error-prone.

Summary

In this post I described Cache-Aside Pattern and its primary implementation in .NET Core. I proposed also augmented design to achieve more elegant solution with a small amount of work. Happy caching! 🙂

Company, don’t waste your programmers time

Introduction

In this post I would like to raise the subject of wasting programmers time which can be easily saved. What’s more, it is said time is money so we can save both! This post is especially for management people (managers, directors, CTOs) but developers should also think about whether they use their potential one hundred percent.

These days it is very hard to hire good-skilled and experienced programmer because there are few of them (comparing the demand on the market) and they cost a lot. So when the company finally manages to hire a programmer, what he can do to use this valuable resource effectively for the benefit of both sides (win-win rule)?

1. Provide him with the right equipment

There is nothing more annoying than a situation where you can’t honestly carry out your duties not because of you. I will mention only a few examples:
– slow, unstable or even not working internet connection.
– not working, not efficient computer.
– small, old, low resolution monitor.
– broken accessories like mouse, keyboard.

Nowadays, IT equipment compared to people’s earnings is very, very cheap. So one of the best thing company can do is investing in high performance hardware. Programmers need this, especially today when we need a lot of applications which have high requirements (not to mention all containerization and virtualization). I think high performance computer with at least 16GB RAM, 256-500GB SSD, i7 CPU and two FULL HD monitors should be a standard hardware for programmer. Trust me, this investment will return very quickly.

2. Provide him with the right tools

This is another misunderstanding for me. There are a lot of amazing applications which can boost people productivity, so why people do not use them? From ignorance, lack of money, maybe both?

I will give you an example of the best .NET extension tool – Resharper. This is plugin for Visual Studio which makes programming easier and more automatic in some cases. In my opinion it can save from 10% to 20% percent of programming time. At the time when I am writing this post it costs 299 euro (lifetime subscription without upgrades). You know the rate for the programmer’s hour so calculation of ROI I will left up to you. 😉

3. Provide him with the right environment

Does your work environment look like this?
The Ford assembly line in 1913. Source.

Work can be divided into two types: algorithmic and heuristic. An algorithmic work is one that is governed by procedures, schemes, standards and repeatability. You know what you need to do, how and with what. For example: work on the assembly line or the work of a cashier in a supermarket is just algorithmic. At the other end of the spectrum there is a heuristic work, one during which there is no clearly defined path, only you have to experiment, demonstrate your imagination and creativity to come up with a solution to a problem.

It’s not hard to guess what type of work is programming. So why companies try to put their employees in environments in which they are ineffective, for example big open spaces? I think something have gone wrong recently. They want to save on space but this is only illusory saving. People who mainly use their mind during their work need:
– quiet.
– peace.
– enough space at the desk.
– daylight.
– own place.
– coffee machine. 😉

If they don’t have the conditions I mentioned above, they will have a problem with concentration, creativity and effectiveness. Summarizing they will be less productive so work will take them a lot more time.

4. Create a development process

Every programmer should know:
– what are his tasks and what are the priorities.
– for what he is responsible, accountable, what should he consult and what to inform. See RACI matrix.
– what he can do and what can not.
– the Big Picture of current project.

Without well-defined process developers will not know these things (certainly not everything). None of the them should wonder what to do now, who to assign the task to or who to consult. It should be defined clear and understandable workflow.

On the other hand, development process should be automated as much as possible. From the management perspective, you should use project management software and try to eliminate ineffective communication channels like emails. For technical reasons, you should use continuous integration/deployment/delivery practices because machines are much better in repetitive tasks (and they cost less as I mentioned earlier), while people should focus on creative and difficult to automate tasks.

5. Organize only useful meetings

A lot of time can be lost at meetings. Meetings are usually too long, they have too many participants and there are often off-topics that are not related to our work. I am not suggesting abandoning meetings now but organizing them in a thoughtful way, i.e. meeting should:
– have a reason.
– be as short as possible.
– have earlier prepared agenda.
– only have interested participants who will give value to this meeting.
– start and end on time.

So remember, one useless half-hour meeting per day with 4 participants and on Thursday whole man-day is lost. Think about it the next time you organize a meeting.

6. Let the programmer to rest

The one of the famous personal development coach, Stephen Covey, described in his book The 7 Habits of Highly Effective People the formula for effectiveness:

Effectiveness lies in the balance – what I call the P/PC Balance. P stands for production of desired results, the golden eggs. PC stands for production capability, the ability or asset that produces the golden eggs.

From this quote follows: you can produce more either by work more or work better/smarter but you have to keep balance. If person is tired because of constant overtimes his production capability (PC) will be very low as well as his efficiency. So it is better in this situation let him to rest to charge the batteries.

7. Give him opportunity to grow

We can constantly improve our production capability through learning and training. But when programmer can do it? Does it always have to be time out of work?

I think it should not be like this. This industry is growing so fast that if programmer want to be up to date he should train himself AND his efforts should be supported by his company. Supporting can be in different ways: sending to courses, buying books, organizing workshops, giving time to research and develop new things. It should be something, I think it is fair for both parties.

Summary

In conclusion, if you, company, don’t want to waste your programmer’s time at all, you must provide him with good hardware and tools, create appropriate supporting environment and development process and you must take care of his development and rest. Are there a lot of requirements? Of course! 🙂 But you do not have to do it right away, you can do it gradually. This is an investment but changes will be noticed quickly.

Start today, not tomorrow.

10 common broken rules of clean code

Introduction

From time to time I am asked to do code review of an application. I like doing this, because it is always learning experience. I can see how others program, what problems they have and how they solve them. Except that I can see code written in various conditions and by people with different skills and experience. I prepared a list of 10 popular “clean code” broken rules which I often encounter during this code reviews and I would like to share this list with you.

Note: If you don’t know what clean code means, I recommend to read Robert C. Martin Clean Code book.

1. Bad and inconsistent naming

Naming is very often neglected (especially by less experienced developers) but it shouldn’t. Good naming sometimes is difficult but it is needed for a good level of code redability and maintainability.

2. Too much comments

Comments from line 1, 4, 7, 18 are not needed, becasue code speaks for itself. Comment block from lines 14-16 are not needed too, this code should be removed – our source control will remember everything:).

Comments are needed only in two cases:
– commenting public API.
– when programmer uses some kind of hack and there is no other way to explain this.

In other cases comments are unnecessary and should be avoided. You should use refactor technique called Extract Method instead.

3. Duplication

The same code occurring more than once. If it is not intended, it should be refactored by moving that code to separate class and/or method. See Don’t Repeat Yourself (DRY) principle.

4. Sloppy formatting

Similar to bad naming. It doesn’t affect execution of our code but it affects readability. Some people will say that is minor issue but I will say it is not. What do you think about book which is not formatted good? What do you think about author and editor of this book? In programming we should have the same rules because we are professionals, aren’t we?

5. Too big classes/methods

Classes and methods should be as small as possible. They should have “only one reason to change” as Single Responsibility Principle says. Common code smells are:
– too much lines of codes
– too many dependencies (constructors/methods parameters)
– code in class concerning different entities, use cases.

6. Bad exceptions handling

We should always catch the most specific exception ( DivideByZeroException for example above). Moreover, we should be careful when we try rethrow given exception, because we can lost whole stack trace information. Below corrected code:

7. No encapsulation

This is most frequently broken rule of clean code and object oriented design paradigm. I see this almost in every codebase. Every class is public with public properties and public methods. This is not object oriented programming. Our duty is to create fully encapsulated objects and hide theirs internals. This is topic for another post why to do this but for know I beg you – stop revealing your classes internals only because your IDE add public keyword by default.

8. Too many conditional statements

Sometimes, when our bussiness logic is complicated, I see a lot nested if/else statements. This is difficult to read and understand.

Often we can get rid part of this statements applying some design pattern and principle (see The Open/Closed Principle and Strategy Pattern for example). If we can’t, we should decompose these conditionals.

9. Unused code

Unused code is created in 2 cases. First case is when code was changed (due to refactoring or requirements change) and some part of that code is not valid. Second case is when programmer wanted to add some additional functionality which is not required at the time of writing but he thinks it may be needed later.

In both cases unused code should not exist. It decreases redadabilty of code, design, adds unnecessary complexity to them – it should be avoided. There is even programming principle for second case – YAGNI.

10. Magic strings and numbers

Programming is enough difficult even without magic. Code with meaningless strings and numbers like Option == 1, order.Type = 'A' is difficult to read, maintain and refactor. Instead of this we should use enums and consts and enjoy good readability and compile-time checks.

Summary

I tried to list all most common issues which I often encounter. I think it is important to remember at least two things when we write the code. Firstly, code is more often read than written. Secondly, working code which even fulfills business requirements is not sufficient if it is written badly. Remembering this and knowing rules of writing good quality code will lead us to software craftsmanship faster.

How to publish and handle Domain Events

Introduction

Domain Event is one of the building blocks of Domain Driven Design. It is something that happened in particular domain and it captures memory of it. We create Domain Events to notify other parts of the same domain that something interesting happened and these other parts potentially can react to.

Domain Event is usually immutable data-container class named in the past tense. For example:

Three ways of publishing domain events

I have seen mainly three ways of publishing domain events.

1. Using static DomainEvents class

This approach was presented by Udi Dahan in his Domain Events Salvation post. In short, there is a static class named DomainEvents with method Raise and it is invoked immediately when something interesting during aggregate method processing occurred. Word immediately is worth emphasizing because all domain event handlers start processing immediately too (even aggregate method did not finish processing).

2. Raise event returned from aggregate method

This is approach when aggregate method returns Domain Event directly to ApplicationService. ApplicationService decides when and how to raise event. You can become familiar with this way of raising events reading Jan Kronquist Don’t publish Domain Events, return them! post.

3. Add event to Events Entity collection.

In this way on every entity, which creates domain events, exists Events collection. Every Domain Event instance is added to this collection during aggregate method execution. After execution, ApplicationService (or other component) reads all Eventscollections from all entities and publishes them. This approach is well described in Jimmy Bogard post A better domain events pattern.

Handling domain events

The way of handling of domain events depends indirectly on publishing method. If you use DomainEvents static class, you have to handle event immediately. In other two cases you control when events are published as well handlers execution – in or outside existing transaction.

In my opinion it is good approach to always handle domain events in existing transaction and treat aggregate method execution and handlers processing as atomic operation. This is good because if you have a lot of events and handlers you do not have to think about initializing connections, transactions and what should be treat in “all-or-nothing” way and what not.

Sometimes, however, it is necessary to communicate with 3rd party service (for example e-mail or web service) based on Domain Event. As we know, communication with 3rd party services is not usually transactional so we need some additional generic mechanism to handle these types of scenarios. So I created Domain Events Notifications.

Domain Events Notifications

There is no such thing as domain events notifications in DDD terms. I gave that name because I think it fits best – it is notification that domain event was published.

Mechanism is pretty simple. If I want to inform my application that domain event was published I create notification class for it and as many handlers for this notification as I want. I always publish my notifications after transaction is committed. The complete process looks like this:

1. Create database transaction.
2. Get aggregate(s).
3. Invoke aggregate method.
4. Add domain events to Events collections.
5. Publish domain events and handle them.
6. Save changes to DB and commit transaction.
7. Publish domain events notifications and handle them.

How do I know that particular domain event was published?

First of all, I have to define notification for domain event using generics:

All notifications are registered in IoC container:

In EventsPublisher we resolve defined notifications using IoC container and after our unit of work is completed, all notifications are published:

This is how whole process looks like presented on UML sequence diagram:

You can think that there is a lot of things to remember and you are right!:) But as you can see whole process is pretty straightforward and we can simplify this solution using IoC interceptors which I will try to describe in another post.

Summary

1. Domain event is information about something which happened in the past in modeled domain and it is important part of DDD approach.
2. There are many ways of publishing and handling domain events – by static class, returning them, exposing by collections.
2. Domain events should be handled within existing transaction (my recommendation).
3. For non-trasactional operations Domain Events Notifications were introduced.

Processing commands with Hangfire and MediatR

In previous post about processing multiple instance aggregates of the same type I suggested to consider using eventual consistency approach. In this post I would like to present one way to do this.

Setup

In the beginning let me introduce stack of technologies/patterns:
1. Command pattern – I am using commands but they do not look like theses described in GoF book. They just simple classes with data and they implement IRequest  marker interface of MediatR.

2. Mediator pattern. I am using this pattern because i want to decouple my clients classes (commands invokers) from commands handlers. Simple but great library created by Jimmy Bogard named MediatR implements this pattern very well. Here is simple usage.

and handler:

3. Hangfire. Great open-source library for processing and scheduling background jobs even with GUI monitoring interface. This is where my commands are scheduled, executed and retried if error occured.

Problem

For some of my uses cases, I would like to schedule processing my commands, execute them parallel with retry option and monitor them. Hangfire gives me all these kind of features but I have to have public method which I have to pass to Hangifre method (for example BackgroundJob.Enqueue). This is a problem – with mediator pattern I cannot (and I do not want) pass public method of handler because I have decoupled it from invoker. So I need special way to integrate MediatR with Hangfire without affecting basic assumptions.

Solution

My solution is to have three additional classes:
1. CommandsScheduler – serializes commands and sends them to Hangfire.

2. CommandsExecutor – responods to Hangfire jobs execution, deserializes commands and sends them to handlers using MediatR.

3. MediatorSerializedObject – wrapper class for serialized/deserialized commands with additional properties – command type and additional description.

Finally with this implementation we can change our client clasess to use CommandsScheduler:

and our commands are scheduled, invoked and monitored by Hangfire. I sketched sequence diagram which shows this interaction:

Processing commands with MediatR and Hanfire

Additionally, we can introduce interface for CommandsSchedulerICommandsScheduler. Second implementation will not use Hangfire at all and only will execute MediatR requests directly – for example in development process when we do not want start Hangfire Server.

Summary

I presented the way of processing commands asynchronously using MediatR and Hangfire. With this approach we have:
1. Decoupled invokers and handlers of commands.
2. Scheduling commands mechanism.
3. Invoker and handler of command may be other processes.
4. Commands execution monitoring.
5. Commands execution retries mechanism.

These benefits are very important during development using eventual consistency approach. We have more control over commands processing and we can react quickly if problem will appear.

Processing multiple aggregates – transactional vs eventual consistency

When we use Domain Driven Design approach in our application, sometimes we have to invoke some method on multiple instances of aggregate of the same type.

For example, in our domain we have customers and when big Black Friday campaign starts we have to recalculate theirs discounts. So in domain model exists Customer aggregate with RecalculateDiscount method and in Application Layer we have DiscountAppService which is responsible for this use case.

There are 2 ways to implement this and similar scenarios:

1. Using transactional consistency

This is the simplest solution, we get all customers aggregates and on every instance the RecalculateDiscount method is invoked. We surrounded our processing with TransactionScope so after that we can be certain that every customer have recalculated discount or none of them. This is transactional consistency – it provides us ACID and sometimes is enough solution, but in many cases (especially while processing multiple aggregates in DDD terms) this solution is very bad approach.

First of all, customers are loaded to memory and we can have performance issue. Of course we can change implementation a little, get only customers identifiers and in foreach loop load customers one by one. But we have worse problem – our transaction holds locks on our aggregates until end of processing and other processes have to wait. For the record – default transaction scope isolation level is Serializable. We can change isolation level but we can’t get rid of locks. In this case application becomes less responsive, we can have timeouts and deadlocks – things we should avoid how we can.

2. Using eventual consistency

In this approach we do not use big transaction. Instead of this, we process every customer aggregate separately. Eventual consistency means that in specified time our system wile be in inconsistent state, but after given time will be consistent. In our example there is a time, that some of customers have discounts recalculated and some of them not. Let’s see the code:

In this case on the beginning we got only customers identifiers and we process customer aggregates one by one asynchronously (and parallel if applicable). We removed problem of locking our aggregates for a long time. The simplest solution is usage of Task.Run() , but using this approach we totally losing control of processing. Better solution is to use some 3rd party library like Hangfire, Quartz.NET or messaging system.

Eventual consistency is a big topic used in distributed computing, encountered together with CQRS. In this article I would like to show only another way of executing batch processing using this approach and its benefits. Sometimes this approach is not a good choice – it can have impact on GUI and users may see stale data for some time. That is why it is important to talk with domain experts because often it is fine for user to wait for update of data but sometimes it is unacceptable.

Summary

Transactional consistency – whole processing is executed in one transaction. It is “all or nothing” approach and sometimes can lead to decrease performance, scalability and availability of our application.

Eventual consistency – processing is divided and not executed in one big transaction. In some time application will be in inconsistent state. It leads to better scalability and availability of application. On the other hand can cause problems with GUI (stale data) and it requires supporting mechanisms which enable parallel processing, retries and sometimes process monitors as well.