I’m working pretty hard this week and early next to deliver the CritterWatch MVP (our new management and observability console for the Critter Stack) to a JasperFx Software client. One of the things we need to do for testing is to fake out several failure conditions in message handlers to be able to test CritterWatch’s “Dead Letter Queue” management and alerting features. To that end, we have some fake systems that constantly process messages, and we’ve rigged up what I’m going to call the world’s crudest Chaos Monkey in Wolverine middleware:

    public static async Task Before(ChaosMonkeySettings chaos)
    {
        // Configurable slow handler for testing back pressure
        if (chaos.SlowHandlerMs > 0)
        {
            await Task.Delay(chaos.SlowHandlerMs);
        }

        if (chaos.FailureRate <= 0) return;

        // Chaos monkey — distribute failure rate equally across 5 exception types
        var perType = chaos.FailureRate / 5.0;
        var next = Random.Shared.NextDouble();

        if (next < perType)
        {
            throw new TripServiceTooBusyException("Just feeling tired at " + DateTime.Now);
        }

        if (next < perType * 2)
        {
            throw new TrackingUnavailableException("Tracking is down at " + DateTime.Now);
        }

        if (next < perType * 3)
        {
            throw new DatabaseIsTiredException("The database wants a break at " + DateTime.Now);
        }

        if (next < perType * 4)
        {
            throw new TransientException("Slow down, you move too fast.");
        }

        if (next < perType * 5)
        {
            throw new OtherTransientException("Slow down, you move too fast.");
        }
    }

And this to control it remotely in tests or just when doing exploratory manual testing:

    private static void MapChaosMonkeyEndpoints(WebApplication app)
    {
        var group = app.MapGroup("/api/chaos")
            .WithTags("Chaos Monkey");

        group.MapGet("/", (ChaosMonkeySettings settings) => Results.Ok(settings))
            .WithSummary("Get current chaos monkey settings");

        group.MapPost("/enable", (ChaosMonkeySettings settings) =>
        {
            settings.FailureRate = 0.20;
            return Results.Ok(new { message = "Chaos monkey enabled at 20% failure rate", settings });
        }).WithSummary("Enable chaos monkey with default 20% failure rate");

        group.MapPost("/disable", (ChaosMonkeySettings settings) =>
        {
            settings.FailureRate = 0;
            return Results.Ok(new { message = "Chaos monkey disabled", settings });
        }).WithSummary("Disable chaos monkey (0% failure rate)");

        group.MapPost("/failure-rate/{rate:double}", (double rate, ChaosMonkeySettings settings) =>
        {
            rate = Math.Clamp(rate, 0, 1);
            settings.FailureRate = rate;
            return Results.Ok(new { message = $"Failure rate set to {rate:P0}", settings });
        }).WithSummary("Set chaos monkey failure rate (0.0 to 1.0)");

        group.MapPost("/slow-handler/{ms:int}", (int ms, ChaosMonkeySettings settings) =>
        {
            ms = Math.Max(0, ms);
            settings.SlowHandlerMs = ms;
            return Results.Ok(new { message = $"Handler delay set to {ms}ms", settings });
        }).WithSummary("Set artificial handler delay in milliseconds (for back pressure testing)");

        group.MapPost("/projection-failure-rate/{rate:double}", (double rate, ChaosMonkeySettings settings) =>
        {
            rate = Math.Clamp(rate, 0, 1);
            settings.ProjectionFailureRate = rate;
            return Results.Ok(new { message = $"Projection failure rate set to {rate:P0}", settings });
        }).WithSummary("Set projection failure rate (0.0 to 1.0)");
    }

In this case, the Before middleware is just baked into the message handlers, but in your development the “chaos monkey” middleware could be applied only in testing with a Wolverine extension.

And I was probably listening to Simon & Garfunkel when I did the first cut at the chaos monkey:

Last week I helped a JasperFx Software client with a use case where they get a steady stream of related events from an upstream system into a downstream system where order of processing is important, but the messages might arrive out of order.

Once again referring to the venerable Enterprise Integration Patterns book, that scenario requires a Resequencer:

How can we get a stream of related but out-of-sequence messages back into the correct order?

EIP ReSequencer

To solve the message ordering challenge, we introduced the new Resequencer Saga feature into Wolverine, and combined that with the existing “Partitioned Sequential Messaging” feature.

For the new built in re-sequencing, we do need you to implement this interface on any message types in that related stream so that Wolverine “knows” what order the message is inside of a related stream:

public interface SequencedMessage
{
    int? Order { get; }
}

The next step is to use a special kind of new Wolverine Saga called ResequencerSaga<T>, where the T is just some sort of common interface for all the message types that are part of this ordered stream and also implements the SequencedMessage shown above. Here’s a simple example I used for the testing:

public record StartMyWorkflow(Guid Id);
public record MySequencedCommand(Guid SagaId, int? Order) : SequencedMessage;
public class MyWorkflowSaga : ResequencerSaga<MySequencedCommand>
{
    public Guid Id { get; set; }
    public static MyWorkflowSaga Start(StartMyWorkflow cmd)
    {
        return new MyWorkflowSaga { Id = cmd.Id };
    }
    public void Handle(MySequencedCommand cmd)
    {
        // This will only be called when messages arrive in the correct order,
        // or when out-of-order messages are replayed after gaps are filled
    }
}

At runtime, when Wolverine gets a message that is handled by that MyWorkflowSaga, there is some middleware that first compares the declared order of that message against the recorded state of the saga so far. In more concrete terms, if…

• It’s the first message in the sequence, Wolverine just processes it as normal and records in the saga state what the last processed message order was so that it “knows” what message sequence should be next
• It’s a later message in the sequence compared to the last message sequence processed, the saga state will just store the current message, persist the saga state, and otherwise skip the normal message processing
• The message is the next in the sequence according to what the saga state says should be processed next, it processes normally. If there are any previously out of order messages that the saga state already knows about that are sequentially next after the current message, Wolverine will re-publish those messages locally — but with the normal Wolverine message sequencing these cascading messages will not go anywhere until the initiating message completes

With this mechanism, Wolverine is able to put the messages arriving from the outside world back into the correct sequential order in its own processing.

Of course though, this processing is very stateful and somewhat likely to be vulnerable to concurrent access problems. Most of the saga storage mechanisms in Wolverine happily support optimistic concurrency around saving saga state, so you could just use some selective retries on concurrency violations. Or better yet, Wolverine users can just about completely side step issues with concurrency by utilizing our newest improvement to partitioned messaging we’re calling “Global Partitioning.”

Let’s say that you have a great deal of operations in your system that have to modify a resource of some sort like an entity, a file, a saga in this case, or an event stream that might be a little bit sensitive to concurrent access. Let’s also say that you have a mix of messages that impact these sensitive resources that come from both external, upstream systems and from cascaded messages within your own system.

The syntax for this next feature was added just today in Wolverine 5.21 as I realized the previous syntax was basically unusable in the course of trying to write this blog post. So it goes.

A “global partitioning” allows you to create a guarantee that messages impacting those resources can be processed sequentially within a message group while allowing for parallel processing between message groups throughout the entire cluster.

Imagine it like this (but know I drew this diagram for someone using Kafka even though the next example is using Rabbit MQ queues):

And with this configuration:

using var host = await Host.CreateDefaultBuilder()
    .UseWolverine(opts =>
    {
        // You'd *also* supply credentials here of course!
        opts.UseRabbitMq();
        // Do something to add Saga storage too!
        opts
            .MessagePartitioning
            // This tells Wolverine to "just" use implied
            // message grouping based on Saga identity among other things
            .UseInferredMessageGrouping()
            .GlobalPartitioned(topology =>
            {
                // Creates 5 sharded RabbitMQ queues named "sequenced1" through "sequenced5"
                // with matching companion local queues for sequential processing
                topology.UseShardedRabbitQueues("sequenced", 5);
                topology.MessagesImplementing<MySequencedCommand>();
            });
    }).StartAsync();

What this does is spread the work out for handling MySequencedCommand messages through five different Rabbit MQ + Local queue pairs, with each pair active on only one single node within your application. Even inside each local queue in this partitioning scheme, Wolverine is parallelizing between message groups.

Now, let’s talk about receiving any message that can be cast to MySequencedCommand. If the message is received at a completely different listener than the “sequenced1/2/3/4/5” queues defined above, like from an external system that knows absolutely nothing about your message partitioning, Wolverine is going to immediately determine the message group identity by inferring that from the saga message handler rules (that’s what the UseInferredMessageGrouping() option does for us), then forwards that message to the proper node that is currently handling that group id. If the current node happens to be assigned that message group id, Wolverine forwards the message directly to the right local queue.

Likewise, if you publish a cascading message inside one of your handlers, Wolverine will determine the message group id for that message type, then try to either route that message locally if that group happens to be assigned to the current node (and it probably would be if you were cascading from your own handlers) or sends it remotely to the right messaging endpoint (Rabbit MQ queue or a Kafka topic or an AWS SQS queue maybe).

The point being, this guarantees that related messages are processed sequentially across the entire application cluster while allowing parallel processing between unrelated messages.

Summary

These are hopefully two powerful new features that will benefit Wolverine users in the near future. Both of these features were built at the behest of JasperFx Software clients to directly support their current work. I’m very happy to just quietly fold in reasonably sized new features for JasperFx support clients without extra cost when those features likely benefit the community as a whole. Contact us at sales@jasperfx.net to find out what we can do to help your software development efforts be more successful.

And just for bragging rights tonight, I did some poking around (okay, I asked Claude to do it for me) to see if any other asynchronous messaging tools offer anything similar to what our global partitioning option does for Wolverine users. While you can certainly achieve the same goals through actor frameworks like AkkaDotNet or Orleans (I consider actor frameworks to be such a different paradigm that I don’t really think of them as direct competitors to Wolverine), it doesn’t appear that there are any equivalents out there to this feature in the .NET space. MassTransit and NServiceBus both have more limited versions of this capability, but nothing that is as easy or flexible as what Wolverine has at this point. Now, granted, we’re at this point because Marten event stream appends can be sensitive to concurrent access so we’ve had to take concurrency maybe a little more seriously than the pure play asynchronous messaging tools that don’t really have an event sourcing component.

Wolverine — the event-driven messaging and HTTP framework for .NET — provides a rich, layered set of options for validating incoming data. Whether you are building HTTP endpoints or message handlers, Wolverine meets you where you are: from zero-configuration inline checks to full Fluent Validation or Data Annotation middleware support for both command handlers and HTTP endpoints.

Let’s maybe over simplify validation scenarios say they’ll fall into two buckets:

1. Run of the mill field level validation rules like required fields or value ranges. These rules are the bread and butter of dedicated validation frameworks like Fluent Validation or Microsoft’s Data Annotations markup.
2. Custom validation rules that are custom to your business domain and might involve checks against the existing state of your system beyond the command messages.

Let’s first look at Wolverine’s Data Annotation integration that is completely baked into the core WolverineFx Nuget. To get started, just opt into the Data Annotations middleware for message handlers like this:

using var host = await Host.CreateDefaultBuilder()
    .UseWolverine(opts =>
    {
        // Apply the validation middleware
        opts.UseDataAnnotationsValidation();
    }).StartAsync();

In message handlers, this middleware will kick in for any message type that has any validation attributes as this example:

public record CreateCustomer(
    // you can use the attributes on a record, but you need to
    // add the `property` modifier to the attribute
    [property: Required] string FirstName,
    [property: MinLength(5)] string LastName,
    [property: PostalCodeValidator] string PostalCode
) : IValidatableObject
{
    public IEnumerable<ValidationResult> Validate(ValidationContext validationContext)
    {
        // you can implement `IValidatableObject` for custom
        // validation logic
        yield break;
    }
};
public class PostalCodeValidatorAttribute : ValidationAttribute
{
    public override bool IsValid(object? value)
    {
        // custom attributes are supported
        return true;
    }
}
public static class CreateCustomerHandler
{
    public static void Handle(CreateCustomer customer)
    {
        // do whatever you'd do here, but this won't be called
        // at all if the DataAnnotations Validation rules fail
    }
}

By default for message handlers, any validation errors are logged, then the current execution is stopped through the usage of the HandlerContinuation value we’ll discuss later.

For Wolverine.HTTP integration with Data Annotations, use:

app.MapWolverineEndpoints(opts =>
{
    // Use Data Annotations that are built
    // into the Wolverine.HTTP library
    opts.UseDataAnnotationsValidationProblemDetailMiddleware();
});

Likewise, this middleware will only apply to HTTP endpoints that have a request input model that contains data annotation attributes. In this case though, Wolverine is using the ProblemDetails specification to report validation errors back to the caller with a status code of 400 by default.

Fluent Validation Middleware

Similarly, the Fluent Validation integration works more or less the same, but requires the WolverineFx.FluentValidation package for message handlers and the WolverineFx.Http.FluentValidation package for HTTP endpoints. There are some Wolverine helpers for discovering and registering FluentValidation validators in a way that applies some Wolverine-specific performance optimizations by trying to register most validators with a Singleton lifetime just to allow Wolverine to generate more optimized code.

It is possible to override how Wolverine handles validation failures, but I’d personally recommend just using the ProblemDetails default in most cases.

I would like to note that the way that Wolverine generates code for the Fluent Validation middleware is generally going to be more efficient at runtime than the typical IoC dependent equivalents you’ll frequently find in the MediatR space.

Explicit Validation

Let’s move on to validation rules that are more specific to your own problem domain, and especially the type of validation rules that would require you to examine the state of your system by exercising some kind of data access. These kinds of rules certainly can be done with custom Fluent Validation validators, but I strongly recommend you put that kind of validation directly into your message handlers or HTTP endpoints to colocate business logic together with the actual message handler or HTTP endpoint happy path.

One of the unique features of Wolverine in comparison to the typical “IHandler of T” application frameworks in .NET is Wolverine’s built in support for a type of low code ceremony Railway Programming, and this turns out to be perfect for one off validation rules.

In message handlers we’ve long had support for returning the HandlerContinuation enum from Validate() or Before() methods as a way to signal to Wolverine to conditionally stop all additional processing:

public static class ShipOrderHandler
{
    // This would be called first
    public static async Task<(HandlerContinuation, Order?, Customer?)> LoadAsync(ShipOrder command, IDocumentSession session)
    {
        var order = await session.LoadAsync<Order>(command.OrderId);
        if (order == null)
        {
            return (HandlerContinuation.Stop, null, null);
        }
        var customer = await session.LoadAsync<Customer>(command.CustomerId);
        return (HandlerContinuation.Continue, order, customer);
    }
    // The main method becomes the "happy path", which also helps simplify it
    public static IEnumerable<object> Handle(ShipOrder command, Order order, Customer customer)
    {
        // use the command data, plus the related Order & Customer data to
        // "decide" what action to take next
        yield return new MailOvernight(order.Id);
    }
}

But of course, with the example above, you could also write that with Wolverine’s declarative persistence like this:

public static class ShipOrderHandler
{
    // The main method becomes the "happy path", which also helps simplify it
    public static IEnumerable<object> Handle(
      ShipOrder command, 
      // This is loaded by the OrderId on the ShipOrder command
      [Entity(Required = true)]
      Order order, 
      // This is loaded by the CustomerId value on the ShipOrder command
      [Entity(Required = true)]
      Customer customer)
    {
        // use the command data, plus the related Order & Customer data to
        // "decide" what action to take next
        yield return new MailOvernight(order.Id);
    }
}

In the code above, Wolverine would stop the processing if either the Order or Customer entity referenced by the command message is missing. Similarly, if this code were in an HTTP endpoint instead, Wolverine would emit a ProblemDetails with a 400 status code and a message stating the data that is missing.

If you were using the code above with the integration with Marten or Polecat, Wolverine can even emit code that uses Marten or Polecat’s batch querying functionality to make your system more efficient by eliminating database round trips.

Likewise in the HTTP space, you could also return a ProblemDetails object directly from a Validate() method like:

public class ProblemDetailsUsageEndpoint
{
    public ProblemDetails Validate(NumberMessage message)
    {
        if (message.Number > 5)
            return new ProblemDetails
            {
                Detail = "Number is bigger than 5",
                Status = 400
            };
        // All good — continue!
        return WolverineContinue.NoProblems;
    }
    [WolverinePost("/problems")]
    public static string Post(NumberMessage message) => "Ok";
}

Even More Lightweight Validation!

When reviewing client code that uses the HandlerContinuation or ProblemDetails syntax, I definitely noticed the code can become verbose and noisy, especially compared to just embedding throw new InvalidOperationException("something is not right here"); code directly in the main methods — which isn’t something I’d like to see people tempted to do.

Instead, Wolverine 5.18 added a more lightweight approach that allows you to just return an array of strings from a Before/Validation() method:

    public static IEnumerable<string> Validate(SimpleValidateEnumerableMessage message)
    {
        if (message.Number > 10)
        {
            yield return "Number must be 10 or less";
        }
    }

    // or

    public static string[] Validate(SimpleValidateStringArrayMessage message)
    {
        if (message.Number > 10)
        {
            return ["Number must be 10 or less"];
        }

        return [];
    }

At runtime, Wolverine will stop a handler if there are any messages or emit a ProblemDetails response in HTTP endpoints.

Summary

Hopefully, Wolverine has you covered no matter what with options. A few practical takeaways:

• Reach for Validate() / ValidateAsync() first whenever IoC services or database queries are involved or the validation logic is just specific to your message handler or HTTP endpoint.
• Use Data Annotations middleware when your model types are already decorated with attributes and you want zero validator classes.
• Use Fluent Validation middleware when you want reusable, composable validators shared across multiple handlers or endpoints.

All three strategies generate efficient, ahead-of-time compiled middleware via Wolverine’s code generation engine, keeping the runtime overhead minimal regardless of which path you choose.