Re-Sequencer and Global Message Partitioning in Wolverine

Last week I helped a JasperFx Software client with a use case where they get a steady stream of related events from an upstream system into a downstream system where order of processing is important, but the messages might arrive out of order.

Once again referring to the venerable Enterprise Integration Patterns book, that scenario requires a Resequencer:

How can we get a stream of related but out-of-sequence messages back into the correct order?

EIP ReSequencer

To solve the message ordering challenge, we introduced the new Resequencer Saga feature into Wolverine, and combined that with the existing “Partitioned Sequential Messaging” feature.

For the new built in re-sequencing, we do need you to implement this interface on any message types in that related stream so that Wolverine “knows” what order the message is inside of a related stream:

			
public interface SequencedMessage
{
    int? Order { get; }
}

The next step is to use a special kind of new Wolverine Saga called ResequencerSaga<T>, where the T is just some sort of common interface for all the message types that are part of this ordered stream and also implements the SequencedMessage shown above. Here’s a simple example I used for the testing:

			
public record StartMyWorkflow(Guid Id);
public record MySequencedCommand(Guid SagaId, int? Order) : SequencedMessage;
public class MyWorkflowSaga : ResequencerSaga<MySequencedCommand>
{
    public Guid Id { get; set; }
    public static MyWorkflowSaga Start(StartMyWorkflow cmd)
    {
        return new MyWorkflowSaga { Id = cmd.Id };
    }
    public void Handle(MySequencedCommand cmd)
    {
        // This will only be called when messages arrive in the correct order,
        // or when out-of-order messages are replayed after gaps are filled
    }
}

		

At runtime, when Wolverine gets a message that is handled by that MyWorkflowSaga, there is some middleware that first compares the declared order of that message against the recorded state of the saga so far. In more concrete terms, if…

It’s the first message in the sequence, Wolverine just processes it as normal and records in the saga state what the last processed message order was so that it “knows” what message sequence should be next
It’s a later message in the sequence compared to the last message sequence processed, the saga state will just store the current message, persist the saga state, and otherwise skip the normal message processing
The message is the next in the sequence according to what the saga state says should be processed next, it processes normally. If there are any previously out of order messages that the saga state already knows about that are sequentially next after the current message, Wolverine will re-publish those messages locally — but with the normal Wolverine message sequencing these cascading messages will not go anywhere until the initiating message completes

With this mechanism, Wolverine is able to put the messages arriving from the outside world back into the correct sequential order in its own processing.

Of course though, this processing is very stateful and somewhat likely to be vulnerable to concurrent access problems. Most of the saga storage mechanisms in Wolverine happily support optimistic concurrency around saving saga state, so you could just use some selective retries on concurrency violations. Or better yet, Wolverine users can just about completely side step issues with concurrency by utilizing our newest improvement to partitioned messaging we’re calling “Global Partitioning.”

Let’s say that you have a great deal of operations in your system that have to modify a resource of some sort like an entity, a file, a saga in this case, or an event stream that might be a little bit sensitive to concurrent access. Let’s also say that you have a mix of messages that impact these sensitive resources that come from both external, upstream systems and from cascaded messages within your own system.

The syntax for this next feature was added just today in Wolverine 5.21 as I realized the previous syntax was basically unusable in the course of trying to write this blog post. So it goes.

A “global partitioning” allows you to create a guarantee that messages impacting those resources can be processed sequentially within a message group while allowing for parallel processing between message groups throughout the entire cluster.

Imagine it like this (but know I drew this diagram for someone using Kafka even though the next example is using Rabbit MQ queues):

And with this configuration:

			
using var host = await Host.CreateDefaultBuilder()
    .UseWolverine(opts =>
    {
        // You'd *also* supply credentials here of course!
        opts.UseRabbitMq();
        // Do something to add Saga storage too!
        opts
            .MessagePartitioning
            // This tells Wolverine to "just" use implied
            // message grouping based on Saga identity among other things
            .UseInferredMessageGrouping()
            .GlobalPartitioned(topology =>
            {
                // Creates 5 sharded RabbitMQ queues named "sequenced1" through "sequenced5"
                // with matching companion local queues for sequential processing
                topology.UseShardedRabbitQueues("sequenced", 5);
                topology.MessagesImplementing<MySequencedCommand>();
            });
    }).StartAsync();

		

What this does is spread the work out for handling MySequencedCommand messages through five different Rabbit MQ + Local queue pairs, with each pair active on only one single node within your application. Even inside each local queue in this partitioning scheme, Wolverine is parallelizing between message groups.

Now, let’s talk about receiving any message that can be cast to MySequencedCommand. If the message is received at a completely different listener than the “sequenced1/2/3/4/5” queues defined above, like from an external system that knows absolutely nothing about your message partitioning, Wolverine is going to immediately determine the message group identity by inferring that from the saga message handler rules (that’s what the UseInferredMessageGrouping() option does for us), then forwards that message to the proper node that is currently handling that group id. If the current node happens to be assigned that message group id, Wolverine forwards the message directly to the right local queue.

Likewise, if you publish a cascading message inside one of your handlers, Wolverine will determine the message group id for that message type, then try to either route that message locally if that group happens to be assigned to the current node (and it probably would be if you were cascading from your own handlers) or sends it remotely to the right messaging endpoint (Rabbit MQ queue or a Kafka topic or an AWS SQS queue maybe).

The point being, this guarantees that related messages are processed sequentially across the entire application cluster while allowing parallel processing between unrelated messages.

Summary

These are hopefully two powerful new features that will benefit Wolverine users in the near future. Both of these features were built at the behest of JasperFx Software clients to directly support their current work. I’m very happy to just quietly fold in reasonably sized new features for JasperFx support clients without extra cost when those features likely benefit the community as a whole. Contact us at sales@jasperfx.net to find out what we can do to help your software development efforts be more successful.

And just for bragging rights tonight, I did some poking around (okay, I asked Claude to do it for me) to see if any other asynchronous messaging tools offer anything similar to what our global partitioning option does for Wolverine users. While you can certainly achieve the same goals through actor frameworks like AkkaDotNet or Orleans (I consider actor frameworks to be such a different paradigm that I don’t really think of them as direct competitors to Wolverine), it doesn’t appear that there are any equivalents out there to this feature in the .NET space. MassTransit and NServiceBus both have more limited versions of this capability, but nothing that is as easy or flexible as what Wolverine has at this point. Now, granted, we’re at this point because Marten event stream appends can be sensitive to concurrent access so we’ve had to take concurrency maybe a little more seriously than the pure play asynchronous messaging tools that don’t really have an event sourcing component.

Natural Keys in the Critter Stack

Just to level set everyone, there are two general categories of identifiers we use in software:

“Surrogate” keys are data elements like Guid values, database auto numbering or sequences, or snowflake generated identifiers that have no real business meaning and just try to be unique values.
“Natural” keys have some kind of business meaning and usually utilize some piece of existing information like email addresses or phone numbers. A natural key could also be an external supplied identifier from your clients. In fact, it’s quite common to have your own tracking identifier (usually a surrogate key) while also having to track a client or user’s own identification for the same business entity.

That very last sentence is where this post takes off. You see Marten can happily track event streams with either Guid identifiers (surrogate key) or string identifiers — or strong typed identifiers that wrap an inner Guid or string, but in this case that’s really the same thing, just with more style I guess. Likewise, in combination with Wolverine for our recommended “aggregate handler workflow” approach to building command handlers, we’ve only supported the stream id or key. Until now!

With the Marten 8.23 and Wolverine 5.18 releases last week (we’ve been very busy and there are newer releases now), you are now able to “tag” Marten (or Polecat!) event streams with a natural key in addition to its surrogate stream id and use that natural key in conjunction with Wolverine’s aggregate handler workflow.

Of course, if you use strings as the stream identifier you could already use natural keys, but let’s just focus on the case of Guid identified streams that are also tagged with some kind of natural key that will be supplied by users in the commands sent to the system.

First, to tag streams with natural keys in Marten, you have to have a strong typed identifier type for the natural key. Next, there’s a little bit of attribute decoration in the targeted document type of a single stream projection, i.e., the “write model” for an event stream. Here’s an example from the Marten documentation:

			
public record OrderNumber(string Value);
public record InvoiceNumber(string Value);
public class OrderAggregate
{
    public Guid Id { get; set; }
    [NaturalKey]
    public OrderNumber OrderNum { get; set; }
    public decimal TotalAmount { get; set; }
    public string CustomerName { get; set; }
    public bool IsComplete { get; set; }
    [NaturalKeySource]
    public void Apply(OrderCreated e)
    {
        OrderNum = e.OrderNumber;
        CustomerName = e.CustomerName;
    }
    public void Apply(OrderItemAdded e)
    {
        TotalAmount += e.Price;
    }
    [NaturalKeySource]
    public void Apply(OrderNumberChanged e)
    {
        OrderNum = e.NewOrderNumber;
    }
    public void Apply(OrderCompleted e)
    {
        IsComplete = true;
    }
}

		

In particular, see the usage of [NaturalKey] which should be self-explanatory. Also see the [NaturalKeySource] attribute that we’re using to mark when a natural key value might change. Marten is starting to use source generators for some projection internals (in place of some nasty, not entirely as efficient as it should have been, Expression-compiled-to-Lambda functions).

And that’s that, really. You’re now able to use the designated natural keys as the input to an “aggregate handler workflow” command handler with Wolverine. See Natural Keys from the Wolverine documentation for more information.

For a little more information:

The natural keys are stored in a separate table, and when using FetchForWriting(), Marten is doing an inner join from the tag table for that natural key type to the mt_streams table in the Marten database
You can change the natural key against the surrogate key
We expect this to be most useful when you want to use the Guid surrogate keys for uniqueness in your own system, but you frequently receive a natural key from API users of your system — or at least this has been encountered by a couple different JasperFx Software customers.
The natural key storage does have a unique value constraint on the “natural key” part of the storage
Really only a curiosity, but this was done in the same wave of development as Marten’s new DCB support

Validation Options in Wolverine

Wolverine — the event-driven messaging and HTTP framework for .NET — provides a rich, layered set of options for validating incoming data. Whether you are building HTTP endpoints or message handlers, Wolverine meets you where you are: from zero-configuration inline checks to full Fluent Validation or Data Annotation middleware support for both command handlers and HTTP endpoints.

Let’s maybe over simplify validation scenarios say they’ll fall into two buckets:

Run of the mill field level validation rules like required fields or value ranges. These rules are the bread and butter of dedicated validation frameworks like Fluent Validation or Microsoft’s Data Annotations markup.
Custom validation rules that are custom to your business domain and might involve checks against the existing state of your system beyond the command messages.

Let’s first look at Wolverine’s Data Annotation integration that is completely baked into the core WolverineFx Nuget. To get started, just opt into the Data Annotations middleware for message handlers like this:

			
using var host = await Host.CreateDefaultBuilder()
    .UseWolverine(opts =>
    {
        // Apply the validation middleware
        opts.UseDataAnnotationsValidation();
    }).StartAsync();

		

In message handlers, this middleware will kick in for any message type that has any validation attributes as this example:

			
public record CreateCustomer(
    // you can use the attributes on a record, but you need to
    // add the `property` modifier to the attribute
    [property: Required] string FirstName,
    [property: MinLength(5)] string LastName,
    [property: PostalCodeValidator] string PostalCode
) : IValidatableObject
{
    public IEnumerable<ValidationResult> Validate(ValidationContext validationContext)
    {
        // you can implement `IValidatableObject` for custom
        // validation logic
        yield break;
    }
};
public class PostalCodeValidatorAttribute : ValidationAttribute
{
    public override bool IsValid(object? value)
    {
        // custom attributes are supported
        return true;
    }
}
public static class CreateCustomerHandler
{
    public static void Handle(CreateCustomer customer)
    {
        // do whatever you'd do here, but this won't be called
        // at all if the DataAnnotations Validation rules fail
    }
}

		

By default for message handlers, any validation errors are logged, then the current execution is stopped through the usage of the HandlerContinuation value we’ll discuss later.

For Wolverine.HTTP integration with Data Annotations, use:

			
app.MapWolverineEndpoints(opts =>
{
    // Use Data Annotations that are built
    // into the Wolverine.HTTP library
    opts.UseDataAnnotationsValidationProblemDetailMiddleware();
});

		

Likewise, this middleware will only apply to HTTP endpoints that have a request input model that contains data annotation attributes. In this case though, Wolverine is using the ProblemDetails specification to report validation errors back to the caller with a status code of 400 by default.

Fluent Validation Middleware

Similarly, the Fluent Validation integration works more or less the same, but requires the WolverineFx.FluentValidation package for message handlers and the WolverineFx.Http.FluentValidation package for HTTP endpoints. There are some Wolverine helpers for discovering and registering FluentValidation validators in a way that applies some Wolverine-specific performance optimizations by trying to register most validators with a Singleton lifetime just to allow Wolverine to generate more optimized code.

It is possible to override how Wolverine handles validation failures, but I’d personally recommend just using the ProblemDetails default in most cases.

I would like to note that the way that Wolverine generates code for the Fluent Validation middleware is generally going to be more efficient at runtime than the typical IoC dependent equivalents you’ll frequently find in the MediatR space.

Explicit Validation

Let’s move on to validation rules that are more specific to your own problem domain, and especially the type of validation rules that would require you to examine the state of your system by exercising some kind of data access. These kinds of rules certainly can be done with custom Fluent Validation validators, but I strongly recommend you put that kind of validation directly into your message handlers or HTTP endpoints to colocate business logic together with the actual message handler or HTTP endpoint happy path.

One of the unique features of Wolverine in comparison to the typical “IHandler of T” application frameworks in .NET is Wolverine’s built in support for a type of low code ceremony Railway Programming, and this turns out to be perfect for one off validation rules.

In message handlers we’ve long had support for returning the HandlerContinuation enum from Validate() or Before() methods as a way to signal to Wolverine to conditionally stop all additional processing:

			
public static class ShipOrderHandler
{
    // This would be called first
    public static async Task<(HandlerContinuation, Order?, Customer?)> LoadAsync(ShipOrder command, IDocumentSession session)
    {
        var order = await session.LoadAsync<Order>(command.OrderId);
        if (order == null)
        {
            return (HandlerContinuation.Stop, null, null);
        }
        var customer = await session.LoadAsync<Customer>(command.CustomerId);
        return (HandlerContinuation.Continue, order, customer);
    }
    // The main method becomes the "happy path", which also helps simplify it
    public static IEnumerable<object> Handle(ShipOrder command, Order order, Customer customer)
    {
        // use the command data, plus the related Order & Customer data to
        // "decide" what action to take next
        yield return new MailOvernight(order.Id);
    }
}

		

But of course, with the example above, you could also write that with Wolverine’s declarative persistence like this:

			
public static class ShipOrderHandler
{
    // The main method becomes the "happy path", which also helps simplify it
    public static IEnumerable<object> Handle(
      ShipOrder command, 
      // This is loaded by the OrderId on the ShipOrder command
      [Entity(Required = true)]
      Order order, 
      // This is loaded by the CustomerId value on the ShipOrder command
      [Entity(Required = true)]
      Customer customer)
    {
        // use the command data, plus the related Order & Customer data to
        // "decide" what action to take next
        yield return new MailOvernight(order.Id);
    }
}

		

In the code above, Wolverine would stop the processing if either the Order or Customer entity referenced by the command message is missing. Similarly, if this code were in an HTTP endpoint instead, Wolverine would emit a ProblemDetails with a 400 status code and a message stating the data that is missing.

If you were using the code above with the integration with Marten or Polecat, Wolverine can even emit code that uses Marten or Polecat’s batch querying functionality to make your system more efficient by eliminating database round trips.

Likewise in the HTTP space, you could also return a ProblemDetails object directly from a Validate() method like:

			
public class ProblemDetailsUsageEndpoint
{
    public ProblemDetails Validate(NumberMessage message)
    {
        if (message.Number > 5)
            return new ProblemDetails
            {
                Detail = "Number is bigger than 5",
                Status = 400
            };
        // All good — continue!
        return WolverineContinue.NoProblems;
    }
    [WolverinePost("/problems")]
    public static string Post(NumberMessage message) => "Ok";
}

		

Even More Lightweight Validation!

When reviewing client code that uses the HandlerContinuation or ProblemDetails syntax, I definitely noticed the code can become verbose and noisy, especially compared to just embedding throw new InvalidOperationException("something is not right here"); code directly in the main methods — which isn’t something I’d like to see people tempted to do.

Instead, Wolverine 5.18 added a more lightweight approach that allows you to just return an array of strings from a Before/Validation() method:

    public static IEnumerable<string> Validate(SimpleValidateEnumerableMessage message)
    {
        if (message.Number > 10)
        {
            yield return "Number must be 10 or less";
        }
    }

    // or

    public static string[] Validate(SimpleValidateStringArrayMessage message)
    {
        if (message.Number > 10)
        {
            return ["Number must be 10 or less"];
        }

        return [];
    }

At runtime, Wolverine will stop a handler if there are any messages or emit a ProblemDetails response in HTTP endpoints.

Summary

Hopefully, Wolverine has you covered no matter what with options. A few practical takeaways:

Reach for Validate() / ValidateAsync() first whenever IoC services or database queries are involved or the validation logic is just specific to your message handler or HTTP endpoint.
Use Data Annotations middleware when your model types are already decorated with attributes and you want zero validator classes.
Use Fluent Validation middleware when you want reusable, composable validators shared across multiple handlers or endpoints.

All three strategies generate efficient, ahead-of-time compiled middleware via Wolverine’s code generation engine, keeping the runtime overhead minimal regardless of which path you choose.

New Improvements to Marten’s LINQ Support

I hold these two facts to be simultaneously true:

Language Integrated Query (LINQ) is the singular best feature in .NET that developers would miss out working in other development platforms
Developing and supporting a LINQ provider is a nastily hard and laborious task for an OSS author and probably my least favorite area of Marten to work in

Alright, on that note, let’s talk about a couple potentially important recent improvements to Marten’s LINQ support. First, we’ve received the message loud and clear that Marten was sometimes hard when what you really need is to fetch data from more than one document type at a time. We’d also got some feedback about the difficultly overall in making denormalized views projected from events with a mix of different streams and potentially different reference documents.

For projections in the event sourcing space, we added the Composite Projection capability. For straight up document database work, Marten 8.23 introduced support for the LINQ GroupJoin operator as shown in this test from the Marten codebase:

			
public class JoinCustomer
{
    public Guid Id { get; set; }
    public string Name { get; set; } = "";
    public string City { get; set; } = "";
}
public class JoinOrder
{
    public Guid Id { get; set; }
    public Guid CustomerId { get; set; }
    public string Status { get; set; } = "";
    public decimal Amount { get; set; }
}
[Fact]
public async Task GroupJoin_select_outer_entity_properties()
{
    // It's just setting up some Customer and Order data in the database
    await SetupData();
    var results = await _session.Query<JoinCustomer>()
        .GroupJoin(
            _session.Query<JoinOrder>(),
            c => c.Id,
            o => o.CustomerId,
            (c, orders) => new { c, orders })
        .SelectMany(
            x => x.orders,
            (x, o) => new { x.c.Name, x.c.City })
        .ToListAsync();
    results.Count.ShouldBe(3);
    results.Count(r => r.City == "Seattle").ShouldBe(2); // Alice's 2 orders
    results.Count(r => r.City == "Portland").ShouldBe(1); // Bob's 1 order
}

		

This is of course brand new, which means there are probably “unknown unknown” bugs in there, but just give us a reproduction in a GitHub issue and we’ll address whatever it is.

Select/Where Hoisting

Without getting into too many details, the giant “hey, let’s rewrite our LINQ support almost from scratch!” effort in Marten V7 a couple years ago made some massive strides in our LINQ provider, but unintentionally “broke” our support for chaining Where clauses *after* Select transforms. To be honest, that’s nothing I even realized you could or would do with LINQ, so I was caught off guard when we got a couple bug reports about that later. No worries now, because you can now do that with Marten as this new test shows:

    [Fact]
    public async Task select_before_where_with_different_type()
    {
        var doc1 = new DocWithInner { Id = Guid.NewGuid(), Name = "one", Inner = new InnerDoc { Value = 10, Text = "low" } };
        var doc2 = new DocWithInner { Id = Guid.NewGuid(), Name = "two", Inner = new InnerDoc { Value = 50, Text = "mid" } };
        var doc3 = new DocWithInner { Id = Guid.NewGuid(), Name = "three", Inner = new InnerDoc { Value = 90, Text = "high" } };

        theSession.Store(doc1, doc2, doc3);
        await theSession.SaveChangesAsync();

        // Select().Where() - the problematic ordering from GH-3009
        var results = await theSession.Query<DocWithInner>()
            .Select(x => x.Inner)
            .Where(x => x.Value > 40)
            .ToListAsync();

        results.Count.ShouldBe(2);
        results.ShouldContain(x => x.Value == 50);
        results.ShouldContain(x => x.Value == 90);
    }

Summary

So you might ask, how did we suddenly get to a point where there are literally no open GitHub issues related to LINQ in the Marten codebase? It turns out that the LINQ provider support is an absolutely perfect place to just let Claude go fix it — but know that that is backed up by about a 1,000 regression tests for LINQ to chew through at the same time.

My limited experience suggests that the AI assisted development really works when you have very well defined and executable acceptance requirements and better yet tests for your AI agent to develop to. Duh.

Fun Five Year Retrospective on Marten Adoption

In the midst of some let’s call it “market” research to better understand how Marten stacks up to a newer competitor, I stumbled back over a GitHub discussion I initiated in 2021 called “What would it take for you to adopt Marten?” long before I was able to found JasperFx Software. I seeded this original discussion with my thoughts about the then forthcoming giant Marten 4.0 release that was meant to permanently put Marten on a solid technical foundation for all time and address all known significant technical shortcomings of Marten.

Narrators’s voice: the V4 release was indeed a major step forward and still shapes a great deal of Marten’s internals, but it was not even remotely the end all, be all of technical releases and V5 came out less than 6 months later to address shortcomings of V4 and to add first class multi-tenancy through separate databases. And arguably, V7 just three years later was nearly as big a change to Marten’s internals.

So now that it’s five years later and Marten’s usage numbers are vastly greater than that moment in time in 2021, let me run through the things we thought needed to change to garner more usage, whether or not and how those ideas took fruit, and whether or not I think those ideas made any difference in the end.

Enterprise Level Support

People frequently told me that they could not seriously consider Marten or later Wolverine without there being commercial support for those tools or at least a company behind them. As of now, JasperFx Software (my company) provides support agreements for any tool under the JasperFx GitHub organization. I would say though that the JasperFx support agreement ends up being more like an ongoing consulting engagement rather than the “here’s an email for support, we’ll response within 72 hours” licensing agreement that you’d be getting from other Event Driven Architecture tools and companies in the .NET space.

Read more about that in Little Diary of How JasperFx Helps Our Clients.

And no, we’re not a big company at all, but we’re getting there and at least “we” isn’t just the royal “we” now:)

I’m hoping that JasperFx is able to expand when we are able to start selling the CritterWatch commercial add on soon.

Moar YouTube Videos

I dragged my feet on this one for a long time and probably still don’t do well enough, but we have the JasperFx Software Channel now with some videos and plenty of live streams. I’ve had mostly positive feedback on the live streams, so it’s just up to me to get back in a groove on this one.

SQL Server or CosmosDb Support in Addition to PostgreSQL

The most common complaint or concern about Marten in its first 5-7 years was that it only supported PostgreSQL as a backing data store. The most common advice we got from the outside was that we absolutely had to have SQL Server support in order to be viable inside the .NET ecosystem where shops do tend to be conservative in technology adoption and also tend to favor Microsoft offerings.

While I’ve always seen the obvious wisdom in supporting SQL Server, I never believed that it was practical to replicate Marten’s functionality with SQL Server. Partially because SQL Server lagged far behind PostgreSQL in its JSON capabilities for a long time and partially just out of sheer bandwidth limitations. I think it’s telling that nobody built a truly robust and widely used event store on top of SQL Server in the mean time.

But it’s 2026 and the math feels very different in many ways:

PostgreSQL has grown in stature and at least in my experience, far more .NET shops are happy to take the plunge into PostgreSQL. It absolutely helps that the PostgreSQL ecosystem has absolutely exploded with innovation and that PostgreSQL has first class managed hosting or even serverless support on every cloud provider of any stature.
SQL Server 2025 introduced a new native JSON type that brought SQL Server at least into the same neighborhood as PostgreSQL’s JSONB type. Using that, the JasperFx Software is getting close to releasing a full fledged port of most of Marten (you won’t miss the parts that were left out, I know I won’t!) called “Polecat” that will be backed by SQL Server 2025. We’ll see how much traction that tool gets, but early feedback has been positive.
While we’re not there yet on Event Sourcing, at least Wolverine does have CosmosDb backed transactional inbox and outbox support as well as other integration into Wolverine handlers. I don’t have any immediate plans for Event Sourcing with CosmosDb other than “wouldn’t that be nice?” kind of thoughts. I don’t hear that many requests for this. I get even less feedback about DynamoDb, but I’ve always assumed we’d get around to that some day too.

Better Support for Cross Document Views or Queries

So, yeah. Document database approaches are awesome when your problem domain is well described by self-contained entities, but maybe not so much if you really need to model a lot of relationships between different first class entities in your system. Marten already had the Include() operator in our LINQ support, but folks aren’t super enthusiastic about it all the time. As Marten really became mostly about Event Sourcing over time, some of this issue went away for folks who could focus on using projections to just write documents out exactly as your use cases needed — which can sometimes happily eliminate the need for fancy JOIN queries and AutoMapper type translation in memory. However, I’ve worked with several JasperFx clients and other users in the past couple years who had real struggles with creating denormalized views with Marten projections, so that needed work too.

While that complaint was made in 2021, we now have or are just about to get in the next wave of releases:

The new “composite projection” model that was designed for easier creation of denormalized event projection views that has already met with some early success (and actionable feedback). This feature was also designed with some performance and scalability tuning in mind as well.
The next big release of Marten (8.23) will include support for the GroupJoin LINQ provider. Finally.
And let’s face it, EF Core will always have better LINQ support than Marten over all and a straight up relational table is probably always going to be more appropriate for reporting. To that end, Marten 8.23 will also have an extension library that adds first class event projections that write to EF Core.

Polecat 1.0 will include all of these new Marten features as well.

Improving LINQ Support

LINQ support was somewhat improved for that V4 release I was already selling in 2021, but much more so for V7 in early 2024 that moved us toward using much more PostgreSQL specific optimizations in JSONB searching as we were able to utilize JSONPath searching or back to the PostgreSQL containment operator.

At this point, it has turned out that recent versions of Claude are very effective at enhancing or fixing issues in the LINQ provider and at this point we have zero open issues related to LINQ for the first time since Marten’s founding back in 2015!

There’s one issue open as I write this that has an existing fix that hasn’t been committed to master yet, so if you go check up on me, I’m not technically lying:)

Open Telemetry Support and other Improved Metadata

Open Telemetry support is table stakes for .NET application framework tools and especially for any kind of Event Driven Architecture or Event Sourcing tool like Marten. We’ve had all that since Marten V7, with occasional enhancements or adjustments since in reaction to JasperFx client needs.

More Sample Applications

Yeah, we could still do a lot better on this front. Sigh.

One thing I want to try doing soon is developing some Claude skills for the Critter Stack in general, and a particular focus on creating instructions for best practices converting codebases to the Critter Stack. As part of that, I’ve identified about a dozen open source sample applications out there that would be good targets for this work. It’s a lazy way to create new samples applications while building an AI offering for JasperFx, but I’m all about being lazy sometimes.

We’ll see how this goes.

Scalability Improvements including Sharding

We’ve done a lot here since that 2021 discussion. Some of the Event Sourcing scalability options are explained here. This isn’t an exhaustive list, but since 2021 we have:

Much better load distribution of asynchronous projection and subscription work within clusters
Support for PostgreSQL read replicas
First class support for managing PostgreSQL native partitioning with Marten
A ton of internal improvements including work to utilize the latest, greatest low level support in Npgsql for query batching

And for that matter, I’ve finishing up a proposal this weekend for a new JasperFx client looking to scale a single Marten system to the neighborhood of 200-300 billion events, so there’s still more work ahead.

Event Streaming Support or Transactional Outbox Integration

This was frequently called out as a big missing feature in Marten in 2021, but with the integration into the full Critter Stack with Wolverine, we have first class event streaming support that was introduced in 2024 for every messaging technology that Wolverine supports today, which is just about everything you could possibly think to use!

Management User Interface

Sigh. Still in flight, but now very heavily in flight with a CritterWatch MVP promised to a JasperFx client by the end of this month. Learn more about that here:

Cloud Hosting Models and Recipes

People have brought this up a bit over the years, but we don’t have much other than some best practices with using Marten inside of Docker containers. I think it helps us that PostgreSQL is almost ubiquitous now and that otherwise a Marten application is just a .NET application.

SignalR + the Critter Stack

It’s early so I should be too cocky, but JasperFx Software is having success in integrating SignalR with both Wolverine and Marten in our forthcoming CritterWatch product. In this post I’ll show you how we’re doing that from the server side C# code all the way down to the client side TypeScript.

Last week I did a live stream talking about many of the details and a way too early demonstration of CritterWatch, JasperFx Software‘s long planned management console for the “Critter Stack” tools (Marten, Wolverine, and soon to be Polecat).

A big technical wrinkle in the CritterWatch approach so far is our utilization of the SignalR messaging support built into Wolverine. Just like with external messaging brokers like Rabbit MQ or Azure Service Bus, Wolverine does a lot of work to remove the technical details of SignalR and let’s you focus on just writing your application code.

In some ways, CritterWatch is kind of a man in the middle between the intended CritterWatch user interface (Vue.js) and the Wolverine enabled applications in your system:

Note that Wolverine will be required for CritterWatch, but if you only today use Marten and want to use CritterWatch to manage just the event sourcing, know that you will be able to use a very minimalistic Wolverine setup just for communication to CritterWatch without having to migrate your entire messaging infrastructure to Wolverine. And for that matter, Wolverine now has a pretty robust HTTP transport for asynchronous messaging that would work fine for CritterWatch integration.

As I said earlier, CritterWatch is going to depend very heavily on two way WebSockets communication between the user interface and the CritterWatch server, and we’re utilizing Wolverine’s SignalR messaging transport (which was purposefully built for CritterWatch in the first place) to get that done. In the CritterWatch codebase, we have this little bit of Wolverine configuration:

    public static void AddCritterWatchServices(this WolverineOptions opts, NpgsqlDataSource postgresSource)
    {
        // Much more of course...
        opts.Services.AddWolverineHttp();
        
        opts.UseSignalR();
        
        // The publishing rule to route any message type that implements
        // a marker interface to the connected SignalR Hub
        opts.Publish(x =>
        {
            x.MessagesImplementing<ICritterStackWebSocketMessage>();
            x.ToSignalR();
        });


        // Really need this so we can handle messages in order for 
        // a particular service
        opts.MessagePartitioning.UseInferredMessageGrouping();
        opts.Policies.AllListeners(x => x.PartitionProcessingByGroupId(PartitionSlots.Five));
    }

And at the bottom of the ASP.Net Core application hosting CritterWatch, we’ll have this to configure the request pipeline:

			
builder.Services.AddWolverineHttp();
var app = builder.Build();
// Little bit more in the real code of course...
app.MapWolverineSignalRHub("/api/messages");
return await app.RunJasperFxCommands(args);

		

As you can infer from the Wolverine publishing rule above, we’re using a marker interface to let Wolverine “know” what messages should always be sent to SignalR:

			
/// <summary>
/// Marker interface for all messages that are sent to the CritterWatch web client
/// via web sockets
/// </summary>
public interface ICritterStackWebSocketMessage : ICritterWatchMessage, WebSocketMessage;

		

We also use that marker interface in a homegrown command line integration to generate TypeScript versions of all those messages with NJsonSchema as well as message types that go from the user interface to the CritterWatch server. Wolverine’s SignalR integration assumes that all messages sent to SignalR or received from SignalR are wrapped in a Cloud Events compliant JSON wrapper, but the only required members are type that should identify what type of message it is and data that holds the actual message body as JSON. To make this easier, when we generate the TypeScript code we also insert a little method like this that we can use to identify the message type sent from the client to the Wolverine powered back end:

			
export class CompactStreamResult implements WebsocketMessage {
    serviceName!: string;
    streamId!: string;
    success!: boolean;
    error!: string | undefined;
    queryId!: string | undefined;
    // THIS method is injected by our custom codegen
    // and helps us communicate with the server as
    // this matches Wolverine's internal identification of
    // this message
    get messageTypeName() : string{
        return "compact_stream_result";
    }
    
    // other stuff...
    init(_data?: any) {
        if (_data) {
            this.serviceName = _data["serviceName"];
            this.streamId = _data["streamId"];
            this.success = _data["success"];
            this.error = _data["error"];
            this.queryId = _data["queryId"];
        }
    }
    static fromJS(data: any): CompactStreamResult {
        data = typeof data === 'object' ? data : {};
        let result = new CompactStreamResult();
        result.init(data);
        return result;
    }
    toJSON(data?: any) {
        data = typeof data === 'object' ? data : {};
        data["serviceName"] = this.serviceName;
        data["streamId"] = this.streamId;
        data["success"] = this.success;
        data["error"] = this.error;
        data["queryId"] = this.queryId;
        return data;
    }
}

		

Most of the code above is generated by NJsonSchema, but our custom codegen inserts in the get messageTypeName() method, that we use in the client side code below to wrap up messages to send back up to our server:

  async function sendMessage(msg: WebsocketMessage) {
    if (conn.state === HubConnectionState.Connected) {
      const payload = 'toJSON' in msg ? (msg as any).toJSON() : msg
      const cloudEvent = JSON.stringify({
        id: crypto.randomUUID(),
        specversion: '1.0',
        type: msg.messageTypeName,
        source: 'Client',
        datacontenttype: 'application/json; charset=utf-8',
        time: new Date().toISOString(),
        data: payload,
      })
      await conn.invoke('ReceiveMessage', cloudEvent)
    }
  }

In the reverse direction, we receive the raw message from a connected WebSocket with the SignalR client, interrogate the expected CloudEvents wrapper, figure out what the message type is from there, deserialize the raw JSON to the right TypeScript type, and generally just relay that to a Pinia store where all the normal Vue.js + Pinia reactive user interface magic happens.

			
export function relayToStore(data: any){
    const servicesStore = useServicesStore();
    const dlqStore = useDlqStore();
    const metricsStore = useMetricsStore();
    const projectionsStore = useProjectionsStore();
    const durabilityStore = useDurabilityStore();
    const eventsStore = useEventsStore();
    const scheduledMessagesStore = useScheduledMessagesStore();
    const envelope = typeof data === 'string' ? JSON.parse(data) : data;
    switch (envelope.type){
        case "dead_letter_details":
            dlqStore.handleDeadLetterDetails(DeadLetterDetails.fromJS(envelope.data));
            break;
        case "dead_letter_queue_summary_results":
            dlqStore.handleDeadLetterQueueSummaryResults(DeadLetterQueueSummaryResults.fromJS(envelope.data));
            break;
        case "all_service_summaries":
            const allSummaries = AllServiceSummaries.fromJS(envelope.data);
            servicesStore.handleAllServiceSummaries(allSummaries);
            if (allSummaries.persistenceCounts) {
                for (const pc of allSummaries.persistenceCounts) {
                    durabilityStore.handlePersistenceCountsChanged(pc);
                }
            }
            if (allSummaries.metricsRollups) {
                metricsStore.handleAllMetricsRollups(allSummaries.metricsRollups);
            }
            break;
        case "summary_updated":
            servicesStore.handleSummaryUpdated(SummaryUpdated.fromJS(envelope.data));
            break;
        case "agent_and_node_state_changed":
            servicesStore.handleAgentAndNodeStateChanged(AgentAndNodeStateChanged.fromJS(envelope.data));
            break;
        case "service_summary_changed":
            servicesStore.handleServiceSummaryChanged(ServiceSummaryChanged.fromJS(envelope.data));
            break;
        case "metrics_rollup":
            metricsStore.handleMetricsRollup(MetricsRollup.fromJS(envelope.data));
            break;
        case "all_metrics_rollups":
            metricsStore.handleAllMetricsRollups(AllMetricsRollups.fromJS(envelope.data));
            break;
        case "shard_states_changed":
            projectionsStore.handleShardStatesChanged(ShardStatesChanged.fromJS(envelope.data));
            break;
        case "persistence_counts_changed":
            durabilityStore.handlePersistenceCountsChanged(PersistenceCountsChanged.fromJS(envelope.data));
            break;
        case "stream_details":
            eventsStore.handleStreamDetails(StreamDetails.fromJS(envelope.data));
            break;
        case "event_query_results":
            eventsStore.handleEventQueryResults(EventQueryResults.fromJS(envelope.data));
            break;
        case "compact_stream_result":
            eventsStore.handleCompactStreamResult(CompactStreamResult.fromJS(envelope.data));
            break;
        // *CASE ABOVE* -- do not remove this comment for the codegen please!
    }
}

		

And that’s really it. I omitted some of our custom codegen code (because it’s hokey), but it doesn’t do much more than find the message types in the .NET code that are marked as going to or coming from the Vue.js client and writes them as matching TypeScript types.

But wait, Marten gets into the act too!

With the Marten + Wolverine integration through this:

        opts.Services.AddMarten(m =>
        {
            // Other stuff...
            m.Projections.Add<ServiceSummaryProjection>(ProjectionLifecycle.Async);
        }).IntegrateWithWolverine(w =>
        {
            w.UseWolverineManagedEventSubscriptionDistribution = true;
        });

Marten can also get into the SignalR act through its support for “Side Effects” in projections. As a certain projection for ServiceSummary is updated with new events in CritterWatch, we can raise messages reflecting the new changes in state to notify our clients with code like this from a SingleStreamProjection:

    public override ValueTask RaiseSideEffects(IDocumentOperations operations, IEventSlice<ServiceSummary> slice)
    {
        var hasShardStates = slice.Events().Any(x => x.Data is ShardStatesUpdated);

        if (hasShardStates)
        {
            var shardEvent = slice.Events().Last(x => x.Data is ShardStatesUpdated).Data as ShardStatesUpdated;
            slice.PublishMessage(new ShardStatesChanged(slice.Snapshot.Id, shardEvent!.States));
        }

        if (slice.Events().All(x => x.Data is IImpactsAgentOrNodes || x.Data is ShardStatesUpdated))
        {
            if (!hasShardStates)
            {
                slice.PublishMessage(new AgentAndNodeStateChanged(slice.Snapshot.Id, slice.Snapshot.Nodes, slice.Snapshot.Agents));
            }
        }
        else
        {
            slice.PublishMessage(new ServiceSummaryChanged(slice.Snapshot));
        }

        return new ValueTask();
    }

The Marten projection itself knows absolutely nothing about where those messages will go or how, but Wolverine kicks in to help its Critter Stack sibling and it deals with all the message delivery. The message types above all implement the ICritterStackWebSocketMessage interface, so they will get routed by Wolverine to SignalR. To rewind, the workflow here is:

CritterWatch constantly receives messages from Wolverine applications with changes in state like new messaging endpoints being used, agents being reassigned, or nodes being started or shut down
CritterWatch saves any changes in state as events to Marten (or later to SQL Server backed Polecat)
The Marten async daemon processes those events to update CritterWatch’s ServiceSummary projection
As pages of events are applied to individual services, Marten calls that RaiseSideEffects() method to relay some state changes to Wolverine, which will..
Send those messages to SignalR based on Wolverine’s routing rules and on to the client side code which…
Relays the incoming messages to the proper Pinia store

Summary

I won’t say that using Wolverine for processing and sending messages via SignalR is justified in every application, but it more than pays off if you have a highly interactive application that sends any number of messages between the user interface and the server.

Sometime last week I said online that no project is truly a failure if you learned something valuable from that effort that could help a later project succeed. When I wrote that I was absolutely thinking about the work shown above and relating that to a failed effort of mine called Storyteller (Redux + early React.js + roll your own WebSockets support on the server) that went nowhere in the end, but taught me a lot of valuable lessons about using WebSockets in a highly interactive application that has directly informed my work on CritterWatch.

On Debugging Problems

This post is from a decade old draft where I thought I would write a grand treatise about every bit of wisdom I could possibly impart to other developers. So let this be proof that I absolutely can still write about software development outside of promoting the “Critter Stack” if you give me a decade to finish the post and a day when my new fangled AI agent is being extremely slow!

A lot of my time over the past 15 years has been spent helping other software developers analyze, debug, and fix problems in their own work. I also spend a lot of time working in some fairly complicated problem domains where I not infrequently have to unwind problems of my own creation. It follows pretty naturally that I’ve spent a significant amount of time thinking both about how I debug problems in code and how I would try to teach other developers to be better at debugging their own problems.

Here then is a few thoughts about the act of debugging problems in software systems. You’ll note that I’m not going to talk really at all about specific tools. I’m happy to leave that to other people and really just focus on how you use your noggin to think your way into determining root causes and fixes.

Do what I say, not what I just did…

And of course while I was writing this I wasn’t practicing what I’m preaching, so let’s look at a micro-cosm of all this advice for a problem I was troubleshooting right after I wrote most of this post. Right now I’m focused on a forthcoming product for JasperFx Software named “CritterWatch.” That product will have to manage communication between Wolverine systems, the centralized CritterWatch service and connected CritterWatch browsers. Last week I was initially struggling with why the initial load of the web pages was missing some data from the backend as shown in this diagram:

I was working pretty late in the afternoon waiting for my daughter to wrap up an after school activity, and I admittedly wasn’t at my best. I followed these steps:

There were custom value types in the information flow from Wolverine to CritterWatch to the CritterWatch user interface, so I first went with the reasonable theory that there were JSON serialization problems that had cropped up since last I’d worked seriously in this codebase.
I was feeling lazy at the end of the day, so I tried to get Claude to “solve” the problem for me and told it to look at serialization issues. Claude claimed to have fixed a problem, but no dice when I ran the whole stack
I wasn’t really getting into the whole flow yet, so I kept trying to just add more tests to verify bidirectional JSON serialization and deserialization of the custom types, and all of those new tests were passing just fine.
I finally listened to what the feedback was telling me and moved off of what had been my initial theory once I had really disproven that theory about serialization problems several ways to Sunday.
I walked away for the night, and came back fresh the next morning
After some rest, I went a lot deeper in the code and got back up to speed on how data flowed throughout and with a little more judicious debugging I was able to determine that the data was flowing incorrectly much earlier in the workflow than I’d originally thought and that the problem was very different than I’d first theorized. To use the analogy I’ll introduce in a bit, I took my mental X-Wing right into the trench to go find the thermal exhaust port.
Knowing where the problem was, I switched to using automated tests that were finer grained than running the while system end to end and much more quickly repeatable
With the faster feedback loop and a new theory about what could be going wrong, I made a change to the internals that turned out to be correct and I got the tests passing before moving on to running the full stack end to end and seeing the whole system function correctly. Finally.
And lastly, I actually did remember to go rip out some of the temporary tracing code I’d put in while diagnosing the problems

Does any of this matter if we have LLMs now?

Definitely for the short term and probably for the longer term too? I’ve been very impressed with my usage of Claude Code so far, but I’ve seen plenty of times when I could have probably have solved an issue much faster myself than letting Claude hit it with brute force. Even with a tool like Claude or Copilot, I’d still say you want to feed it what you think the likely cause is to speed up its own work and whatever facts you know. And I guess at the end of the day I just don’t believe an LLM is going to be better than a capable human at everything, especially when you’re facing a novel problem.

I have found it advantageous to have the LLM tools retrofit tests to verify or disprove theories about why code is broken. It’s also been helpful to get an LLM to create summaries of a code subsystem including dumping that out to a markdown file with requested Mermaid diagrams explaining the flow.

Oh, and I absolutely think it’s important for you to understand how the fancy AI tool was able to fix the issue later anyway.

15/15 Minute Rule

Before we get into the real meat of this post, there are two unhealthy behaviors I see from other developers asking for help that drive me absolutely bananas:

Not making the slightest attempt to solve their problem on their own before they ask for help, but I think this is commonly caused by developers who can’t do what I’m going to call step 1 below
Spending hours banging their head against the wall trying to solve something that I could have helped them get through much more quickly

I really wish I’d made of note who wrote this first — and we can happily quibble about what exactly the time duration should be — but I’d highly recommend the “15/15 Rule” of debugging. You should spend at least 15 minutes trying to debug a problem on your own both to develop your own skills and to keep from overloading your colleagues and maybe causing them to resent you. After that though, spend no more than 15 more minutes trying to solve the problem before you lift your head up and ask someone else for help.

It’s been years since I have been an active development lead, but I vividly remember how frustrating either extreme can be. At the time I started this post I was a remote “architect” who was on call to help anyone who could reach me on Hipchat (I think? Long before Slack), and I tried to take that role seriously. Now, one very important thing I should have taken more seriously then and that I have tried to impart to technical leaders since is that you should try really hard to teach the other people on your team how you debugged the problem and found the eventual answer. That could include knowing to check certain log files, database tables, or just imparting more about how the system works.

And of course, there were days when I wished fewer people needed my help on any given day. Then a few years later that company hired a lot more people in the main office with a few senior folks mixed in and people stopped asking me for help altogether. And if you are assuming that probably led to me no longer working there, you’d probably be at least partially right.

Step 1: Believing that you can figure it out

I despise touchy feely, kumbayah statements in regards to software development, but I do actually think that believing that you can understand the problem you’re facing and fix it is the all important first step. If you stay on the outside and just randomly try different things without understanding what you’re dealing with or try to set Debugger breakpoints in what you hope might be a useful spot, you might get lucky, but in my experience that’s not a sure thing.

Look, I’m solidly GenX, so the analogy I’m going to make is trying to destroy the Death Star. Sure, you can try to lob shots from the outside, the only sure way is to go fly your X-Wing right through that trench until you find the thermal exhaust port that will blow up the whole thing.

Long story short, I think you need to be ready to roll your sleeves up and learn enough about whatever it is you’re working on to have a good enough mental model of how information flows and where the points are that the process could be failing.

Oh, and another important point. Experience helps tremendously in debugging problems, and the only way to get experience is to try to debug problems yourself or at least just not ask the older dev on your team to just fix it for you.

Create a Mental Model

Okay, so once we actually believe that we can ultimately understand the code, the problem, and the eventual solution, I suggest you create or refresh your own mental model of how the code that’s failing works. I’d say to first focus on:

Where, when, and how system state is mutated
The workflow of the code or system or systems in play. How does information pass between elements of the code or the system?

What I think specifically you’re looking for is the most likely place in that workflow where things are going wrong and that you should…

One thing I already like AI for is to quickly say “make me a summary of XYZ code in markdown with a Mermaid sequence diagram for ABC.”

Have a Theory, then Prove or Disprove It

Formulate the most likely theory about why the functionality is failing and devise some way to verify or disprove that theory. Sometimes you can do this through targeted Debugger usage, but I’ll also highly recommend trying to build automated tests to verify intermediate steps as that’s almost always going to be a much faster feedback cycle than manually running through a system or series of systems. To put this more bluntly, always be looking to shorten your debugging session by looking for faster, finer grained feedback cycles that can tell you something valuable. And of course, absolutely leave behind automated test coverage as regression tests against whatever the problem turns out to be.

Having a working theory will provide some structure to how you’re trying to debug and fix your broken code. However, you need to be able to drop that working theory and look for a new one as soon as you get feedback to the contrary. Several of the longest debugging sessions I’ve undergone were prolonged by me refusing to give up on my initial explanation for what was wrong.

I was joking online the other day that Extreme Programming was the fount of all knowledge about software development. One of the ideas I learned from XP was that having to frequently use a Debugger tool meant that you should be adding more finer-grained automated tests and that you generally don’t want to spend a long time fussing with a Debugger if you can help it.

Pay Attention to Trends and Correlation

“Correlation isn’t Causation” necessarily, but it will absolutely suggest issues with certain types of input or configuration and tell you something that deserves further investigation and be an input into creating a viable theory.

Select isn’t Broken

The saying “Select isn’t Broken” comes from the seminal Pragmatic Programmer book that every aspiring developer should read at some point (it’s pretty short). I also like the Coding Horror take on this from years ago.

The gist of this is that it’s far more likely that the source of your problem is in your freshly written code and not in the much more mature tools you may be using. As a .Net developer, I know that it’s far more likely that a bug in my system is much more likely caused by my code than the core CLR code underneath it that’s used by millions of developers.

This isn’t necessarily saying that you’re a bad developer, but it can definitely save you time to assume that the problem is in your code and start from there.

Another way to put this is to always apply…

Occam’s Razor

From Wikipedia’s entry on Occam’s Razor:

Occam’s Razor is a problem-solving principle that, when presented with competing hypothetical answers to a problem, one should select the one that makes the fewest assumptions

This one is simple, look for the most likely explanation for your problem and focus in on that in your testing and debugging. Sorry to say this, but Occam’s Razor is frequently going to point at your code.

I probably shouldn’t tell anybody this, but an easy “tell” that I’m getting annoyed or impatient with somebody that I’m trying to help solve a problem is if I use either the phrase “Occam’s Razor says…” or something to the effect of “Dude, SELECT isn’t broken…”

I wrote that last sentence a decade ago but it’s still completely true today.

Don’t be so quick to blame the weird thing

So here’s the situation, you just got assigned to a project that uses some kind of technology that is completely new to you. Or maybe you pulled down some hotshot OSS library for the first time. Either way, you might find yourself pounding your head on your desk not being able to understand why some piece of code that uses that weird new thing isn’t working.

The common reaction is to blame the weird new thing, but you might just be completely wrong. Even when faced with some kind of novel technology, you still need to check for the normal, banal problems. I’ve fallen prey to this myself several times in the past, only to realize that my code that used the strange new thing was wrong. In particular, I remember feeling pretty stupid when I realized that I had a file path in the code wrong. It wasn’t the strange, new thing, it was just the common kind of mistake that I’ve made and quickly fixed several hundred times before.

Another way to put this is to always blame your code first. I say that based on the fact that it should be easiest and quickest to troubleshoot your own code to eliminate any possible problems there before getting into the strange new thing.

This is closely related to the initial “Believing that you can figure it out” rule, because blaming the “weird thing” allows you to push off all responsibility instead of diving in to try to understand what’s not working.

This section was written a decade ago after an incident where I happened to be in the main office when a developer said in a stand up meeting that he had a “FubuMVC problem.” As the primary author of FubuMVC, I jumped in and said I’d pair with him to try to fix whatever it was. When we sat down, he showed me the exception he was getting, and a quick glance at the inner exception pointed at it being just a run of the mill issue with a collection in his code not being initialized before the code tried to modify it. Easy money. But of course, the next day I popped into their stand up again and he told them that we had fixed the “FubuMVC problem.” Grr.

As the author of many OSS tools meant for other developers, I sometimes get the brunt of this issue. I’ve got some thoughts and lessons learned about better or worse stack trace and exception messages from my time writing tools and frameworks for other developers that I tackle in a later section — but, the main thing I want to tell other developers sometimes is to…

In no small part, Wolverine’s runtime architecture was purposely designed to streamline Exception stack traces compared to how FubuMVC at that time (or ASP.Net Core today for that matter) would add an absurd amount of framework noise to stack traces. I would argue that this is a significant advantage to Wolverine over the middleware strategies of other tools in the .NET space today.

Read the Exception Message Carefully

Yeah, this one is self explanatory. But yet this section is here because oftentimes the most important information you need to pay attention is buried in an inner exception. Or just don’t jump to incorrect conclusions about what the exception message and stack trace is telling you.

And also, the absolutely most important debugging rule. If you jump online and flag me down to help you solve your problem, don’t ever just say “I had an Exception” but instead post the whole damn stack trace (please)!

Just Flat Out Walk Away

Yeah, I’m not kidding. Sometimes if you’re really stuck and you can get away with this, just walk away and go do something else and hopefully recharge. It’s not a perfectly reliable or predictable strategy at all, but your subconscious will frequently throw up the eventual answer — or at least a new theory — at some random time while you’re walking the dog or doing dishes or whatever your daily routines are.

Probably more importantly, just getting some mental rest and coming back with a fresh mind and hopefully ready to entertain new ideas about what’s going wrong and how to solve that is frequently helpful as compared to banging your head on your desk and hoping this time you’ll notice something in the Debugger that tips you off to the problem.

The Secret of Management

As I was wrapping this up, I realized that I recreated this classic episode of News Radio (i.e., one of the greatest sitcoms ever even if it gave us Joe Rogan and RIP Phil Hartman):

Using the Azure Service Bus Emulator for Local Wolverine Development

As I wrote last week, I finally got on the AI bus and started using Claude Code. One of the things I’ve been doing with that so far is ripping through “chore” tasks that I’ve long wanted to do, but sounded too time consuming. One of those things was converting Wolverine’s own test suite for its Azure Service Bus integration into using the Azure Service Bus Emulator — which turned out to be extremely fortunate timing as the emulator recently gained support for the Azure Service Bus Management API that finally made the emulator usable.

The emulator is already turning out to be very useful for Wolverine development, especially in areas where we needed 3-5 namespaces just to test features like “namespace per tenant” and named broker features inside of Wolverine. A JasperFx client just happened to ask about that this week, so I lazily had Claude build a new section in the Wolverine docs to explain how that’s working for us and how you might use the emulator for your own local testing. Maybe in the next Wolverine (5.17) we’ll add some syntactical sugar to make this a little easier.

In the meantime, here’s how we’re using the emulator for Wolverine testing:

The Azure Service Bus Emulator allows you to run integration tests against a local emulator instance instead of a real Azure Service Bus namespace. This is exactly what Wolverine uses internally for its own test suite.

Docker Compose Setup

The Azure Service Bus Emulator requires a SQL Server backend. Here is a minimal Docker Compose setup:

			
networks:
  sb-emulator:
services:
  asb-sql:
    image: "mcr.microsoft.com/azure-sql-edge"
    environment:
      - "ACCEPT_EULA=Y"
      - "MSSQL_SA_PASSWORD=Strong_Passw0rd#2025"
    networks:
      sb-emulator:
  asb-emulator:
    image: "mcr.microsoft.com/azure-messaging/servicebus-emulator:latest"
    volumes:
      - ./docker/asb/Config.json:/ServiceBus_Emulator/ConfigFiles/Config.json
    ports:
      - "5673:5672"   # AMQP messaging
      - "5300:5300"   # HTTP management
    environment:
      SQL_SERVER: asb-sql
      MSSQL_SA_PASSWORD: "Strong_Passw0rd#2025"
      ACCEPT_EULA: "Y"
      EMULATOR_HTTP_PORT: 5300
    depends_on:
      - asb-sql
    networks:
      sb-emulator:

		

TIP

The emulator exposes two ports: the AMQP port (5672) for sending and receiving messages, and an HTTP management port (5300) for queue/topic administration. These must be mapped to different host ports.

Emulator Configuration File

The emulator reads a Config.json file on startup. A minimal configuration that lets Wolverine auto-provision everything it needs:

			
{
  "UserConfig": {
    "Namespaces": [
      {
        "Name": "sbemulatorns"
      }
    ],
    "Logging": {
      "Type": "File"
    }
  }
}

		

You can also pre-configure queues and topics in this file if needed:

			
{
  "UserConfig": {
    "Namespaces": [
      {
        "Name": "sbemulatorns",
        "Queues": [
          {
            "Name": "my-queue",
            "Properties": {
              "MaxDeliveryCount": 3,
              "LockDuration": "PT1M",
              "RequiresSession": false
            }
          }
        ],
        "Topics": [
          {
            "Name": "my-topic",
            "Subscriptions": [
              {
                "Name": "my-subscription",
                "Properties": {
                  "MaxDeliveryCount": 3,
                  "LockDuration": "PT1M"
                }
              }
            ]
          }
        ]
      }
    ],
    "Logging": {
      "Type": "File"
    }
  }
}

		

Connection Strings

The emulator uses standard Azure Service Bus connection strings with UseDevelopmentEmulator=true:

			
// AMQP connection for sending/receiving messages
var messagingConnectionString =
    "Endpoint=sb://localhost:5673;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=SAS_KEY_VALUE;UseDevelopmentEmulator=true;";
// HTTP connection for management operations (creating queues, topics, etc.)
var managementConnectionString =
    "Endpoint=sb://localhost:5300;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=SAS_KEY_VALUE;UseDevelopmentEmulator=true;";

		

WARNING

The emulator uses separate ports for messaging (AMQP) and management (HTTP) operations. In production Azure Service Bus, a single connection string handles both, but the emulator requires you to configure these separately.

Configuring Wolverine with the Emulator

The key to using the emulator with Wolverine is setting both the primary connection string (for AMQP messaging) and the ManagementConnectionString (for HTTP administration) on the transport:

			
var builder = Host.CreateApplicationBuilder();
builder.UseWolverine(opts =>
{
    opts.UseAzureServiceBus(messagingConnectionString)
        .AutoProvision()
        .AutoPurgeOnStartup();
    // Required for the emulator: set the management connection string
    // to the HTTP port since it differs from the AMQP port
    var transport = opts.Transports.GetOrCreate<AzureServiceBusTransport>();
    transport.ManagementConnectionString = managementConnectionString;
    // Configure your queues, topics, etc. as normal
    opts.ListenToAzureServiceBusQueue("my-queue");
    opts.PublishAllMessages().ToAzureServiceBusQueue("my-queue");
});

		

Creating a Test Helper

Wolverine’s own test suite uses a static helper extension method to standardize emulator configuration across all tests. Here’s the pattern:

			
public static class AzureServiceBusTesting
{
    // Connection strings pointing at the emulator
    public static readonly string MessagingConnectionString =
        "Endpoint=sb://localhost:5673;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=SAS_KEY_VALUE;UseDevelopmentEmulator=true;";
    public static readonly string ManagementConnectionString =
        "Endpoint=sb://localhost:5300;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=SAS_KEY_VALUE;UseDevelopmentEmulator=true;";
    private static bool _cleaned;
    public static AzureServiceBusConfiguration UseAzureServiceBusTesting(
        this WolverineOptions options)
    {
        // Delete all queues and topics on first usage to start clean
        if (!_cleaned)
        {
            _cleaned = true;
            DeleteAllEmulatorObjectsAsync().GetAwaiter().GetResult();
        }
        var config = options.UseAzureServiceBus(MessagingConnectionString);
        var transport = options.Transports.GetOrCreate<AzureServiceBusTransport>();
        transport.ManagementConnectionString = ManagementConnectionString;
        return config.AutoProvision();
    }
    public static async Task DeleteAllEmulatorObjectsAsync()
    {
        var client = new ServiceBusAdministrationClient(ManagementConnectionString);
        await foreach (var topic in client.GetTopicsAsync())
        {
            await client.DeleteTopicAsync(topic.Name);
        }
        await foreach (var queue in client.GetQueuesAsync())
        {
            await client.DeleteQueueAsync(queue.Name);
        }
    }
}

		

Writing Integration Tests

With the helper in place, integration tests become straightforward:

			
public class when_sending_messages : IAsyncLifetime
{
    private IHost _host;
    public async Task InitializeAsync()
    {
        _host = await Host.CreateDefaultBuilder()
            .UseWolverine(opts =>
            {
                opts.UseAzureServiceBusTesting()
                    .AutoPurgeOnStartup();
                opts.ListenToAzureServiceBusQueue("send_and_receive");
                opts.PublishMessage<MyMessage>()
                    .ToAzureServiceBusQueue("send_and_receive");
            }).StartAsync();
    }
    public async Task DisposeAsync()
    {
        await _host.StopAsync();
    }
    [Fact]
    public async Task send_and_receive_a_single_message()
    {
        var message = new MyMessage("Hello");
        var session = await _host.TrackActivity()
            .IncludeExternalTransports()
            .Timeout(30.Seconds())
            .SendMessageAndWaitAsync(message);
        session.Received.SingleMessage<MyMessage>()
            .Name.ShouldBe("Hello");
    }
}

		

TIP

Use .IncludeExternalTransports() on the tracked session so Wolverine waits for messages that travel through Azure Service Bus rather than only tracking in-memory activity.

Disabling Parallel Test Execution

Because the emulator is a shared resource, tests that create and tear down queues or topics can interfere with each other when run in parallel. Wolverine’s own test suite disables parallel execution for its Azure Service Bus tests:

			
// Add to a file like NoParallelization.cs in your test project
[assembly: CollectionBehavior(CollectionBehavior.CollectionPerAssembly)]

Critter Stack Roadmap Update for 1st Quarter 2026

That is an American Polecat (black-footed ferret), our avatar for our newest Critter Stack project.

Mostly for my own sake to collect my own thoughts, I wanted to do a little update on the Critter Stack roadmap as it looks right now. This is an update on Critter Stack Roadmap for 2026 from December. Things have changed a little bit, or really just become more clear. While the rest of the Critter Stack core team has been early adopters of AI tools, I was late to the party, but two weeks into my own adoption of Claude Code, my ambition for the year has hugely expanded and this new update will reflect that.

Also, we’ve delivered an astonishing amount of new functionality in the first six weeks for 2026:

Marten’s new composite projection capability that is already getting usage. This feature is going to hopefully make it much easier to create denormalized “query model” projections with Marten to support reporting and dashboard screens
Wolverine got rate limiting middleware support (community built feature)
Wolverine’s options for transactional middleware, inbox, outbox, and scheduled messaging support grew to include Oracle, MySql, Sqlite, and CosmosDb. Weasel support for Critter Stack style “it just works” migrations were added for Oracle, MySql, and Sqlite as well

Short to Medium Term Roadmap

I think we are headed toward a Marten 9.0 and Wolverine 6.0 release this year, but I think that’s 2nd or even 3rd quarter this year.

CritterWatch

My personal focus (i.e. JasperFx’s) is switching to CritterWatch as of today. We have a verbal agreement with a JasperFx Software client to have a functional CritterWatch MVP in their environment by the end of March 2026, so here we go! More on this soon as I probably do quite a bit of thinking and analysis on how this should function out loud. The MVP scope is still this:

A visualization and explanation of the configuration of your Critter Stack application
Performance metrics integration from both Marten and Wolverine
Event Store monitoring and management of projections and subscriptions
Wolverine node visualization and monitoring
Dead Letter Queue querying and management
Alerting – but I don’t have a huge amount of detail yet. I’m paying close attention to the issues JasperFx clients see in production applications though, and using that to inform what information Critter Watch will surface through its user interface and push notifications

Marten 8.*

I think that Marten 8.* has just about played out and there’s only a handful of new features I’m personally thinking about before we effectively turn the page on Marten 8.*:

First Class EF Core Projections. Just the ability to use an EF Core DbContext to write projected data with EF Core. I’ve thought that this would further help Marten users with reporting needs.
An ability to tag event streams with user-defined “natural keys”, and efficient mechanisms to use those natural keys in APIs like FetchForWriting() and FetchLatest(). This will be done in conjunction with Wolverine’s “aggregate handler workflow.” This has been occasionally requested and on our roadmap for a couple years, but it moves up now because of some ongoing client work

Add in some ongoing improvements to the new “composite projection” feature and some improvements to the robustness of the Async Daemon subsystem and I think that’s a wrap on Marten 8.

One wild card is that Marten will gain some kind of model for Dynamic Consistency Boundaries (DCB) this year. I’m not sure whether I think that could or should be done in 8.* or wait for 9.0 though. I was initially dubious about DCB because it largely seemed to be a workaround for event store tools that can’t support strong consistency between event streams the way that Marten can. I’ve come around to DCB a little bit more after reviewing some JasperFx client code where they need to do quite a few cross-stream operations and seeing some opportunity to reduce repetitive code. This will be part of an ongoing process of improving the full Critter Stack’s ability to express cross-stream commands and will involve the integration into Wolverine as well.

Wolverine 5.*

Wolverine has exploded in development and functionality over the past three months, but I think that’s mostly played out as well. Looking at the backlog today, it’s mostly small ball refinements here and there. As mentioned before, I think Wolverine will be part of the improvements to cross-stream operations with Marten as well.

Wolverine gets a lot of community contributions though, and that could continue as a major driver of new features.

Introducing Polecat!

After 10 years of people sagely telling us that Marten would be much more popular if only it supported SQL Server, let’s all welcome Polecat to the Critter Stack. Polecat is going to be a SQL Server Backed Event Store and Document Db tool within the greater Critter Stack ecosystem. As you can imagine, Polecat is very much based on Marten with some significant simplifications. Right now the very basic event sourcing capabilities are already in place, but there’s plenty more to do before I’d suggest using it in a production application.

The key facts about its approach so far:

Supply a robust Event Store functionality using SQL Server as the storage mechanism
Mimics Marten’s API, and it’s likely some of the public API ends up being standardized between the two tools
Uses the same JasperFx.Events library for event abstractions and projection or subscription base types
Uses Weasel.SqlServer for automatic database migrations similar to Marten
Supports the bigger Critter Stack “stateful resource” model with Weasel to build out schema objects
Support both conjoined and separate database multi-tenancy
Projections will be based on the model in JasperFx.Events and supply SingleStreamProjection, MultiStreamProjection, EventProjection, and FlatTableProjection right out of the box
STJ only for the serialization. No Newtonsoft support this time
QuickAppend will be the default event appending approach
Only support .NET 10
Only support Sql Server 2025 (v17)
Utilize the new Sql Server JSON type much like Marten uses the PostgreSQL JSONB
Strictly using source generators instead of the Marten code generation model — but let’s call this an experiment for now that might end up moving to Marten 9.0 later on

I blew a tremendous amount of time in late 2024 and throughout 2025 getting ready to do this work by pulling out much of the guts of Marten Event Sourcing into potentially reusable libraries, and Polecat is the result.

Selfishly, the CritterWatch approach requires its own event sourced persistence, and I’m hoping that Polecat and SQL Server could be used as an alternative to Marten and PostgreSQL for shops that are interested in CritterWatch but don’t today use PostgreSQL.

Marten 9.0 and Wolverine 6.0

There will be major version releases of the two main critters later this year. The main goal of these releases will be all about optimizing the cold start time of the two tools and at least moving closer to true AOT compliance. We’ll be reevaluating the code generation model of both tools as part of this work.

The only other concrete detail we know is that these releases will dump .NET 8.0 support.

Summary

The road map changes all the time based on what issues clients and users are hitting and sometimes because we just have to stop and respond to something Microsoft or other technologies are doing. But at least for this moment, this is what the Critter Stack core team and I are thinking about.

Big Critter Stack Releases

The Critter Stack had a big day today with releases for both Marten and Wolverine.

First up, we have Marten 8.22 that included:

Lots of bug fixes, including several old LINQ related bugs and issues related to full text search that finally got addressed
Some improvements for the newer Composite Projections feature as users start to use it in real project work. Hat tip to Anne Erdtsieck on this one (and a JasperFx client needing an addition to it as well)
Some optimizations, including a potentially big one as Marten can now use a source generator to build some of the projection code that before depended on not perfectly efficient Expression compilation. This will impact “self aggregating” snapshot projections that use the Apply / Create / ShouldDelete conventions

Next, a giant Wolverine 5.16 release that brings:

Many, many bug fixes
Several small feature requests for our HTTP support
Improved resiliency for Kafka especially but also for any usage of external message brokers with Wolverine. See Sending Error Handling. Plus better error handling for durable listener endpoints when the transactional inbox database is unavailable
Wait, what? Wolverine has experimental support for CosmosDb as a transactional inbox/outbox and all of Wolverine’s declarative persistence helpers?
The ability to mark some message handlers or HTTP endpoints as opting out of automatic transactional middleware (for a JasperFx client). See this, but it applies to all persistence options.
Modular monolith usage improvements for a pair of JasperFx clients who are helping us stretch Wolverine to yet more use cases.
More to come on this, but we’ve recently slipped in Sqlite and Oracle support for Wolverine

Summary

Fluent Validation Middleware

Explicit Validation

Even More Lightweight Validation!

Summary

Select/Where Hoisting

Summary

Enterprise Level Support

More Professional Documentation

Moar YouTube Videos

SQL Server or CosmosDb Support in Addition to PostgreSQL

Better Support for Cross Document Views or Queries

Improving LINQ Support

Open Telemetry Support and other Improved Metadata

More Sample Applications

Scalability Improvements including Sharding

Event Streaming Support or Transactional Outbox Integration

Management User Interface

Cloud Hosting Models and Recipes

But wait, Marten gets into the act too!

Summary

Do what I say, not what I just did…

Does any of this matter if we have LLMs now?

15/15 Minute Rule

Step 1: Believing that you can figure it out

Create a Mental Model

Have a Theory, then Prove or Disprove It

Pay Attention to Trends and Correlation

Select isn’t Broken

Occam’s Razor

Don’t be so quick to blame the weird thing

Read the Exception Message Carefully

Just Flat Out Walk Away

The Secret of Management

Docker Compose Setup

Emulator Configuration File

Connection Strings

Configuring Wolverine with the Emulator

Creating a Test Helper

Writing Integration Tests

Disabling Parallel Test Execution

Short to Medium Term Roadmap

Marten 9.0 and Wolverine 6.0

Summary