Marten’s async daemon is the subsystem that runs projections and subscriptions in the background. It’s also been historically been the single hardest thing in Marten to test. Every test has to publish events, stand up a daemon (sometimes two, in hot/cold mode), wait for shards to catch up, and then check what the projections wrote. The DaemonTests project has also been for years one of our worst sources of flaky tests and I never trusted it to run in CI until earlier this year.

So when Bobcat needed a real test suite to prove itself against before 1.0, DaemonTests was one of the obvious guinea pigs (another was Wolverine’s leadership compliance tests that I’ll write about next). If projected specifications and the Supervisor could make that suite readable and reliable, they could hopefully do it anywhere.

This post is about how that went: the mechanics of how the tests were decorated, what the specifications look like in preview and in results, where we got real bugs out of it, and how the suite runs in CI today under the Supervisor’s retry budget.

I’ve already introduced Bobcat and the Bobcat Supervisor in earlier posts, so now let’s move on to a “what it looks like on a real codebase” follow up. All of the code is public in the Marten repository, and I’ll link to the actual files as I go.

Fair warning: a lot of the Bobcat work on DaemonTests was done by AI agents working from Bobcat issues, with me reviewing the pull requests. I’m going to quote the agents’ own write ups in a couple of places because, honestly, they documented the trade offs better than I would have.

No need to rewrite tests

Bobcat was initially going to be an alternative Gherkin tool with possibly a more code centric model where specifications could be “projected” from the raw test code — with a 3rd mode using a fluent interface kind of approach where you embedded Lambda functions inside of Given/When/Then() methods. It quickly became clear that the “projected” model was going to be the most usable, so we simplified Bobcat by dropping the early Gherkin model. And the less said about the Fluent Interface + nested Lambda approach, that better. I despise tools like that. Even the ones that I might have written myself years ago.

The constraint that shaped all of Bobcat’s “projected specification” model is that a big, old test suite has to be able to adopt it one class at a time, with no mandatory base class, no signature changes, and no rewrite. DaemonTests is exactly that kind of suite. It’s got hundreds of tests, a lot of them written years apart by different people, and nobody was going to sign up to rewrite them in a new DSL.

The first pull request back in September rendered two of the test classes as specifications. Eighteen tests started reporting themselves as Given/When/Then steps, and not one test body was edited. All of the steps came from decorating five helper methods on the shared DaemonContext base class and marking the two classes with one attribute.

The second pull request in October moved that work to Bobcat 1.0, routed the projection waits through Bobcat’s catch-up diagnostics, added a handful of brand new specifications written with the Critter Stack vocabulary, and rendered three more classes. Today the suite references Bobcat 1.0.0 and WolverineFx.Bobcat 6.55.0, and bobcat preview lists 30 specifications across six features:

Feature: Building an aggregate projection
Feature: Catching up projections and subscriptions
Feature: Rebuilding an inline projection
Feature: Resuming a failed rebuild
Feature: Skipping events that fail to apply
Feature: The async daemon keeps read models current

That’s a small fraction of the 393 tests in the project, but you have to start somewhere!

Mechanics, part one: decorate the helpers you already call

Almost every daemon test calls the same handful of helper methods on DaemonContext: start the daemon, publish the events, wait for a shard to reach a sequence number, rebuild a projection, compare every expected aggregate against what the projection actually wrote. Those helpers are the vocabulary of the suite. So that’s where the Bobcat attributes went:

[When("the projection daemon is running")]
internal async Task<IProjectionDaemon> StartDaemon()
{
    var daemon = theStore.Tenancy.Default.Database.As<MartenDatabase>()
        .StartProjectionDaemon(theStore);
    await daemon.StartAllAsync();
    _daemon = daemon;
    return daemon;
}
[Given("the events are published")]
internal async Task PublishSingleThreaded()
{
    await PublishSingleThreaded<Trip>();
}
[Given("the events are published on {threads} threads")]
internal Task PublishMultiThreaded(int threads)
{
    // ...
}
[Then("the {shardName} shard has reached sequence {sequence}")]
internal async Task<ShardState> WaitForShard(IProjectionDaemon daemon, string shardName, long sequence,
    Type readModel, TimeSpan timeout = default)
{
    // ...
}
[When("the {projectionName} projection is rebuilt")]
internal Task RebuildProjection(IProjectionDaemon daemon, string projectionName)
{
    return daemon.RebuildProjectionAsync(projectionName, CancellationToken.None);
}
[Then("every expected aggregate matches the actual")]
internal async Task CheckAllExpectedAggregatesAgainstActuals()
{
    // ...
}

[Given], [When], [Then], [Check] and the keywordless [Step] are all the same StepAttribute with a different keyword. The {placeholder} names in the sentence bind to the method’s parameters by name, and they’re filled in from the argument at each call site. A Type argument renders as the type’s short name, so WaitForCatchUp(typeof(Trip)) reads as “the Trip read model has caught up”. The docs on step helpers cover the binding rules, and the placeholder rules are worth reading because a Guid or a record renders differently than you might expect.

Then the test class gets one attribute:

[BobcatFeature("Building an aggregate projection")]
public class build_aggregate_projection: DaemonContext
{
    // 17 existing [Fact] tests, untouched
}

And a test like this one, which has not changed, now renders as a five step specification:

[Fact]
public async Task rebuild_the_projection()
{
    NumberOfStreams = 10;
    Logger.LogDebug("The expected number of events is {NumberOfEvents}", NumberOfEvents);
    StoreOptions(x => x.Projections.Add(new TripProjectionWithCustomName(), ProjectionLifecycle.Async));
    var agent = await StartDaemon();
    await PublishSingleThreaded();
    await WaitForShard(agent, "TripCustomName:All", NumberOfEvents, typeof(Trip), 30.Seconds());
    Logger.LogDebug("About to rebuild TripCustomName:All");
    await RebuildProjection(agent, "TripCustomName");
    Logger.LogDebug("Done rebuilding TripCustomName:All");
    await CheckAllExpectedAggregatesAgainstActuals();
}

And the results look like this in the console (we use Spectre.Console under the covers, so there’s color coded results and just take my word for it that it’s pretty!):

  rebuild the projection OK
  ────────────────────────────────
    ✓ When  the projection daemon is running (120ms)
    ✓ Given the events are published (67ms)
    ✓ Then  the TripCustomName:All shard has reached sequence 231 (248ms)
    ✓ When  the TripCustomName projection is rebuilt (25ms)
    ✓ Then  every expected aggregate matches the actual (21ms)

Yes, the When comes before the Given in that one, because that’s the order the test actually does things in. Bobcat renders what ran. It doesn’t editorialize. To be honest, I’ve never liked the Given/When/Then nomenclature anyway. To each their own.

The trick that makes this work with zero changes to the test bodies is that Bobcat.Generators is a source generator that emits C# interceptors for every call to a decorated helper. There’s no reflection or stack walking at runtime. Each call site is known at build time, which is also what lets bobcat preview show the specification before anything has run.

An interceptor has to be an extension method, and an extension method can’t see a protected member. So every decorated helper on DaemonContext had to be widened from protected to internal. That’s a C# language rule and not a Bobcat design choice, and Bobcat reports a protected helper as a BOBCAT039 diagnostic so you find out at build time rather than wondering why a step is missing. In Marten’s case that was eight helpers, and InternalsVisibleTo already existed for a sibling project, so nothing new was exposed. The Bobcat docs call this out because it’s the one thing you’ll actually have to touch.

Mechanics, part two: marker comments

Decorated helpers give you the work a test does. They don’t give you the narrative, and a lot of daemon tests have a middle section that’s just plain Marten code with no helper to hang a step on. For those, Bobcat reads marker comments: any // comment that opens with Given, When, Then, And or But as a whole word is a declared step. Every other comment is left alone, which is the property that makes this usable on a real suite that’s already full of explanatory comments like // Gotta do this to have it mixed up.

The clearest example in DaemonTests is when_skipping_events_in_daemon, which specifies what happens when a projection throws on some events and the daemon is configured to skip them:

[BobcatFeature("Skipping events that fail to apply")]
public class when_skipping_events_in_daemon : DaemonContext
{
    // ...
    [Given("the names are published to a running daemon, the bad ones failing to apply")]
    internal async Task<IProjectionDaemon> PublishTheEvents()
    {
        var daemon = await StartDaemon();
        var events = theNames.Select(name => new NameEvent {Name = name})
            .OfType<object>()
            .ToArray();
        theSession.Events.StartStream(Guid.NewGuid(), events);
        await theSession.SaveChangesAsync();
        await daemon.Tracker.WaitForHighWaterMark(theNames.Length);
        await WaitForShard(daemon, "CollateNames:All", theNames.Length, typeof(NamesByLetter));
        await WaitForShard(daemon, "NamedDocuments:All", theNames.Length, typeof(NamedDocument), 5.Minutes());
        return daemon;
    }
    [Fact]
    public async Task see_the_dead_letter_events()
    {
        var daemon = await PublishTheEvents();
        // When the daemon stops, draining the dead letter events it queued up
        await daemon.StopAllAsync();
        // Then each skipped event is a dead letter for the projection that skipped it
        var skipped = await theSession.Query<DeadLetterEvent>().ToListAsync(TestContext.Current.CancellationToken);
        var namedDocumentsSkipped = skipped.Where(x => x.ProjectionName == "NamedDocuments" && x.ShardName == "All")
            .Select(x => x.EventSequence).OrderBy(x => x).ToList();
        namedDocumentsSkipped.ShouldBe([4L, 5, 6, 7, 11, 14]);
        var collateNamesSkipped = skipped.Where(x => x.ProjectionName == "CollateNames" && x.ShardName == "All")
            .Select(x => x.EventSequence).OrderBy(x => x).ToList();
        collateNamesSkipped.ShouldBe([4L, 5, 6, 7]);
    }
}

There are three different Bobcat mechanisms in that one test, and they compose:

1. PublishTheEvents() is a [Given] helper that lives on the test class itself rather than on DaemonContext. It calls StartDaemon() and WaitForShard(), which are decorated too, so those render nested underneath the Given with their own timings.
2. The two // comments are marker comments. They declare what the test is doing in the stretch of code that calls no helper.
3. The two ShouldBe() calls are projected assertions, which is the next section.

Here’s what that renders as from bobcat run:

  see the dead letter events OK
  ────────────────────────────────────
    ✓ Given the names are published to a running daemon, the bad ones failing to apply (306ms)
      ✓ When  the projection daemon is running (15ms)
      ✓ Then  the CollateNames:All shard has reached sequence 15 (44ms)
      ✓ And   the NamedDocuments:All shard has reached sequence 15
    · When  the daemon stops, draining the dead letter events it queued up — declared
    ✓ Then  each skipped event is a dead letter for the projection that skipped it (4ms)
      ✓ And   namedDocumentsSkipped should be [4L, 5, 6, 7, 11, 14] (4ms)
      ✓ And   collateNamesSkipped should be [4L, 5, 6, 7]

Notice the When the daemon stops line is marked declared with no tick and no timing. Nothing decorated ran inside that comment’s region, so Bobcat has nothing to report about it other than that the test claims to do it. The documentation is pretty strict about keeping “declared” and “recorded” separate, and I think that’s right. A step that borrowed a verdict from its neighbors would be exactly the kind of inference that makes test reports lie to you.

Marker comments are also how we narrated the longest, most involved daemon test in the suite, rebuild_mode_start_resumes_after_a_failure, which proves that a failed rebuild can be restarted from its committed progress without folding any event twice. That one is pure Marten and Postgres code with no shared helpers, and the comments carry the whole specification:

// Given a hundred streams of one event each
// When a Rebuild-mode run fails on event 60
// Then the committed progress and the failure are recorded by extended progression tracking
// And every event at or below the committed mark is in a document already
// When the shard is stopped, the fault cleared, and the Rebuild-mode run started again
// Then every event is folded into its document exactly once
// And the failure is cleared
// And no event below the first run's committed mark was folded again by the second run
// And the restarted run keeps following the store past where the rebuild started

Some of those comments, like the “Given a hundred streams” line, aren’t “meaningful” because they map to code. They’re meaningful because they’re the sentence a maintainer needs in order to understand what the next ten lines are for. Bobcat just happens to also render them.

One gotcha I tripped over while grabbing screenshots for this post: a marker comment wraps onto the // lines directly below it, up to a blank line or a line of code (dogfooding for the win I guess). That’s handy for long sentences, but it also means an explanatory comment sitting right under a marker becomes part of the step’s sentence. A couple of the steps in that rebuild test render with a trailing explanation glued on for that reason. Leave a blank line between the marker and the explanation and it stays a comment.

Mechanics, part three: your Shouldly assertions are steps too

We’ll add support for other assertion libraries later if there’s any demand, but I prefer Shouldly and that’s what the Critter Stack code already uses, so here we are.

DaemonTests has one line in its csproj that does a surprising amount of work:

<!-- A statement-level Shouldly assertion inside a [BobcatFeature] class renders as its own
     Then step, and a run of consecutive assertions is all evaluated before the test fails,
     so a red daemon spec shows every disagreement rather than the first. Classes without
     [BobcatFeature] are untouched. -->
<BobcatProjectAssertions>true</BobcatProjectAssertions>

With that on, any statement-level Shouldly call inside a [BobcatFeature] class becomes a Then step with a sentence built from the C# expression as written, a verdict, a duration, and an expected/actual cell where the comparison allows it. jNames.Names.OrderBy(x => x).ShouldBe([...]) renders as “jNames.Names.OrderBy(x => x) should be [“Jack”, “Jane”, “Jeremy”, “Jill”]” with no attribute and no helper. Nothing moves.

The part that matters more than the rendering is that a run of consecutive assertions is all evaluated before the test fails. A plain Shouldly test throws on the first ShouldBe and never reaches the rest. With projected assertions, every assertion in the run executes, and the test fails once at the end with all of the wrongs in the failure. For daemon tests that check three or four things about shard state after a long setup, that’s the difference between one slow red/green cycle and four of them. The Shouldly page in the Bobcat docs is the whole story, including the closed set of comparisons that get a cell and the ones (ShouldBeTrue, ShouldBeOfType) that render as a step with no cell because any cell would be making something up.

Two things to know from using this on a real suite:

• An assertion inside a loop is a step per iteration, because each one really ran. The “the shards should still be running” spec renders shard.Status should be AgentStatus.Running once per shard. If you don’t want that, assert once over the whole collection outside the loop, and the docs say the same. The rebuild-resume test does exactly that, and says why in a comment:

// One assertion over the whole set rather than one per event: with BobcatProjectAssertions on, a
// per-event ShouldBe would render as sixty steps
var foldedAgain = committedSequences
    .Select(e => (Index: ((RestartableIncrement)e.Data).Index, e.Sequence))
    .Where(x => RestartableTallyProjection.Applications[x.Index] != 1)
    .Select(x => $"event {x.Index} at sequence {x.Sequence}")
    .ToList();
foldedAgain.ShouldBeEmpty();

• Only a statement is projected. x.ShouldNotBeNull().Name.ShouldBe("a") consumes the first assertion’s result, so Bobcat leaves a fluent chain alone rather than risk reporting a NullReferenceException in place of the real failure.

There’s also a comment in when_skipping_events_in_daemon explaining why each subject is pulled into a local first: on Bobcat 1.0.0-alpha.1 a projected assertion whose subject spanned lines broke the build because the generator didn’t escape the newlines in the subject string. That’s the kind of thing that dogfooding on a real suite is for, and it’s why the comments in these files tend to say why and not just what.

All the Bobcat-specific bits in one place

For the whole DaemonTests project, this is the entire Bobcat surface area:

That’s it. No base class was imposed, and catch_up_timeout_diagnostics deliberately isn’t a [BobcatFeature] because those tests expect the error a recorded specification would report, which is a nice illustration that opting in is per class.

The one dependency cost worth being honest about: Bobcat.Xunit 1.0 is built on xunit.v3 4.x over Microsoft.Testing.Platform 2, and the two xunit lines can’t share a test project. Marten’s other 23 test projects are still on xunit.v3 3.x, so DaemonTests takes 4.x through a VersionOverride and the rest of the repo moves on its own schedule. The compatibility page spells out what pairs with what.

Waiting on the daemon

The most useful change in the second PR wasn’t a rendering change at all. Daemon tests spend most of their lives waiting for a projection shard to catch up, and when a wait times out, Marten’s own exception used to say roughly “the shards haven’t started within the timeout.” That tells you nothing about which shard, where it stood, or whether the daemon even ran.

DaemonContext now has two waits that route through Bobcat’s catch-up diagnosis:

/// <summary>
/// Wait for every async projection in <c>theStore</c> to catch up before reading
/// <paramref name="readModel"/>. On a timeout this throws Bobcat's
/// <see cref="ProjectionCatchUpTimeoutException"/>, which names each shard that was behind and
/// where it stood (<c>'Trip:All' at 412 of 1000</c>), rather than the bare TimeoutException
/// Marten's own wait throws. A stalled daemon test then says which shard stalled.
/// </summary>
[Given("the {readModel} read model has caught up")]
internal async Task WaitForCatchUp(Type readModel, TimeSpan timeout = default)
{
    if (timeout == default) timeout = 30.Seconds();
    try
    {
        await EventStores.WaitForNonStaleProjectionsAsync(theStore, timeout,
            token: TestContext.Current.CancellationToken);
    }
    catch (TimeoutException e) when (e is not ProjectionCatchUpTimeoutException)
    {
        throw await catchUpTimedOut($"the {readModel.Name} read model has caught up", readModel, timeout, e);
    }
}

We have pre-canned helpers for this in our Wolverine.Bobcat package that’s meant to be our specifications library for event sourced applications with the Critter Stack.

And when one of those waits times out, this is the message you get, from a test in catch_up_timeout_diagnostics that forces the timeout on purpose by never starting the daemon:

Waiting for 'the TripCustomName:All shard has reached sequence 43' timed out after 2s, before reading the Trip.
Behind: 'TripCustomName:All' at 0 of 43. This is the environment (a daemon behind, stopped or not running), not
evidence about the Trip itself. The store said: TimeoutException: Shard TripCustomName:All did not reach
sequence number 43 in the time allowed

“This is the environment, not evidence about the Trip itself” is the sentence I care about. When an AI agent, or a tired human, reads that failure, the next move is to look at the daemon and not at the projection code.

And it paid for itself before the PR was even reviewed. From the agent’s write up:

The first two full gate runs failed the new specs with Behind: 'ShipmentSummary:All' at 0 of 4, which meant the hosted daemon had never run its shard. The cause was that the fixture cleaned event data after the Solo daemon started. That restarted the sequence underneath a shard that had already loaded its old progress, so the shard sat “ahead” of every new event. Isolated runs won that race and full runs lost it.

That’s a real test fixture bug of exactly the kind that produces a “flaky” test that nobody can reproduce on their own machine. The fix was a one liner (clean the store before StartAsync()), and the comment in ShipmentsHost now explains the race for the next person. The diagnosis is what made it findable.

New specifications with the Critter Stack vocabulary

Everything above is about retrofitting existing tests. For new daemon specifications, we used CritterStackSpec, the store-only base class from WolverineFx.Bobcat, against an application that hosts its own daemon with AddAsyncDaemon(DaemonMode.Solo). This is from async_daemon_specifications:

[BobcatFeature("The async daemon keeps read models current")]
public class async_daemon_specifications: CritterStackSpec, IClassFixture<ShipmentsHost>
{
    public async_daemon_specifications(ShipmentsHost app): base(app.Host)
    {
        ProjectionTimeout = 30.Seconds();
    }
    [Fact]
    public async Task an_async_projection_catches_up_with_an_appended_event()
    {
        var id = Guid.NewGuid();
        await GivenEvents<ShipmentSummary>(id, new ShipmentPacked(id, 3));
        await WhenActing("the shipment is shipped", services => append(services, id, new ShipmentShipped(id)));
        ThenEvents(new ShipmentShipped(id));
        await ThenReadModel<ShipmentSummary>(id, Specify<ShipmentSummary>()
            .With(x => x.Items, 3)
            .With(x => x.Shipped, true));
    }
    [Fact]
    public async Task an_async_projection_deletes_the_read_model_its_stream_cancels()
    {
        var id = Guid.NewGuid();
        await GivenEvents<ShipmentSummary>(id, new ShipmentPacked(id, 2));
        await WhenActing("the shipment is cancelled", services => append(services, id, new ShipmentCancelled(id)));
        ThenEvents(new ShipmentCancelled(id));
        await ThenNoReadModel<ShipmentSummary>(id);
    }
}

Nothing in there waits by hand. ThenReadModel opens with Bobcat’s own catch-up step, which is the same diagnosis as above if it times out. And this is where the “data centric” side of Bobcat shows up, because the events and the read model render as grids:

The third spec in that file, a rebuild through the running daemon, had to go on a host of its own, and the class comment says why: JasperFx’s rebuild stops the shard’s continuous agent, rebuilds with dedicated agents, and doesn’t start the continuous agent again, so any spec that ran after the rebuild on a shared host timed out at “‘ShipmentSummary:All’ at 2 of 4”. Whether a hosted daemon should resume the shard after a rebuild is a question for Marten rather than for the specification, and I’d rather have that written down in the test than silently worked around.

What it looks like: preview, run, and the UI

Because every step is known at build time, you can see what the suite specifies without running it. bobcat preview prints the whole thing to the console with the templates unfilled and the helper each step binds to:

  rebuild the projection
  ────────────────────────────────
      ○ When  the projection daemon is running
        ↳ DaemonContext.StartDaemon
      ○ Given the events are published
        ↳ DaemonContext.PublishSingleThreaded
      ○ Then  the {shardName} shard has reached sequence {sequence}
        ↳ DaemonContext.WaitForShard
      ○ When  the {projectionName} projection is rebuilt
        ↳ DaemonContext.RebuildProjection
      ○ Then  every expected aggregate matches the actual
        ↳ DaemonContext.CheckAllExpectedAggregatesAgainstActuals

The same preview is available in the Bobcat UI once you’ve got a resident runner attached for the suite. This is bobcat ui plus bobcat resident src/DaemonTests/bin/Debug/net10.0/DaemonTests, and every feature and specification gets its own run button:

I like this view a lot more than I expected to. Seventeen tests about building an aggregate projection, readable in one screen by somebody who has never opened the C#.

Running a couple of features with bobcat run src/DaemonTests --feature "The async daemon keeps read models current" "Skipping events that fail to apply" gives you the run page, with the passing specs folded to one line each:

And when something’s wrong, the failed scenario is opened on its specification with the disagreement right there. I broke one expectation on purpose for this screenshot, so the aggregate is expected to contain a “Joe” that was never published:

The whole run also lands in a JSON results file with a published schema, and bobcat run --html writes a self contained results page you can attach to a CI artifact. Those are what an agent reads when it’s working on a red build, and the bobcat tool page covers the rest of the commands.

While grabbing these screenshots I noticed the UI’s scenario page renders the marker-comment steps ahead of the recorded helper steps and flattens the nesting, where the console gets the order right. That’s bobcat#810, and it’s a reminder that 1.0 means the model is stable, not that I’m done finding things.

The Supervisor, retry budgets, and the flaky test that earned its retry

Now the other half of the story. Readable specifications don’t help if the suite doesn’t run reliably, and DaemonTests was historically the least reliable project in Marten. It had its own workflow (daemon.yml) that shelled straight out to dotnet test, which meant the most timing-sensitive suite in the repo was the one with no retry story at all.

Today every Marten test target runs through the Bobcat Supervisor from the Nuke build, and the daemon suite is one job in the matrix of tests.yml. The whole configuration lives in build/SupervisedTests.cs, and the interesting part is small:

csharp

const int MaxRetriesPerRun = 25;
const int MaxAttemptsPerTest = 3;
static readonly TimeSpan StallThreshold = TimeSpan.FromMinutes(5);
void configureSupervisor(Supervisor supervisor, TestLedger ledger)
{
    supervisor.RetryBudget = DisableTestRetry
        ? RetryBudget.None
        : new RetryBudget { MaxAttemptsPerTest = MaxAttemptsPerTest, MaxRetriesPerRun = MaxRetriesPerRun };
    // The policy only ever retries in a fresh process, so idle lane workers are released
    // before the retry passes. Without it, workers+1 test hosts sit resident at once — which is
    // what OOM-killed 16GB GitHub runners on Wolverine, both times during a retry.
    supervisor.ReleaseIdleLanes = true;
    // Report-only observability. A test in flight past the threshold is named at once (a wedged
    // daemon test used to be visible only as a job hitting its 20 minute cap), and a one-line
    // heartbeat every 30s says how far a run got and what has been running longest.
    supervisor.StallThreshold = StallThreshold;
    supervisor.StallAction = StallAction.Report;
    supervisor.HeartbeatInterval = TimeSpan.FromSeconds(30);
    // Longest-processing-time-first lane balancing from the previous run's ledger, keyed by
    // display name (bobcat#730). No ledger, or no match, degrades to balancing by test count,
    // and the run report's first lines say how many durations matched.
    supervisor.KnownTestDurations = ledger.KnownDurations();
}

A few things about that policy that I think are important, and that took us a while to get right:

Retries are opt-in, per test, and always in a fresh process. Bobcat’s default failure policy only retries a test that carries [Trait("Retry", "N")]. Everything else that fails, fails. Up until we made that change, every failure in Marten’s CI was retried, which made “flaky” and “broken” impossible to tell apart without going and opening the log. A retry runs in a fresh process because a failed daemon test usually leaves a half-built store, schema objects, or a running daemon behind it. The fresh process is the reset.

The retry budget is the operator’s ceiling, not the test’s. MaxAttemptsPerTest = 3 is the most any test gets, including the first attempt. A test’s own [Trait("Retry", "N")] can lower that but never raise it. MaxRetriesPerRun = 25 caps the whole run, and the build’s own comment says how to read it: a run sitting at the cap isn’t “25 flaky tests,” it’s 25 flaky tests plus an unknown tail.

Stall detection is report-only. A test in flight past five minutes is named in the log, the job summary, and the run report, but nothing gets killed. Before this, a wedged daemon test was only visible as a job hitting its 20 minute cap with no indication of which test wedged. Nothing in Marten’s suites legitimately runs that long as one test.

A pass on a retry is never laundered into a green tick. This is the RetryLedger, and its header comment is the best summary of the problem I’ve read:

Nothing in the GitHub UI distinguishes a job that passed cleanly from a job that passed after retrying nine tests. Both render as a green tick, and a Serilog warning buried in the log of a job that PASSED is a warning nobody has a reason to go read. Without this, “we added retries” quietly becomes “we stopped being able to tell whether the suite works”.

So every supervised run writes its retry count to the GitHub job summary, emits a ::warning annotation when the count is nonzero so it shows up on a green run, and uploads a JSON ledger that a flakiness roll-up job aggregates across every job in the workflow. Deliberately not in there is failing the build past a retry threshold. The value is in the visibility, not in a cliff that turns master red on a bad day.

What the ledger says about the daemon suite right now

Here’s the daemon job’s ledger from the most recent green master run as I write this (November 2nd):

The one test that passed on a retry was Bug_catching_up_apply_errors.should_catch_apply_event_exceptions, and it’s the best example I have of how the opt-in policy is supposed to work. That test carries [Trait("Retry", "3")], and the comment above it explains how it earned that:

// Retry is opt-in under the supervisor (JasperFx/bobcat#565), and this test earned it with
// evidence: in 33 green master runs (2026-09 to 2026-10-15) its first attempt failed in 13 —
// "Task should throw System.AggregateException but did not" — and passed on a fresh-process
// retry every time. It is still reported as FLAKY in the retry ledger and the job summary;
// the trait keeps a known intermittent from turning the job red, it does not hide it. The
// projection throws deterministically, so a CatchUpAsync that returns cleanly is worth an
// investigation of its own (state left by an earlier test in the same process is the lead).
[Fact]
[Trait("Retry", "3")]

That’s the whole philosophy in one comment. The ledger gave us the evidence (13 first-attempt failures in 33 runs, same message every time, always passes on a fresh process). The trait keeps a known intermittent from turning the job red. The ledger still reports it as flaky on every run so it can’t quietly disappear. And the comment leaves the next investigator a lead. A second test in the same file picked up the same trait a couple of weeks later after one miss turned a whole job red for a pull request that couldn’t possibly have touched it.

One worker, and a local gate

The daemon job runs supervised at one worker. The issue that drove this work measured two lanes green locally, 374 of 374 twice, at 5m10s against 7m09s with retries off, but raising the worker count in CI is a separate decision we want to make on CI evidence rather than on a laptop. Marten’s heavier suites do split across lanes with a database per lane, and the parallel-ready suites page is the checklist we run a suite through before we let it.

The last piece is that the same supervisor configuration runs locally. ./build.sh LocalGate runs the logical suites of tests.yml on your own machine through Bobcat’s local gate, with the same opt-in retry policy, the same stall report, and the same lane balancing from the ledger, before you push. The implementation is in build/LocalGate.cs and the header comment there is, again, the documentation.

What I’d tell you to do with your own suite

If you’ve got a big integration suite with a lot of history, this is the order I’d go in, because it’s the order that worked on DaemonTests:

1. Put the suite under the Supervisor first, with retries off (--disable-test-retry in Marten’s build), and read the ledger for a couple of weeks. You’ll find out which tests are actually flaky and which are just broken.
2. Give the proven-flaky tests a [Trait("Retry", "N")] with a comment that says what the evidence was. Keep the ledger reporting them.
3. Pick the test classes that explain how the system is supposed to behave, decorate the three or four shared helpers they all call, and mark the classes with [BobcatFeature]. Don’t touch the test bodies.
4. Add marker comments where a test’s narrative isn’t carried by a helper, and turn on BobcatProjectAssertions.
5. Replace your bare TimeoutException waits with something that says what was behind. If you’re on Marten, Polecat or Fisher, WolverineFx.Bobcat has that built in.
6. Write the new specifications with CritterStackSpec or WolverineSpec, and let the existing tests stay existing tests.

The code for all of this is public:

• The suite: src/DaemonTests in Marten, with DaemonContext.cs holding the decorated helpers and Specifications/ holding the CritterStackSpec specs and the catch-up diagnostics
• The two pull requests: #5363 (the first rendering, no test bodies edited) and #5690 (Bobcat 1.0, catch-up diagnostics, CritterStackSpec), plus the Bobcat issue that drove the second one
• The build: build/SupervisedTests.cs, build/RetryLedger.cs, build/LocalGate.cs and .github/workflows/tests.yml
• The Bobcat docs: Writing Specifications, Shouldly assertions as steps, the bobcat tool, the Bobcat UI, Reliable Integration Testing and Continuous Integration for the Supervisor, and Local Gates

Bobcat is MIT licensed, 1.0 is out, and DaemonTests is the suite I’d point you at if you want to see what it looks like on code that wasn’t written for it.

Bobcat 1.0, the Critter Stack’s new tool for executable specifications (automated tests that are human readable for easier review) was released just yesterday. As I wrote about in the release post yesterday, Bobcat is the “spiritual” successor to a mostly forgotten OSS project of mine called Storyteller that was itself trying to be a better version of FitNesse which was trying to solve the same problems of making human readable but executable specifications that has mostly been popularized by Gherkin tools like Cucumber.

I’ve admittedly just lifted this from the documentation website, but here’s an introduction to authoring a specification with Bobcat:

What you will build: one business rule, specified three ways in plain C# against a plain .NET class, with no Critter Stack and nothing hosted. You start from an xUnit test you could have written yesterday, make it a specification without touching its body, give it a vocabulary so the next test reads the same way, and then write the data-heavy cases as tables. By the end you have seen what the run prints for a pass, a wrong value, a wrong row and a thrown exception.

Bobcat specifications are code. There is no second file and no binding layer between the specification and the test that executes it: the test is the specification. It keeps running on its own runner, in your IDE’s test explorer, and Bobcat renders it as readable steps with a verdict per step.

That is a deliberate compromise. A Gherkin or FitNesse tool theoretically lets a non-technical domain expert author a specification directly, in a file of their own; a projected test is C#, so authoring stays with the developers and the domain expert reviews the rendered specification instead.

There is a little bit of an exemption though for being able to express table centric tests through Markdown tables directly embedded inside of C# (we’ll try to add F# later, but for right now, we’re using C# specific features). The folks who did use Storyteller effectively usually mention the effectiveness of Storyteller’s support for table driven testing for data intensive tests, the “set verification” feature, and the ability to express decision tree tests in Storyteller and now Bobcat with declarative markdown tables.

I should point out that the source generator with Bobcat can (hopefully) validate the Markdown table text at compile time to help eliminate those pesky binding problems that were so nasty with Gherkin or FitNesse.

The project setup is Getting Started: the three packages, the runner adapter, and global.json. This page assumes that and starts at the first test. Every sample on it is compiled and run in Bobcat’s own repository, in src/Bobcat.Xunit.Samples, and every output block is what that run printed.

The rule

A bank account that may not be overdrawn:

public record Transaction(string Kind, decimal Amount);
public class Account
{
    public decimal Balance { get; private set; }
    public List<Transaction> Transactions { get; } = new();
    public void Deposit(decimal amount)
    {
        Balance += amount;
        Transactions.Add(new Transaction("Deposit", amount));
    }
    public bool TryWithdraw(decimal amount)
    {
        if (amount > Balance) return false;
        Balance -= amount;
        Transactions.Add(new Transaction("Withdrawal", amount));
        return true;
    }
}

Step 1: a test you already have, as a specification

Here is the test as most teams would write it, with Shouldly:

[Fact]
public void a_withdrawal_larger_than_the_balance_is_refused_as_most_teams_write_it()
{
    var account = new Account();
    account.Deposit(100);
    var accepted = account.TryWithdraw(200);
    accepted.ShouldBeFalse();
    account.Balance.ShouldBe(100);
}

Make it a specification without changing a line of its logic. Mark the class [BobcatFeature("Withdrawals")], and say what each part of the body does in a comment that opens with Given, When or Then:

[Fact]
public void a_withdrawal_larger_than_the_balance_is_refused_narrated()
{
    // Given an account with a balance of 100
    var account = new Account();
    account.Deposit(100);
    // When 200 is withdrawn
    var accepted = account.TryWithdraw(200);
    // Then the withdrawal is refused and the balance is still 100
    accepted.ShouldBeFalse();
    account.Balance.ShouldBe(100);
}

On xUnit, [BobcatFeature] is all it takes: the Bobcat.Xunit package opens a scenario around every test in the class and closes it with the verdict xUnit reported. On TUnit, NUnit and MSTest, put [BobcatScenario] beside it. (The sample project references Bobcat by project rather than by package, so it states [BobcatScenario] itself.) Run dotnet test in a terminal and the run prints the specification it produced:

Feature: Withdrawals
═════════════════════
  a withdrawal larger than the balance is refused narrated OK
  ────────────────────────────────────────────────────────────
    · Given an account with a balance of 100 — declared
    · When  200 is withdrawn — declared
    ✓ Then  the withdrawal is refused and the balance is still 100 (3ms)
      ✓ Then  accepted should be false (1ms)
      ✓ And   account.Balance should be 100 (2ms)

The declared is honest. A marker comment declares a step; nothing observed the Given or the When, so they have no duration and no verdict of their own. The Then is different only because the sample project opts into projecting Shouldly assertions as steps, so the two calls under the comment rendered beneath it with their own verdicts. Without that opt-in the Then would read declared as well, and when a Shouldly assertion threw, the scenario would be red with the exception and nothing narrower: no expected and actual pair, and the second assertion never reached. That is the limit of this style, and also its whole appeal. Opting an existing suite in costs one attribute per class and some comments (Marker comments, Projecting the assertions you already wrote).

Keep the comments, add the helpers

The two styles compose. A comment gives a test its narrative, and a decorated helper called underneath it renders nested under that sentence, with its own timing and verdict (They are different steps, and they nest). Most suites end up with both.

Step 2: a vocabulary

The second withdrawal spec should read like the first, and the tenth like the second. Move the mechanics into step helpers on a class of their own: methods decorated with [Given], [When] or [Then], carrying a sentence whose {placeholders} are filled from the arguments of those names:

private readonly Account _account = new();
private bool _accepted;
[Given("an account with a balance of {balance}")]
internal void AnAccountWithABalanceOf(decimal balance) => _account.Deposit(balance);
[When("{amount} is withdrawn")]
internal void IsWithdrawn(decimal amount) => _accepted = _account.TryWithdraw(amount);
[Then("the withdrawal is {outcome}")]
internal void TheWithdrawalIs(string outcome)
    => SpecAssert.Check("outcome", _accepted ? "accepted" : "refused", outcome);
[Then("the balance is {balance}")]
internal void TheBalanceIs(decimal balance) => SpecAssert.Check("balance", _account.Balance, balance);

The specification is now four calls:

[Fact]
public void a_withdrawal_larger_than_the_balance_is_refused()
{
    _steps.AnAccountWithABalanceOf(100);
    _steps.IsWithdrawn(200);
    _steps.TheWithdrawalIs("refused");
    _steps.TheBalanceIs(100);
}

Every call to a decorated helper is a recorded step: Bobcat’s source generator intercepts the call at build time, so there is no reflection and no registration. Two details carry weight:

• The helpers are internal, not private. The generated interceptor is an extension method, which cannot reach a private member. That one keyword is the whole adoption cost on an existing helper class (BOBCAT039).
• SpecAssert.Check records instead of throwing. It writes an expected/actual cell onto the step in progress and returns, so the specification keeps going and reports every disagreement it has. The adapter fails the test once, at the end, with all of them (Checks that gather instead of throwing).

Now write a specification whose expectations are wrong. A withdrawal of 30 from 100 is accepted and leaves 70, and this spec says otherwise, twice:

[Fact]
public void a_withdrawal_within_the_balance_is_refused()
{
    _steps.AnAccountWithABalanceOf(100);
    _steps.IsWithdrawn(30);
    _steps.TheWithdrawalIs("refused");
    _steps.TheBalanceIs(100);
}

  a withdrawal within the balance is refused FAILED
  ────────────────────────────────────────────────────
    ✓ Given an account with a balance of 100
    ✓ When  30 is withdrawn
    ✗ Then  the withdrawal is expected 'refused', got 'accepted' (1ms)
    ✗ And   the balance is expected '100', got '70' (1ms)
  Failed with Rights: 2, Wrongs: 2, Errors: 0

Both wrongs are there. A cell named for the placeholder it belongs to renders in the sentence, where the value sits, and the test runner’s own failure message says the same thing:

failed Bobcat.Xunit.Samples.Specs.WithdrawalSpecs.a_withdrawal_within_the_balance_is_refused (38ms)
  Xunit.MicrosoftTestingPlatform.XunitException: Bobcat.Engine.SpecAssertionException : 2 specification steps failed:
    Then the withdrawal is refused => outcome: expected 'refused', got 'accepted'
    And the balance is 100 => balance: expected '100', got '70'

A helper that throws is different. An exception is an error, not a wrong, and it stops the scenario: the step is marked !, the exception is printed once at the bottom, and every step after it renders as ○ … — not run with its placeholders unfilled, because its arguments were never evaluated (Report everything, then stop).

Step 3: the boundary as a decision table

The interesting cases of the overdraft rule are the ones at the edge: exactly the balance, a penny over. Four specs of four lines each would say less than one table. A helper that takes a StepTable receives the whole table from the call site, and TableRunner.Run calls a row method once per row:

[Then("withdrawals are decided as")]
internal Task WithdrawalsAreDecidedAs(StepTable table) => TableRunner.Run(this, nameof(decide), table);
private string decide(decimal balance, decimal amount)
{
    var account = new Account();
    account.Deposit(balance);
    return account.TryWithdraw(amount) ? "accepted" : "refused";
}

The balance and amount columns bind to the parameters of those names. The returned value is compared against the one column no parameter names, outcome, so the table is a decision table with a verdict per row:

[Fact]
public Task the_overdraft_boundary() => _steps.WithdrawalsAreDecidedAs("""
    | balance | amount | outcome  |
    | 100     | 99.99  | accepted |
    | 100     | 100    | accepted |
    | 100     | 100.01 | refused  |
    """);

The table is pipe-delimited text, read by an implicit conversion to StepTable. A markdown table pastes in unchanged, alignment row and all, so the table in the pull request and the table in the test are the same text. Give one row the wrong answer and the grid says which:

[Fact]
public Task the_overdraft_boundary_with_a_wrong_row() => _steps.WithdrawalsAreDecidedAs("""
    | balance | amount | outcome  |
    | 100     | 99.99  | accepted |
    | 100     | 100    | refused  |
    | 100     | 100.01 | refused  |
    """);

  the overdraft boundary with a wrong row FAILED
  ─────────────────────────────────────────────────
    ✗ Then  withdrawals are decided as (4ms)
╭───┬─────────┬────────┬────────────────────────────────────┬─────────╮
│ # │ balance │ amount │ outcome                            │ Verdict │
├───┼─────────┼────────┼────────────────────────────────────┼─────────┤
│ 1 │ 100     │ 99.99  │ accepted                           │   OK    │
│ 2 │ 100     │ 100    │ expected 'refused', got 'accepted' │  FAIL   │
│ 3 │ 100     │ 100.01 │ refused                            │   OK    │
╰───┴─────────┴────────┴────────────────────────────────────┴─────────╯
  Failed with Rights: 2, Wrongs: 1, Errors: 0

A row that throws is that row’s failure and the remaining rows still run. out parameters are output columns too, so one row method can judge several results (Decision tables).

Step 4: the history as a set verification

The account’s transaction list is a collection, and a collection claim is a set verification: the expected rows are a table, the system’s rows are matched against them, and every row gets its own verdict.

[Then("the transactions are")]
internal void TheTransactionsAre(StepTable expected)
    => SetVerificationComparer.Verify(_account.Transactions, expected, ordered: true);

[Fact]
public void the_history_records_what_happened()
{
    _steps.AnAccountWithABalanceOf(100);
    _steps.IsWithdrawn(30);
    _steps.IsWithdrawn(200);
    _steps.TheTransactionsAre("""
        | Kind       | Amount |
        | Deposit    | 100    |
        | Withdrawal | 30     |
        """);
}

The refused withdrawal must leave no trace, and the table says so by not listing it. Now a specification that forgets a withdrawal that did happen:

[Fact]
public void the_history_the_specification_did_not_expect()
{
    _steps.AnAccountWithABalanceOf(100);
    _steps.IsWithdrawn(30);
    _steps.TheTransactionsAre("""
        | Kind    | Amount |
        | Deposit | 100    |
        """);
}

  the history the specification did not expect FAILED
  ──────────────────────────────────────────────────────
    ✓ Given an account with a balance of 100
    ✓ When  30 is withdrawn
    ✗ Then  the transactions are (4ms)
╭───┬────────────┬────────┬─────────╮
│ # │ Kind       │ Amount │ Verdict │
├───┼────────────┼────────┼─────────┤
│ 1 │ Deposit    │ 100    │   OK    │
│ 2 │ Withdrawal │ 30     │  EXTRA  │
╰───┴────────────┴────────┴─────────╯
  Failed with Rights: 4, Wrongs: 1, Errors: 0

EXTRA is a row the system produced that the specification does not describe. MISSING is a row the specification describes that the system never produced, shown with the values it expected. FAIL is a matched row with a wrong cell, and ORDER is a row in the wrong place when order was part of the claim. Name keyColumns: to say which columns identify a row, and leave ordered: out when order is not the claim (Set verification).

Step 5: one specification per example

A decision table judges many rows inside one specification, with one verdict. When each example is something a person may want to run, see fail or track on its own, make each row its own specification instead. [BobcatTheory] is xUnit’s [Theory] plus the scenario bracket, and [BobcatExamples] writes its rows as the same table:

[BobcatTheory]
[BobcatExamples("""
    | balance | amount | outcome  |
    | 100     | 100    | accepted |
    | 100     | 100.01 | refused  |
    """)]
public void a_withdrawal_is_decided(decimal balance, decimal amount, string outcome)
{
    _steps.AnAccountWithABalanceOf(balance);
    _steps.IsWithdrawn(amount);
    _steps.TheWithdrawalIs(outcome);
}

Each row runs as its own test with its own identity and its own rendered steps:

  a withdrawal is decided (balance: 100, amount: 100, outcome: accepted) OK
  ────────────────────────────────────────────────────────────
    ✓ Given an account with a balance of 100
    ✓ When  100 is withdrawn
    ✓ Then  the withdrawal is accepted (1ms)

Columns bind to parameters by name, and because the table is a constant, the build checks it: a column naming no parameter or a cell the parameter’s type cannot read is a compile error rather than a surprise at run time (A theory’s examples as a table).

Reading and running

• The run prints its specifications when a terminal is attached and nothing is listening on the monitor wire. BOBCAT_SPEC_CONSOLE=1 prints anyway, into CI’s captured output; BOBCAT_SPEC_CONSOLE_FILE=spec.txt writes the report to a file instead (Reading the specification the run produced).
• bobcat run renders the specification tree in the terminal, and bobcat preview shows every specification without executing anything, because the steps are known at build time (The bobcat tool).
• Your IDE’s test explorer shows each specification as the ordinary test it is, so you run and debug it there (Integrating Bobcat with Your IDE).

Three things to know before you start

• xUnit v3, TUnit, NUnit or MSTest (3.10+). xUnit v2 has no adapter and is not waiting for one. Its BeforeAfterTestAttribute cannot hand over a test result, which is the seam the adapter needs (xUnit v2 is not supported).
• Use the shipped runner adapter, Bobcat.Xunit, Bobcat.TUnit, Bobcat.NUnit or Bobcat.MSTest. Marten’s hand-rolled one never set the verdict, so every scenario it published was a clean pass, red tests included (What you have to add).
• Declared is not executed. A marker comment describes what the test does; it does not make the test do it. The two can drift, and a comment can never carry a cell. When a step is worth a verdict of its own, it is worth a helper (Declared is not executed).