broker / queue / durable delivery

Useful messaging without broker ceremony.

Fibril is a lightweight message broker for durable work queues and fan-out streams. A publish is confirmed once it is on disk. Workers lease messages, then complete, retry or delay them. Broker state stays plain and inspectable during incidents.

Start running Read docs GitHub

Early-stage. Durability is core and tested. Production readiness is still earned.

Publish
Message and enqueue state reach disk
Only then is it confirmed
A worker leases it, then acks or retries

With replication, the confirm also waits until the required replicas persist it.

view

use fibril_client::SubEvent;
use std::time::Duration;

while let SubEvent::Delivery(msg) = sub.recv().await {
    match send_email(msg.text()?).await {
        Ok(()) => msg.complete().await?,
        Err(err) if err.is_temporary() => {
            msg.retry_after(Duration::from_secs(30)).await?
        }
        Err(_) => msg.fail().await?,
    };
}

await foreach (var msg in sub.Deliveries())
{
    try
    {
        await SendEmailAsync(msg.Text());
    }
    catch (EmailException error)
    {
        if (error.IsTemporary)
            msg.RetryAfter(TimeSpan.FromSeconds(30));
        else
            msg.Fail();
        continue;
    }
    msg.Complete();
}

for msg := range sub.Deliveries {
    sendErr := sendEmail(msg.Text())
    var settleErr error
    switch {
    case sendErr == nil:
        settleErr = msg.Complete()
    case isTemporary(sendErr):
        settleErr = msg.RetryAfter(30 * time.Second)
    default:
        settleErr = msg.Fail()
    }
    if settleErr != nil {
        return settleErr
    }
}

for await (const msg of sub) {
  try {
    await sendEmail(msg.text());
  } catch (error) {
    if (isTemporary(error)) {
      await msg.retryAfter(30_000); // milliseconds
    } else {
      await msg.fail();
    }
    continue;
  }
  await msg.complete();
}

async for msg in sub:
    try:
        await send_email(msg.text())
    except TemporaryEmailError:
        await msg.retry_after(30)  # seconds
    except EmailError:
        await msg.fail()
    else:
        await msg.complete()

Worker loops with manual acknowledgements. Email sending and error types are application code.

inside a cluster

Copied before it is confirmed. Recovered after a crash.

Clustering is experimental. These scenes walk through the real protocol one step at a time. Timing is illustrative, not measured.

INSIDE FIBRIL / THE MESSAGE PATHCurrent main

Follow one message. Watch the waits overlap.

Illustrative time units, not benchmark results. The replicated examples require owner + one durable follower. Payload and enqueue-event dependencies both matter. Polling/wakeup and batching delays are simplified.

Follow the story

Read the story, step by step

Durable replication · Current main

A publish enters the owner
The owner must preserve both the payload and its enqueue event.
Local persistence is in progress
Appending and fsyncing establish the owner’s durable boundary.
The follower can read the durable batch
This animation shows the data response. The current transport is follower-pull.
The follower persists its own copy
The payload-bearing queue log must be durable and applied before reporting progress.
Durable progress reaches the owner
The exact payload/enqueue dependency and assignment epoch are checked.
The durability requirement is satisfied
Delivery and publisher confirmation can proceed independently.
The worker acknowledges processing
Processing ACK and publisher confirmation are different signals.

Early replication · Experimental · opt-in setting

A publish enters the owner
The durable confirmation contract stays the same.
Local I/O starts
The owner’s fsync is still pending.
A written batch becomes readable early
The experimental path can begin replica work before local fsync finishes.
Two persistence windows overlap
Overlap can remove serial waiting. Network, batching and progress-report costs remain.
The follower has durable evidence
The owner still verifies both dependencies and its own durability.
Confirm and deliver after the durable boundary
Early replication changes scheduling, not the required durable copies.
The worker sends a processing ACK
This signal does not retroactively replace the durability policy.

INSIDE FIBRIL / FAILOVER & RETURNImplemented · experimental clustering

An owner falls. The queue finds its way.

One enrolled queue, three replicas, majority-durable confirms. The metadata leader survives. Steps are explanatory. Their duration is not a latency prediction.

Follow the story

One enrolled queue, three replicas, majority-durable confirms. The metadata leader survives. Steps are explanatory. Their duration is not a latency prediction.

Read the story, step by step

An owner falls. The queue finds its way.

01 / Every copy has a role
Clients use A for this queue. Followers pull its durable payload and enqueue-event history. C also hosts the metadata controller: a different role from queue ownership.
02 / A stops responding
A stops moving. The client loses its connection, but B has not become the owner yet. A transport failure is evidence to investigate. It is not permission to serve.
03 / The controller proposes a replacement
With eager detection enabled, explicit connection loss and failed reconnect verification can start placement before heartbeat expiry. Silent failures still use expiry. The new assignment remains fenced.
04 / Freeze the evidence
Surviving replicas authorize the exact transition and seal their history. Each source can be inspected after its own seal completes. The worker collects the required evidence before selecting a source.
05 / Find an agreed boundary
An accepted checkpoint can provide a verified base. Recovery compares the required suffix and validates live payloads. A large live backlog still costs work. The diagram does not hide that behind a checkpoint.
06 / Prepare an installed quorum
Compatible retained bytes can be reused after verification. Missing data is copied, state is installed durably, and exact process/storage receipts are collected. B is still fenced.
07 / Activate the exact assignment
Consensus commits the validated replacement. The required quorum and process/storage identities bind this activation. Local admission enables B to serve under epoch 9.
08 / Clients find the new owner
The client refreshes topology and reattaches to B. Delivery and new confirmations resume. Failure detection, worker recovery and client reconnection are separate parts of the outage.
09 / A returns as a learner
A comes back with its old history. It cannot reclaim ownership. It catches up in the background while B continues serving. Learner progress does not count as admitted quorum evidence.
10 / A earns its place again
After verified catch-up and admission, A is a follower. B remains the owner. The recovered topology preserves the confirmation contract rather than silently reducing the required copies.

When evidence is unavailable

05 / Evidence is insufficient
A is down and C cannot provide the required evidence. B stays fenced. The worker can retry transient failures. Contradictory authoritative history needs investigation. There is no unsafe promotion to keep the picture moving.

More scenes in the docs: placement, checkpoints, rebalancing →

crash test

Kill it mid-flight. The message is still there.

fibrilctl prints the offset only after the broker has the message on disk. Kill the broker with SIGKILL, start it again, and the message is waiting. In a cluster the confirm also waits for the required replicas, so a surviving node can take over.

~/fibril-demo

Recorded from fibril-server and fibrilctl built from source, on a scratch data dir with the default config. Log lines are trimmed: no timestamps, thread ids or file paths, module paths cut to their last segment, low-value fields dropped.

node-1 / raft 1 / :9876

node-2 / raft 2 / :9877

node-3 / raft 3 / :9878

client

Recorded with scripts/record-crash-demo.mjs from a local three-node cluster: FIBRIL_COORDINATION_MODE=ganglion, a shared FIBRIL_CLUSTER_SECRET, FIBRIL_COORDINATION_PEERS for raft ids 1 to 3 and FIBRIL_COORDINATION_ASSIGNMENT_DURABILITY=replica_durable:2. The queue was declared first with fibrilctl queue declare. Broker ports as in compose.cluster.example.yaml. Default timeouts: brokers heartbeat every 3 s and drop out of placement after 9 s (coordination.ganglion.liveness_ttl_ms). With eager failover on (runtime_seed.replication.eager_failover), the same scenario activated node-2 about 6 s after the kill. Most of that was electing a new raft leader, since node-1 also led raft. Times are seconds after the kill. Log lines are trimmed: no timestamps or file paths, module paths cut to their last segment, low-value fields dropped.

Durable by default

Queue publish confirmations wait for the message and the state needed to recover it to cross the persistence boundary.

Clients that read like intent

Complete, retry, delay, or fail work without building broker rituals into your app.

Lease under load

Pull-based delivery and bounded prefetch make pressure visible.

Fan-out streams

Plexus streams broadcast each record to every subscriber, alongside the work queues, with durability tiers you opt into.

current shape

Working parts, plainly stated.

The broker has a useful baseline today. The table separates available behavior from areas that still need tooling or client polish.

Read project status →

Feature	Status	Notes
Durable queues	available	Append-only log, snapshot, and replay.
Publish and subscribe	available	Custom TCP protocol with Rust, TypeScript, Python, Go, and C# clients.
Explicit settlement	available	ACK, fail, immediate retry, and delayed retry paths.
Delayed publish	available	Broker path and Rust, TypeScript, Python, Go, and C# client methods are wired.
Message TTL	available	Per-message or per-queue default. Expired messages drop or dead-letter.
Dead lettering	available	Global and per-queue policy are configurable. Replay tooling is still early.
Backpressure	available	Pull-based delivery and bounded prefetch.
Sparse queues	available	Lazy loading and idle eviction are wired. Observability is still growing.
Message inspection	available	Browse active queue messages from admin tooling, with optional payload previews.
Partitioned queues	available	Multiple partitions per queue, with client-side key routing and transparent fan-in.
Plexus streams	available	Fan-out channel: every subscriber receives every record, with durable named cursors and durability tiers you opt into.
Wildcard subscribe	available	Subscribe to every queue or stream matching a wildcard, and auto-attach new ones as they appear. Client-side.
Partition ownership	experimental	Cluster nodes split a queue's active traffic by owning different partitions. Experimental.
Replication	experimental	Followers keep partition copies for failover and recovery. Experimental.
Transactions	out of scope	Not planned. Transactional workflows are intentionally excluded.

Durable queues

available

Append-only log, snapshot, and replay.

Publish and subscribe

available

Custom TCP protocol with Rust, TypeScript, Python, Go, and C# clients.

Explicit settlement

available

ACK, fail, immediate retry, and delayed retry paths.

Delayed publish

available

Broker path and Rust, TypeScript, Python, Go, and C# client methods are wired.

Message TTL

available

Per-message or per-queue default. Expired messages drop or dead-letter.

Dead lettering

available

Global and per-queue policy are configurable. Replay tooling is still early.

Backpressure

available

Pull-based delivery and bounded prefetch.

Sparse queues

available

Lazy loading and idle eviction are wired. Observability is still growing.

Message inspection

available

Browse active queue messages from admin tooling, with optional payload previews.

Partitioned queues

available

Multiple partitions per queue, with client-side key routing and transparent fan-in.

Plexus streams

available

Fan-out channel: every subscriber receives every record, with durable named cursors and durability tiers you opt into.

Wildcard subscribe

available

Subscribe to every queue or stream matching a wildcard, and auto-attach new ones as they appear. Client-side.

Partition ownership

experimental

Cluster nodes split a queue's active traffic by owning different partitions. Experimental.

Replication

experimental

Followers keep partition copies for failover and recovery. Experimental.

Transactions

out of scope

Not planned. Transactional workflows are intentionally excluded.

durable queue throughput

Persisted. Confirmed. Still fast.

By default, a queue publish confirmation means the message and its enqueue state have crossed the persistence boundary through fsync. Replicated durable policies also wait for the required replicas to persist them.

Explore confirmation guarantees → See how durable writes stay fast →

250k+messages/sec ingress

250k+messages/sec egress

Observed on a single node with 1 KiB payloads on a Ryzen 5950X, using the durable queue path. These figures measure ingress and delivery throughput. The historical workload does not await a confirmation for every publish. Workload, results and measurement details →