Skip to content
Fibril

Tuning ledger

Every change here was measured against the build before it. Some were clear wins, some became settings because they trade one thing for another, and some lost and were removed. Each chart starts at zero, with the earlier build in grey and the change in colour: green where it helped, red where it cost.

Unless an entry says otherwise, numbers come from three brokers sharing one consumer NVMe drive with durable 1 KiB messages, two or more runs per arm alternating, and ranges are run to run. Dates are when the measurement was taken. The optimization log holds the full rows with commits and conditions, and the storage write path animates the storage changes.

kept2026-10-25

Zero-ahead for preallocated segments

A shared worker writes and syncs zeros a little ahead of each writer, so syncs no longer wait for the filesystem to record new blocks.

owner fsync p50 4.9 → 0.73 ms 0 1k msg/s p50 9.4–9.5 → 2.6–3.6 ms 0 1k msg/s CPU per message 100 → 135 % of before 0
trade-off

More and smaller fsyncs raised CPU per message at 1k msg/s by about 35%. Zeroing in 1 MiB chunks pushed p99 to 34–38 ms, so it zeroes in 4 MiB chunks. Its writes share the drive, so mid-size bursts gained about 3 ms of p99.

verdict

Merged. The low-rate floor was the biggest latency cost left, and the CPU rise is partly won back by loaded fsync spacing.

kept2026-10-25

Rollover off the writer, truncate off the writer, spare reuse

Rollover syncs only the outgoing data and hands the header, directory and index syncs to the next fsync. Truncation runs on the manifest worker, and up to 4 freed segments per log are reused.

10k msg/s p50 30 → 9 ms 0 10k msg/s p99 83–95 → 28–33 ms 0 CPU per million messages 54 → 68 s 0
trade-off

Cheaper syncs made fsync groups smaller, about 3.3 times more fsync jobs per message, so CPU per message rose about 25%.

verdict

Merged. Tail latency fell to a third, and later spacing work targets the CPU.

kept2026-10-25

Loaded fsync spacing

Under load, a new fsync waits up to 2 ms after the previous one so more records share it. Idle logs sync at once. It is the runtime setting storage.loaded_fsync_spacing_ms.

fixed 3 ms floor, 10k msg/s CPU per million 60–63 → 47–49 s 0 fixed 3 ms floor, 100 msg/s p50 5.8–7.4 → 8–8.1 ms 0
trade-off

A fixed floor saved CPU under load but added latency at low rates, as the second chart shows. Applying it only under load keeps the saving and drops that cost.

verdict

Merged as a runtime setting, 2 ms by default. With it Fibril led RabbitMQ at 100 and 1k msg/s in the same harness.

kept2026-10-25

Raw replication bytes

The owner sends stored records to followers as they are on disk, and the follower decodes once, instead of decoding and re-encoding on the owner.

1 KiB throughput 100 → 105 % of before 0 4 KiB CPU per message 100 → 92 % of before 0
trade-off

None measured. It changes the replication frame contents, which both sides of a rolling upgrade must understand.

verdict

Merged.

kept2026-10-26

Tail cache on the owner only, 16 MiB

Followers stopped filling a read cache they never read from, and the owner's budget dropped to 16 MiB.

follower memory at 10k msg/s 184–187 → 105–109 MiB 0 saturation throughput 142–143 → 142 k msg/s 0
trade-off

A consumer more than 16 MiB behind reads from disk instead of memory. 4 MiB measured the same, and 16 MiB keeps headroom for lagging readers.

verdict

Merged.

kept2026-10-26

Smaller writer buffers

The default writer buffer factor dropped from 128 to 32, which still leaves 2,048 channel slots per log for bursts.

memory per node, 200 idle queues 940–959 → 744–774 MiB 0
trade-off

Factor 8 saved another 80 MiB, at a quarter of the burst headroom. Saturation and p99 did not move between 8 and 128.

verdict

Merged at 32, a balance between idle memory and bursts.

kept2026-10-26

Fsync and notify threads on demand

Each log starts its fsync and notify threads when it needs them and retires them after 10 s unused.

threads per node, 200 idle queues 644 → 240 0 10k msg/s p99 35.9–37.7 → 35.4–35.9 ms 0
trade-off

None measured in memory, CPU or latency. Reading the fsync loop for this change also found that merged fsync groups skipped a roll's deferred syncs, now fixed.

verdict

Merged.

kept2026-10-20

One publisher sink per partition

Every publisher handle of a partition shares one sink and confirm loop instead of one per connection.

CPU per million, 16 conns, durable 10k/s 214 → 113 s 0 durable saturation 114 → 120 k msg/s 0
trade-off

Tighter coalescing timers cut buffered 10k msg/s p50 from about 2.5 to 1.4–1.6 ms, for 12–25% more CPU, and plain yields lost 7% durable saturation. Those variants stayed on a branch.

verdict

Merged with the ordinary Tokio timer.

kept2026-10-27

mimalloc v2 instead of v3

Moved off mimalloc 3.3.2, which has open reports of rare multi-threaded crashes, to the long-stable 2.3.2.

memory per node, 200 idle queues 831–885 → 439–494 MiB 0 saturation throughput 135–138 → 132–132 k msg/s 0
trade-off

About 2–3% less peak throughput and a few ms more p99 at 10k msg/s.

verdict

Merged for reliability, with a large idle memory gain.

setting2026-10-24

Publish batching gap mode

A batch of publishes closes after 100 µs without a new request, on a microsecond timer.

buffered p50 at 10k msg/s, 16 connections 0.99 → 0.37 ms 0 buffered CPU per message 100 → 171 % of before 0
trade-off

Durable publishes gained nothing and paid up to 31% more CPU at 10k msg/s, with a wider p99 at 50k msg/s.

verdict

A per-durability setting, on for buffered publishes only.

setting2026-09-27

Separate follower push credit

Gives the follower push stream its own credit window instead of the per-read byte budget.

4 KiB saturation 100 → 133 % of before 0
trade-off

No gain at 1 KiB, and more CPU and memory per message at 4 KiB.

verdict

An off-by-default runtime setting, to retest on single-log queues.

not merged2026-10-26

Push frames built from smaller reads

Fill each owner push frame from 256 KiB or 1 MiB reads with a yield between them, to shorten long task polls.

1 KiB long-poll time per 60 s 7.6 → 1.5 s 0 4 KiB throughput 42.4–42.7 → 41–41.1 k msg/s 0
trade-off

At 4 KiB the follower waits on the owner, so a slower frame build costs throughput.

verdict

Not merged. The poll gain did not show up in latency.

not merged2026-10-26

Pipelined push frames

Read and encode the next push frame while waiting for credit.

1 KiB throughput 141–142 → 137–139 k msg/s 0
trade-off

It removed most of the follower's idle gap at 4 KiB without raising 4 KiB throughput, and the read-ahead cost the owner at 1 KiB.

verdict

Not merged. The 4 KiB limit sits in the credit window and the follower apply.

not merged2026-09-26

Batched follower apply

Apply several replicated batches per follower actor step.

split-drive saturation 100 → 110 % of before 0
trade-off

Paced latency got worse.

verdict

Not merged. A throughput gain that taxes every paced workload is not a default.

not merged2026-09-20

Speculative delivery

Deliver before durability under an explicit contract, confirming once the message is durable or processed.

tmpfs saturation 100 → 90–92 % of before 0
trade-off

Confirm and delivery p99 fell, but it needs its correctness gates on single-log queues, and benchmarks with instant consumers overstate it.

verdict

Kept on its branch.

setting2026-10-13

Buffered queues (AfterFlush)

A queue can confirm and deliver once every required copy has written the message to the operating system, before the fsync. A background sync makes it durable, every 100 ms by default.

10k msg/s p99, no sync vs 100 ms sync 96.6–110 → 11.2–11.7 ms 0
trade-off

A weaker guarantee: power loss or a storage failure on every required node at once can lose confirmed messages from the last sync window. The first broker attempt in September stalled, because replication and delivery read only durable records, and without a background sync it had a repeatable tail and CPU regression.

verdict

Merged on 13 October as an experimental per-queue mode, once Keratin tracked written and durable progress separately and replication carried both. Durable stays the default, and a queue's mode cannot be changed after its first declare yet.

not merged2026-09-22

A much larger tail cache

Raised the read cache from 64 MiB to 4 GiB.

broker memory 0.44 → 4.87 GiB 0 delivery throughput 419–422 → 419–422 k msg/s 0
trade-off

Cache hits reached 100% and delivery did not move.

verdict

Default unchanged. Reads were never the limit.

removed2026-10-25

Stage-then-commit

Stage writer input immediately in bounded chunks and decide fsync submission after staging.

1k msg/s CPU per million 439–477 → 965–1,057 s 0 1k msg/s p99 16–17 → 25–31 ms 0
trade-off

It had won on an older build. Against current main it cost about twice the CPU and a worse low-rate p99 for about 1% at saturation.

verdict

Removed, along with its switch.

removed2026-09-27

Close a batch when writer input drains

End the pending append batch as soon as the writer's input queue is empty.

8 KiB median delivery 86–87 → 72–73 ms 0 fsync jobs 100 → 118 % of before 0
trade-off

About 11% more CPU, no early closures at 1 KiB, and later no effect at 100 or 1k msg/s.

verdict

Removed with the other storage switches.

pending2026-10-27

jemalloc instead of mimalloc v2

An allocator comparison of mimalloc v2, jemalloc, snmalloc and tcmalloc on the same build.

memory per node, 200 idle queues 395–513 → 296–322 MiB 0 saturation throughput 131–135 → 139–139 k msg/s 0
trade-off

tcmalloc used the least memory but more idle CPU and comes from a small wrapper. snmalloc used the most idle memory.

verdict

On a branch, waiting on longer stability soaks before it replaces mimalloc v2.