Skip to content
Fibril

Storage write path

Every queue partition is a Keratin log, and a log is a series of segment files. A durable publish is confirmed once an fsync covers its record, so the cost and timing of those syncs set the latency floor for durable queues. This page animates how a segment is written, synced, rolled over and reused, and the changes made in October 2026 that cut the per-sync cost from 4.9 ms to about 0.7 ms and the rollover stall from about 18 ms to about 1 ms.

Each slab below is one segment file. The lighter shade is data written to the file, the darker shade is data an fsync has made durable. Measured figures come from three brokers sharing one consumer NVMe drive with durable 1 KiB messages, and the fsync times in the bubbles are illustrative values at that scale. The last scene is a proposed design from resource limits.

writtensyncedsegment 7file grows as it is written fsync 4.8 msrecords →
10.0 / 10 s
synced (durable) written, not yet synced reserved, never written zeros written ahead old data in a reused file spare file
Appends move the written frontier, an fsync moves the synced one The writer appends records and the file grows with them. An fsync makes everything written before it durable, so one sync covers every append since the last one. That group commit is why the synced shade jumps in steps. A confirm waits for the sync that covers its record. Every append here extends the file, so each sync also waits for the filesystem to record the new blocks in its journal. Measured at 1k msg/s: each owner fsync took a flat 4.9 ms and carried about 4 records, set by those journal commits.
Transcript of every scene
  1. Appends move the written frontier, an fsync moves the synced one. The writer appends records and the file grows with them. An fsync makes everything written before it durable, so one sync covers every append since the last one. That group commit is why the synced shade jumps in steps. A confirm waits for the sync that covers its record. Every append here extends the file, so each sync also waits for the filesystem to record the new blocks in its journal. Measured at 1k msg/s: each owner fsync took a flat 4.9 ms and carried about 4 records, set by those journal commits.
  2. Rolling to a new file used to stall the writer. When a segment fills, the writer opens the next one. It used to do every step inline: sync the outgoing segment, create the file, write and sync its header, sync the directory so the new name is durable, then write and sync the sealed index. Appends waited behind all of it. Measured: owner rollover held the writer for about 18 ms.
  3. Reserving the file up front did not make syncs cheaper. Preallocating the segment reserves its full size at once, so the file stops growing. The space is reserved but never written, and the filesystem still records each block the first time it is written, with a journal commit. The slab is full size but hollow, and the fsync cost stays where it was. fallocate alone left the per-sync floor unchanged.
  4. A background worker writes zeros ahead of the writer. One shared worker writes and syncs zeros into the reserved space ahead of each log's writer. When the writer gets there, the blocks already exist on disk, so its fsync only flushes data. The lead is a sawtooth: when it drops below half the window, the worker zeroes 4 MiB chunks until it is a full window ahead again, never further. The window is the preallocation size, 8 MiB by default, so the lead stays between about 4 and 8 MiB. Zeroing starts once a segment holds 64 KiB, all segments share a 64 MiB/s cap, and a writer that overtakes its zeros turns zeroing off for that segment, since at that rate it would only double the disk traffic. Measured: owner fsync p50 4.9 → 0.73 ms. At 1k msg/s end to end, p50 9.5 → 2.6–3.6 ms and p99 28–30 → 13–16 ms.
  5. Rollover hands its syncs to the next fsync. The writer now only switches files and keeps appending. The outgoing segment's last data, the new file's header and the directory entry are synced by the next fsync, together with the new appends. No record in the new segment counts as durable before that sync, and recovery drops a new segment whose header never made it to disk. Measured: owner roll time 18 → 1 ms, throughput +2–5%.
  6. Consumed segments become spares and come back at the next roll. Once every message in a segment is acknowledged, the log truncates it off the writer's path. A freed file is kept as a spare, up to 4 per log, instead of being deleted. At the next roll the writer renames a spare into place. Its blocks were all written before, so syncs stay cheap and the zero worker only has to run past the file's old end. Reused files skip both the journal growth and the zeroing for the length they already had.
  7. One spare pool per node, safe across logs (proposed). Spares would move between logs through one pool per filesystem. A reused file can hold old records from another queue whose offsets happen to fit, so each file carries a generation number in its header that grows on every use, and every record is stamped with it. Recovery accepts only records that match the header. When a file enters the pool, its header is rewritten with the next generation and synced in the background, before any log can take it. Proposed design, part of the resource limits work. No zeroing and no extra syncs on the write path.

Where the numbers come from

Section titled “Where the numbers come from”

The measurements, commits and conditions behind each scene are rows in the optimization log. The tuning ledger puts these changes beside the ones that did not pay off, with the trade-offs of each.