Handmade PostgreSQL 3/5 — The Storage Engine
Part three of the Handmade PostgreSQL campaign: the database stops living in memory. Pages on disk, a log that is written before the answer is spoken, and a recovery pass that turns a half-finished directory back into a database.
The two commands, unchanged
serve: <command> started as: <command> <port> <datadir>
sql: <command> started as: <command> <port>
The server listens on TCP <port>, keeps its data under <datadir> and runs
until it is killed. The client connects to 127.0.0.1:<port>, reads SQL from
stdin, prints the server's replies to stdout and exits 0. The reply
format was frozen in part one — acknowledgement lines, rows joined by a
single |, ERROR: for anything that fails — and it is not restated here
because it has not moved.
This part continues parts one and two: many clients at once, one shared
database, a session that survives a bad statement, and the query engine you
built — WHERE, ORDER BY, UPDATE, DELETE. The first check re-earns all
of it before anything new is asked of you.
What this part adds to the contract
BEGIN;printsBEGIN,COMMIT;printsCOMMIT,ROLLBACK;printsROLLBACK. The statements between aBEGINand itsCOMMITare one unit — all of them or none of them — andROLLBACKundoes everything since theBEGIN: inserts vanish, updated rows come back exactly as they were. A transaction belongs to the connection that opened it.- A statement outside a transaction autocommits, and it must be durable the
moment its acknowledgement line is printed. The ack is a promise: once
INSERT 1has reached stdout, that row survives anything — including the process dying on the very next byte. - One statement exists only for the test harness:
CRASH;terminates the server process instantly, mid-flight, with no clean shutdown — no flushing of pending work, no tidy save-on-exit. It simulates the power cord leaving the wall, and the next server started on that data directory must recover: everything acknowledged outside a transaction and every committed transaction is there, everything uncommitted is gone.
That is the whole trick the checks play: they cannot see inside your engine, so they kill it and read what is left. Write the intent to a log, flush it, acknowledge, then apply — and every check below passes. Save-on-exit passes none of the ones that matter.
Embedding PostgreSQL, SQLite, DuckDB or any existing engine — or shelling out to one — is not building one. The pages, the log and the recovery are yours.
The ladder
- Set up: re-earn parts one and two (10)
- Data survives a restart (40)
- The schema survives too (20)
- The data directory holds the data (20)
BEGINandCOMMITin one breath (20)ROLLBACKleaves no trace (40)- The acknowledgement is a promise (60)
- Uncommitted work dies with the process (40)
- Recovery replays once, not twice (20)
- Review: how you built the storage (150, judged)
0
Public
Reinvent the Wheel
handmade-postgresql-3-storage
30 min
~24 per session
No
10–150
- database
- storage
- durability
- handmade-postgresql
- campaign
1
Set up the project and re-earn parts one and two
+10 pts per passing check · +10 for completing the task
T 10
pts / check
+10 pts per passing check · +10 for completing the task
Continue the database you built in parts one and two — same repository,
same two commands:serve: started as:
sql: started as:Starting from an empty folder is allowed; your previous work is fetched
for you when the session opens. Either way AGENTS.md (or README.md) has
to carry the three lines the platform captures into session memory:serve:,sql:andtest:. AGENTS.md wins when both files declare
one.Before this part asks for anything new, it re-checks what the earlier
parts earned: a server that boots and answers,CREATE TABLEandINSERTthat acknowledge, and aWHEREand anORDER BYanswered
correctly over a second connection to the same running server.Then the work starts: pages on disk,
BEGIN/COMMIT/ROLLBACK, a
write-ahead log, and recovery from aCRASH;.2
Data survives a restart
+40 pts per passing check · +10 for completing the task
40
pts / check
+40 pts per passing check · +10 for completing the task
The headline rung of this part: rows outlive the process that wrote
them. One server takes a table and some rows into its data directory
and is then stopped; a brand-new server, started on the same port and
the same data directory, reads them back — same values, same order.Where and how the rows live inside the directory is your call: one file
per table, a page file, a log, a heap. What is not your call is the
outcome — data written by a server that has since exited is data the
next server serves, and a second lifetime can keep writing to the same
table without losing what the first one put there.Nothing in memory counts. A process that only saves on exit will pass
this rung and fail the crash rungs later, so it is worth building the
real thing now.3
The schema survives too
+20 pts per passing check · +10 for completing the task
20
pts / check
+20 pts per passing check · +10 for completing the task
Rows are not the only thing on disk. The catalog — which tables exist,
what columns they declare, in what order — belongs in the data
directory as well, and a server that starts on a directory it did not
create must come up knowing exactly what is in it.Both directions are checked. A table created by one server can be
inserted into by the next one without being created again; and a table
nobody ever created is still unknown after a restart, so writing to it
answers with anERROR:line and the session carries on. Recovering
the schema means recovering the absences too.4
The data directory holds the data
+20 pts per passing check · +10 for completing the task
20
pts / check
+20 pts per passing check · +10 for completing the task
The data directory is not decoration. After a server has taken writes
and stopped, that directory contains the database: files with your
pages, your catalog, your log. An engine that keeps everything in the
process and rebuilds from somewhere else has not stored anything.The other half of the same claim is isolation. A server started on a
different, empty data directory is a different database: it shares no
tables, no rows and no memory with the one next door. Nothing may live
beside the binary — no file in the repository root, no cache in the
home directory, no singleton in a temp folder. The directory the server
was handed is the whole world it can see.5
BEGIN and COMMIT in one breath
+20 pts per passing check · +10 for completing the task
20
pts / check
+20 pts per passing check · +10 for completing the task
Implement
BEGIN;andCOMMIT;. Each acknowledges with its own
keyword — the lineBEGIN, the lineCOMMIT. Statements between the
two still print their normal acknowledgements, and a transaction
belongs to the connection that opened it: another connection is not in
it.After the
COMMITthe work is permanent in both senses the checks can
reach. A later connection to the same running server sees the rows, and
so does a server started afterwards on the same data directory.
Statements outside a transaction keep autocommitting exactly as they
always have.6
ROLLBACK leaves no trace
+40 pts per passing check · +10 for completing the task
40
pts / check
+40 pts per passing check · +10 for completing the task
Implement
ROLLBACK;. It acknowledges with the lineROLLBACKand
undoes everything since theBEGIN— not only inserts: anUPDATE
taken back restores the values the rows had before it.The undo has to hold in every place the checks can look: the
SELECT
right after theROLLBACKon the same connection, another connection
to the same running server, and a server started later on the same data
directory. A rolled-back row that reappears after a restart is a
rollback that only happened in memory — which means the undo was never
part of what you wrote to disk.7
The acknowledgement is a promise
+60 pts per passing check · +10 for completing the task
60
pts / check
+60 pts per passing check · +10 for completing the task
Implement
CRASH;— a statement that exists only for the test harness.
It terminates the server process instantly, mid-flight: no flush of
pending work, no tidy save-on-exit, nothing that would not survive the
power cord leaving the wall (abort()is the honest spelling). The
port goes dead with it; the client that sent it may see anything or
nothing.Then make the contract's promise real: a statement outside a
transaction is durable the moment its acknowledgement line is
printed. The check inserts rows, seesINSERT 1on the wire, kills
the server withCRASH;from another connection, starts a new server
on the same data directory and expects everything it was promised.That forces the write-ahead order — append the intent to a log, flush
it to disk, and only then acknowledge. An ack only counts if it reached
the client before the death, so flush the socket as you answer, not at
exit.8
Uncommitted work dies with the process
+40 pts per passing check · +10 for completing the task
40
pts / check
+40 pts per passing check · +10 for completing the task
The other half of the crash bargain: work inside a transaction that
never reached itsCOMMITmust vanish when the server dies. The
acknowledgements printed inside an open transaction were provisional —INSERT 1after aBEGINis a receipt, not a promise; onlyCOMMIT
turns receipts into promises.Here the
CRASH;arrives on the very connection that holds the open
transaction, so the server dies with work in flight. Recovery has to
tell the two cases apart: a transaction that was never committed leaves
nothing behind, and a transaction whoseCOMMITwas acknowledged is
there in full. A log replayer has to know where a transaction started
and whether it ever finished.9
Recovery replays once, not twice
+20 pts per passing check · +10 for completing the task
20
pts / check
+20 pts per passing check · +10 for completing the task
Recovery has to be idempotent. After a crash, every server started on
that data directory recovers — and the second one must see exactly what
the first one saw. A replayer that re-applies the log on every start
without remembering what it already applied inserts every surviving row
twice; a recovery that rewrites the log carelessly loses rows on the
second pass.The check crashes a server holding both autocommitted rows and a
committed transaction, then opens the data directory twice in a row —
two full server lifetimes, one read each. Identical output both times.
Checkpoint after the replay, or make the replay naturally idempotent.10
Review: how you built the storage
Open-ended — a panel of 5 judges splits a 150-pt budget
P
T 150
pt budget
Open-ended — a panel of 5 judges splits a 150-pt budget
The rungs above proved the storage works. This one asks how it is
built, and a panel of judges reads the repository to answer:
architecture, performance, code craft, tests, and the way the project
is run.Nothing new to implement. Spend the time you have left where the panel
looks:- Architecture. Is there a storage layer at all — pages, a buffer,
a log, a catalog — or is durability sprinkled through the executor?
Could the on-disk format change without rewriting the SQL, and could
a new statement be added without touching the log? - Performance. What does an
fsynccost you, how many do you do
per acknowledgement, and what did you decide to buy with them? Does a
read touch the disk every time or a buffer you keep? Say what the
trade-off is instead of hoping nobody asks. - Code quality. Duplication, dead branches, functions that do four
things. The measurement probes report duplication; the judge reads
the rest. - Tests. Is recovery tested, or only the happy path? A suite that
never kills a server mid-write has not tested this part — crash
points, replay, an interrupted transaction, a data directory opened
twice. - Technical governance. Write down the on-disk format: what a page
looks like, what a log record holds, how recovery decides what to
replay, what the catalog stores. A successor should be able to read a
file of yours with your document open. Declare your conventions in
tooling — a formatter, a linter, thetest:command that really
runs.
When you are done, write
.ololo/storage-done.mdwith a short
description of what you built and the decisions you made (at least 10
words).- Architecture. Is there a storage layer at all — pages, a buffer,