Handmade PostgreSQL 4/5 — Indexes and Plans

Part four of the Handmade PostgreSQL campaign: the database gets fast, and learns to say how. Parts one through three gave you a server, an engine and the statements that read and change rows. This part adds the two things that separate a table scan from a database — an index, and a plan that admits which one it used.

The two commands, unchanged

serve: <command>     started as:  <command> <port> <datadir>
sql:   <command>     started as:  <command> <port>

The server binds TCP <port>, keeps its data under <datadir>, serves many clients at once and runs until killed. The client connects to 127.0.0.1:<port>, reads SQL from stdin, prints the replies to stdout and exits 0 — and exits non-zero with an ERROR: line when nothing is listening.

The reply format was frozen in part one and is not restated here: one acknowledgement line per statement, SELECT prints its rows |-joined first, a failure prints ERROR: <text> and the session carries on. Everything parts one through three earned is re-checked on the first rung, so a codebase carried over honestly pays from the first probe.

The bulk loader on the last rungs sends many rows per statement — INSERT INTO t VALUES (…),(…),(…); — and reads back the single INSERT <n> the frozen contract already promises. That is not a new statement, only the shape of one you already reply to.

What this part adds to the contract

Two additions, and nothing else:

CREATE INDEX <name> ON <table> (<column>); replies with exactly CREATE INDEX. Index names are unique per database: creating one that already exists is an ERROR: line, and the session continues. An index survives a restart, like everything else under <datadir>.

EXPLAIN <select>; prints the plan it would run: one line per plan node, outermost node first, each line starting with the node name in capitals — SEQ SCAN, INDEX SCAN, FILTER, SORT, LIMIT, AGGREGATE — followed by a space and whatever detail you want to give. After the plan lines comes the trailer EXPLAIN <n>, where n is the number of plan lines printed. Only the leading node keyword is graded; the detail after it is yours to design.

EXPLAIN SELECT * FROM t WHERE k = 42;
INDEX SCAN t using ix_t_k (k = 42)
EXPLAIN 1

The plan has to be the truth. When an index covers the column a query filters on by equality, the plan says INDEX SCAN and the lookup really goes through the index; when no index covers it, the plan says SEQ SCAN. A planner that prints INDEX SCAN unconditionally is graded the same as one that prints it never — the rungs ask both questions of the same table.

Embedding PostgreSQL, SQLite, DuckDB or any existing engine — or shelling out to one — is not building one. The index structure, the planner and the storage are yours.

The ladder

  • Set up: carry parts one to three forward (10)
  • CREATE INDEX, twice is an error, and it survives a restart (20)
  • EXPLAIN a sequential scan (20)
  • The planner picks the index — and knows when not to (40)
  • Indexed lookups are exact, before and after the index (40)
  • The index stays honest through UPDATE and DELETE (40)
  • Range scans over an index (40)
  • A hundred thousand rows and a hundred lookups (60)
  • Eight readers and one writer at the same time (40)
  • Review: how you made it fast (150, judged)
Sessions

0

Visibility

Public

Category

Reinvent the Wheel

Slug

handmade-postgresql-4-indexes

Duration

30 min

Judge reviews

~24 per session

Active session

No

Points

10–150

Tags
  • database
  • indexes
  • performance
  • handmade-postgresql
  • campaign
  • 1

    Set up the project and carry parts one to three forward

    10

    pts / check

    +10 pts per passing check · +10 for completing the task

    Same two commands as the rest of the campaign, same declarations:

    serve: started as:
    sql: started as:

    Start in the folder where part three was built, or in an empty one with
    your previous work fetched for you. Either way, an AGENTS.md (or
    README.md) documents the stack and carries the three lines the platform
    captures into session memory: serve:, sql: and test:. AGENTS.md
    wins when both files declare one.

    This rung re-checks what parts one through three already earned — the
    server binds, rows go in, the data is on disk and comes back after a
    restart, WHERE filters and ORDER BY sorts. Nothing new. If this rung
    pays, the codebase you carried over is intact and the index work can
    start on solid ground.

    Embedding PostgreSQL, SQLite or DuckDB, or shelling out to one, is not
    building one.

  • 2

    CREATE INDEX

    20

    pts / check

    +20 pts per passing check · +10 for completing the task

    The first addition to the contract:

    CREATE INDEX ON (); -> CREATE INDEX

    One acknowledgement line, spelled exactly CREATE INDEX. Index names are
    unique per database, so a second CREATE INDEX under a name already taken
    is a failure: an ERROR: line, and the session carries on as always.

    An index is data, not a runtime decision. It lives under <datadir>
    alongside the rows, so stopping the server and starting it again on the
    same directory finds it still there — still refusing its own name, still
    answering the queries it covers with exactly the same rows.

    Nothing here says the index has to be fast yet, or that anything has to use
    it. It has to exist, be acknowledged, be unique, and survive.

  • 3

    EXPLAIN a sequential scan

    20

    pts / check

    +20 pts per passing check · +10 for completing the task

    The second addition to the contract. EXPLAIN <select>; does not run the
    query — it prints the plan, one line per plan node, outermost node first:

    EXPLAIN SELECT * FROM t;
    SEQ SCAN t (4 rows)
    EXPLAIN 1

    Each plan line starts with the node name in capitals — SEQ SCAN,
    INDEX SCAN, FILTER, SORT, LIMIT, AGGREGATE — then a space, then
    whatever detail you find useful. Only the leading keyword is graded; the
    rest of the line is your design. After the plan lines comes the trailer
    EXPLAIN <n>, where n is the number of plan lines you just printed.

    This rung asks the easy case: a table with no index at all. There is only
    one honest way to read every row of it, and the plan has to say so.

  • 4

    The planner picks the index

    40

    pts / check

    +40 pts per passing check · +10 for completing the task

    One table, two integer columns, an index on exactly one of them. Two
    queries with the same shape:

    EXPLAIN SELECT * FROM t WHERE k = 42; -- k is indexed
    EXPLAIN SELECT * FROM t WHERE j = 42; -- j is not

    The first plan has to contain an INDEX SCAN line. The second has to
    contain a SEQ SCAN line and no INDEX SCAN at all. Same table, same
    statement shape, same run — so the answer cannot come from a template. The
    planner has to look at what indexes exist and decide.

    The rule this part grades is simple and absolute: when an index covers the
    column an equality filter names, the plan is an INDEX SCAN; when nothing
    covers it, the plan is a SEQ SCAN. Whatever else your plan prints —
    FILTER, SORT, cost estimates, row estimates — is yours, and only the
    leading keyword of each line is read. The trailer EXPLAIN <n> still has
    to count the plan lines you printed.

  • 5

    Indexed lookups are exact

    40

    pts / check

    +40 pts per passing check · +10 for completing the task

    A fast wrong answer is worth nothing. Now that the planner reaches for the
    index, the index has to give back exactly what a full scan would have.

    Three questions of the same table:

    • a key that was already there when the index was built;
    • a key inserted after the index was built — an index is not a
      snapshot, it is maintained on every write;
    • a key that is not in the table at all, which is the single line
      SELECT 0 and not an error.

    Row shape and reply format are the frozen ones from part one: values
    |-joined, then SELECT <n>.

  • 6

    The index stays honest through UPDATE and DELETE

    40

    pts / check

    +40 pts per passing check · +10 for completing the task

    An index that is only correct until somebody writes is a cache, not an
    index. Rows move and rows vanish, and every entry pointing at them has to
    move or vanish with them — in the same statement, not on the next restart.

    Two writes and four questions:

    • UPDATE an indexed key to a new value. The old key must now find
      nothing (SELECT 0), and the new key must find the row with all of its
      other columns intact.
    • DELETE a row by its indexed key. That key must now find nothing.
    • A key nobody touched must still find its row, unchanged.
    • And a full SELECT ... ORDER BY k must return exactly the surviving
      rows — an index that quietly drops a row from the table is worse than
      no index.

    UPDATE and DELETE reply with their row counts, as frozen in part one.

  • 7

    Range scans over an index

    40

    pts / check

    +40 pts per passing check · +10 for completing the task

    Point lookups are the easy half. An index earns its keep on ranges:

    SELECT * FROM t WHERE k > 100 AND k < 400 ORDER BY k;

    Three ranges over the same indexed column, all of them with an explicit
    ORDER BY so the comparison is exact:

    • a wide one that catches several rows;
    • a narrow one that catches exactly one;
    • one whose bounds sit either side of nothing at all, which is SELECT 0.

    Both bounds are strict. The endpoints themselves are real keys in the
    table and must not come back — an off-by-one at the edge of a range is
    the classic index bug, and it is exactly what is being asked here.

    Whether you walk the index in order or scan and sort is your decision to
    make and to defend in the review; what is graded here is that the rows
    come back, all of them, only them, in ascending key order.

  • 8

    A hundred thousand rows

    60

    pts / check

    +60 pts per passing check · +10 for completing the task

    Everything so far fits in a handful of rows, where a linear scan and a
    B-tree are indistinguishable. This rung is the one that tells them apart.

    A loader pushes roughly a hundred thousand rows into one table, batched —
    many rows per statement, INSERT INTO t VALUES (…),(…),(…);, answered by
    the single INSERT <n> the contract already promises. Then an index goes
    on the key column, and a second client connects and asks about a
    hundred random keys, one point lookup each. Every answer has to be exactly
    right; a single wrong or missing row fails the rung.

    The check will not time your algorithm to the millisecond, but it will
    notice a database that grinds. There is a generous ceiling on the whole
    probe — load, index build and a hundred lookups together — and a design
    that re-reads the table from disk for every lookup, or re-parses the whole
    datadir on every statement, will not fit inside it. A design with a real
    index and a sane write path will fit with room to spare.

    Worth thinking about before you start: how many times does one row get
    copied on its way in, how big is one index node, and does a point lookup
    touch the table file once or once per row.

  • 9

    Eight readers and one writer

    40

    pts / check

    +40 pts per passing check · +10 for completing the task

    Part one proved the server can hold eight connections open. This rung asks
    what those eight connections do to an index.

    Eight clients connect at once and each runs its own set of indexed point
    lookups against a loaded table, while a ninth client inserts new rows into
    that same table for the whole duration. Every reader has to come away with
    its own correct answers: no crash, no truncated reply, no rows from one
    lookup arriving on another connection, and nobody left waiting behind a
    lock until the deadline expires.

    The writer never touches the keys the readers ask about, so there is one
    right answer for every lookup regardless of interleaving. What is graded
    is that a shared index under concurrent read and write stays a correct
    index — whatever you use to get there, a lock, a reader-writer split,
    copy-on-write nodes or a single-threaded queue that is simply fast enough.

  • 10

    Review: how you made it fast

    150

    pt budget

    Open-ended — a panel of 5 judges splits a 150-pt budget

    The rungs above proved the index is correct and the planner is honest.
    This one asks what is underneath, and a panel of judges reads the
    repository to answer: architecture, performance, code craft, tests, and
    the way the project is run. Performance carries the most weight in this
    part — it is the part where the database stopped being a toy.

    Nothing new to implement. Spend the time you have left where the panel
    looks:

    • The index structure. What is it — a B-tree, a hash table, a sorted
      array rebuilt on write, a skip list? What is the cost curve of a point
      lookup, a range scan and an insert in that structure, and where does it
      stop being acceptable? An index that is a linear scan wearing a hat is
      a finding, not a design.
    • Performance. Where does the time actually go on a hundred thousand
      rows — parsing, allocation, syscalls, the write path? What is a page,
      how big, and how many of them does one lookup touch? What did you do
      about the load that you would not have done for four rows?
    • Is the planner a decision or a hardcode? A planner that pattern
      matches on the word WHERE and prints INDEX SCAN passes two probes
      and fails the review. Show where the choice is made, what it looks at,
      and what would have to change to add a second index or a second rule.
    • What you measured. Numbers, not adjectives. A benchmark, a timing
      harness, a row-count-versus-milliseconds table in the notes — anything
      that shows you looked instead of guessed, and anything that shows what
      you tried and rejected.
    • Tests. Do they cover the index under writes, the empty range, the
      key that is not there? A suite that only tests happy-path lookups has
      not tested an index.
    • Technical governance. Write down the decisions a successor would
      otherwise have to guess: the index structure and why, what the planner
      is allowed to assume, the on-disk layout. Declare your conventions in
      tooling — a formatter, a linter, the test: command that really runs.

    When you are done, write .ololo/indexes-done.md with a short
    description of what you built and the decisions you made (at least 10
    words).