Build a Database from Zero (Part 10): Get Data Onto the Disk — Persistence

Database

PostgreSQL

Build from Zero

2 hours ago1199 words·6 min read

What You'll Get Out of This

Take the in-memory machine (heap + index) and make it survive a restart — the moment "a toy" becomes "a database";
Learn a devastating truth the hard way: writing the whole thing to one file is a disaster, once data grows;
See why a real database writes in many small pieces, not one big blob — which sets up Part 11's "pages."

Before We Start

Know Part 9: you have rows (a heap) and an index pointing at TIDs;
Have localStorage / JSON.stringify experience — that's about to look suspiciously like "a naive database."

The Problem With Being All in Memory

Everything so far — the array, the heap, the B+ tree, the TIDs — lives in memory. The instant the program quits, it's gone. That's not a database; that's a spreadsheet that resets.

So the natural next step: save the data somewhere it doesn't evaporate. You reached for the same tool you've used a thousand times.

Step 1: The Naive Attempt — Dump It All to One File

The obvious first try: serialize the whole table to one file, and read it back.

// save: write everything to one file
async function saveAll(table, path) {
  await Deno.writeTextFile(path, JSON.stringify({
    rows: table.rows,
    index: table.index
  }));
}

// load: read it back
async function loadAll(path) {
  const text = await Deno.readTextFile(path);
  return JSON.parse(text);
}

It works — data survives a restart. But hold this up against what you learned in Parts 1–9. There's a deeper problem hiding: you wrote the whole thing.

Step 2: The Devastating Truth — Whole-File Rewrites Are a Disaster

Imagine your table grows to 10 million rows. Now here's what saveAll does, every single time:

txt

write EVERY row to disk, even though you only changed ONE  ← O(n) for a 1-row change
load EVERY row back to memory, even though you only needed ONE

For a toy that's fine. But think about the real world:

Amazon checkout: one UPDATE to change a price. With whole-file rewrites, that single change means writing the entire catalog to disk.
Multiply that by thousands of orders — it's not slow, it's catastrophic.

This is the insight a real database is built around: you must not rewrite the whole table just because one row changed. You need to write only the part that changed. And that demands a smaller unit of storage.

Step 3: The Fix Is a Smaller Unit — Say Hello to the "Page"

The solution is the same idea as the B+ tree in Part 8: break the blob into pieces. A database divides its file into fixed-size chunks, and reads/writes one chunk at a time, not the whole file.

Let's invent that chunk — the page (you already met it in the TL;DR story). A page is a fixed amount of bytes (say a few KB). The heap file becomes a sequence of pages:

txt

heap file on disk:
[ PAGE 0 ][ PAGE 1 ][ PAGE 2 ][ PAGE 3 ][ PAGE 4 ][ ... ]
   rows       rows      rows      rows      rows

Now, UPDATE on a row doesn't rewrite the world — it only rewrites the one page that row lives in.

txt

naive:   write ALL of file (~O(n))   for ANY change
on disk:  write ONLY the affected page (small, fixed) for a change

This is the entire reason databases have pages: to make a localized change a localized write.

Step 4: Wire Up "Read One Page / Write One Page"

Let's give our toy a page layer. Say a page holds up to PAGE_SIZE bytes, and the heap is an array of pages. Reading/writing works on one page:

const PAGE_SIZE = 512;          // tiny, just so the toy is readable
const pages = [];               // each is a chunk of the heap

// read a single page's bytes from the file at a given offset
async function readPage(path, pageIndex) {
  const f = await Deno.open(path, { read: true });
  const bytes = new Uint8Array(PAGE_SIZE);
  const n = await f.read(bytes, { offset: pageIndex * PAGE_SIZE });
  await f.close();
  return bytes;                  // the one page we wanted
}

Notice: we don't slurp the whole file — we skip to the offset of the page we care about, and read exactly PAGE_SIZE bytes. That's the seed of "the database reads in pages."

Step 5: What This Costs You (and Why It's Worth It)

Writing per-page buys you locality, but it reveals a new constraint: a row can't straddle two pages.

txt

[ page i: ...| need 60 bytes but only 30 left |... ]   ← row doesn't fit

So a database keeps track of how much free space each page has, and picks a page where a new row fits. You might recognize this as the free space map (FSM) from PostgreSQL. Not because the toy needed it — because you just discovered why a real one does.

The lesson is the whole point of this part: persistence didn't just mean "survive a restart." It means "write only what changed." And to write only what changed, you need chunks — pages.

Hands-On Practice

Feel the O(n) pain: make saveAll write 1 million rows, then change one row and save again — time it. Now watch what happens if a single UPDATE is that slow in a hot loop.
Write one page: implement writePage(path, pageIndex, bytes) and update a single row by rewriting just its page. Confirm only that region changed.
Think about the free-space map: if a page has 30 bytes left and a new row needs 60, where do you put it? Sketch a tiny free-space map (page → free bytes) and use it to pick a page.

Summary and Keywords

Persistence means surviving a restart — but done naively, it's a whole-file rewrite that's O(n) per change;
The fix: break the file into pages, and read/write one page at a time — localized change = localized write;
A row can't span two pages, so the database keeps a free-space map (page → free bytes) to place new rows;
This is the seed of every "read from disk" story you've heard: the database reads in chunks, not the whole thing.

Keywords: persistence、serialize、write-ahead、page、chunk、free space map (FSM)、locality、random access

What's Next

You now have pages on disk. But notice something: pages are just chunks of bytes — so rough, so low-level, and so unlike the neat rows you first imagined. Almost as if a database needs to understand "a row" as something that lives inside a page, and "a page" as something it reads from disk as a unit.

That's Part 11: we look at what a page actually looks like inside, why the database reads by page and not by row, and why "page size" (8KB in PostgreSQL) is such a famous number. You're about to see the disk story from the ground up.