makr.dk

Backup and disaster recovery

For site owners. Kartotek keeps everything it stores — the database, uploaded media, archived Link PDFs and WARCs, and every Published Post's PDF, JSON, and WARC exports — on one persistent disk. That's durable against a bad deploy, but not against losing the disk itself. This doc covers what Kartotek exports for a regular offsite, offline copy, and how to actually fetch and verify one.

What's included

Media, archived Links, and Post PDF/JSON/WARC exports are assembled under one stable folder, /data/kartotek-export/:

kartotek-export/
  media/    — every uploaded/fetched Image, Video, and Audio source file
  links/    — every archived external Link PDF and WARC
  posts/
    pdf/    — the current PDF for every Published Post
    json/   — the current native JSON and JSON-LD exports for every Published Post
    warc/   — every Published Post Version's own WARC capture (permanent per Version)
  manifest.json   — a checksummed index of every file above, plus the database artifacts described below

media/, links/, and posts/ aren't copies — they point at the same files Kartotek already stores and serves, so nothing on the site's disk is duplicated, and neither of those directories is ever versioned. Disposable things (resized image variant caches, logs, build output) are deliberately left out — they're cheap to regenerate and not worth backing up.

The database itself lives in a separate /data/database-exports/ folder, dated directly in the filename: kartotek-<YYYY-MM-DD>.sqlite and kartotek-<YYYY-MM-DD>.json — a native SQLite snapshot, and the same snapshot again as structured, human-readable JSON.

The JSON half is currently suspended. Since 2026-08-26 only the .sqlite snapshot is produced. Building the JSON copy repeatedly exhausted the server's memory and restarted the site, so it is switched off until that is understood and fixed; the Backups page shows a JSON size of nothing while this is the case. Your database is still backed up daily and is still fully restorable — the SQLite snapshot is the complete, authoritative copy, and the restore process uses it rather than the JSON. What is missing is the tool-agnostic, human-diffable second format, not the backup itself. Once per calendar day (UTC), Kartotek builds a fresh pair, validates it (a SQLite integrity check, plus a structural check on the JSON), and only then replaces what was there — the previous day's pair stays completely untouched until the new one has actually passed validation. Once a new pair is in place, any older dated pair is deleted, so only the current day's pair remains on disk at any time. manifest.json reports the current pair's size and checksum (as database/kartotek.sqlite/database/kartotek.json, alongside everything else).

Since the database-backed Admin settings (/_admin/settings) live in this same database, they travel with it automatically — no separate export or restore step, and no extra configuration needed on a different host to carry over a portable setting an admin already configured.

This check runs automatically — at boot and on an hourly interval thereafter — so it doesn't depend on a deploy happening or any request staying open; a day with no deploy at all still gets its own export within an hour of the date turning over. If today's pair already exists and validates, every later check that day is a fast no-op.

The heavy part of the export — verifying the copy, which on a database this size takes about two minutes — runs in a separate background process, so building a backup no longer makes the site slow or unresponsive while it happens. Until 2026-09-03 it did: the daily export blocked the site for roughly two minutes each time, and a page requested in that window could take over a minute and a half to load.

Manual download over SFTP (disaster recovery)

Day-to-day off-site copies are handled automatically by the direct Dropbox backup (see below) — there is no routine manual download step. SFTP remains the direct, manual path onto the server's own disk: useful for a one-off full download, for fetching a copy without involving Dropbox, or as the disaster-recovery fallback if Dropbox itself is ever unavailable. Render — the platform hoejriis.dk runs on — supports downloading files off a paid disk-backed service over SSH/SFTP. Full reference: render.com/docs/disks#transferring-files.

To pull a copy manually:

  1. Connect using a dedicated SSH key added to the Render service (not a key used anywhere else).
  2. Recursively download /data/kartotek-export/ (media, archived Links (PDF and WARC), and Post PDF/JSON/WARC — most SFTP clients follow the directory symlinks inside it automatically; if a particular client can't follow symlinks, download media/, links/, and posts/'s real target directories directly instead) and /data/database-exports/ (the current dated SQLite/JSON pair).
  3. Prefer a client that only transfers new or changed files, so a repeat download after the first is quick.
  4. Store the verified copy wherever you keep offline backups — OneDrive, iCloud Drive, an external disk.

Verifying a download

manifest.json lists every file — including the two database artifacts — with its size, last-modified time, and a SHA-256 checksum, plus the export's timestamp, the deployed Release/commit (when known), and counts of each kind of file. Recompute each downloaded file's checksum yourself and compare it against the manifest. The manifest also lists warnings — for example, a Published Post whose PDF hasn't finished generating yet — worth a second look before you consider that day's copy complete.

Checking backup health without downloading anything

The admin sidebar's Backups section (/_admin/backups) shows the current export date, when it was validated, its file sizes, the most recent attempt and any error, and the same counts and warnings as the manifest. It also has a "Check / run now" button for an out-of-band check, useful right after a change you want reflected sooner than the next automatic check.

Before attempting anything, each check first makes sure there's comfortably enough free disk space for a SQLite copy and a JSON export of the live database (estimated from its current size, so the margin grows along with it); if there isn't, the check fails cleanly with a clear message rather than risking a failure partway through a large write. If a previous check is still running when the next one would otherwise start, the new one is skipped rather than run alongside it.

Getting notified of a failure by email

A failing daily export used to be visible only by opening the Admin Dashboard or the Backups section — easy to miss for a day or more. Now, the moment a check's outcome changes from healthy to failing, Kartotek emails the configured admin address with what failed, the error, and a link back to the Backups section. It won't email again for the same ongoing failure on every later hourly retry — once it recovers (the next check that succeeds), a single follow-up email confirms that. This uses the same outbound mail setup as the admin login link, so it works wherever that already does.

The main Admin Dashboard also surfaces backup health at a glance: Last successful DB export, Files synced to Dropbox, and Files not synced to Dropbox tiles under Operations. A tile moves to Needs attention when the database export is more than 25 hours old (or has never succeeded), or when one or more files have gone unsynced for more than two hours — a newly changed file gets a two-hour grace period before it counts, so an ordinary in-progress sync never falsely reads as a problem. Selecting any of these tiles opens the Backup files matrix (/_admin/backup-matrix, also reachable from the Backup sidebar section) — every file the Dropbox sync tracks, its size, when it was last modified locally and last synced, its current sync state, and any error, filterable and sortable the same way other Admin lists are.

Direct Dropbox backups

This is the ongoing off-site backup: Kartotek pushes a copy directly to your own Dropbox automatically, with no Mac, SFTP client, or manual step involved — the manual SFTP path above stays in reserve for disaster recovery. This covers the complete backup set: the current dated database pair, every media file, every archived Link PDF and WARC, and every Post's PDF/JSON/WARC exports.

Setup

  1. In /_admin/dropbox-backups, click Connect Dropbox. This opens Dropbox's own sign-in/approval page in your browser — Kartotek never sees your Dropbox password, only a token Dropbox issues once you approve.
  2. Once connected, click Choose destination… to browse your Dropbox account from within Kartotek. You can navigate into an existing folder or create a new one, then confirm it as the backup destination. Kartotek checks that it can actually list and upload to the folder before saving your choice.
  3. That's it — uploads happen automatically from then on (see "How it runs," below). Upload now triggers an out-of-band sync immediately, useful right after setup to confirm everything works without waiting for the next automatic run.

This requires two things to already be configured on the Render side before Connect Dropbox does anything — see docs/Internal/architecture.md's Configuration section (or ask whoever manages the Render deployment) if the button reports Dropbox isn't configured yet.

How it runs

Once connected with a destination selected, a sync runs automatically right after each day's database export is ready, and again on its own periodic schedule in between — so ordinary day-to-day media/PDF changes still make it to Dropbox without waiting for the next daily export. A sync only ever adds or updates files in your chosen Dropbox folder, at that file's own exact relative path — never a newly timestamped or renamed copy: a new file uploads, a changed file safely replaces the old copy at the same path, and an unchanged file is skipped. Nothing is ever deleted from Dropbox because a file disappeared from Kartotek's own disk — for example, once the daily rotation removes yesterday's database pair from Render, that same pair stays in Dropbox indefinitely, giving you a longer history there than Kartotek itself ever keeps locally. SQLite's own temporary/journal files and a genuinely empty (zero-byte) capture — for example, an interrupted Link-PDF render — are never uploaded in the first place, so Dropbox only ever accumulates real, finished content.

If an upload fails partway through (a network hiccup, a temporary Dropbox error), the next sync — automatic or manually triggered via Upload now/Retry failed — picks up exactly where it left off; nothing needs to restart from scratch.

Status, and disconnecting

/_admin/dropbox-backups shows the connected account, the selected destination, and the last sync's counts (checked/uploaded/skipped/failed) and any error. Test connection confirms Kartotek can still reach your Dropbox account right now, without doing a full sync. If Dropbox access is ever revoked or expires (e.g. you removed the app's access from Dropbox's own settings), this will show clearly rather than silently failing — reconnect the same way as the original setup.

Disconnect removes Kartotek's stored access to your Dropbox account — it never deletes anything already uploaded there. Changing the destination folder later only affects future uploads; it never moves or deletes what's already at the previous destination.

Cleaning up erroneous artifacts

An earlier version of the Dropbox sync could leave behind two kinds of junk on Dropbox: SQLite's own temporary/journal debris, and zero-byte files from an interrupted Link-PDF capture. Both are excluded going forward (see "How it runs," above), but anything already uploaded before that fix needs a one-time cleanup — Kartotek never deletes from Dropbox on its own, so this is a deliberate, explicitly-confirmed action you run yourself, from a machine with access to the instance's admin session:

  1. BASE_URL=<your instance's URL> npx tsx scripts/dropboxCleanup.ts scan (or npm run dropbox-cleanup -- scan from the repo root) — lists your connected Dropbox destination and reports exactly what it would remove: the file's path, size, and why it was flagged. This changes nothing.
  2. Review the report. Only two kinds of file are ever flagged — temp/journal debris and zero-byte captures — and a real, legitimate dated database export is never touched, regardless of its size. Anything the tool isn't confident about is left alone rather than guessed at.
  3. If you're satisfied, npx tsx scripts/dropboxCleanup.ts delete --confirm --token=<the token the scan printed> actually removes exactly those files from Dropbox. The token is single-use and expires after 10 minutes, and is checked against a fresh re-scan at the moment of deletion — if anything on Dropbox changed since your scan, it refuses and asks you to scan again, rather than deleting against a stale list.
  4. npx tsx scripts/dropboxCleanup.ts run walks through scan → review → confirm interactively in one guided command, if you'd rather not copy the token by hand.

Every scan and delete attempt — including which files failed to delete, if any — is recorded and viewable via npx tsx scripts/dropboxCleanup.ts audit-log.

Restoring from a Dropbox backup

You can always fall back to a manual copy: download the dated kartotek-<date>.sqlite/.json pair, plus the media/, links/, and posts/ folders, from your chosen Dropbox destination — the same relative layout Kartotek's own /data export uses (see "What's included," above).

For a genuine disaster recovery — a fresh Kartotek instance with nothing on it yet — Kartotek also ships a guided restore tool that does this for you directly from Dropbox, with no manual download/upload step:

  1. Deploy a fresh Kartotek instance with persistent storage and the usual environment variables/secrets configured, and let it boot normally (a brand-new instance is never literally empty — it always has its own small starter Category/Post baseline — but has none of your real content yet).
  2. Connect Dropbox the same way as any other instance, via /_admin/dropbox-backups → Connect Dropbox, and confirm it's pointed at the same destination folder your original instance was backing up to.
  3. Run BASE_URL=<your fresh instance's URL> npx tsx scripts/restore.ts run (or npm run restore -- run from the repo root) from a machine with gh-style access to the instance's admin session — see scripts/restore.ts's own header for every subcommand. This lists the dated database snapshots available on Dropbox, lets you preview one (a genuine dry run — downloads and validates everything, checksums against Dropbox's own reported content hash, runs a SQLite integrity check, but changes nothing on the target), and then, once you confirm, actually restores the database, media, archived Link PDFs/WARCs, and every Post's PDF/JSON/WARC export to their expected locations.
  4. Restoring into an instance that already has real content of its own requires an extra, explicit confirmation step (a short-lived token minted by the preview you just ran) — Kartotek refuses an overwrite of a non-empty installation otherwise.
  5. Every run — preview or apply — ends with an integrity report: what was restored, what was missing, what failed validation, and what was skipped. Restart the Kartotek process afterward (the running process doesn't pick up a replaced database file on its own) — the restored database then goes through the exact same idempotent startup migrations any other deploy does, no separate migration step needed.

A restored snapshot that isn't the very latest one still pairs that day's database content with Dropbox's current media/PDF/JSON/WARC files, not a point-in-time set for those — see "Direct Dropbox backups," above, for why (only the database pair is dated; everything else is a single, continuously-overwritten copy). The restore tool calls this out explicitly whenever a non-latest snapshot is selected.

Personal Media Archive and backup/recovery

The Media Archive's catalogue (which Dropbox folders are selected as roots, every catalogued item's metadata and scan state, and any Category/Tags/notes you've added) lives in Kartotek's own primary database, so it's covered by the daily database export and Dropbox backup above like any other data — no separate backup step. Cached thumbnails aren't backed up: they're small, reproducible derivatives (regenerated on demand from Dropbox via each item's Retry action), never an authoritative copy of anything. The original photos themselves are never stored in Kartotek at all — Dropbox remains the one authoritative copy, unaffected by anything covered on this page.

After restoring a database snapshot onto a fresh or different instance, reconnect Dropbox and revisit Media Archive → Roots: a root whose folder no longer exists, has moved, or belongs to a different Dropbox account than the one now connected shows a clear status rather than failing silently, and a plain Rescan reconciles the catalogue against Dropbox's current state (new/changed/moved/missing items resolved the same way an ordinary rescan already handles them) without changing any item's stable archive identity.

What this doesn't cover

Kartotek doesn't manage how long you keep old copies on your own devices, or encrypt anything beyond what Dropbox's own storage already provides. Media, archived Links, and Post PDF/JSON/WARC exports aren't individually dated/versioned on Dropbox (see above) — restoring an older database snapshot doesn't roll those back to match.

View raw markdown