Links and web preservation

For administrators. How Kartotek preserves a Post's own page and every external Link it carries, checks whether a Link's destination is still reachable, and offers visitors an honest fallback when it isn't. See Welcome to Kartotek for the vocabulary this doc assumes, and Link for the Link Content Element's own fields and inline rendering.

Archival

Kartotek independently preserves content in several ways, all automatic and all asynchronous — publishing or saving is never delayed waiting on any of them. Wayback submission is queued only once at least one Domain is configured (see Domains, UIDs and addresses) — until one is, automatic Wayback preservation is disabled and reported as such in the Admin sidebar, though any capture already made stays available. Local WARC capture doesn't depend on a configured Domain at all.

A Post's own page, on every associated Domain. Every time a Published Post gets a new Version, its own public page is submitted to the Internet Archive Wayback Machine for a snapshot — separately, on every Domain configured on this deployment, not only the Primary one, since a Kartotek installation's content is shared identically across every configured Domain. Beyond the initial publish submission, each (Post, Domain) pairing is automatically resubmitted roughly every three months, keeping the Wayback snapshot current for content that's still up. Nothing is submitted for a PrePub, Hidden, or Private Post.

External Link URLs. Every external Link added to a Post, or whose destination changes, is independently preserved the moment it's saved — including for a PrePub or Hidden Post, since only the already-public external site is touched. Three independent actions are queued: the URL is submitted to the Wayback Machine, the linked page is rendered to a local Kartotek PDF snapshot, and the linked page (plus the resources needed to replay it — stylesheets, scripts, images, and similar) is captured to a local, standards-compatible WARC file. Each addition of a Link is captured separately — the same URL used twice gets three independent sets of captures.

A Post's own WARC. Alongside its PDF and native JSON export (see Posts and publishing and Import, export and external publishing), every eligible Published Post Version also gets a local, standards-compatible WARC capture of its own canonical page — including same-origin stylesheets, scripts, images, and other resources needed to replay it faithfully offline in a standards-compatible archive-replay tool (e.g. ReplayWeb.page). Unlike the PDF and native JSON exports, which always reflect only the Post's current live Version, a Post's WARC is permanent per Version — publishing a later Version creates its own new WARC without ever touching or replacing an earlier Version's own capture, at that Version's own permanent address (domain/<uid>/v/<n>, see Domains, UIDs and addresses).

Every capture is bounded and safe. A capture (Post or Link) never follows a live page's own outbound links beyond the resources needed to render it, is bounded by a time limit, a maximum number of resources, and a maximum total size, and never reaches into any private/internal network address — a resource that would require that is simply skipped and recorded as such rather than fetched.

Link health and archive fallback

Beyond the one-time captures above, Kartotek periodically checks whether each Link's live destination is still actually reachable, and offers a visitor an honest fallback when it isn't.

Checking, not scanning. A Link's live status is checked in the background on a bounded schedule (never triggered by a visitor simply loading a page) and cached for 24 hours. From a Link's own standalone page, anyone can click Check now for an up-to-the-minute result — limited to once every 10 minutes per Link to keep this from becoming a way to hammer either the target site or Kartotek itself.

Six honest states, never collapsed into a generic "broken":

• Healthy — the live destination works.
• Restricted — it exists but requires authentication Kartotek doesn't have.
• Limited/temporary — a timeout, rate-limit, or server error; may well recover on its own.
• Missing — the destination genuinely returned "not found" or "gone."
• Invalid/unsafe — not a supported or safe address to check at all.
• Unknown — never checked yet, or the check itself couldn't reach a conclusion (e.g. a DNS failure).

When a Link is Missing, Kartotek separately checks whether a Web Archive already has an accessible copy of it (independent of whether Kartotek's own Wayback submission above ever succeeded), and records that too. This checks the Wayback Machine, Arquivo.pt, and Common Crawl — Wayback is preferred whenever it has a usable capture; otherwise the best available capture from another provider is used. A provider that's temporarily unreachable or rate-limited never blocks the others or erases a capture already confirmed from a previous check.

What a visitor sees: a Healthy destination is always the primary, plain link, marked with a small green status icon. Once it's anything else, if a Kartotek-captured PDF exists it's offered as a clearly labelled Archived copy, with its capture date and an explicit note that it's historical — preferred over a Web Archive snapshot when both exist, though a Web Archive snapshot (when one exists) is always still reachable from the Link's own standalone page, which lists every confirmed provider's capture, Wayback Machine first whenever it has one. If neither exists, the destination is described honestly as currently unreachable rather than hidden or silently left as a dead link. Every rendered Link — not only a Missing one — carries this same compact coloured status icon (a letter in a circle, with an accessible hover label) immediately after it, opening that Link's own standalone page when clicked; see Link for the full colour/letter mapping. A destination that recovers to Healthy later goes right back to being the primary link — any archive stays reachable from the standalone page regardless. That standalone page also lists the Link's own local WARC capture (alongside its PDF sibling) whenever one exists.

HTTPS upgrade. For a Link whose stored destination is http://, an HTTPS check runs alongside the ordinary health check, against the https:// equivalent of that same address — a Link already stored as https://, or one whose HTTPS equivalent isn't reachable, simply shows nothing here. When the HTTPS equivalent is reachable, an Upgrade to HTTPS action re-verifies it's still reachable at the moment it's clicked and, if so, updates the Link to the https:// address. This never happens automatically — a Link only ever moves to https:// when explicitly requested, or when the Link evidence backfill (see "Admin visibility," below) applies the identical, freshly-re-verified rule on your behalf.

Archived copies and History. The Link's own standalone page goes further than the single preferred snapshot above: an Archived copies section lists every individually confirmed Web Archive capture across every checked provider — not just the one Kartotek picks as the primary fallback — each with its own provider name and capture date, newest first, with a long capture history revealed a few at a time rather than all at once. This never triggers a fresh provider lookup on page load — it's built from the same background discovery pass that already populates the preferred snapshot, so a slow or temporarily unreachable provider never holds up the page itself. Below that, a History section shows a concise, durable timeline of the Link's own operational lifecycle — when it was added, first checked, and first confirmed working, plus every later transition between working and unavailable and every canonical web-address change (the Upgrade to HTTPS action included) — without listing every individual routine check that found nothing new. This history is a genuine, persisted record, not something recomputed from the Link's current state on the fly, so it survives an HTTP→HTTPS upgrade intact rather than starting over. Where an exact historical date can't be established for a Link that predates this feature, Kartotek says so plainly rather than guessing.

Admin visibility

The sidebar's Links group expands to three entries: Rotten links, Evidence matrix, and Local artifacts.

Rotten links is the same view the Dashboard's own "Needs attention" tile opens into (see Dashboard and maintenance): sortable/searchable/filterable columns for the owning Post, destination, Health status, PDF/Wayback preservation state, and last-checked/last-healthy/first-failure dates, plus Retry and History actions per Link. Summary tiles break the total down by Health status. Any combination of column filters can be picked directly (for example, Health = Missing and PDF = anything but Succeeded reproduces "missing, no preserved copy").

Evidence matrix is a broader, filterable/resizable grid — one row per canonical Link, covering every evidence type at once: overall state (Queued/Partial/Complete/Failed), live status, PDF, WARC, Wayback, other Web Archive providers, and when it was last verified. This is what every Dashboard "Link evidence" tile opens into, pre-filtered.

Local artifacts lists any local PDF/WARC file discovered on disk that doesn't (yet) have a matching capture record — normally only possible after an interrupted capture. A candidate names exactly one Link whose destination hostname matches and which still lacks that evidence kind, with Associate (creates the capture record, keeping the file's own real modified date rather than inventing a capture time) and Reject actions; an orphan has no confident match at all and is shown separately, never guessed at.

Every capture is kept, not just the latest. Retrying a Link's archival never discards or overwrites an earlier capture — it makes a brand-new attempt alongside every previous one, all individually visible in that Link's own capture history from the Rotten links view. If a retry itself fails, the Link's best-available archived copy (PDF or Wayback) stays exactly whichever one last succeeded, never blanked out by a later failed attempt.

Two corpus-wide Backfill operations (see Dashboard and maintenance for the shared Backfill design) handle Link catch-up: Link health checks, an explicit, resumable re-check of every current Link's live health with a live count of checked/skipped/failed/remaining; and Link evidence discovery, a broader pass that, for every canonical Link, brings its live check, HTTP→HTTPS upgrade, local PDF/WARC capture, and Wayback/other-provider lookups all current at once — starting, pausing, resuming, or explicitly retrying only the Links currently Failed, without redoing already-covered work. It also reconciles any local PDF/WARC file that exists without a matching capture record — offered for review rather than ever silently linked up automatically. A brand-new deployment, or one enabling archival for the first time, doesn't need every eligible Post and Link to wait for its own next edit: the same Backfill design covers a Post WARC & Wayback archival operation, generating any missing Post WARC and submitting every associated-Domain URL to Wayback in one resumable pass. Running any of these repeatedly is always safe — an already fully-evidenced Link or Post is a fast no-op, never duplicated work.

A Post's own metadata/details view (see Posts and publishing) similarly shows its WARC capture status and, per associated Domain, its Wayback submission status, timestamp, and resulting Wayback link, each with its own retry action.