Back up and recover

Create a consistent checkpoint for an HA deployment, restore it at a separate site, and reopen traffic after verification.

On this page

1. Export control state and verify the replica

For an HA cluster, the packaged hourly systemd timer exports models, published revisions, roles, and other control state from the shared cluster configuration. Install the timer on the designated backup host with the released package, the same cluster configuration, and credentials to read its control state. For catalog-backed local tables, keep GRAPHLAKE_BACKUP_CATALOG_DIR pointed at the real catalog in the backup unit.

Shell
sudo install -m 0644 /opt/graphlake/current/graphlake-backup.service /etc/systemd/system/
sudo install -m 0644 /opt/graphlake/current/graphlake-backup.timer /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl start graphlake-backup.service
sudo systemctl enable --now graphlake-backup.timer
sudo systemctl list-timers graphlake-backup.timer

Sync the completed checkpoint and its related files to an independent site. Treat the archive as credential-bearing data. On the replica, verify the completed copy before treating it as recoverable.

Shell
/opt/graphlake/current/bin/graphlake-admin backup verify \
  --file /srv/graphlake-replica/checkpoint-001/control.json

2. Bind source data to the same checkpoint

For a recovery point that includes external sources, pause changes to model control state and the mapped data at the same business checkpoint. Export control state, run pg_dump -Fc for the PostgreSQL data if used, and copy the exact Parquet files referenced by your models. Use the fingerprint command for each PostgreSQL relation in the manifest.

Shell
/opt/graphlake/current/bin/graphlake-admin backup fingerprint-postgres \
  --dsn-env LEDGER_DSN --relation ledger.accounts --relation ledger.transfers

Place manifest.json next to control.json. Include postgres.dump when the manifest contains PostgreSQL and keep Parquet paths relative to the checkpoint. Keep any exported local-<id> directories next to it. Use actual lowercase SHA-256 digests and the published active_revision. Record at least one known-result query with the exact rows expected on the recovered data.

JSON
{
  "format": 1,
  "control_sha256": "<control.json sha256 field>",
  "postgres": {
    "dump_sha256": "<postgres.dump SHA-256>",
    "dsn_env": "LEDGER_DSN",
    "relations": {
      "ledger.accounts": "<accounts fingerprint>",
      "ledger.transfers": "<transfers fingerprint>"
    }
  },
  "parquet": {
    "payments.parquet": "<payments.parquet SHA-256>"
  },
  "canaries": [
    {
      "model": "ledger",
      "revision": "<published active_revision>",
      "language": "cypher",
      "query": "MATCH (:Account)-[:Transfer]->(:Account) RETURN count(*) AS transfers",
      "expected_rows": [
        [
          "4"
        ]
      ]
    }
  ]
}

Omit the postgres or parquet block when the checkpoint has no such external source. For models that use only local copies, the control archive records the associated local files. See the manifest contract for checkpoint details.

3. Import at the target site

Use a fresh cluster name and S3 prefix at the target, with its own administrator credentials. Configure the same recovery settings on all candidate Coordinators. Set GRAPHLAKE_RESTORE_INBOX to a read-only directory where each checkpoint is a separate subdirectory. Provide a writable source mount and a compatible empty PostgreSQL database if needed. Set GRAPHLAKE_RESTORE_POSTGRES_DSN_ENV to the target DSN variable name used in the manifest.

Set a dedicated GRAPHLAKE_RECOVERY_TOKEN shared by the target Coordinators. Start the target cluster and its TLS recovery route. From a trusted client, open the target HTTPS /recovery page, authenticate with the recovery token, select the checkpoint, and run the check and import. If import stops after writing begins, retry with the same checkpoint.

4. Pass acceptance and reopen traffic

Run acceptance from the target's /recovery page. It verifies the checkpoint, the restored data, each published model revision, and the exact canary result. Check readiness through the actual client HTTPS origin before cutting traffic. Keep source writes paused until verification and cutover finish.

Restore memberships and issue fresh application credentials as required. Restored user and service-account access is disabled and old sessions are cleared. Verify old credentials are rejected, new roles are enforced, and each client protocol and monitoring check recovers through production ingress.