Monitor and troubleshoot

Check each layer of a deployment, then set up continuous checks for query results, failover, and data freshness.

On this page

Check the service and ingress

Run the first two probes through the same HTTPS endpoint that clients use. Query the private status port on each Coordinator host to check its role. Set --status-port when starting the process, as shown in the cluster installation.

Shell
curl -fsS https://graphlake.example/healthz
curl -fsS https://graphlake.example/readyz
curl -fsS http://127.0.0.1:7338/status
sudo journalctl -u graphlake -u graphlake-worker@7441 -n 100 --no-pager
/healthz responds but /readyz fails
Check the Coordinator's role, etcd, shared storage, Worker reachability, and ingress forwarding.
Service ready, model fails
Open Runtime Settings → Health Checks, run a check on the published model, and inspect the source and Worker results. Then query the same model through the client ingress.
Only one client protocol fails
Check the real Bolt handshake or Gremlin WebSocket upgrade through the ingress. Compare it with the Coordinator listener on the private route.

Configure continuous checks

Use the packaged graphlake-monitor on a host with Python 3 and access to the real client endpoints. Start from monitor.json. Set the enabled protocols under ingress, give each Coordinator and Worker a stable name, and use private forwards for their status and identity endpoints.

Create a dedicated query-only service account for the published models under test. For each model, set a bounded read-only canary with a known result. Sort multi-row results explicitly and use the API's string or null values for expected cells. Set the service account token and Worker probe token in root-owned /etc/graphlake/monitor.env with mode 0600.

From the directory with your finished monitor.json, install the packaged service and verify its metrics.

Shell
sudo install -m 0640 -o root -g graphlake monitor.json /etc/graphlake/monitor.json
sudo install -m 0644 /opt/graphlake/current/graphlake-monitor.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now graphlake-monitor
curl -fsS http://127.0.0.1:9337/metrics

Track freshness, resources, and recovery

Scrape the monitor on 127.0.0.1:9337 under the graphlake-monitor job. If also scraping active production metrics, use the graphlake-active job. Load the packaged Prometheus rules and configure Alertmanager routing. Track local table observation and reconciliation age, CSR generation freshness, free disk space, and verified off-site backup age.

For queue saturation, inspect slots, latency, and spill usage before changing query concurrency. For a freshness alert, check the model's last successful load and the current CSR generation, then rerun a canary after repair. The recovery guide covers control exports and acceptance tests. Use telemetry.yaml to configure OTLP logs, metrics, and traces per node.