Monitor and troubleshoot
Check each layer of a deployment, then set up continuous checks for query results, failover, and data freshness.
On this page
Check the service and ingress
Run the first two probes through the same HTTPS endpoint that clients use. Query the private status port on each Coordinator host to check its role. Set --status-port when starting the process, as shown in the cluster installation.
curl -fsS https://graphlake.example/healthz
curl -fsS https://graphlake.example/readyz
curl -fsS http://127.0.0.1:7338/status
sudo journalctl -u graphlake -u graphlake-worker@7441 -n 100 --no-pager/healthzresponds but/readyzfails- Check the Coordinator's role, etcd, shared storage, Worker reachability, and ingress forwarding.
- Service ready, model fails
- Open Runtime Settings → Health Checks, run a check on the published model, and inspect the source and Worker results. Then query the same model through the client ingress.
- Only one client protocol fails
- Check the real Bolt handshake or Gremlin WebSocket upgrade through the ingress. Compare it with the Coordinator listener on the private route.
Configure continuous checks
Use the packaged graphlake-monitor on a host with Python 3 and access to the real client endpoints. Start from monitor.json. Set the enabled protocols under ingress, give each Coordinator and Worker a stable name, and use private forwards for their status and identity endpoints.
Create a dedicated query-only service account for the published models under test. For each model, set a bounded read-only canary with a known result. Sort multi-row results explicitly and use the API's string or null values for expected cells. Set the service account token and Worker probe token in root-owned /etc/graphlake/monitor.env with mode 0600.
From the directory with your finished monitor.json, install the packaged service and verify its metrics.
sudo install -m 0640 -o root -g graphlake monitor.json /etc/graphlake/monitor.json
sudo install -m 0644 /opt/graphlake/current/graphlake-monitor.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now graphlake-monitor
curl -fsS http://127.0.0.1:9337/metricsTrack freshness, resources, and recovery
Scrape the monitor on 127.0.0.1:9337 under the graphlake-monitor job. If also scraping active production metrics, use the graphlake-active job. Load the packaged Prometheus rules and configure Alertmanager routing. Track local table observation and reconciliation age, CSR generation freshness, free disk space, and verified off-site backup age.
For queue saturation, inspect slots, latency, and spill usage before changing query concurrency. For a freshness alert, check the model's last successful load and the current CSR generation, then rerun a canary after repair. The recovery guide covers control exports and acceptance tests. Use telemetry.yaml to configure OTLP logs, metrics, and traces per node.