Install a cluster

Deploy a Coordinator and Workers, verify distributed queries, then extend the same setup to a multi-host cluster with failover.

On this page

1. Prepare the hosts

The Coordinator serves the console, publishes models, and routes supported HTTP query, preview, and export work to Workers. Bolt and Gremlin driver sessions run on the Coordinator. Workers fetch the published model and read its sources, so give both roles access to the same source data and credentials.

For a first deployment, use a Debian 12 host with systemd and an etcd v3 HTTP/JSON gateway listening on 127.0.0.1:2379. Follow the download and service installation sections of Install on Linux on each host. Complete package extraction and unit installation before proceeding to the cluster startup below. Use the same GraphLake build on every node.

2. Configure a single-host cluster

Save the following as cluster.yaml. This example uses two loopback Workers and a private local storage directory. The Coordinator uses ports 7337 for the workbench and 7338 for process status.

YAML
name: finance-lab
metadata: http://127.0.0.1:2379
storage: /var/lib/graphlake/cluster-data
token_env: GRAPHLAKE_CLUSTER_TOKEN
distributed_partitions: 4
memory_mib: 256
workers:
  - http://127.0.0.1:7441
  - http://127.0.0.1:7442

Install the configuration and set GRAPHLAKE_CLUSTER_TOKEN in the environment file to a value generated by openssl rand -hex 32. Use the same value for the Coordinator and Workers. Put any source credential variables referenced by your models in the same environment file.

Shell
curl -fsS http://127.0.0.1:2379/health
sudo install -d -m 0750 -o root -g graphlake /etc/graphlake
sudo install -d -m 0700 -o graphlake -g graphlake /var/lib/graphlake/cluster-data
sudo install -m 0640 -o root -g graphlake cluster.yaml /etc/graphlake/cluster.yaml
sudo touch /etc/graphlake/graphlake.env
sudo chown root:root /etc/graphlake/graphlake.env
sudo chmod 0600 /etc/graphlake/graphlake.env
sudoedit /etc/graphlake/graphlake.env

3. Start the Workers and Coordinator

Save the Worker template as /etc/systemd/system/graphlake-worker@.service. The instance name selects the loopback port. Keep source files under the catalog path in the example or change --source-dir to match your mount.

systemd
[Unit]
Description=GraphLake Worker on port %i
After=network-online.target
Wants=network-online.target

[Service]
User=graphlake
Group=graphlake
EnvironmentFile=/etc/graphlake/graphlake.env
ExecStart=/opt/graphlake/current/bin/graphlake-admin --cluster-config /etc/graphlake/cluster.yaml --port %i worker --source-dir /var/lib/graphlake/catalog
Restart=on-failure
UMask=0077
NoNewPrivileges=yes

[Install]
WantedBy=multi-user.target

When systemctl edit graphlake opens the editor, paste the following override. The empty ExecStart= clears the packaged standalone startup command.

systemd
[Service]
EnvironmentFile=/etc/graphlake/graphlake.env
ExecStart=
ExecStart=/opt/graphlake/current/bin/graphlake-admin --catalog-dir /var/lib/graphlake/catalog --assets-dir /opt/graphlake/current/web --cluster-config /etc/graphlake/cluster.yaml --status-port 7338 --query-slots 3

For a fresh catalog, initialize the administrator password once. If the catalog already has an administrator, start at the second command.

Shell
sudo -u graphlake /opt/graphlake/current/bin/graphlake-admin \
  --catalog-dir /var/lib/graphlake/catalog auth init
sudo systemctl edit graphlake
sudo systemctl daemon-reload
sudo systemctl enable --now graphlake-worker@7441 graphlake-worker@7442
sudo systemctl enable --now graphlake
curl -fsS http://127.0.0.1:7337/readyz
curl -fsS http://127.0.0.1:7338/status

Open the workbench at http://127.0.0.1:7337/, using an SSH tunnel for remote access. Publish a model, check it in Runtime Settings → Health Checks, and run a query with a known result. For queries that use distributed stages, inspect distributed in the raw response for participating Workers and stage counts. A small canary can pass even when its plan runs on one Worker.

4. Move to multiple hosts

Use synchronized clocks, an etcd cluster with quorum, and an S3-compatible bucket with a dedicated prefix. Deploy the same cluster configuration on each node. Give the service account access to that bucket prefix, and use read-only credentials for mapped source data. Use shared object storage for Parquet or mount the same files on every node that reads them.

YAML
name: finance
metadata:
  - https://etcd-a.example:2379
  - https://etcd-b.example:2379
  - https://etcd-c.example:2379
storage: s3://graphlake-state/finance
token_env: GRAPHLAKE_CLUSTER_TOKEN
ca_file: /etc/graphlake/tls/cluster-ca.pem
identity_file: /etc/graphlake/tls/client-identity.pem
distributed_partitions: 4
memory_mib: 256
workers:
  - https://worker-a.example:7443
  - https://worker-b.example:7443

Replace the example addresses and storage prefix. List up to three members of the same etcd cluster. The cluster CA file must trust the private TLS endpoints. Each node's identity_file is a PEM certificate and private key at the configured path, readable by the service account.

Start one Worker instance per host. Its listener stays on IPv4 loopback. Put an mTLS reverse proxy in front of each Worker, verify client certificates, and forward only /internal/v1/jobs, /internal/v1/stages, /internal/v1/csr, /internal/v1/identity, and /internal/v1/readiness/{model}/{revision}. The workers entries name these private proxies. Confirm access to etcd, shared storage, and mapped sources on every host.

Set distributed_partitions from 2 to 8 for eligible distributed query stages, or 0 to dispatch each query to one Worker. Review the query execution and memory reference before tuning.

5. Enable Coordinator failover

Give the two Coordinator hosts the same build, cluster namespace, and bootstrap catalog, including the same administrator password hash. Add these flags to both Coordinator service commands. Set the public URL to the actual HTTPS origin used by the browser.

Shell flags
--coordinator-failover --public-url https://graphlake.example

Start one Coordinator and wait for initial import to complete, then start the standby. Use a TLS ingress that forwards to the private Coordinator listeners and routes HTTP to the instance passing /coordinator/active. The packaged HAProxy configuration shows HTTPS, Gremlin WebSocket, Bolt handshake, and recovery routing. Enable and test the Bolt listener only if clients use Bolt.

From the client endpoint, verify HTTPS readiness and a known-result query. In a controlled failover drill, stop the active Coordinator and verify the standby becomes active, then repeat the same query and reconnect each enabled driver. Observe both nodes through their private status ports. Upgrade Workers before Coordinators and retain a pre-upgrade recovery checkpoint. See the redundant ingress reference when deploying HAProxy and Keepalived together.