HugeGraph PD

September 9, 2026 ยท View on GitHub

License Version

Overview

HugeGraph PD (Placement Driver) is a meta server that provides cluster management and coordination services for HugeGraph distributed deployments. It serves as the central control plane responsible for:

  • Service Discovery: Automatic registration and discovery of Store and Server nodes
  • Partition Management: Dynamic partition allocation, balancing, and rebalancing across Store nodes
  • Metadata Storage: Centralized storage of cluster metadata, configuration, and state information
  • Node Scheduling: Intelligent scheduling and load balancing of graph operations
  • Health Monitoring: Continuous health checks and failure detection via heartbeat mechanism

PD uses SOFA-jraft for Raft consensus and RocksDB for persistent metadata storage, ensuring high availability and consistency in distributed environments.

Architecture

HugeGraph PD is a Maven multi-module project consisting of 8 modules:

ModuleDescription
hg-pd-grpcgRPC protocol definitions (.proto files) and generated Java stubs for inter-service communication
hg-pd-commonShared utilities, constants, and helper classes used across PD modules
hg-pd-coreCore PD logic: Raft integration, metadata stores, partition allocation, store monitoring, task scheduling
hg-pd-servicegRPC service implementations and REST API (Spring Boot) for management and metrics
hg-pd-clientJava client library for applications to communicate with PD cluster
hg-pd-cliCommand-line utilities for PD administration and debugging
hg-pd-testUnit and integration tests for all PD components
hg-pd-distDistribution assembly: packaging, configuration templates, startup scripts

For detailed architecture and design, see Architecture Documentation.

Quick Start

Prerequisites

  • Java: 11 or higher
  • Maven: 3.5 or higher
  • Disk Space: At least 1GB for PD data directory

Build

From the project root (build PD and all dependencies):

mvn clean package -pl hugegraph-pd -am -DskipTests

Or build from the hugegraph-pd directory:

cd hugegraph-pd
mvn clean install -DskipTests

The assembled distribution will be available at:

hugegraph-pd/hg-pd-dist/target/hugegraph-pd-<version>.tar.gz

Run

Extract the distribution package and start PD:

tar -xzf hugegraph-pd-<version>.tar.gz
cd hugegraph-pd-<version>

# Start PD server
bin/start-hugegraph-pd.sh

# Stop PD server
bin/stop-hugegraph-pd.sh

Startup Options

bin/start-hugegraph-pd.sh [-g GC_TYPE] [-j "JVM_OPTIONS"] [-y ENABLE_OTEL] [-d DAEMON]
  • -g: GC type (g1 or ZGC, default: g1)
  • -j: Custom JVM options (e.g., -j "-Xmx4g -Xms4g")
  • -y: Enable OpenTelemetry tracing (true or false, default: false)
  • -d: Daemon mode (true = daemon, false = foreground; default: true). Set to false when running under Docker or a process supervisor so the container exits if Java dies.

Configuration

Key configuration file: conf/application.yml

Core Settings

ParameterDefaultDescription
grpc.host127.0.0.1gRPC server bind address (use actual IP for production)
grpc.port8686gRPC server port
server.port8620REST API port for management and metrics
raft.address127.0.0.1:8610Raft service address for this PD node
raft.peers-list127.0.0.1:8610Comma-separated list of all PD nodes in the Raft cluster
pd.data-path./pd_dataDirectory for storing PD metadata and Raft logs
auth.secret-keynone (required)Password required by the REST API with an internal service name (hg, store, hubble, vermeer) via HTTP Basic auth. No default is shipped; generate one per deployment and configure every REST client (e.g. Hubble's operations.pd.password) with the same value

Single-Node Example

grpc:
  host: 127.0.0.1
  port: 8686

server:
  port: 8620

raft:
  address: 127.0.0.1:8610
  peers-list: 127.0.0.1:8610

pd:
  data-path: ./pd_data

3-Node Cluster Example

For a production 3-node PD cluster, configure each node:

Node 1 (192.168.1.10):

grpc:
  host: 192.168.1.10
  port: 8686
raft:
  address: 192.168.1.10:8610
  peers-list: 192.168.1.10:8610,192.168.1.11:8610,192.168.1.12:8610

Node 2 (192.168.1.11):

grpc:
  host: 192.168.1.11
  port: 8686
raft:
  address: 192.168.1.11:8610
  peers-list: 192.168.1.10:8610,192.168.1.11:8610,192.168.1.12:8610

Node 3 (192.168.1.12):

grpc:
  host: 192.168.1.12
  port: 8686
raft:
  address: 192.168.1.12:8610
  peers-list: 192.168.1.10:8610,192.168.1.11:8610,192.168.1.12:8610

For detailed configuration options and production tuning, see Configuration Guide.

Docker Bridge Network Example

When running PD in Docker with bridge networking (e.g., docker/docker-compose-3pd-3store-3server.yml), configuration is injected via environment variables instead of editing application.yml directly. Container hostnames are used instead of IP addresses. HG_PD_AUTH_SECRET_KEY is required by the image and must be the same value on every PD node and every PD REST client; generate it once (openssl rand -hex 24) and keep it:

pd0 container:

HG_PD_GRPC_HOST=pd0
HG_PD_AUTH_SECRET_KEY=<the same secret on every node>
HG_PD_RAFT_ADDRESS=pd0:8610
HG_PD_RAFT_PEERS_LIST=pd0:8610,pd1:8610,pd2:8610
HG_PD_INITIAL_STORE_LIST=store0:8500,store1:8500,store2:8500

pd1 container:

HG_PD_GRPC_HOST=pd1
HG_PD_AUTH_SECRET_KEY=<the same secret on every node>
HG_PD_RAFT_ADDRESS=pd1:8610
HG_PD_RAFT_PEERS_LIST=pd0:8610,pd1:8610,pd2:8610
HG_PD_INITIAL_STORE_LIST=store0:8500,store1:8500,store2:8500

pd2 container:

HG_PD_GRPC_HOST=pd2
HG_PD_AUTH_SECRET_KEY=<the same secret on every node>
HG_PD_RAFT_ADDRESS=pd2:8610
HG_PD_RAFT_PEERS_LIST=pd0:8610,pd1:8610,pd2:8610
HG_PD_INITIAL_STORE_LIST=store0:8500,store1:8500,store2:8500

See docker/README.md for the full environment variable reference.

Verify Deployment

Check if PD is running:

# Check process
ps aux | grep hugegraph-pd

# Test REST API
curl http://localhost:8620/actuator/health

# Check logs
tail -f logs/hugegraph-pd.log

gRPC API

PD exposes several gRPC services for cluster management. Key services include:

  • PD Service (PDGrpc): Store registration, partition queries, leader election
  • KV Service (KvServiceGrpc): Distributed key-value operations for metadata
  • Watch Service (HgPdWatchGrpc): Watch for partition and store changes
  • Pulse Service (HgPdPulseGrpc): Heartbeat and health monitoring

Proto definitions are located in:

hugegraph-pd/hg-pd-grpc/src/main/proto/

For API reference and usage examples, see API Reference.

Testing

Run PD tests:

# All PD tests
mvn test -pl hugegraph-pd/hg-pd-test -am

# Specific test class
mvn test -pl hugegraph-pd/hg-pd-test -am -Dtest=YourTestClass

Docker

Build PD Docker image:

# From project root
docker build -f hugegraph-pd/Dockerfile -t hugegraph/pd:latest .

# Generate the REST secret once and keep it: every PD REST client needs this same value, and a new one silently breaks the clients already using the old one. Store it somewhere durable rather than only in this shell.
export HG_PD_AUTH_SECRET_KEY="$(openssl rand -hex 24)"

# Run container
docker run -d \
  -p 8620:8620 \
  -p 8686:8686 \
  -p 8610:8610 \
  -e HG_PD_AUTH_SECRET_KEY="${HG_PD_AUTH_SECRET_KEY}" \
  -e HG_PD_GRPC_HOST=<your-ip> \
  -e HG_PD_RAFT_ADDRESS=<your-ip>:8610 \
  -e HG_PD_RAFT_PEERS_LIST=<your-ip>:8610 \
  -e HG_PD_INITIAL_STORE_LIST=<store-ip>:8500 \
  -v /path/to/data:/hugegraph-pd/pd_data \
  --name hugegraph-pd \
  hugegraph/pd:latest

For Docker Compose examples with HugeGraph Store and Server, see:

docker/docker-compose-3pd-3store-3server.yml

Documentation

Production Deployment Notes

Cluster Size

  • Minimum: 3 nodes (Raft quorum requirement)
  • Recommended: 3 or 5 nodes for production (odd numbers for Raft election)

High Availability

  • PD uses Raft consensus for leader election and data replication
  • Cluster can tolerate up to (N-1)/2 node failures (e.g., 1 failure in 3-node cluster)
  • Leader handles all write operations; followers handle read operations

Network Requirements

  • Ensure low latency (<5ms) between PD nodes for Raft consensus
  • Open required ports: 8620 (REST), 8686 (gRPC), 8610 (Raft)

Security

  • Keep all three ports on a trusted network. The REST API on 8620 includes management endpoints that mutate the cluster (peer changes, store removal, data movement), and the gRPC and Raft ports carry no authentication.
  • REST requests need HTTP Basic auth: one of the internal service names (hg, store, hubble, vermeer) with the auth.secret-key value as the password. Health probes (/v1/health, /v1/ready, /actuator/**, /v1/prom/targets/*) stay unauthenticated.
  • auth.secret-key has no shipped default, because a secret in the source tree is published to everyone. Generate one per deployment (openssl rand -hex 24) and set it in the config file, or through HG_PD_AUTH_SECRET_KEY, which the Docker image requires. Give every REST client the same value: the Server's bin/wait-storage.sh reads PD_AUTH_PASSWORD (and PD_AUTH_USER, default store), and Hubble reads operations.pd.password. A client left on a stale secret gets 401, and wait-storage.sh aborts the Server's startup on the first one rather than waiting out WAIT_STORAGE_TIMEOUT_S.
  • An existing conf/application.yml carried over from an earlier release has no auth block, and still carries management.endpoints.web.exposure.include: "*". PD then starts with an empty secret and refuses every authenticated REST request, logging an error that names auth.secret-key, while /actuator/env, /actuator/configprops and /actuator/beans stay anonymously readable on 8620. Before upgrading, add auth.secret-key and narrow that exposure to health,metrics,prometheus. PD refuses to start if the key is set to the placeholder value that earlier revisions of this repository carried.

Monitoring

PD exposes metrics via REST API at:

  • Health check: http://<pd-host>:8620/actuator/health
  • Liveness: http://<pd-host>:8620/v1/health (REST listener is up)
  • Readiness: http://<pd-host>:8620/v1/ready (200 only while the PD sees a raft leader)
  • Metrics: http://<pd-host>:8620/actuator/metrics

Community

Contributing

Contributions are welcome! Please read our Development Guide and follow the Apache HugeGraph contribution guidelines.