README.md

June 26, 2026 · View on GitHub

duplicacy-exporter banner

Prometheus exporter for Duplicacy backup metrics -- real-time progress, speed, and post-run summaries for your Grafana dashboards.

PyPI GitHub Release Docker Hub Grafana Dashboard License Python 3.13 Image Size Coverage

Overview

duplicacy-exporter bridges Duplicacy backups and Prometheus, giving you full observability over backup operations. It works with both Duplicacy CLI (via log tailing) and Duplicacy Web UI (via webhook), exposing metrics that Prometheus scrapes and Grafana visualizes.

Key capabilities

  • Real-time metrics -- backup speed (bytes/sec), progress (0-100%), chunks uploaded/skipped, updated per chunk
  • Post-run summaries -- duration, file counts, bytes uploaded, exit codes, revision numbers
  • Prune tracking -- monitors prune operations with completion timestamps
  • Two collection modes -- log_tail for CLI users, webhook for Web UI users
  • Smart label resolution -- automatic snapshot ID, storage target, and machine name detection from logs
  • Storage host mapping -- translates IPs and Tailscale FQDNs into human-readable names
  • Lightweight -- single Python file, one dependency (prometheus_client), Alpine image (~30 MB)

Architecture

+---------------------+         +----------------------+         +------------+
|                     |  logs   |                      | scrape  |            |
|  Duplicacy CLI      +-------->+  duplicacy-exporter  +<--------+ Prometheus |
|  (Docker / file)    |  tail   |                      |  :9750  |            |
+---------------------+         +----------+-----------+         +------+-----+
                                           |                            |
+---------------------+  POST   |          |                            |
|  Duplicacy Web UI   +-------->+  /webhook endpoint   |         +------v-----+
|  (report_url)       |         |                      |         |  Grafana   |
+---------------------+         +----------------------+         +------------+

Log tail mode connects to the Docker Engine API over a Unix socket (or tails a log file) and parses Duplicacy output line-by-line. It extracts chunk-level progress in real time and summary statistics at completion.

Webhook mode receives JSON payloads from Duplicacy Web UI's report_url setting, extracting the same summary metrics without needing Docker socket access.

Quick Start

Deploy alongside your Duplicacy CLI container using a shared log volume:

services:
  duplicacy-exporter:
    image: drumsergio/duplicacy-exporter:0.6.0
    container_name: duplicacy-exporter
    restart: unless-stopped
    environment:
      - MODE=log_tail
      - LOG_FILE=/logs/duplicacy.log
      - LISTEN_PORT=9750
    volumes:
      - duplicacy-logs:/logs:ro
    ports:
      - "9750:9750"

volumes:
  duplicacy-logs:

Note: Mount the same duplicacy-logs volume in your Duplicacy container, writing output to /logs/duplicacy.log. This avoids exposing the Docker socket.

Docker Compose -- Webhook Mode (for Web UI)

services:
  duplicacy-exporter:
    image: drumsergio/duplicacy-exporter:0.6.0
    container_name: duplicacy-exporter
    restart: unless-stopped
    environment:
      - MODE=webhook
      - LISTEN_PORT=9750
    ports:
      - "9750:9750"

Then set report_url in Duplicacy Web UI to: http://duplicacy-exporter:9750/webhook

Docker Compose -- Log File Mode

If you write Duplicacy logs to a file instead of using Docker:

services:
  duplicacy-exporter:
    image: drumsergio/duplicacy-exporter:0.6.0
    container_name: duplicacy-exporter
    restart: unless-stopped
    environment:
      - MODE=log_tail
      - LOG_FILE=/logs/duplicacy.log
      - LISTEN_PORT=9750
    volumes:
      - /path/to/duplicacy/logs:/logs:ro
    ports:
      - "9750:9750"

Configuration

All configuration is done through environment variables:

VariableDefaultDescription
MODElog_tailCollection mode: log_tail or webhook
DOCKER_CONTAINER_NAMEduplicacy-cli-cronContainer name to tail logs from (log_tail mode)
LOG_FILE(empty)Path to log file; alternative to Docker socket (log_tail mode)
LISTEN_PORT9750Port for the metrics and webhook HTTP server
WEBHOOK_PATH/webhookPath for the webhook POST endpoint (webhook mode)
MACHINE_NAME(empty)Machine name label. In log_tail mode it must be set (or learned from a DUPLICACY_META / notification line) before backup metrics are emitted.
SNAPSHOT_ID(empty)Snapshot id for log_tail users whose logs have no section headers / DUPLICACY_META (e.g. stock duplicacy backup). Lets post-run summary metrics resolve.
TAILSCALE_DOMAINmango-alpha.ts.netTailscale domain suffix to strip from storage URLs
STORAGE_HOST_MAP(empty)JSON object mapping hostname/IP to display name
REPLAY_HOURS25Hours of Docker log history to replay on startup
TIMESTAMP_FILE/data/duplicacy_exporter_last_tsFile to persist last-seen log timestamp (avoids counter double-count on restart). Co-located with STATE_FILE under /data so one volume persists both.
MAX_LOG_BUFFER1048576Maximum Docker log buffer size in bytes before discarding partial data (1 MB)
LOG_LEVELINFOLogging verbosity: DEBUG, INFO, WARNING, ERROR
PERSIST_ENABLEDtrueSave the last completed backup/storage/prune values to disk and reload them on startup, so metrics survive a restart (see Persistence). Set to false to opt out.
STATE_FILE/data/duplicacy_exporter_state.jsonWhere the durable metric state is stored. Mount a volume at its directory so state survives container re-creation, not just restarts.
PERSIST_INTERVAL15Seconds between state snapshots. A snapshot is only written when a value actually changed.
POLLER_ENABLEDfalseEnable the optional storage poller. Truthy values: 1, true, yes. Off by default.
POLLER_INTERVAL86400Seconds between storage poller cycles (default 24h)
POLLER_REPOSITORIES(empty)JSON list of repositories to poll. Each item: {"path": "...", "storage": "...", "snapshot_id": "..."} (path required; storage defaults to default; snapshot_id optional)
DUPLICACY_BINduplicacyPath to the duplicacy CLI binary used by the poller
POLLER_TIMEOUT1800Per-command timeout in seconds for duplicacy list / check

Storage host mapping example

Map raw IPs or hostnames to friendly names:

STORAGE_HOST_MAP='{"192.168.10.100":"watchtower","192.168.20.5":"geiserct"}'

Persistence across restarts

Prometheus metrics live only in memory, so without persistence a restart wipes them: in webhook mode the values stay gone until the next backup reports, which can leave Home Assistant sensors unavailable/unknown for hours.

Persistence is on by default. The last completed backup, storage-poller and prune values are snapshotted to STATE_FILE (default /data/duplicacy_exporter_state.json) and reloaded on startup, so /metrics re-serves them immediately. Only durable "last completed" values are persisted — the real-time progress gauges (*_running, *_speed_*, *_progress_*, live chunk counts) are not, since they reflect an in-flight backup and settle from live data within one cycle.

Mount a volume at the state file's directory so it also survives container re-creation (image upgrades), not just a docker restart:

    volumes:
      - duplicacy-exporter-data:/data   # or a bind mount, e.g. /mnt/user/appdata/duplicacy-exporter:/data

If the directory is not writable the exporter logs one warning and continues without persistence (metrics still work, they just won't survive a restart).

Metrics

All backup metrics carry labels: snapshot_id, storage_target, machine. All prune metrics carry labels: storage_target, machine.

Each distinct (snapshot_id, storage_target) is its own series (and its own device in duplicacy-ha), so multiple backups are tracked independently. In webhook mode snapshot_id comes from the Web UI report's backup id, falling back to the source directory name — so two backups on one machine never collapse into one. In log_tail mode it comes from a DUPLICACY_META snapshot_id=… line, a --- Backup -> … (id) --- section header, or the SNAPSHOT_ID env var.

Real-time (updated per chunk during backup)

MetricTypeDescription
duplicacy_backup_runningGauge1 if backup is in progress, 0 otherwise
duplicacy_backup_speed_bytes_per_secondGaugeCurrent backup speed
duplicacy_backup_progress_ratioGaugeProgress from 0.0 to 1.0
duplicacy_backup_chunks_uploadedGaugeChunks uploaded in current run
duplicacy_backup_chunks_skippedGaugeChunks skipped in current run

Post-run summary

MetricTypeDescription
duplicacy_backup_last_success_timestamp_secondsGaugeUnix timestamp of last successful backup
duplicacy_backup_last_duration_secondsGaugeDuration of last backup in seconds
duplicacy_backup_last_files_totalGaugeTotal files in last backup
duplicacy_backup_last_files_newGaugeNew files in last backup
duplicacy_backup_last_bytes_uploadedGaugeBytes uploaded in last backup
duplicacy_backup_last_bytes_newGaugeNew bytes in last backup
duplicacy_backup_last_chunks_newGaugeNew chunks in last backup
duplicacy_backup_last_files_size_bytesGaugeTotal logical size of all files in the last backup snapshot (webhook: total_file_size)
duplicacy_backup_last_chunks_size_bytesGaugeTotal size of chunks referenced by the last backup, compressed and not deduplicated across revisions (webhook: total_chunk_size)
duplicacy_backup_last_exit_codeGaugeExit code: 0 = success, 1 = failure
duplicacy_backup_last_revisionGaugeRevision number of last backup
duplicacy_backup_bytes_uploaded_totalCounterCumulative bytes uploaded across all runs

Prune

MetricTypeDescription
duplicacy_prune_runningGauge1 if prune is in progress
duplicacy_prune_last_success_timestamp_secondsGaugeUnix timestamp of last successful prune

Diagnostics

MetricTypeDescription
duplicacy_exporter_last_activity_timestamp_secondsGaugeUnix timestamp of the last log line parsed (alert if it goes stale)
duplicacy_exporter_backups_seen_totalCounterCompleted backups detected, including any dropped for a missing snapshot_id/storage_target/machine. If this climbs while labelled series stay empty, the exporter is seeing backups it can't label — set SNAPSHOT_ID/MACHINE_NAME.

Storage poller (optional, opt-in)

Only populated when the storage poller is enabled. Storage metrics carry labels storage_target, machine; snapshot metrics carry snapshot_id, storage_target, machine.

MetricTypeDescription
duplicacy_storage_total_size_bytesGaugeTotal size of all chunks in the storage, from duplicacy check. Approximate — reconstructed from Duplicacy's human-readable size formatting (worst case ~0.65% low).
duplicacy_storage_total_chunksGaugeTotal number of chunks in the storage, from duplicacy check
duplicacy_snapshot_revisionsGaugeNumber of revisions for a snapshot id, from duplicacy list
duplicacy_snapshot_last_revisionGaugeHighest (latest) revision number for a snapshot id, from duplicacy list
duplicacy_poller_last_success_timestamp_secondsGaugeUnix timestamp of the last fully successful poller cycle
duplicacy_poller_errors_totalCounterPoller errors (missing binary, timeout, or parse failure) across all cycles

Webhook payload

In webhook mode the exporter consumes Duplicacy Web UI's report_url POST. That report is a single flat JSON object and is sent only for backups (not for prune, copy, or check). It carries these 26 fields:

FieldMeaning
computerMachine name (used for the machine label)
directorySource directory backed up (the per-backup differentiator → snapshot_id)
start_time, end_timeUnix timestamps; their difference is the duration
result"Success" or "Error" (capitalized)
storage, storage_urlDestination storage URL (used for the storage_target label)
total_files, new_filesFile counts (total / new this revision)
total_file_size, new_file_sizeLogical file bytes (total / new)
total_chunks, new_chunksChunk counts (total / new)
total_chunk_size, new_chunk_sizeChunk bytes after compression (total / new)
total_file_chunks, new_file_chunksFile-content chunk counts
total_file_chunk_size, new_file_chunk_sizeFile-content chunk bytes
total_metadata_chunks, new_metadata_chunksMetadata chunk counts
total_metadata_chunk_size, new_metadata_chunk_sizeMetadata chunk bytes
upload_chunk_sizeBytes actually uploaded this run (note: no "d" — upload, not uploaded)
upload_file_chunk_size, upload_metadata_chunk_sizeUploaded file / metadata chunk bytes

There is no id, snapshot_id, revision, prune, or storage-size field in this payload. The exporter differentiates backups by directory, and uses the poller (below) for storage size and revision counts.

Storage poller (optional)

The Web UI webhook cannot carry storage size or revision counts. The optional storage poller fills that gap by periodically running the duplicacy CLI:

  • duplicacy -log list -all → revision count and latest revision per snapshot id
  • duplicacy -log check -tabular -stats → total chunk count and total storage size

Enable it with POLLER_ENABLED=true and a POLLER_REPOSITORIES JSON list:

environment:
  - POLLER_ENABLED=true
  - POLLER_INTERVAL=86400          # once a day
  - POLLER_REPOSITORIES=[{"path":"/repos/photos","storage":"watchtower","snapshot_id":"photos"}]
volumes:
  - /srv/duplicacy/photos:/repos/photos   # an initialised duplicacy repository

⚠️ Security & cost. The poller runs the duplicacy binary against your storage, so it needs the binary (bundled in the image) plus storage credentials and an initialised repository inside the exporter container (mount the repo's .duplicacy directory or provide credentials via the environment). It is opt-in and off by default for this reason. check can be slow and costly on remote storage (it lists every chunk), so keep POLLER_INTERVAL large. The exporter still runs fine without the poller — the bundled binary simply enables it.

What lives where

You want…Use
Per-run speed, progress, files, uploaded byteslog_tail or webhook
Storage size + revision countspoller (not in the webhook)
Prune completion trackinglog_tail (not in the webhook, not in the poller)

The storage-size value is approximate: Duplicacy's check prints sizes in a lossy human format (e.g. 5,120M), which the exporter converts back to bytes.

Endpoints

PathMethodDescription
/metricsGETPrometheus metrics endpoint
/webhookPOSTDuplicacy Web UI report endpoint
/healthGETHealth check (returns 200 OK)

Prometheus Configuration

Add the exporter as a scrape target:

scrape_configs:
  - job_name: "duplicacy"
    static_configs:
      - targets: ["duplicacy-exporter:9750"]
        labels:
          instance: "my-server"

Example alerting rule

groups:
  - name: duplicacy
    rules:
      - alert: DuplicacyBackupFailed
        expr: duplicacy_backup_last_exit_code != 0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Duplicacy backup failed for {{ $labels.snapshot_id }}"

      - alert: DuplicacyBackupStale
        expr: time() - duplicacy_backup_last_success_timestamp_seconds > 86400
        for: 1h
        labels:
          severity: critical
        annotations:
          summary: "No successful Duplicacy backup in 24h for {{ $labels.snapshot_id }}"

Grafana Dashboard

A ready-to-import dashboard is included in dashboard.json and published on Grafana.com (#25089).

Import it in Grafana via Dashboards → Import → Upload JSON file or use the dashboard ID 25089.

Troubleshooting

Exporter starts but no metrics appear

  • Log tail mode: Verify the Docker socket is mounted (/var/run/docker.sock:/var/run/docker.sock:ro) and the DOCKER_CONTAINER_NAME matches your Duplicacy container exactly.
  • Log file mode: Confirm the log file path is correct and the volume mount provides read access.
  • Webhook mode: Ensure report_url in Duplicacy Web UI points to http://<exporter-host>:9750/webhook. The exporter must be reachable from the Web UI container.

Metrics show but labels are wrong or missing

  • Set LOG_LEVEL=DEBUG to see how each log line is parsed and which labels are resolved.
  • If storage targets show as raw IPs, use STORAGE_HOST_MAP to map them to friendly names.
  • If machine name is missing, set MACHINE_NAME explicitly.

Metrics (or HA sensors) disappear after restarting the exporter

  • Persistence is on by default — confirm STATE_FILE's directory is a writable mounted volume (default /data). Without a volume the state is lost when the container is re-created on an image upgrade.
  • Check the logs for Disabling metric persistence; cannot write …: the directory isn't writable. Mount a volume or set STATE_FILE to a writable path.
  • See Persistence across restarts for details.

Docker socket permission denied

The exporter process runs as root inside the container by default. If you run it as a non-root user, ensure the user has access to the Docker socket (typically group docker, GID 999 or similar).

Webhook returns 404

Verify the WEBHOOK_PATH environment variable matches the path you configured in Duplicacy Web UI. The default is /webhook.

License

GPL-3.0