Customizações Duplicati

• Duplicati Monitor
• Duplicati Dashboard
• Duplicati Dashboard2

Duplicati Monitor

Link: https://github.com/RafaMunoz/duplicati-monitor     git clone https://github.com/RafaMunoz/duplicati-monitor.git

Duplicati Monitor

Duplicati Monitor is a docker-packaged monitoring solution for Duplicati.

It allows you to control the status of your backups and statistics for each report and notifies you by any means supported by Apprice. It is meant to be self-hosted and powered by docker.

Quick Start

Notifications

To send notifications and satisfy the end user, it has been implemented with the Apprise library, which allows you to send a notification to almost all the most popular notification services available today, such as: Telegram, Discord, Slack, Amazon SNS, Gotify, etc.

You can check all the services available in the official documentation.

Templates

You can create your own templates to receive the information you exactly need.

By default we use the following two:

# Succes backup
💾 🟢 <Extra.backup-name>

# Error backup 
💾 🔴 <Extra.backup-name>

You can create your own templates in a simple way. You only have to use the symbols < and > delimit the fields and separating keys with dots . to send along with the message that you want.

In the docs/examples_report folder of this repository you have two examples of the JSON reports that Duplicati sends and in which you can see the fields that compose it.

For example, with the following template, the result shown in the telegram image would be obtained.

💾 🟢 Backup: <Extra.backup-name>\n - Examined Files: <Data.ExaminedFiles>\n - Duration: <Data.Duration>\n - Status: *<Data.TestResults.ParsedResult>*

Environment Variables

ENVIRONMENT VARIABLE	TYPE	DEFAULT	DESCRIPTION
URI_NOTIFICATION	required		URI in Apprise format where the notifications will be sent.
TEMPLATE_SUCCESS	opcional	💾 🟢 <Extra.backup-name>	Message template to be sent when the backup is successful.
TEMPLATE_ERROR	opcional	💾 🔴 <Extra.backup-name>	Message template that will be sent when the backup is executed in an erroneous way.
PORT	opcional	8000	Listening port on which the service is set up to receive the reports.

Docker Run

To start the container you can do it with the following command.

docker run -d --name=duplicati-monitor -p 8000:8000 -e URI_NOTIFICATION=tgram://<TOKEN_TELEGRAM_BOT>/<CHANEL_ID>/?format=markdown rafa93m/duplicati-monitor

Or you can also use the following docker compose.

version: "3"
services:
    duplicati-monitor:
        container_name: duplicati-monitor
        image: rafa93m/duplicati-monitor
        ports:
            - 8000:8000
        environment:
            URI_NOTIFICATION: "tgram://<TOKEN_TELEGRAM_BOT>/<CHANEL_ID>/?format=markdown"
            TEMPLATE_SUCCESS: "💾 🟢 Backup: <Extra.backup-name>"
            TEMPLATE_ERROR: "💾 🔴 Backup <Extra.backup-name> failed 🔥 🔥"
        restart: unless-stopped

Setup Duplicati

Add theses two options for each backup you want to monitor:

• send-http-result-output-format: json
• send-http-url: http://IP_ADDRESS:PORT/report

Duplicati Dashboard

Link: https://github.com/fabien-github/duplicati_dashboard?tab=readme-ov-file#demo  
git clone https://github.com/fabien-github/duplicati_dashboard.git

Duplicati Dashboard is a monitoring solution for Duplicati.

It allows you to monitor your backups status, collects stats for each reports and alerts you by email when a backup fails. It is intended to be self-hosted and works with docker-compose.

Everything is already pre-configured and ready to be deployed.

• Duplicati Dashboard
• Demo
• Quick Start
  • Running with docker-compose
  • Setup Duplicati
  • Connect to your dashboard
• Configuration
  • Env file
• Notes
  • Grafana configuration locked
  • Backup over more than 30 days rotation
  • Alerting graph
  • Deleting removed backup data
  • Docker-compose
• Other informations
• License

Demo

The right side of the video is not integrated in the dashboard. You can't control your backup with it.

Quick Start

Running with docker-compose

git clone https://github.com/fabien-github/duplicati_dashboard.git
cd duplicati_dashboard
docker-compose up -d

Setup Duplicati

Add theses two options for each backup you want to monitor:

• send-http-result-output-format: json
• send-http-url: http://localhost:8080

This assumes that your Duplicati instance is on the same host as your Duplicati Dashboard.

Connect to your dashboard

http://localhost:3000

Login: admin Password: password

For email alerting, you need to configure a SMTP relay. See Configuration > Env file

Configuration

Env file

The file config.env is used to configure some options. It will be shared between the 3 containers.

Only use this default configuration for testing purposes.

Influxdb variable will create and setup the database only on the first startup.

Variables	Default	Description
DOCKER_INFLUXDB_INIT_MODE	setup	Automatically bootstrap the system
DOCKER_INFLUXDB_INIT_USERNAME	telegraf_user	Influxdb superadmin user
DOCKER_INFLUXDB_INIT_PASSWORD	telegraf_password	Influxdb superadmin password
DOCKER_INFLUXDB_INIT_ORG	telegraf_org	Influxdb Organization (used by influxdb / telegraf / grafana)
DOCKER_INFLUXDB_INIT_BUCKET	telegraf	Influxdb bucket to store reports (used by influxdb / telegraf / grafana)
DOCKER_INFLUXDB_INIT_ADMIN_TOKEN	telegraf_token	Influxdb superadmin token (used by influxdb / telegraf / grafana)
DOCKER_INFLUXDB_INIT_RETENTION		Influxdb data retention, default will retain forever
INFLUXD_REPORTING_DISABLED	false	Disable InfluxData telemetry
TELEGRAF_LISTENER_PORT	8080	Port used by http_listener_v2 input, endpoint for the reports sent by Duplicati
TELEGRAF_LISTENER_PATH	/	Path to listen to
GF_SECURITY_ADMIN_USER	admin	Grafana superadmin user
GF_SECURITY_ADMIN_PASSWORD	password	Grafana superadmin password
GF_SERVER_ROOT_URL	http://localhost:3000	Grafana URL, used in some templates like email notifications
GF_DASHBOARDS_DEFAULT_HOME_DASHBOARD_PATH	/etc/grafana/provisioning/dashboards/duplicati_dashboard.json	Force Duplicati dashboard by default on home page
GF_SMTP_ENABLED	false	Set to true for email notifications
GF_SMTP_HOST	localhost:25	SMTP relay server. [host]:[port]
GF_SMTP_FROM_NAME	Grafana	Name of the email sender
GF_SMTP_USER		In case of SMTP auth
GF_SMTP_PASSWORD		In case of SMTP auth
GF_SMTP_FROM_ADDRESS	admin@grafana.localhost	Address used when sending out emails
GF_SMTP_EHLO_IDENTITY	${HOSTNAME}	Name to be used as client identity for EHLO in SMTP dialog (Default will be the container ID)
GF_SMTP_STARTTLS_POLICY		“OpportunisticStartTLS”, “MandatoryStartTLS”, “NoStartTLS”
NOTIFIER_EMAIL_RECIPIENT	example@example.com	Recipients for email notification (separated by a semicolon)
NOTIFIER_EMAIL_REMINDER_ENABLE	true	Re-send an email if alerts are still active
NOTIFIER_EMAIL_REMINDER_FREQUENCY	2h	Delay between email reminders

Notes

Grafana configuration locked

The dashboard is locked by the Grafana provisioning system. You can't edit the datasource, the dashboard or the alert notifier from the UI. You will need to copy the dashboard or disable the provisioning configuration.

Dashboard path: ./grafana/provisioning/dashboards/duplicati_dashboard.json

The idea is to keep the stack easy to deploy for everyone without investing time to learn Grafana configuration.

Feel free to fork the project or directly edit files on your own.

More information here.

Backup over more than 30 days rotation

• Grafana will discover your backups name from the reports but only over the last 90 days. So if your backups are scheduled for more than 90 days, you will need to edit the request of the variable Backup in the dashboard configuration:

from(bucket: v.defaultBucket)
  |> range(start: -90d)
  |> filter(fn: (r) => true)
  |> toString()
  |> group(columns: ["backup-name"])
  |> distinct(column: "backup-name")
  |> keep(columns: ["_value"])

• Last reported status / Last reported variations / Alerting graph are based over the past 30 days. You will need to adapt each panel requests if your backups are scheduled over more than 30 days.

import "influxdata/influxdb/schema"

from(bucket: v.defaultBucket)
  |> range(start: -30d)
  ...

• In v2, alerts have their own provisioning file and query range can be edited here.

Alerting graph

The section "Alerting graph" is only used to trigger an alert when a backup fails. This is due to the lack of grafana alert support on other panel type. #6983

Backups status will be checked every minutes. An alert will be triggered after a pending status of 10min. Same delays are used on the recovery.

Warning reports don't trigger an alert.

Deleting removed backup data

After getting inside the influxdb container:

influx delete \
  --org telegraf_org \
  --bucket telegraf \
  --token "telegraf_token" \
  --start 1970-01-01T00:00:00Z \
  --stop $(date +"%Y-%m-%dT%H:%M:%SZ") \
  --predicate '_measurement="duplicati" AND "backup-name"="backup_to_delete"'

Docker-compose

• Telegraf: Receive JSON reports from Duplicati.

• Influxdb: Store reports converted by Telegraf.

• Grafana: Requests Influxdb to generate dashboard and alerts.

Telegraf endpoint provides a limited HTTP autentification.
The configuration file is located here : ./telegraf/telegraf.conf

Feel free to add a proxy like traefik or nginx to protect the stack on an unsecure network. (TLS, IP Restrictions, ...)

Other informations

• Not sure of the scalabilty, requests to the database are not efficients. Timeseries databases are not really adapted for this kind of data. This is mainly due to the nested and uneven json format from the reports and the variation time between reports.
• Features are limited directly by the stack itself. For example, it's nearly impossible to add a managment system for the backups.
• This project has no link with the development of Duplicati and his team.

License

Distributed under the GNU General Public License v3.0 License. See LICENSE for more information.

Duplicati Dashboard2

Link: https://github.com/wchorski/duplicati-dashboard  git clone https://github.com/wchorski/duplicati-dashboard.git

A NodeJS based server that collects JSON data from Duplicati backup logs
📦 Tech

    Frontend: NextJS
    API: NextJS
    Database: InfluxDB

    [!warning] Backups sharing the same name will cause issues. The backup's name must be unique across all Duplicati instances, it will be used as an ID for logging. MUST BE URL FRIENDLY example: "Laptop--Home_Folder_Backup", "Desktop--Home_Folder_Backup" is a good naming convention.

Usecase

Initally this was just some middle ware that serves as an endpoint for JSON friendly monitoring apps, but I also built a simple UI so it could be used as a standalone app.
Duplicati Setup

you can either add these settings for the globally or per backup in the Advanced options

    send-http-result-output-format = json
    send-http-url = "http://APPSDOMAIN/api/backups"

API

Here is a breakdown of what endpoints and search parameters that can be passed through.
URL Breakdown

http://APPDOMAIN/backups/BACKUP_ID?start=-5h&first=true

    BACKUP_ID => the id (or name) of the backup saved
    stuff after the "?" search query sets range of time of pulled data
        start => how far back to you want to start pulling data i.e.
            -40d 40 days ago [the default]
            -5h 5 hours ago
            -60m 60 miniutes ago
            1999-12-31T00:00:00 starting on December 31st, 1999
            1694117521 starting on this UNIX time (seconds)
        stop => the end of the range
            now() => my current time [the default]
            all the other examples above as long as the date is after the start
            both query parameters can be omitted
        last => set to true if you'd like the last recorded point in the table
            can also use the http://localhost:3000/api/backups/last/BACKUP_ID endpoint for cleaner GET (this endpoint also takes start and stop query parameters)
        first => same as last but returns the first point of recoreded data within the range

    [!note] make sure relative dates have a negative i.e. -5h as your are looking back in time. Positive time values will cause errors

Examples
query     url
all backup stats in database     http://APPDOMAIN/backups
single backup stats     http://APPDOMAIN/backups/BACKUP_ID
last recorded backup stat     http://APPDOMAIN/backups/last/BACKUP_ID
same as above     http://APPDOMAIN/backups/BACKUP_ID?last=true
last recorded backup stat in the last 5 hours     http://APPDOMAIN/backups/last/BACKUP_ID?start=-5h
⚙️ Development

    git clone https://github.com/wchorski/duplicati-dashboard.git && cd duplicati-dashboard
    cp .env.template .env.local
    set up InfluxDB instance
    get InfluxDB API Token for .env.local
    yarn install
    yarn dev

🏭 Production

    git clone https://github.com/wchorski/duplicati-dashboard.git && cd duplicati-dashboard
    cp .env.template .env
    the INFLUX_TOKEN in .env should be a long ~88 character string
    docker compose up -d

Home Assistant

rest: 
- authentication: basic
  username: "admin"
  password: "password"
  scan_interval: 86400
  resource: http://APPDOMAIN.lan/api/backups/last/DUPLICATI_ID
  sensor:
    - name: "duplicati-DUPLICATI_ID-status"
      value_template: "{{ value_json.status }}"
    - name: "duplicati-DUPLICATI_ID-time"
      value_template: >
        {% set thistime = value_json.time %}
        {{ as_timestamp(thistime) | timestamp_custom("%Y %M, %d %H:%M") }}

#Todo

    create dynamic nav based on unique duplicati_ids from database
    add FAQ as a page inside the app
    mobile friendly (almost there)
    graph trends in app
    Home Assistant Template sensor
    Don't be lazy and figure out Types in TableClient.tsx component
    get real data for screenshots
    why is bg tile image weird when scrolling on mobile?
    human readable bytes formatter (gb tb)
    human readable duration formatter