diff --git a/README.md b/README.md index 800c373..4650b40 100644 --- a/README.md +++ b/README.md @@ -16,10 +16,11 @@ It handles __recurring or one-off backups of Docker volumes__ to a __local direc - [One-off backups using Docker CLI](#one-off-backups-using-docker-cli) - [Configuration reference](#configuration-reference) - [How to](#how-to) - - [Stopping containers during backup](#stopping-containers-during-backup) + - [Stop containers during backup](#stop-containers-during-backup) - [Automatically pruning old backups](#automatically-pruning-old-backups) - [Send email notifications on failed backup runs](#send-email-notifications-on-failed-backup-runs) - [Customize notifications](#customize-notifications) + - [Run custom commands before / after backup](#run-custom-commands-before--after-backup) - [Encrypting your backup using GPG](#encrypting-your-backup-using-gpg) - [Restoring a volume from a backup](#restoring-a-volume-from-a-backup) - [Set the timezone the container runs in](#set-the-timezone-the-container-runs-in) @@ -36,6 +37,7 @@ It handles __recurring or one-off backups of Docker volumes__ to a __local direc - [Running on a custom cron schedule](#running-on-a-custom-cron-schedule) - [Rotating away backups that are older than 7 days](#rotating-away-backups-that-are-older-than-7-days) - [Encrypting your backups using GPG](#encrypting-your-backups-using-gpg) + - [Using mysqldump to prepare the backup](#using-mysqldump-to-prepare-the-backup) - [Running multiple instances in the same setup](#running-multiple-instances-in-the-same-setup) - [Differences to `futurice/docker-volume-backup`](#differences-to-futuricedocker-volume-backup) @@ -278,6 +280,27 @@ You can populate below template according to your requirements and use it as you # BACKUP_STOP_CONTAINER_LABEL="service1" +########### EXECUTING COMMANDS IN CONTAINERS PRE/POST BACKUP + +# It is possible to define commands to be run in any container before and after +# a backup is conducted. The commands themselves are defined in labels like +# `docker-volume-backup.exec-pre=/bin/sh -c 'mysqldump [options] > dump.sql'. +# Several options exist for controlling this feature: + +# By default, any output of such a command is suppressed. If this value +# is configured to be "true", command execution output will be forwarded to +# the backup container's stdout and stderr. + +# EXEC_FORWARD_OUTPUT="true" + +# Without any further configuration, all commands defined in labels will be +# run before and after a backup. If you need more fine grained control, you +# can use this option to set a label that will be used for narrowing down +# the set of eligible containers. When set, an eligible container will also need +# to be labeled as `docker-volume-backup.exec-label=database`. + +# EXEC_LABEL="database" + ########### NOTIFICATIONS # Notifications (email, Slack, etc.) can be sent out when a backup run finishes. @@ -336,7 +359,7 @@ You can work around this by either updating `docker-compose` or unquoting your c ## How to -### Stopping containers during backup +### Stop containers during backup In many cases, it will be desirable to stop the services that are consuming the volume you want to backup in order to ensure data integrity. This image can automatically stop and restart containers and services (in case you are running Docker in Swarm mode). @@ -436,6 +459,63 @@ Overridable template names are: `title_success`, `body_success`, `title_failure` For a full list of available variables and functions, see [this page](https://github.com/offen/docker-volume-backup/blob/master/docs/NOTIFICATION-TEMPLATES.md). +### Run custom commands before / after backup + +In certain scenarios it can be required to run specific commands before and after a backup is taken (e.g. dumping a database). +When mounting the Docker socket into the `docker-volume-backup` container, you can define pre- and post-commands that will be run in the context of the target container. +Such commands are defined by specifying the command in a `docker-volume-backup.exec-[pre|post]` label. + +Taking a database dump using `mysqldump` would look like this: + +```yml +version: '3' + +services: + # ... define other services using the `data` volume here + database: + image: mariadb + volumes: + - backup_data:/tmp/backups + labels: + - docker-volume-backup.exec-pre=/bin/sh -c 'mysqldump --all-databases > /backups/dump.sql' + +volumes: + backup_data: +``` + +Due to Docker limitations, you currently cannot use any kind of redirection in these commands unless you pass the command to `/bin/sh -c` or similar. +I.e. instead of using `echo "ok" > ok.txt` you will need to use `/bin/sh -c 'echo "ok" > ok.txt'`. + +If you need fine grained control about which container's commands are run, you can use the `EXEC_LABEL` configuration on your `docker-volume-backup` container: + +```yml +version: '3' + +services: + database: + image: mariadb + volumes: + - backup_data:/tmp/backups + labels: + - docker-volume-backup.exec-pre=/bin/sh -c 'mysqldump --all-databases > /tmp/volume/dump.sql' + - docker-volume-backup.exec-label=database + + backup: + image: offen/docker-volume-backup:latest + environment: + EXEC_LABEL: database + volumes: + - data:/backup/dump:ro + - /var/run/docker.sock:/var/run/docker.sock:ro + +volumes: + backup_data: +``` + + +The backup procedure is guaranteed to wait for all `pre` commands to finish. +However there are no guarantees about the order in which they are run, which could also happen concurrently. + ### Encrypting your backup using GPG The image supports encrypting backups using GPG out of the box. @@ -739,6 +819,32 @@ volumes: data: ``` +### Using mysqldump to prepare the backup + +```yml +version: '3' + +services: + database: + image: mariadb:latest + labels: + - docker-volume-backup.exec-pre=/bin/sh -c 'mysqldump -psecret --all-databases > /tmp/dumps/dump.sql' + volumes: + - app_data:/tmp/dumps + backup: + image: offen/docker-volume-backup:latest + environment: + BACKUP_FILENAME: db.tar.gz + BACKUP_CRON_EXPRESSION: "0 2 * * *" + volumes: + - ./local:/archive + - data:/backup/data:ro + - /var/run/docker.sock:/var/run/docker.sock + +volumes: + data: +``` + ### Running multiple instances in the same setup ```yml @@ -780,12 +886,12 @@ This image is heavily inspired by `futurice/docker-volume-backup`. We decided to - The original image is based on `ubuntu` and requires additional tools, making it heavy. This version is roughly 1/25 in compressed size (it's ~12MB). -- The original image uses a shell script, when this version is written in Go, which makes it easier to extend and maintain (more verbose too). +- The original image uses a shell script, when this version is written in Go. - The original image proposed to handle backup rotation through AWS S3 lifecycle policies. This image adds the option to rotate away old backups through the same command so this functionality can also be offered for non-AWS storage backends like MinIO. Local copies of backups can also be pruned once they reach a certain age. - InfluxDB specific functionality from the original image was removed. - `arm64` and `arm/v7` architectures are supported. - Docker in Swarm mode is supported. -- Notifications on failed backups are supported -- IAM authentication through instance profiles is supported +- Notifications on finished backups are supported. +- IAM authentication through instance profiles is supported. diff --git a/cmd/backup/config.go b/cmd/backup/config.go index 2beca14..c1f481f 100644 --- a/cmd/backup/config.go +++ b/cmd/backup/config.go @@ -39,4 +39,6 @@ type Config struct { WebdavPath string `split_words:"true" default:"/"` WebdavUsername string `split_words:"true"` WebdavPassword string `split_words:"true"` + ExecLabel string `split_words:"true"` + ExecForwardOutput bool `split_words:"true"` } diff --git a/cmd/backup/exec.go b/cmd/backup/exec.go new file mode 100644 index 0000000..003ea5d --- /dev/null +++ b/cmd/backup/exec.go @@ -0,0 +1,122 @@ +// Copyright 2022 - Offen Authors +// SPDX-License-Identifier: MPL-2.0 + +package main + +import ( + "bytes" + "context" + "fmt" + "io/ioutil" + "os" + "strings" + "sync" + + "github.com/cosiner/argv" + "github.com/docker/docker/api/types" + "github.com/docker/docker/api/types/filters" + "github.com/docker/docker/pkg/stdcopy" +) + +func (s *script) exec(containerRef string, command string) ([]byte, []byte, error) { + args, _ := argv.Argv(command, nil, nil) + execID, err := s.cli.ContainerExecCreate(context.Background(), containerRef, types.ExecConfig{ + Cmd: args[0], + AttachStdin: true, + AttachStderr: true, + }) + if err != nil { + return nil, nil, fmt.Errorf("exec: error creating container exec: %w", err) + } + + resp, err := s.cli.ContainerExecAttach(context.Background(), execID.ID, types.ExecStartCheck{}) + if err != nil { + return nil, nil, fmt.Errorf("exec: error attaching container exec: %w", err) + } + defer resp.Close() + + var outBuf, errBuf bytes.Buffer + outputDone := make(chan error) + + go func() { + _, err := stdcopy.StdCopy(&outBuf, &errBuf, resp.Reader) + outputDone <- err + }() + + select { + case err := <-outputDone: + if err != nil { + return nil, nil, fmt.Errorf("exec: error demultiplexing output: %w", err) + } + break + } + + stdout, err := ioutil.ReadAll(&outBuf) + if err != nil { + return nil, nil, fmt.Errorf("exec: error reading stdout: %w", err) + } + stderr, err := ioutil.ReadAll(&errBuf) + if err != nil { + return nil, nil, fmt.Errorf("exec: error reading stderr: %w", err) + } + + res, err := s.cli.ContainerExecInspect(context.Background(), execID.ID) + if err != nil { + return nil, nil, fmt.Errorf("exec: error inspecting container exec: %w", err) + } + + if res.ExitCode > 0 { + return stdout, stderr, fmt.Errorf("exec: running command exited %d", res.ExitCode) + } + + return stdout, stderr, nil +} + +func (s *script) runLabeledCommands(label string) error { + f := []filters.KeyValuePair{ + {Key: "label", Value: label}, + } + if s.c.ExecLabel != "" { + f = append(f, filters.KeyValuePair{ + Key: "label", + Value: fmt.Sprintf("docker-volume-backup.exec-label=%s", s.c.ExecLabel), + }) + } + containersWithCommand, err := s.cli.ContainerList(context.Background(), types.ContainerListOptions{ + Quiet: true, + Filters: filters.NewArgs(f...), + }) + if err != nil { + return fmt.Errorf("runLabeledCommands: error querying for containers", err) + } + + if len(containersWithCommand) == 0 { + return nil + } + + wg := sync.WaitGroup{} + wg.Add(len(containersWithCommand)) + + var cmdErrors []error + for _, container := range containersWithCommand { + go func(c types.Container) { + cmd, _ := c.Labels[label] + s.logger.Infof("Running %s command %s for container %s", label, cmd, strings.TrimPrefix(c.Names[0], "/")) + stdout, stderr, err := s.exec(c.ID, cmd) + if err != nil { + cmdErrors = append(cmdErrors, err) + } + if s.c.ExecForwardOutput { + os.Stderr.Write(stderr) + os.Stdout.Write(stdout) + } + wg.Done() + }(container) + } + + wg.Wait() + if len(cmdErrors) != 0 { + return join(cmdErrors...) + } + return nil +} diff --git a/cmd/backup/main.go b/cmd/backup/main.go index fef06f9..352cd4f 100644 --- a/cmd/backup/main.go +++ b/cmd/backup/main.go @@ -38,6 +38,13 @@ func main() { }() s.must(func() error { + runPostCommands, err := s.runCommands() + defer func() { + s.must(runPostCommands()) + }() + if err != nil { + return err + } restartContainers, err := s.stopContainers() // The mechanism for restarting containers is not using hooks as it // should happen as soon as possible (i.e. before uploading backups or diff --git a/cmd/backup/script.go b/cmd/backup/script.go index bb6c85f..9b4854b 100644 --- a/cmd/backup/script.go +++ b/cmd/backup/script.go @@ -212,6 +212,22 @@ func newScript() (*script, error) { return s, nil } +func (s *script) runCommands() (func() error, error) { + if s.cli == nil { + return noop, nil + } + + if err := s.runLabeledCommands("docker-volume-backup.exec-pre"); err != nil { + return noop, fmt.Errorf("runCommands: error running pre commands: %w", err) + } + return func() error { + if err := s.runLabeledCommands("docker-volume-backup.exec-post"); err != nil { + return fmt.Errorf("runCommands: error running post commands: %w", err) + } + return nil + }, nil +} + // stopContainers stops all Docker containers that are marked as to being // stopped during the backup and returns a function that can be called to // restart everything that has been stopped. diff --git a/go.mod b/go.mod index ea5d2a7..5321020 100644 --- a/go.mod +++ b/go.mod @@ -18,6 +18,7 @@ require ( require ( github.com/Microsoft/go-winio v0.4.17 // indirect github.com/containerd/containerd v1.5.5 // indirect + github.com/cosiner/argv v0.1.0 // indirect github.com/docker/distribution v2.7.1+incompatible // indirect github.com/docker/go-connections v0.4.0 // indirect github.com/docker/go-units v0.4.0 // indirect diff --git a/go.sum b/go.sum index 4496260..1b7ec47 100644 --- a/go.sum +++ b/go.sum @@ -208,6 +208,8 @@ github.com/coreos/go-systemd/v22 v22.1.0/go.mod h1:xO0FLkIi5MaZafQlIrOotqXZ90ih+ github.com/coreos/go-systemd/v22 v22.3.2/go.mod h1:Y58oyj3AT4RCenI/lSvhwexgC+NSVTIJ3seZv2GcEnc= github.com/coreos/pkg v0.0.0-20160727233714-3ac0863d7acf/go.mod h1:E3G3o1h8I7cfcXa63jLwjI0eiQQMgzzUDFVpN/nH/eA= github.com/coreos/pkg v0.0.0-20180928190104-399ea9e2e55f/go.mod h1:E3G3o1h8I7cfcXa63jLwjI0eiQQMgzzUDFVpN/nH/eA= +github.com/cosiner/argv v0.1.0 h1:BVDiEL32lwHukgJKP87btEPenzrrHUjajs/8yzaqcXg= +github.com/cosiner/argv v0.1.0/go.mod h1:EusR6TucWKX+zFgtdUsKT2Cvg45K5rtpCcWz4hK06d8= github.com/cpuguy83/go-md2man/v2 v2.0.0-20190314233015-f79a8a8ca69d/go.mod h1:maD7wRr/U5Z6m/iR4s+kqSMx2CaBsrgA7czyZG/E6dU= github.com/cpuguy83/go-md2man/v2 v2.0.0/go.mod h1:maD7wRr/U5Z6m/iR4s+kqSMx2CaBsrgA7czyZG/E6dU= github.com/creack/pty v1.1.7/go.mod h1:lj5s0c3V2DBrqTV7llrYr5NG6My20zk30Fl46Y7DoTY= diff --git a/test/cli/run.sh b/test/cli/run.sh index 269ef60..4f1f695 100755 --- a/test/cli/run.sh +++ b/test/cli/run.sh @@ -44,7 +44,7 @@ docker run --rm \ --env BACKUP_FILENAME=test.tar.gz \ --env "BACKUP_FROM_SNAPSHOT=true" \ --entrypoint backup \ - offen/docker-volume-backup:$TEST_VERSION + offen/docker-volume-backup:${TEST_VERSION:-canary} docker run --rm -it \ -v backup_data:/data alpine \ diff --git a/test/commands/.gitignore b/test/commands/.gitignore new file mode 100644 index 0000000..4083037 --- /dev/null +++ b/test/commands/.gitignore @@ -0,0 +1 @@ +local diff --git a/test/commands/docker-compose.yml b/test/commands/docker-compose.yml new file mode 100644 index 0000000..0119fcc --- /dev/null +++ b/test/commands/docker-compose.yml @@ -0,0 +1,36 @@ +version: '3.8' + +services: + database: + image: mariadb:10.7 + deploy: + restart_policy: + condition: on-failure + environment: + MARIADB_ROOT_PASSWORD: test + MARIADB_DATABASE: backup + labels: + - docker-volume-backup.exec-pre=/bin/sh -c 'mysqldump -ptest --all-databases > /tmp/volume/dump.sql' + - docker-volume-backup.exec-post=/bin/sh -c 'echo "post" > /tmp/volume/post.txt' + - docker-volume-backup.exec-label=test + volumes: + - app_data:/tmp/volume + + backup: + image: offen/docker-volume-backup:${TEST_VERSION:-canary} + deploy: + restart_policy: + condition: on-failure + environment: + BACKUP_FILENAME: test.tar.gz + BACKUP_CRON_EXPRESSION: 0 0 5 31 2 ? + EXEC_LABEL: test + EXEC_FORWARD_OUTPUT: "true" + volumes: + - archive:/archive + - app_data:/backup/data:ro + - /var/run/docker.sock:/var/run/docker.sock + +volumes: + app_data: + archive: diff --git a/test/commands/run.sh b/test/commands/run.sh new file mode 100644 index 0000000..72d2374 --- /dev/null +++ b/test/commands/run.sh @@ -0,0 +1,62 @@ +#!/bin/sh + +set -e + +cd $(dirname $0) + + +docker-compose up -d +sleep 30 # mariadb likes to take a bit before responding + +docker-compose exec backup backup +sudo cp -r $(docker volume inspect --format='{{ .Mountpoint }}' commands_archive) ./local + +tar -xvf ./local/test.tar.gz +if [ ! -f ./backup/data/dump.sql ]; then + echo "[TEST:FAIL] Could not find file written by pre command." + exit 1 +fi +echo "[TEST:PASS] Found expected file." + +if [ -f ./backup/data/post.txt ]; then + echo "[TEST:FAIL] File created in post command was present in backup." + exit 1 +fi +echo "[TEST:PASS] Did not find unexpected file." + +docker-compose down --volumes +sudo rm -rf ./local + + +echo "[TEST:INFO] Running commands test in swarm mode next." + +docker swarm init + +docker stack deploy --compose-file=docker-compose.yml test_stack + +while [ -z $(docker ps -q -f name=backup) ]; do + echo "[TEST:INFO] Backup container not ready yet. Retrying." + sleep 1 +done + +sleep 20 + +docker exec $(docker ps -q -f name=backup) backup + +sudo cp -r $(docker volume inspect --format='{{ .Mountpoint }}' test_stack_archive) ./local + +tar -xvf ./local/test.tar.gz +if [ ! -f ./backup/data/dump.sql ]; then + echo "[TEST:FAIL] Could not find file written by pre command." + exit 1 +fi +echo "[TEST:PASS] Found expected file." + +if [ -f ./backup/data/post.txt ]; then + echo "[TEST:FAIL] File created in post command was present in backup." + exit 1 +fi +echo "[TEST:PASS] Did not find unexpected file." + +docker stack rm test_stack +docker swarm leave --force diff --git a/test/compose/docker-compose.yml b/test/compose/docker-compose.yml index 7ee40f8..f216b3f 100644 --- a/test/compose/docker-compose.yml +++ b/test/compose/docker-compose.yml @@ -22,7 +22,7 @@ services: - webdav_backup_data:/var/lib/dav backup: &default_backup_service - image: offen/docker-volume-backup:${TEST_VERSION} + image: offen/docker-volume-backup:${TEST_VERSION:-canary} hostname: hostnametoken depends_on: - minio diff --git a/test/notifications/docker-compose.yml b/test/notifications/docker-compose.yml index 8b05484..a8981a6 100644 --- a/test/notifications/docker-compose.yml +++ b/test/notifications/docker-compose.yml @@ -1,8 +1,8 @@ version: '3' services: - backup: &default_backup_service - image: offen/docker-volume-backup:${TEST_VERSION} + backup: + image: offen/docker-volume-backup:${TEST_VERSION:-canary} restart: always environment: BACKUP_FILENAME: test.tar.gz diff --git a/test/swarm/docker-compose.yml b/test/swarm/docker-compose.yml index ba780ea..71f9280 100644 --- a/test/swarm/docker-compose.yml +++ b/test/swarm/docker-compose.yml @@ -18,8 +18,8 @@ services: volumes: - backup_data:/data - backup: &default_backup_service - image: offen/docker-volume-backup:${TEST_VERSION} + backup: + image: offen/docker-volume-backup:${TEST_VERSION:-canary} depends_on: - minio deploy: