Add option to run pre/post commands for any container (#73)

* Add option to run pre commands on arbitrary container

* Correctly handle quoted args in commands

* Provide defaults for test version arg

* Allow filtering of target containers

* Add documentation on exec commands

* Use mysqldump in exec test

* Add mysqldump section to recipes

* Also run commands test in swarm mode

* Use name instead of id

* Add syntax highlighting

* Add missing license headers
This commit is contained in:
Frederik Ring 2022-02-22 07:53:33 +01:00 committed by GitHub
parent 3ded77448c
commit 0504a92a1f
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
14 changed files with 366 additions and 11 deletions

116
README.md
View File

@ -16,10 +16,11 @@ It handles __recurring or one-off backups of Docker volumes__ to a __local direc
- [One-off backups using Docker CLI](#one-off-backups-using-docker-cli) - [One-off backups using Docker CLI](#one-off-backups-using-docker-cli)
- [Configuration reference](#configuration-reference) - [Configuration reference](#configuration-reference)
- [How to](#how-to) - [How to](#how-to)
- [Stopping containers during backup](#stopping-containers-during-backup) - [Stop containers during backup](#stop-containers-during-backup)
- [Automatically pruning old backups](#automatically-pruning-old-backups) - [Automatically pruning old backups](#automatically-pruning-old-backups)
- [Send email notifications on failed backup runs](#send-email-notifications-on-failed-backup-runs) - [Send email notifications on failed backup runs](#send-email-notifications-on-failed-backup-runs)
- [Customize notifications](#customize-notifications) - [Customize notifications](#customize-notifications)
- [Run custom commands before / after backup](#run-custom-commands-before--after-backup)
- [Encrypting your backup using GPG](#encrypting-your-backup-using-gpg) - [Encrypting your backup using GPG](#encrypting-your-backup-using-gpg)
- [Restoring a volume from a backup](#restoring-a-volume-from-a-backup) - [Restoring a volume from a backup](#restoring-a-volume-from-a-backup)
- [Set the timezone the container runs in](#set-the-timezone-the-container-runs-in) - [Set the timezone the container runs in](#set-the-timezone-the-container-runs-in)
@ -36,6 +37,7 @@ It handles __recurring or one-off backups of Docker volumes__ to a __local direc
- [Running on a custom cron schedule](#running-on-a-custom-cron-schedule) - [Running on a custom cron schedule](#running-on-a-custom-cron-schedule)
- [Rotating away backups that are older than 7 days](#rotating-away-backups-that-are-older-than-7-days) - [Rotating away backups that are older than 7 days](#rotating-away-backups-that-are-older-than-7-days)
- [Encrypting your backups using GPG](#encrypting-your-backups-using-gpg) - [Encrypting your backups using GPG](#encrypting-your-backups-using-gpg)
- [Using mysqldump to prepare the backup](#using-mysqldump-to-prepare-the-backup)
- [Running multiple instances in the same setup](#running-multiple-instances-in-the-same-setup) - [Running multiple instances in the same setup](#running-multiple-instances-in-the-same-setup)
- [Differences to `futurice/docker-volume-backup`](#differences-to-futuricedocker-volume-backup) - [Differences to `futurice/docker-volume-backup`](#differences-to-futuricedocker-volume-backup)
@ -278,6 +280,27 @@ You can populate below template according to your requirements and use it as you
# BACKUP_STOP_CONTAINER_LABEL="service1" # BACKUP_STOP_CONTAINER_LABEL="service1"
########### EXECUTING COMMANDS IN CONTAINERS PRE/POST BACKUP
# It is possible to define commands to be run in any container before and after
# a backup is conducted. The commands themselves are defined in labels like
# `docker-volume-backup.exec-pre=/bin/sh -c 'mysqldump [options] > dump.sql'.
# Several options exist for controlling this feature:
# By default, any output of such a command is suppressed. If this value
# is configured to be "true", command execution output will be forwarded to
# the backup container's stdout and stderr.
# EXEC_FORWARD_OUTPUT="true"
# Without any further configuration, all commands defined in labels will be
# run before and after a backup. If you need more fine grained control, you
# can use this option to set a label that will be used for narrowing down
# the set of eligible containers. When set, an eligible container will also need
# to be labeled as `docker-volume-backup.exec-label=database`.
# EXEC_LABEL="database"
########### NOTIFICATIONS ########### NOTIFICATIONS
# Notifications (email, Slack, etc.) can be sent out when a backup run finishes. # Notifications (email, Slack, etc.) can be sent out when a backup run finishes.
@ -336,7 +359,7 @@ You can work around this by either updating `docker-compose` or unquoting your c
## How to ## How to
### Stopping containers during backup ### Stop containers during backup
In many cases, it will be desirable to stop the services that are consuming the volume you want to backup in order to ensure data integrity. In many cases, it will be desirable to stop the services that are consuming the volume you want to backup in order to ensure data integrity.
This image can automatically stop and restart containers and services (in case you are running Docker in Swarm mode). This image can automatically stop and restart containers and services (in case you are running Docker in Swarm mode).
@ -436,6 +459,63 @@ Overridable template names are: `title_success`, `body_success`, `title_failure`
For a full list of available variables and functions, see [this page](https://github.com/offen/docker-volume-backup/blob/master/docs/NOTIFICATION-TEMPLATES.md). For a full list of available variables and functions, see [this page](https://github.com/offen/docker-volume-backup/blob/master/docs/NOTIFICATION-TEMPLATES.md).
### Run custom commands before / after backup
In certain scenarios it can be required to run specific commands before and after a backup is taken (e.g. dumping a database).
When mounting the Docker socket into the `docker-volume-backup` container, you can define pre- and post-commands that will be run in the context of the target container.
Such commands are defined by specifying the command in a `docker-volume-backup.exec-[pre|post]` label.
Taking a database dump using `mysqldump` would look like this:
```yml
version: '3'
services:
# ... define other services using the `data` volume here
database:
image: mariadb
volumes:
- backup_data:/tmp/backups
labels:
- docker-volume-backup.exec-pre=/bin/sh -c 'mysqldump --all-databases > /backups/dump.sql'
volumes:
backup_data:
```
Due to Docker limitations, you currently cannot use any kind of redirection in these commands unless you pass the command to `/bin/sh -c` or similar.
I.e. instead of using `echo "ok" > ok.txt` you will need to use `/bin/sh -c 'echo "ok" > ok.txt'`.
If you need fine grained control about which container's commands are run, you can use the `EXEC_LABEL` configuration on your `docker-volume-backup` container:
```yml
version: '3'
services:
database:
image: mariadb
volumes:
- backup_data:/tmp/backups
labels:
- docker-volume-backup.exec-pre=/bin/sh -c 'mysqldump --all-databases > /tmp/volume/dump.sql'
- docker-volume-backup.exec-label=database
backup:
image: offen/docker-volume-backup:latest
environment:
EXEC_LABEL: database
volumes:
- data:/backup/dump:ro
- /var/run/docker.sock:/var/run/docker.sock:ro
volumes:
backup_data:
```
The backup procedure is guaranteed to wait for all `pre` commands to finish.
However there are no guarantees about the order in which they are run, which could also happen concurrently.
### Encrypting your backup using GPG ### Encrypting your backup using GPG
The image supports encrypting backups using GPG out of the box. The image supports encrypting backups using GPG out of the box.
@ -739,6 +819,32 @@ volumes:
data: data:
``` ```
### Using mysqldump to prepare the backup
```yml
version: '3'
services:
database:
image: mariadb:latest
labels:
- docker-volume-backup.exec-pre=/bin/sh -c 'mysqldump -psecret --all-databases > /tmp/dumps/dump.sql'
volumes:
- app_data:/tmp/dumps
backup:
image: offen/docker-volume-backup:latest
environment:
BACKUP_FILENAME: db.tar.gz
BACKUP_CRON_EXPRESSION: "0 2 * * *"
volumes:
- ./local:/archive
- data:/backup/data:ro
- /var/run/docker.sock:/var/run/docker.sock
volumes:
data:
```
### Running multiple instances in the same setup ### Running multiple instances in the same setup
```yml ```yml
@ -780,12 +886,12 @@ This image is heavily inspired by `futurice/docker-volume-backup`. We decided to
- The original image is based on `ubuntu` and requires additional tools, making it heavy. - The original image is based on `ubuntu` and requires additional tools, making it heavy.
This version is roughly 1/25 in compressed size (it's ~12MB). This version is roughly 1/25 in compressed size (it's ~12MB).
- The original image uses a shell script, when this version is written in Go, which makes it easier to extend and maintain (more verbose too). - The original image uses a shell script, when this version is written in Go.
- The original image proposed to handle backup rotation through AWS S3 lifecycle policies. - The original image proposed to handle backup rotation through AWS S3 lifecycle policies.
This image adds the option to rotate away old backups through the same command so this functionality can also be offered for non-AWS storage backends like MinIO. This image adds the option to rotate away old backups through the same command so this functionality can also be offered for non-AWS storage backends like MinIO.
Local copies of backups can also be pruned once they reach a certain age. Local copies of backups can also be pruned once they reach a certain age.
- InfluxDB specific functionality from the original image was removed. - InfluxDB specific functionality from the original image was removed.
- `arm64` and `arm/v7` architectures are supported. - `arm64` and `arm/v7` architectures are supported.
- Docker in Swarm mode is supported. - Docker in Swarm mode is supported.
- Notifications on failed backups are supported - Notifications on finished backups are supported.
- IAM authentication through instance profiles is supported - IAM authentication through instance profiles is supported.

View File

@ -39,4 +39,6 @@ type Config struct {
WebdavPath string `split_words:"true" default:"/"` WebdavPath string `split_words:"true" default:"/"`
WebdavUsername string `split_words:"true"` WebdavUsername string `split_words:"true"`
WebdavPassword string `split_words:"true"` WebdavPassword string `split_words:"true"`
ExecLabel string `split_words:"true"`
ExecForwardOutput bool `split_words:"true"`
} }

122
cmd/backup/exec.go Normal file
View File

@ -0,0 +1,122 @@
// Copyright 2022 - Offen Authors <hioffen@posteo.de>
// SPDX-License-Identifier: MPL-2.0
package main
import (
"bytes"
"context"
"fmt"
"io/ioutil"
"os"
"strings"
"sync"
"github.com/cosiner/argv"
"github.com/docker/docker/api/types"
"github.com/docker/docker/api/types/filters"
"github.com/docker/docker/pkg/stdcopy"
)
func (s *script) exec(containerRef string, command string) ([]byte, []byte, error) {
args, _ := argv.Argv(command, nil, nil)
execID, err := s.cli.ContainerExecCreate(context.Background(), containerRef, types.ExecConfig{
Cmd: args[0],
AttachStdin: true,
AttachStderr: true,
})
if err != nil {
return nil, nil, fmt.Errorf("exec: error creating container exec: %w", err)
}
resp, err := s.cli.ContainerExecAttach(context.Background(), execID.ID, types.ExecStartCheck{})
if err != nil {
return nil, nil, fmt.Errorf("exec: error attaching container exec: %w", err)
}
defer resp.Close()
var outBuf, errBuf bytes.Buffer
outputDone := make(chan error)
go func() {
_, err := stdcopy.StdCopy(&outBuf, &errBuf, resp.Reader)
outputDone <- err
}()
select {
case err := <-outputDone:
if err != nil {
return nil, nil, fmt.Errorf("exec: error demultiplexing output: %w", err)
}
break
}
stdout, err := ioutil.ReadAll(&outBuf)
if err != nil {
return nil, nil, fmt.Errorf("exec: error reading stdout: %w", err)
}
stderr, err := ioutil.ReadAll(&errBuf)
if err != nil {
return nil, nil, fmt.Errorf("exec: error reading stderr: %w", err)
}
res, err := s.cli.ContainerExecInspect(context.Background(), execID.ID)
if err != nil {
return nil, nil, fmt.Errorf("exec: error inspecting container exec: %w", err)
}
if res.ExitCode > 0 {
return stdout, stderr, fmt.Errorf("exec: running command exited %d", res.ExitCode)
}
return stdout, stderr, nil
}
func (s *script) runLabeledCommands(label string) error {
f := []filters.KeyValuePair{
{Key: "label", Value: label},
}
if s.c.ExecLabel != "" {
f = append(f, filters.KeyValuePair{
Key: "label",
Value: fmt.Sprintf("docker-volume-backup.exec-label=%s", s.c.ExecLabel),
})
}
containersWithCommand, err := s.cli.ContainerList(context.Background(), types.ContainerListOptions{
Quiet: true,
Filters: filters.NewArgs(f...),
})
if err != nil {
return fmt.Errorf("runLabeledCommands: error querying for containers", err)
}
if len(containersWithCommand) == 0 {
return nil
}
wg := sync.WaitGroup{}
wg.Add(len(containersWithCommand))
var cmdErrors []error
for _, container := range containersWithCommand {
go func(c types.Container) {
cmd, _ := c.Labels[label]
s.logger.Infof("Running %s command %s for container %s", label, cmd, strings.TrimPrefix(c.Names[0], "/"))
stdout, stderr, err := s.exec(c.ID, cmd)
if err != nil {
cmdErrors = append(cmdErrors, err)
}
if s.c.ExecForwardOutput {
os.Stderr.Write(stderr)
os.Stdout.Write(stdout)
}
wg.Done()
}(container)
}
wg.Wait()
if len(cmdErrors) != 0 {
return join(cmdErrors...)
}
return nil
}

View File

@ -38,6 +38,13 @@ func main() {
}() }()
s.must(func() error { s.must(func() error {
runPostCommands, err := s.runCommands()
defer func() {
s.must(runPostCommands())
}()
if err != nil {
return err
}
restartContainers, err := s.stopContainers() restartContainers, err := s.stopContainers()
// The mechanism for restarting containers is not using hooks as it // The mechanism for restarting containers is not using hooks as it
// should happen as soon as possible (i.e. before uploading backups or // should happen as soon as possible (i.e. before uploading backups or

View File

@ -212,6 +212,22 @@ func newScript() (*script, error) {
return s, nil return s, nil
} }
func (s *script) runCommands() (func() error, error) {
if s.cli == nil {
return noop, nil
}
if err := s.runLabeledCommands("docker-volume-backup.exec-pre"); err != nil {
return noop, fmt.Errorf("runCommands: error running pre commands: %w", err)
}
return func() error {
if err := s.runLabeledCommands("docker-volume-backup.exec-post"); err != nil {
return fmt.Errorf("runCommands: error running post commands: %w", err)
}
return nil
}, nil
}
// stopContainers stops all Docker containers that are marked as to being // stopContainers stops all Docker containers that are marked as to being
// stopped during the backup and returns a function that can be called to // stopped during the backup and returns a function that can be called to
// restart everything that has been stopped. // restart everything that has been stopped.

1
go.mod
View File

@ -18,6 +18,7 @@ require (
require ( require (
github.com/Microsoft/go-winio v0.4.17 // indirect github.com/Microsoft/go-winio v0.4.17 // indirect
github.com/containerd/containerd v1.5.5 // indirect github.com/containerd/containerd v1.5.5 // indirect
github.com/cosiner/argv v0.1.0 // indirect
github.com/docker/distribution v2.7.1+incompatible // indirect github.com/docker/distribution v2.7.1+incompatible // indirect
github.com/docker/go-connections v0.4.0 // indirect github.com/docker/go-connections v0.4.0 // indirect
github.com/docker/go-units v0.4.0 // indirect github.com/docker/go-units v0.4.0 // indirect

2
go.sum
View File

@ -208,6 +208,8 @@ github.com/coreos/go-systemd/v22 v22.1.0/go.mod h1:xO0FLkIi5MaZafQlIrOotqXZ90ih+
github.com/coreos/go-systemd/v22 v22.3.2/go.mod h1:Y58oyj3AT4RCenI/lSvhwexgC+NSVTIJ3seZv2GcEnc= github.com/coreos/go-systemd/v22 v22.3.2/go.mod h1:Y58oyj3AT4RCenI/lSvhwexgC+NSVTIJ3seZv2GcEnc=
github.com/coreos/pkg v0.0.0-20160727233714-3ac0863d7acf/go.mod h1:E3G3o1h8I7cfcXa63jLwjI0eiQQMgzzUDFVpN/nH/eA= github.com/coreos/pkg v0.0.0-20160727233714-3ac0863d7acf/go.mod h1:E3G3o1h8I7cfcXa63jLwjI0eiQQMgzzUDFVpN/nH/eA=
github.com/coreos/pkg v0.0.0-20180928190104-399ea9e2e55f/go.mod h1:E3G3o1h8I7cfcXa63jLwjI0eiQQMgzzUDFVpN/nH/eA= github.com/coreos/pkg v0.0.0-20180928190104-399ea9e2e55f/go.mod h1:E3G3o1h8I7cfcXa63jLwjI0eiQQMgzzUDFVpN/nH/eA=
github.com/cosiner/argv v0.1.0 h1:BVDiEL32lwHukgJKP87btEPenzrrHUjajs/8yzaqcXg=
github.com/cosiner/argv v0.1.0/go.mod h1:EusR6TucWKX+zFgtdUsKT2Cvg45K5rtpCcWz4hK06d8=
github.com/cpuguy83/go-md2man/v2 v2.0.0-20190314233015-f79a8a8ca69d/go.mod h1:maD7wRr/U5Z6m/iR4s+kqSMx2CaBsrgA7czyZG/E6dU= github.com/cpuguy83/go-md2man/v2 v2.0.0-20190314233015-f79a8a8ca69d/go.mod h1:maD7wRr/U5Z6m/iR4s+kqSMx2CaBsrgA7czyZG/E6dU=
github.com/cpuguy83/go-md2man/v2 v2.0.0/go.mod h1:maD7wRr/U5Z6m/iR4s+kqSMx2CaBsrgA7czyZG/E6dU= github.com/cpuguy83/go-md2man/v2 v2.0.0/go.mod h1:maD7wRr/U5Z6m/iR4s+kqSMx2CaBsrgA7czyZG/E6dU=
github.com/creack/pty v1.1.7/go.mod h1:lj5s0c3V2DBrqTV7llrYr5NG6My20zk30Fl46Y7DoTY= github.com/creack/pty v1.1.7/go.mod h1:lj5s0c3V2DBrqTV7llrYr5NG6My20zk30Fl46Y7DoTY=

View File

@ -44,7 +44,7 @@ docker run --rm \
--env BACKUP_FILENAME=test.tar.gz \ --env BACKUP_FILENAME=test.tar.gz \
--env "BACKUP_FROM_SNAPSHOT=true" \ --env "BACKUP_FROM_SNAPSHOT=true" \
--entrypoint backup \ --entrypoint backup \
offen/docker-volume-backup:$TEST_VERSION offen/docker-volume-backup:${TEST_VERSION:-canary}
docker run --rm -it \ docker run --rm -it \
-v backup_data:/data alpine \ -v backup_data:/data alpine \

1
test/commands/.gitignore vendored Normal file
View File

@ -0,0 +1 @@
local

View File

@ -0,0 +1,36 @@
version: '3.8'
services:
database:
image: mariadb:10.7
deploy:
restart_policy:
condition: on-failure
environment:
MARIADB_ROOT_PASSWORD: test
MARIADB_DATABASE: backup
labels:
- docker-volume-backup.exec-pre=/bin/sh -c 'mysqldump -ptest --all-databases > /tmp/volume/dump.sql'
- docker-volume-backup.exec-post=/bin/sh -c 'echo "post" > /tmp/volume/post.txt'
- docker-volume-backup.exec-label=test
volumes:
- app_data:/tmp/volume
backup:
image: offen/docker-volume-backup:${TEST_VERSION:-canary}
deploy:
restart_policy:
condition: on-failure
environment:
BACKUP_FILENAME: test.tar.gz
BACKUP_CRON_EXPRESSION: 0 0 5 31 2 ?
EXEC_LABEL: test
EXEC_FORWARD_OUTPUT: "true"
volumes:
- archive:/archive
- app_data:/backup/data:ro
- /var/run/docker.sock:/var/run/docker.sock
volumes:
app_data:
archive:

62
test/commands/run.sh Normal file
View File

@ -0,0 +1,62 @@
#!/bin/sh
set -e
cd $(dirname $0)
docker-compose up -d
sleep 30 # mariadb likes to take a bit before responding
docker-compose exec backup backup
sudo cp -r $(docker volume inspect --format='{{ .Mountpoint }}' commands_archive) ./local
tar -xvf ./local/test.tar.gz
if [ ! -f ./backup/data/dump.sql ]; then
echo "[TEST:FAIL] Could not find file written by pre command."
exit 1
fi
echo "[TEST:PASS] Found expected file."
if [ -f ./backup/data/post.txt ]; then
echo "[TEST:FAIL] File created in post command was present in backup."
exit 1
fi
echo "[TEST:PASS] Did not find unexpected file."
docker-compose down --volumes
sudo rm -rf ./local
echo "[TEST:INFO] Running commands test in swarm mode next."
docker swarm init
docker stack deploy --compose-file=docker-compose.yml test_stack
while [ -z $(docker ps -q -f name=backup) ]; do
echo "[TEST:INFO] Backup container not ready yet. Retrying."
sleep 1
done
sleep 20
docker exec $(docker ps -q -f name=backup) backup
sudo cp -r $(docker volume inspect --format='{{ .Mountpoint }}' test_stack_archive) ./local
tar -xvf ./local/test.tar.gz
if [ ! -f ./backup/data/dump.sql ]; then
echo "[TEST:FAIL] Could not find file written by pre command."
exit 1
fi
echo "[TEST:PASS] Found expected file."
if [ -f ./backup/data/post.txt ]; then
echo "[TEST:FAIL] File created in post command was present in backup."
exit 1
fi
echo "[TEST:PASS] Did not find unexpected file."
docker stack rm test_stack
docker swarm leave --force

View File

@ -22,7 +22,7 @@ services:
- webdav_backup_data:/var/lib/dav - webdav_backup_data:/var/lib/dav
backup: &default_backup_service backup: &default_backup_service
image: offen/docker-volume-backup:${TEST_VERSION} image: offen/docker-volume-backup:${TEST_VERSION:-canary}
hostname: hostnametoken hostname: hostnametoken
depends_on: depends_on:
- minio - minio

View File

@ -1,8 +1,8 @@
version: '3' version: '3'
services: services:
backup: &default_backup_service backup:
image: offen/docker-volume-backup:${TEST_VERSION} image: offen/docker-volume-backup:${TEST_VERSION:-canary}
restart: always restart: always
environment: environment:
BACKUP_FILENAME: test.tar.gz BACKUP_FILENAME: test.tar.gz

View File

@ -18,8 +18,8 @@ services:
volumes: volumes:
- backup_data:/data - backup_data:/data
backup: &default_backup_service backup:
image: offen/docker-volume-backup:${TEST_VERSION} image: offen/docker-volume-backup:${TEST_VERSION:-canary}
depends_on: depends_on:
- minio - minio
deploy: deploy: