Commit Graph

66 Commits

Author SHA1 Message Date
c3daeacecb
Improve Swarm support (#333)
* Query for labeled services as well

* Try scaling down services

* Scale services back up

* Use progress tool from Docker CLI

* In test, label both services

* Clean up error and log messages

* Document scale-up/down approach in docs

* Downgrade Docker CLI to match client

* Document services stats

* Do not rely on PreviousSpec for storing desired replica count

* Log warnings from Docker when updating services

* Check whether container and service labels collide

* Document script behavior on label collision

* Add additional check if all containers have been removed

* Scale services concurrently

* Move docker interaction code into own file

* Factor out code for service updating

* Time out after five minutes of not reaching desired container count

* Inline handling of in-swarm container level restart

* Timer is more suitable for timeout race

* Timeout when scaling down services should be configurable

* Choose better filename

* Reflect changes in naming

* Rename and deprecate BACKUP_STOP_CONTAINER_LABEL

* Improve logging

* Further simplify logging
2024-01-31 12:17:41 +01:00
4d7d2e50cf
Guard DinD docker info call with timeout (#315)
* Add debug logging

* Guard DinD docker info with timeout

* Remove debug logging from test runner
2023-12-12 20:28:13 +01:00
0325889ac4
Pruning method logs nonsensical configuration values (#301)
* Pruning method logs nonsensical configuration values

* Adjust test assertion about log output
2023-11-04 12:19:44 +01:00
e7631d8d53
Retry creation of sandbox container (#276)
* Extend sleep when Docker daemon seems ready

* Pruning test shall not write to local fs

* Make sure container names do not conflict
2023-09-19 14:45:09 +02:00
MaxJa4
336c5bed71
Replace Gzip with PGzip (#266)
* Replace Gzip with optimized PGzip. Add concurrency option.

* Add shortened timeout for 'dc down' too.

* Add NaturalNumberZero to allow zero.

* Add test for concurrency=0

* Rename to GZIP_PARALLELISM

* Fix block size. Fix compression level. Fix CI.

* Refactor compression writer fetching. Renamed WholeNumber
2023-09-03 16:49:52 +02:00
1e39ac41f4
Run tests Docker in Docker (#261)
* Try running tests in Docker

* Spawn new container for each test

* Store test artifacts outside of mount

* When requested, build up to date image in test script

* sudo is unneccessary in containerized test env

* Skip azure test

* Backdate fixture file in JSON database

* Pin versions for azure tools

* Mount temp volume for /var/lib/docker to prevent dangling ones created by VOLUME instruction

* Fail backdating tests with message

* Add some documentation on test setup

* Cache images

* Run compose stacks with shortened default timeout
2023-09-02 15:17:46 +02:00
MaxJa4
43c4961116
Support _FILE based configuration values (#264)
* Replace envconfig with env

* Adjust config options and processing

* Added _FILE variant for all password vars.

* Try pathenvconfig

* Revert everything so far

* Use our fork of envconfig with custom lookup

* Use our fork of envconfig with custom lookup

* Test compose timeout option

* Remove secret resolving and specific _FILE config

* Fix timing issue in swarm tests

* Revert "Test compose timeout option"

This reverts commit ab50b21748, reversing
changes made to 0282514b2b.

Revert "Test compose timeout option"

This reverts commit 0282514b2b.

* Use offen/envconfig v1.5.0

* Add info about _FILE in README

* Value > File. Panic on file error. Panic on duplicate presence.

* Test panic on duplicate vars and panic on file error.
2023-08-30 20:14:24 +02:00
MaxJa4
ad4e2af83f
Exclude specific backends from pruning (#262)
* Skip backends while pruning

* Add pruning test step and silence download log for better readability

* Add test cases for pruning in all backends

Also add -q or --quiet-pull to all tests.

* Add test case for skipping backends while pruning

* Adjusted test logging, generate new test spec file

* Gitignore for temp test file
2023-08-27 19:19:11 +02:00
88a4794083
Zstd test does not need to spin up MinIO server (#258) 2023-08-24 22:14:04 +02:00
7011261dc5 Pin version of mock openapi server used in Dropbox test 2023-08-24 22:01:07 +02:00
9ba8143be2 Add license headers to vendored Dropbox swagger spec 2023-08-24 19:52:33 +02:00
MaxJa4
e08a3303bf
Add new storage backend: Dropbox (#103) (#251)
* Add new storage backend: Dropbox (#103)

* Remove duplicate check

* Add concurrency level for parallel upload to dropbox.

* Fixed some instabilites. Changed default concurrency to 6.

* Added some env config vars to readme. WIP

* Wrap errors for storage backend creation.

* Fixed token issue, added OAuth2 including recipe and docs.

* Readme typo fix

* Test for dropbox integration

* Update info and TOC

* Missed a file

* Docker-compose fix

* Fix endpoint connection

* Fix container names

* Fix log fetching

* Fix log fetching (again)

* Print command output to logs

* Addressing comments part 1

* Address comments part 2

* OpenAPI Mock spec path adjusted
* Dropbox FileMetadata reflection refactored
* NaturalNumber type added

* Add OAuth2 mock server for CI testing

* Fix env name of oauth2 endpoint

* Remove hostname

* Add forgotten change to commit...

* Fix oauth2 endpoint

"Worked on my machine"

* Try again

* Try suggested hostname again

* Fix docker internal DNS resolving issues (as suggested by oauth2 mock docs)

* Add docker network, remove hostname

* Network not external

* Last hostname try

* Add more delay, add oauth2 endpoint log

* Temp CI log output of command even when failing

* Try different config and method

* Add custom server-hostname. Rename test folder to accellerate debugging

* Try that fix again

* Adding quotes

* Port fix attempt

* Try localhost

* Try extra hosts

* Change network mode

* Undo some changes

* Use static IP

* Remove specific IP binding

* Change to default net driver

* Fix static IP

* Squash for revert

* Revert "Squash for revert"

This reverts commit e9b617be9a.

* Actual fix for CI testing from #257
2023-08-24 19:33:47 +02:00
Michal Middleton
67e7288855
Add support for zstd compression (#249)
Co-authored-by: Michal Middleton <jafa81@gmail.com>
2023-08-19 19:20:13 +02:00
ba-tno
07afa53cd3
Update shoutrrr to 0.7 (#213)
* Replace docker-compose reference with docker[space]compose

* Update shoutrrr only to 0.7.1

* modules after go mod tidy

* Refer to v0.7 docs of shoutrrr

* Replace docker-compose reference with docker[space]compose

* Update shoutrrr only to 0.7.1

* modules after go mod tidy

* Refer to v0.7 docs of shoutrrr

* Remove 'v' from shoutrrr doc link
2023-04-29 20:14:04 +02:00
808cf8f82d
Local directory can be used instead of volume for storing test artifact (#204) 2023-04-02 19:41:00 +02:00
Diulgher Artiom
7f20036b15
Possibility to use -u (user) option in docker exec (#203)
* Add user option for docker exec

* Add test for user option

* Return test version for image

* remove gitea config file

* refactor tests

* remove comments & fix image name

* add docs

* cleanup

* Update README.md with suggested correction

Co-authored-by: Frederik Ring <frederik.ring@gmail.com>

* fix backup command & bind folder instead of volume

---------

Co-authored-by: tao <generaltao.md@gmail.com>
Co-authored-by: Frederik Ring <frederik.ring@gmail.com>
2023-04-02 11:12:10 +02:00
aee802cb09
Migrate CI setup to GitHub Actions, also publish to GHCR (#199)
* Run tests in GitHub actions

* Do not try to allocate a pseudo TTY when running compose commands

* Try hard disabling TTY allocation

* Use compose plugin

* Test scripts shall not try to allocate a TTY

* Pass correct base version

* Check whether env var is even needed

* Stop running tests in CircleCI

* Run releases from GitHub actions as well

* Manually construct tags to be pushed on release
2023-03-16 19:32:44 +01:00
5c04e11f10
Add support for Azure Blob Storage (#171)
* Scaffold Azure storage backend that does nothing yet

* Implement copy for Azure Blob Storage

* Set up automated testing for Azure Storage

* Implement pruning for Azure blob storage

* Add documentation for Azure Blob Storage

* Add support for remote path

* Add azure to notifications doc

* Tidy go.mod file

* Allow use of managed identity credential

* Use volume in tests

* Auto append trailing slash to endpoint if needed, clarify docs, tidy mod file
2023-01-11 21:40:48 +01:00
9b7af67a26
Run tests using compose v2 (#178) 2023-01-07 18:50:27 +01:00
63961cd826
Pass file location to lifecycle commands (#173)
* Add test case for extending image and calling through to rsync

* Keep backup file location env var

* Add documentation

* Work against untared content in test
2022-12-30 16:07:34 +01:00
9534cde7d9
Allow use of a custom ca cert when working against S3 storages (#170) 2022-12-22 14:37:51 +01:00
92b888e72c Remove debugging remnant from test 2022-10-17 20:41:10 +02:00
3925ac1ee0 Special characters in password do not break GPG test case 2022-10-17 19:42:38 +02:00
5c7856feb3 Consider failed casting to error response, use established minio bootstrap in tests 2022-10-13 19:40:41 +02:00
pixxon
b5cc1262e2
add aws secret handling (#161)
* add aws secret handling

* make it look go-ish

* fix tests

* whitespace

* sleep a bit
2022-10-12 19:14:57 +02:00
26c8ba971f
Add test case for exec label (#132) 2022-07-15 09:34:01 +02:00
b441cf3e2b
Fine grained labels (#115)
* Refactor label command mechanism to be more flexible

* Run all steps wrapped in labeled commands

* Rename methods to be in line with lifecycle

* Deprecate exec-pre and exec-post labels

* Add documentation

* Use type alias for lifecycle phases

* Fix bad imports

* Fix command lookup for deprecated labels

* Use more generic naming for lifecycle phase

* Fail on erroneous post command

* Update documentation
2022-07-10 10:36:56 +02:00
342ae5910e
Add env template helper (#121) 2022-07-06 13:16:32 +02:00
c2a8cc92fc
Untangle tests (#112)
* Isolate S3 test case

* Isolate webdav test case

* Isolate SSH test case

* Isolate local storage test case

* Isolate gpg test case

* Add missing volume mount

* Fix file locations for local test case

* Remove compose test case, use utils

* Use test utils throughout

* Use dedicated tmp dir

* Fix link location that is being tested

* Use dedicated tmp_dirs when working on host fs

* Force delete artifact

* Fix expected filename

* Provide helpful messages on failing tests

* Fix filename

* Use proper volume names

* Fix syntax error, use large resource class

* Use named Docker volumes when referencing them in test scripts

* Add name of test case to logging output
2022-06-23 14:40:29 +02:00
1892d56ff6
Change default value for SSH identity file (#108)
* Change default value for SSH identity file

* Force remove write protected file in tests
2022-06-17 11:28:29 +02:00
Ä°brahim Akyel
0b205fe6dc
SSH Backup Storage Support (#107)
* SSH Client implemented

* Private key auth implemented
Code refactoring

* Refactoring

* Passphrase renamed to IdentityPassphrase
Default private key location changed to .ssh/id
2022-06-17 11:06:15 +02:00
b52b271bac
Allow for the exclusion of files from backups (#100)
* Hoist walking of files so it can be used for features other than archive creation

* Add option to ignore files from backup using glob patterns

* Use Regexp instead of glob for exclusion

* Ignore artifacts

* Add teardown to test

* Allow single Re for filtering only

* Add documentation

* Use MatchString on re, add bad input to message in case of error
2022-05-08 11:20:38 +02:00
94a1edc4ad
Allow disabling of certificate verification for WebDAV (#98) 2022-04-20 14:16:59 +02:00
da8c63f755
Support identical cron schedule (#87)
* Retry on lock being unavailable

* Refactor locking to return plain error

* Collect LockedTime in stats

* Add test case

* Add documentation for LOCK_TIMEOUT

* Log in case lock needs to be awaited

* Release resources created for awaiting lock
2022-03-25 18:26:34 +01:00
508bc07b4f
Disable healthcheck in swarm test (#85) 2022-03-17 11:13:07 +01:00
c9ebb9e14e
Allow multiple schedules in the same container (#78)
* Allow mounting of config directory for multiple schedules

* Add docs for conf.d feature

* Fix behavior on multiple files

* Define default case first in entrypoint script
2022-03-04 13:51:26 +01:00
0504a92a1f
Add option to run pre/post commands for any container (#73)
* Add option to run pre commands on arbitrary container

* Correctly handle quoted args in commands

* Provide defaults for test version arg

* Allow filtering of target containers

* Add documentation on exec commands

* Use mysqldump in exec test

* Add mysqldump section to recipes

* Also run commands test in swarm mode

* Use name instead of id

* Add syntax highlighting

* Add missing license headers
2022-02-22 07:53:33 +01:00
3ded77448c
Do not skip directories when creating tar archive (#72)
* Update targz library to include potential ownership fix

* Move archive logic to main repo

* Remove assertions for debugging

* Use relative path in assertion

* Strip local part from archive location

* Log when extracting in tests

* Fix trimming of prfix

* Add license info to archive.go file

* Undo change in test assertion

* Add test checking for preserved file ownership

* use same postgres version in tests

* Wrap errors when archiving, handle deletion at script layer
2022-02-22 07:49:24 +01:00
180438f1fc
Update ubuntu image used for running integration tests (#67) 2022-02-15 21:05:03 +01:00
3e17d1b123
Ensure end time is recorded for unsuccessful runs too (#62)
* Ensure end time is also recorded for unsuccessful runs

* Clean up integration tests
2022-02-13 09:41:36 +01:00
e6af6efd8a
Add test setup for notification feature (#61)
* Add test case for notification feature

* Fix template data

* bash is needed to interpret test

* Do not use bashisms in test

* Only print FullPath

* Fix assertion
2022-02-12 20:28:38 +01:00
Mauro Molin
8dfdd14527
Added custom notification messages using text/template (#60)
* Added custom notification messages using text/template

* Change notification template path and removed automatic newline trim

* Added stats and changed structure of template params

* Stat file hotfix

* Embedded and fixed default notification templates


Fix

* Changed Output to LogOutput

* Changed stats integer to unsigned

* Bytes formatting in template func


fix

* Changed Archives to Storages

* Removed unecessary sleep for pruning leeway

* Set EndTime after pruning is completed

* Added custom notifications documentation

* Added 5s sleep in swarm test

* Fixed documentation

* Dockerfile copies all files in cmd/backup
2022-02-11 20:05:16 +01:00
Kaerbr
6ded00aa06
Support Nextcloud / WebDav (#48)
* add studio-b12/gowebdav to be able to upload to webdav server

* make sure all env variables are present for webdav upload

* implement file upload to WebDav server

directory defaults to the base directory

* docs: add the new feature to the documentation

* if no WebDav env variable are given throw no error

* docs: use more elegant english :D

Co-authored-by: Frederik Ring <frederik.ring@gmail.com>

* docs: use official spelling of "WebDAV"

* perf: golang likes to return early instead of having an else block

* use WEBDAV_PATH instead of WEBDAV_DIRECTORY

* use split_words for more convenience

like shown here: https://github.com/kelseyhightower/envconfig#struct-tag-support

* simplify

* feat: add pruning of files in WebDAV remote

Based on / Inspired by the minio/S3 implementation of pruning remote files.

* remove logging from the development

* test: first try implementing tests

Sandly I have to use the remote pipeline -- local wont work for me.

* test: adapt used volume names

* test: specify image only once!

* test: minio AND webdav data should be present

* test: backups on WebDAV remote are laying in the root directory

* test: the webdav server stores date in /var/lib/dav

* trying with data subfolder

* test: 1 container was added so the number raised from 3 to 4

* webdav  subfolder is "data" not "backup"

* fix: password AND username must be defined

not password OR username

* improve logging

* feat: if the given path on the server isnt preset it will be created

* test: add creation of new folder for webdav to tests

Co-authored-by: Frederik Ring <frederik.ring@gmail.com>
2022-01-22 13:29:21 +01:00
1f9582df51
Fix handling of empty directories (#44)
* Add test checking whether empty directories are included in backups

* Update targz library to include fix
2021-12-29 10:10:12 +01:00
c062710ce8
Allow for env substitution in backup filename (#39) 2021-12-22 14:39:46 +01:00
3c06bf8102 run cli test using BACKUP_FROM_SNAPSHOT 2021-11-08 08:44:59 +01:00
c59b40f2df automatically create symlink to latest local backup if configured 2021-10-01 18:19:24 +02:00
184b7a1e18 add docs on one off backups using docker cli 2021-09-11 11:21:48 +02:00
5922820ada add test for checking behavior on symlinks 2021-09-04 10:30:34 +02:00
e0c4adc563 move handling of config to script layer 2021-08-24 09:01:44 +02:00