Commit Graph

135 Commits

Author SHA1 Message Date
2d37e08743
Use go 1.20, join errors using stdlib (#182)
* Use go 1.20, join errors using stdlib

* Use go 1.20 proper
2023-02-02 21:07:25 +01:00
5c04e11f10
Add support for Azure Blob Storage (#171)
* Scaffold Azure storage backend that does nothing yet

* Implement copy for Azure Blob Storage

* Set up automated testing for Azure Storage

* Implement pruning for Azure blob storage

* Add documentation for Azure Blob Storage

* Add support for remote path

* Add azure to notifications doc

* Tidy go.mod file

* Allow use of managed identity credential

* Use volume in tests

* Auto append trailing slash to endpoint if needed, clarify docs, tidy mod file
2023-01-11 21:40:48 +01:00
63961cd826
Pass file location to lifecycle commands (#173)
* Add test case for extending image and calling through to rsync

* Keep backup file location env var

* Add documentation

* Work against untared content in test
2022-12-30 16:07:34 +01:00
9534cde7d9
Allow use of a custom ca cert when working against S3 storages (#170) 2022-12-22 14:37:51 +01:00
5c7856feb3 Consider failed casting to error response, use established minio bootstrap in tests 2022-10-13 19:40:41 +02:00
pixxon
b5cc1262e2
add aws secret handling (#161)
* add aws secret handling

* make it look go-ish

* fix tests

* whitespace

* sleep a bit
2022-10-12 19:14:57 +02:00
00c83dfac7 Fix more error strings 2022-09-15 10:49:45 +02:00
a6ec128cab
Run copying and pruning against multiple storages in parallel (#152) 2022-09-01 14:38:04 +02:00
b60c747448 Fix WebDAV spelling, remove some inconsistencies (#143)
* Simplify logging, fix WebDAV spelling

* Define options types per package

* Move util functions that are not used cross package

* Add per file license headers

* Rename config type
2022-08-18 12:37:45 +02:00
MaxJa4
279844ccfb Added abstract helper interface for all storage backends (#135)
* Added abstract helper interface and implemented it for all storage backends

* Moved storage client initializations also to helper classes

* Fixed ssh init issue

* Moved script parameter to helper struct to simplify script init.

* Created sub modules. Enhanced abstract implementation.

* Fixed config issue

* Fixed declaration issues. Added config to interface.

* Added StorageProviders to unify all backends.

* Cleanup, optimizations, comments.

* Applied discussed changes. See description.

Moved modules to internal packages.
Replaced StoragePool with slice.
Moved conditional for init of storage backends back to script.

* Fix docker build issue

* Fixed accidentally removed local copy condition.

* Delete .gitignore

* Renaming/changes according to review

Renamed Init functions and interface.
Replaced config object with specific config values.
Init func returns interface instead of struct.
Removed custom import names where possible.

* Fixed auto-complete error.

* Combined copy instructions into one layer.

* Added logging func for storages.

* Introduced logging func for errors too.

* Missed an error message

* Moved config back to main. Optimized prune stats handling.

* Move stats back to main package

* Code doc stuff

* Apply changes from #136

* Replace name field with function.

* Changed receiver names from stg to b.

* Renamed LogFuncDef to Log

* Removed redundant package name.

* Renamed storagePool to storages.

* Simplified creation of new storage backend.

* Added initialization for storage stats map.

* Invert .dockerignore patterns.

* Fix package typo
2022-08-18 12:37:45 +02:00
MaxJa4
2b7f0c52c0
Print more error info for minio (#136)
* Print more error info for minio

* Unpacked error info
2022-08-15 09:25:32 +02:00
b441cf3e2b
Fine grained labels (#115)
* Refactor label command mechanism to be more flexible

* Run all steps wrapped in labeled commands

* Rename methods to be in line with lifecycle

* Deprecate exec-pre and exec-post labels

* Add documentation

* Use type alias for lifecycle phases

* Fix bad imports

* Fix command lookup for deprecated labels

* Use more generic naming for lifecycle phase

* Fail on erroneous post command

* Update documentation
2022-07-10 10:36:56 +02:00
Erwan LE PRADO
3661a4b49b
feat: Add storage class header (#119)
* feat: Add storage class header

* doc: change the readme

* chore: Remove the unnecessary default  value
2022-07-06 13:18:12 +02:00
342ae5910e
Add env template helper (#121) 2022-07-06 13:16:32 +02:00
1892d56ff6
Change default value for SSH identity file (#108)
* Change default value for SSH identity file

* Force remove write protected file in tests
2022-06-17 11:28:29 +02:00
Ä°brahim Akyel
0b205fe6dc
SSH Backup Storage Support (#107)
* SSH Client implemented

* Private key auth implemented
Code refactoring

* Refactoring

* Passphrase renamed to IdentityPassphrase
Default private key location changed to .ssh/id
2022-06-17 11:06:15 +02:00
b52b271bac
Allow for the exclusion of files from backups (#100)
* Hoist walking of files so it can be used for features other than archive creation

* Add option to ignore files from backup using glob patterns

* Use Regexp instead of glob for exclusion

* Ignore artifacts

* Add teardown to test

* Allow single Re for filtering only

* Add documentation

* Use MatchString on re, add bad input to message in case of error
2022-05-08 11:20:38 +02:00
94a1edc4ad
Allow disabling of certificate verification for WebDAV (#98) 2022-04-20 14:16:59 +02:00
1b1fc4856c
List objects recursively when selecting candidates from S3 (#92) 2022-04-15 11:05:52 +02:00
e81c34b8fc
Consider S3 Path when selecting candidates for pruning (#91) 2022-04-13 17:09:37 +02:00
51af8c3c77
Deprecate BACKUP_FROM_SNAPSHOT (#81) 2022-03-25 18:28:58 +01:00
da8c63f755
Support identical cron schedule (#87)
* Retry on lock being unavailable

* Refactor locking to return plain error

* Collect LockedTime in stats

* Add test case

* Add documentation for LOCK_TIMEOUT

* Log in case lock needs to be awaited

* Release resources created for awaiting lock
2022-03-25 18:26:34 +01:00
b8f71b04a1
Use errgroup for running commands in parallel (#83) 2022-03-10 11:09:39 +01:00
5f3832d621
Consider prefix rules when pruning WebDAV storages (#79) 2022-03-05 13:33:15 +01:00
ae50a3ac4f Add attribution to code taken from moby repository 2022-03-04 16:40:34 +01:00
9155b4d130 Add missing print directive, fix go.mod 2022-02-23 10:12:57 +01:00
0504a92a1f
Add option to run pre/post commands for any container (#73)
* Add option to run pre commands on arbitrary container

* Correctly handle quoted args in commands

* Provide defaults for test version arg

* Allow filtering of target containers

* Add documentation on exec commands

* Use mysqldump in exec test

* Add mysqldump section to recipes

* Also run commands test in swarm mode

* Use name instead of id

* Add syntax highlighting

* Add missing license headers
2022-02-22 07:53:33 +01:00
3ded77448c
Do not skip directories when creating tar archive (#72)
* Update targz library to include potential ownership fix

* Move archive logic to main repo

* Remove assertions for debugging

* Use relative path in assertion

* Strip local part from archive location

* Log when extracting in tests

* Fix trimming of prfix

* Add license info to archive.go file

* Undo change in test assertion

* Add test checking for preserved file ownership

* use same postgres version in tests

* Wrap errors when archiving, handle deletion at script layer
2022-02-22 07:49:24 +01:00
58b42b9036
Supporting proxied Docker APIs through DOCKER_HOST (#70) 2022-02-18 09:08:21 +01:00
Mauro Molin
30265c14ba
Fixed TookTime (#66) 2022-02-14 17:32:05 +01:00
a57e93d01e
Split source into multiple files, deduplicate pruning logic, do not parse templates when notifications are not used (#63)
* Split code into multiple files

* Deduplicate logic for pruning backups against different storages

* Only parse templates when notifications are enabled

* Use better description
2022-02-13 10:52:19 +01:00
3e17d1b123
Ensure end time is recorded for unsuccessful runs too (#62)
* Ensure end time is also recorded for unsuccessful runs

* Clean up integration tests
2022-02-13 09:41:36 +01:00
Mauro Molin
8dfdd14527
Added custom notification messages using text/template (#60)
* Added custom notification messages using text/template

* Change notification template path and removed automatic newline trim

* Added stats and changed structure of template params

* Stat file hotfix

* Embedded and fixed default notification templates


Fix

* Changed Output to LogOutput

* Changed stats integer to unsigned

* Bytes formatting in template func


fix

* Changed Archives to Storages

* Removed unecessary sleep for pruning leeway

* Set EndTime after pruning is completed

* Added custom notifications documentation

* Added 5s sleep in swarm test

* Fixed documentation

* Dockerfile copies all files in cmd/backup
2022-02-11 20:05:16 +01:00
6fe629ce87
Allow path to be set for bucket storage (#52) 2022-01-25 21:16:16 +01:00
1db896f7cf Tweak README, improve client naming, tidy go.mod file 2022-01-22 13:35:13 +01:00
Kaerbr
6ded00aa06
Support Nextcloud / WebDav (#48)
* add studio-b12/gowebdav to be able to upload to webdav server

* make sure all env variables are present for webdav upload

* implement file upload to WebDav server

directory defaults to the base directory

* docs: add the new feature to the documentation

* if no WebDav env variable are given throw no error

* docs: use more elegant english :D

Co-authored-by: Frederik Ring <frederik.ring@gmail.com>

* docs: use official spelling of "WebDAV"

* perf: golang likes to return early instead of having an else block

* use WEBDAV_PATH instead of WEBDAV_DIRECTORY

* use split_words for more convenience

like shown here: https://github.com/kelseyhightower/envconfig#struct-tag-support

* simplify

* feat: add pruning of files in WebDAV remote

Based on / Inspired by the minio/S3 implementation of pruning remote files.

* remove logging from the development

* test: first try implementing tests

Sandly I have to use the remote pipeline -- local wont work for me.

* test: adapt used volume names

* test: specify image only once!

* test: minio AND webdav data should be present

* test: backups on WebDAV remote are laying in the root directory

* test: the webdav server stores date in /var/lib/dav

* trying with data subfolder

* test: 1 container was added so the number raised from 3 to 4

* webdav  subfolder is "data" not "backup"

* fix: password AND username must be defined

not password OR username

* improve logging

* feat: if the given path on the server isnt preset it will be created

* test: add creation of new folder for webdav to tests

Co-authored-by: Frederik Ring <frederik.ring@gmail.com>
2022-01-22 13:29:21 +01:00
32575c831e Also expand env vars in pruning prefix if configured 2021-12-23 09:22:56 +01:00
c062710ce8
Allow for env substitution in backup filename (#39) 2021-12-22 14:39:46 +01:00
4207146fb6 Refactor calling of hooks on exit 2021-12-18 10:31:12 +01:00
1f727f698f Run hooks in order of severity 2021-12-18 10:31:12 +01:00
88c90a206c Use int comparison for checking hooks 2021-12-18 10:31:12 +01:00
8bad0656b3 Enable notifications on multiple levels 2021-12-18 10:31:12 +01:00
08d78a0bd6 allow sending notifications to multiple channels 2021-12-18 10:31:12 +01:00
5a6ce81b58 update github.com/otai/copy, use PreserveOwner option 2021-11-29 08:40:55 +01:00
7bc5b2ccef fix minor error scoping mistakes 2021-11-28 20:06:24 +01:00
210c7d4540
Reuse hook mechanism for scheduling clean up tasks (#33)
* reuse hook mechanism for scheduling clean up tasks

* register hooks before creating files or dirs

* fix logging order

* use typed hook levels
2021-11-08 19:10:10 +01:00
schwannden
411c39ee72
create a snapshot before creating tar archive (#32)
* create a snapshot before creating tar archive

* safeguard snapshot removal and make snapshot optional

* fix typo, make sure remove snapshot failure triggers failure hook

Co-authored-by: Schwannden Kuo <schwannden@mobagel.com>
2021-11-08 08:39:18 +01:00
0c666d0c88 use lstat when checking whether file is a symlink 2021-11-03 18:07:55 +01:00
a0402b407d fix fileinfo mode comparison when checking for symlinks 2021-11-03 18:03:44 +01:00
3193e88fc0 os.FileInfo cannot be used for deleting files as it does not contain a full path 2021-11-02 06:40:37 +01:00
f946f36fb0 exclude symlinks from candidates when pruning local files
Previously, symlinks would be included in the set of candidates, but would
be skipped when pruning. This could lead to a wrong number of candidates
being printed in the log messages.
2021-10-29 09:00:37 +02:00
5245b5882f update README, save some indentation 2021-10-28 19:55:39 +02:00
schwannden
7f0f173115
adding option to skip tls verification error (#30)
* adding option to skip tls verification error

* merge options

* removed merged option from README

Co-authored-by: Schwannden Kuo <schwannden@mobagel.com>
2021-10-28 19:51:35 +02:00
c59b40f2df automatically create symlink to latest local backup if configured 2021-10-01 18:19:24 +02:00
bd73a2b5e4 allow s3 authentication via IAM role 2021-09-30 19:24:43 +02:00
53c257065e ensure script always tries to remove local artifacts even when backup failed 2021-09-12 10:48:19 +02:00
160a47e90b allow registering hooks at different levels 2021-09-09 16:55:49 +02:00
59660ec5c7 include exit log message in notification 2021-09-09 11:08:05 +02:00
88368197c1 implement email notifications on failed backup runs 2021-09-09 09:00:23 +02:00
e46968ed79 call error hooks on script failure 2021-09-09 08:12:07 +02:00
2c06f81503 collect all log output in buffer so it could be used in notifications 2021-09-09 07:24:18 +02:00
8aba98c012 use forked version of package targz 2021-09-04 10:08:06 +02:00
ede94bcd88 display all error messages instead of first one 2021-08-29 19:39:51 +02:00
aae97a5617 try restarting even when stopping some containers failed 2021-08-29 18:51:05 +02:00
825cbb50ef always use background context directly 2021-08-29 18:26:40 +02:00
bea203af3d improve documentation 2021-08-29 18:16:04 +02:00
6034e6a902 print proper local archive in log message 2021-08-29 08:36:45 +02:00
d0eca0a179 fix container stop execution order 2021-08-26 16:22:24 +02:00
a0fe2cf42d handle errors on container restart 2021-08-26 12:50:22 +02:00
5334ff1a5a refactor script initialization 2021-08-25 07:48:20 +02:00
e73256ad70 do not use start time as deadline 2021-08-24 09:15:43 +02:00
e0c4adc563 move handling of config to script layer 2021-08-24 09:01:44 +02:00
2469597848 fix lockfile mechanism 2021-08-23 18:46:49 +02:00
b1c4bee85d use buffered reader to write to encryption mechanism 2021-08-23 17:34:13 +02:00
ec87bd27e7 do not use scanner to write file in chunks 2021-08-23 15:19:50 +02:00
f4f4fa9e74 use full filepath when pruning local backups 2021-08-23 14:56:04 +02:00
7086c6e645 read backup in small chunks when encrypting 2021-08-23 14:48:33 +02:00
411a62a6c7 shorten log messages 2021-08-23 14:48:33 +02:00
5a2bf48ec6 make sure backup also runs when socket isn't present 2021-08-23 14:48:33 +02:00
07b06cf0ba read all configuration in init 2021-08-23 14:48:33 +02:00
4c80494433 use go native strftime version 2021-08-23 14:48:33 +02:00
7244725c5b fix location of success message for having created local backup 2021-08-23 14:48:33 +02:00
d195e8967f improve logging messages 2021-08-23 14:48:33 +02:00
188c14c00f add insecure option, update docs 2021-08-23 14:48:33 +02:00
da9458724f adapt repo layout to go 2021-08-23 14:48:33 +02:00