Commit Graph

25 Commits

Author SHA1 Message Date
baf34ec1f7
Allow authentication using connection string when targeting Azure Blob Storage (#383)
* Allow authentication using connection string when targeting Azure Blob Storage

* Bail on ambiguous configuration
2024-03-08 20:23:30 +01:00
52c22a1891 Auto prepend caller when wrapping errors 2024-02-16 20:19:58 +01:00
83fa0aae48 Refactor handling of runtime configuration to prepare for reloading 2024-02-16 20:19:58 +01:00
pixxon
1d45062100 Move cron scheduling inside application (#338)
* Move cron scheduling inside application

* Make envvar a fallback and check for errors

* Panic significantly less

* propagate error out of runBackup

* Add structured logging

* FIx error propagation to exit

* Enable the new scheduler by default

* Review fixes

* Added docs and better error propagation
2024-02-12 16:04:12 +01:00
c3daeacecb
Improve Swarm support (#333)
* Query for labeled services as well

* Try scaling down services

* Scale services back up

* Use progress tool from Docker CLI

* In test, label both services

* Clean up error and log messages

* Document scale-up/down approach in docs

* Downgrade Docker CLI to match client

* Document services stats

* Do not rely on PreviousSpec for storing desired replica count

* Log warnings from Docker when updating services

* Check whether container and service labels collide

* Document script behavior on label collision

* Add additional check if all containers have been removed

* Scale services concurrently

* Move docker interaction code into own file

* Factor out code for service updating

* Time out after five minutes of not reaching desired container count

* Inline handling of in-swarm container level restart

* Timer is more suitable for timeout race

* Timeout when scaling down services should be configurable

* Choose better filename

* Reflect changes in naming

* Rename and deprecate BACKUP_STOP_CONTAINER_LABEL

* Improve logging

* Further simplify logging
2024-01-31 12:17:41 +01:00
MaxJa4
336c5bed71
Replace Gzip with PGzip (#266)
* Replace Gzip with optimized PGzip. Add concurrency option.

* Add shortened timeout for 'dc down' too.

* Add NaturalNumberZero to allow zero.

* Add test for concurrency=0

* Rename to GZIP_PARALLELISM

* Fix block size. Fix compression level. Fix CI.

* Refactor compression writer fetching. Renamed WholeNumber
2023-09-03 16:49:52 +02:00
MaxJa4
43c4961116
Support _FILE based configuration values (#264)
* Replace envconfig with env

* Adjust config options and processing

* Added _FILE variant for all password vars.

* Try pathenvconfig

* Revert everything so far

* Use our fork of envconfig with custom lookup

* Use our fork of envconfig with custom lookup

* Test compose timeout option

* Remove secret resolving and specific _FILE config

* Fix timing issue in swarm tests

* Revert "Test compose timeout option"

This reverts commit ab50b21748, reversing
changes made to 0282514b2b.

Revert "Test compose timeout option"

This reverts commit 0282514b2b.

* Use offen/envconfig v1.5.0

* Add info about _FILE in README

* Value > File. Panic on file error. Panic on duplicate presence.

* Test panic on duplicate vars and panic on file error.
2023-08-30 20:14:24 +02:00
MaxJa4
ad4e2af83f
Exclude specific backends from pruning (#262)
* Skip backends while pruning

* Add pruning test step and silence download log for better readability

* Add test cases for pruning in all backends

Also add -q or --quiet-pull to all tests.

* Add test case for skipping backends while pruning

* Adjusted test logging, generate new test spec file

* Gitignore for temp test file
2023-08-27 19:19:11 +02:00
MaxJa4
5fcc96edf9
Cleanup: Lint warnings and deprecated packages (#263)
* Fix lint warnings and std lib deprecations

* Replace deprecated std lib with maintained drop-in replacement fork

Backwartds compatible with original package and suggested by std lib due to security and stability issues.

* OAuth2 is now a direct dependency due to Dropbox

* Undo change

* Revert "Replace deprecated std lib with maintained drop-in replacement fork"

This reverts commit 2887bd409f.

* Update channel handling

* Add linter for PRs

* Rename CI, fetch all issues, add govet
2023-08-27 18:14:55 +02:00
MaxJa4
e08a3303bf
Add new storage backend: Dropbox (#103) (#251)
* Add new storage backend: Dropbox (#103)

* Remove duplicate check

* Add concurrency level for parallel upload to dropbox.

* Fixed some instabilites. Changed default concurrency to 6.

* Added some env config vars to readme. WIP

* Wrap errors for storage backend creation.

* Fixed token issue, added OAuth2 including recipe and docs.

* Readme typo fix

* Test for dropbox integration

* Update info and TOC

* Missed a file

* Docker-compose fix

* Fix endpoint connection

* Fix container names

* Fix log fetching

* Fix log fetching (again)

* Print command output to logs

* Addressing comments part 1

* Address comments part 2

* OpenAPI Mock spec path adjusted
* Dropbox FileMetadata reflection refactored
* NaturalNumber type added

* Add OAuth2 mock server for CI testing

* Fix env name of oauth2 endpoint

* Remove hostname

* Add forgotten change to commit...

* Fix oauth2 endpoint

"Worked on my machine"

* Try again

* Try suggested hostname again

* Fix docker internal DNS resolving issues (as suggested by oauth2 mock docs)

* Add docker network, remove hostname

* Network not external

* Last hostname try

* Add more delay, add oauth2 endpoint log

* Temp CI log output of command even when failing

* Try different config and method

* Add custom server-hostname. Rename test folder to accellerate debugging

* Try that fix again

* Adding quotes

* Port fix attempt

* Try localhost

* Try extra hosts

* Change network mode

* Undo some changes

* Use static IP

* Remove specific IP binding

* Change to default net driver

* Fix static IP

* Squash for revert

* Revert "Squash for revert"

This reverts commit e9b617be9a.

* Actual fix for CI testing from #257
2023-08-24 19:33:47 +02:00
Michal Middleton
67e7288855
Add support for zstd compression (#249)
Co-authored-by: Michal Middleton <jafa81@gmail.com>
2023-08-19 19:20:13 +02:00
Erwan LE PRADO
5ea9a7ce15
feat: add better handler for part size (#214)
* feat: add better handler for part size


fix: use local file 


fix: try with another path


fix: use bytes 


chore: go back


go back readme


goback


goback


goback

* chore: better handling

* fix: typo readme

* chore: wrong comparaison

* fix: typo
2023-06-02 16:30:02 +02:00
5c04e11f10
Add support for Azure Blob Storage (#171)
* Scaffold Azure storage backend that does nothing yet

* Implement copy for Azure Blob Storage

* Set up automated testing for Azure Storage

* Implement pruning for Azure blob storage

* Add documentation for Azure Blob Storage

* Add support for remote path

* Add azure to notifications doc

* Tidy go.mod file

* Allow use of managed identity credential

* Use volume in tests

* Auto append trailing slash to endpoint if needed, clarify docs, tidy mod file
2023-01-11 21:40:48 +01:00
9534cde7d9
Allow use of a custom ca cert when working against S3 storages (#170) 2022-12-22 14:37:51 +01:00
5c7856feb3 Consider failed casting to error response, use established minio bootstrap in tests 2022-10-13 19:40:41 +02:00
pixxon
b5cc1262e2
add aws secret handling (#161)
* add aws secret handling

* make it look go-ish

* fix tests

* whitespace

* sleep a bit
2022-10-12 19:14:57 +02:00
MaxJa4
279844ccfb Added abstract helper interface for all storage backends (#135)
* Added abstract helper interface and implemented it for all storage backends

* Moved storage client initializations also to helper classes

* Fixed ssh init issue

* Moved script parameter to helper struct to simplify script init.

* Created sub modules. Enhanced abstract implementation.

* Fixed config issue

* Fixed declaration issues. Added config to interface.

* Added StorageProviders to unify all backends.

* Cleanup, optimizations, comments.

* Applied discussed changes. See description.

Moved modules to internal packages.
Replaced StoragePool with slice.
Moved conditional for init of storage backends back to script.

* Fix docker build issue

* Fixed accidentally removed local copy condition.

* Delete .gitignore

* Renaming/changes according to review

Renamed Init functions and interface.
Replaced config object with specific config values.
Init func returns interface instead of struct.
Removed custom import names where possible.

* Fixed auto-complete error.

* Combined copy instructions into one layer.

* Added logging func for storages.

* Introduced logging func for errors too.

* Missed an error message

* Moved config back to main. Optimized prune stats handling.

* Move stats back to main package

* Code doc stuff

* Apply changes from #136

* Replace name field with function.

* Changed receiver names from stg to b.

* Renamed LogFuncDef to Log

* Removed redundant package name.

* Renamed storagePool to storages.

* Simplified creation of new storage backend.

* Added initialization for storage stats map.

* Invert .dockerignore patterns.

* Fix package typo
2022-08-18 12:37:45 +02:00
Erwan LE PRADO
3661a4b49b
feat: Add storage class header (#119)
* feat: Add storage class header

* doc: change the readme

* chore: Remove the unnecessary default  value
2022-07-06 13:18:12 +02:00
1892d56ff6
Change default value for SSH identity file (#108)
* Change default value for SSH identity file

* Force remove write protected file in tests
2022-06-17 11:28:29 +02:00
İbrahim Akyel
0b205fe6dc
SSH Backup Storage Support (#107)
* SSH Client implemented

* Private key auth implemented
Code refactoring

* Refactoring

* Passphrase renamed to IdentityPassphrase
Default private key location changed to .ssh/id
2022-06-17 11:06:15 +02:00
b52b271bac
Allow for the exclusion of files from backups (#100)
* Hoist walking of files so it can be used for features other than archive creation

* Add option to ignore files from backup using glob patterns

* Use Regexp instead of glob for exclusion

* Ignore artifacts

* Add teardown to test

* Allow single Re for filtering only

* Add documentation

* Use MatchString on re, add bad input to message in case of error
2022-05-08 11:20:38 +02:00
94a1edc4ad
Allow disabling of certificate verification for WebDAV (#98) 2022-04-20 14:16:59 +02:00
da8c63f755
Support identical cron schedule (#87)
* Retry on lock being unavailable

* Refactor locking to return plain error

* Collect LockedTime in stats

* Add test case

* Add documentation for LOCK_TIMEOUT

* Log in case lock needs to be awaited

* Release resources created for awaiting lock
2022-03-25 18:26:34 +01:00
0504a92a1f
Add option to run pre/post commands for any container (#73)
* Add option to run pre commands on arbitrary container

* Correctly handle quoted args in commands

* Provide defaults for test version arg

* Allow filtering of target containers

* Add documentation on exec commands

* Use mysqldump in exec test

* Add mysqldump section to recipes

* Also run commands test in swarm mode

* Use name instead of id

* Add syntax highlighting

* Add missing license headers
2022-02-22 07:53:33 +01:00
a57e93d01e
Split source into multiple files, deduplicate pruning logic, do not parse templates when notifications are not used (#63)
* Split code into multiple files

* Deduplicate logic for pruning backups against different storages

* Only parse templates when notifications are enabled

* Use better description
2022-02-13 10:52:19 +01:00