Commit Graph

134 Commits

Author SHA1 Message Date
nick comer
44ad3bbda2
feat: allow backups to be encrypted with age (#432)
GPG is known to have usability issues and is generally cumbersome to
use. age [0] is a modern alternative to GPG that is designed by a
cryptographer that has worked and continues to work on Golang's crypto
packages for years.

Allowing age to be used to encrypt backups dramatically simplifies the
backup process.

[0]: https://age-encryption.org/
2024-08-19 22:49:49 +02:00
Lennart
8a64da4b0b
Feature: PGP Asymmetric Encryption (#456)
* feat: asym encryption

* tests

* docs

* refactor

* logs & errs

* comment

* Update docs/reference/index.md

use correct env var in example

Co-authored-by: Frederik Ring <frederik.ring@gmail.com>

* Update cmd/backup/encrypt_archive.go

use errwarp for initial error msg

Co-authored-by: Frederik Ring <frederik.ring@gmail.com>

* rm orphaned code in encryption functions

* inline readArmoredKeys

* naming -GPG_PUBLIC_KEYS- to GPG_PUBLIC_KEY_RING

* add eror handling for closing func

* use dynamically generated keys for testing

* rm explicit gpg-agent start

* rm unnecessary private_key export

* pass PASSPHRASE correctly to the decryption command

* capture defer errors

* log & err msg

---------

Co-authored-by: Frederik Ring <frederik.ring@gmail.com>
2024-08-11 10:11:23 +02:00
J. Zebedee
f97ce11734
Add "none" compression type (#457)
* Add "none" compression type

* Add "none" compression to docs

* Use passThroughWriteCloser for "none" compression

* Add test for none compression

---------

Co-authored-by: Frederik Ring <frederik.ring@gmail.com>
2024-08-11 10:11:09 +02:00
336e12f874
Add support for Azure storage access tiers (#452) 2024-08-09 15:37:27 +02:00
dependabot[bot]
e4170addb6
Bump github.com/docker/cli from 26.1.4+incompatible to 27.0.3+incompatible (#437)
* Bump github.com/docker/cli

Bumps [github.com/docker/cli](https://github.com/docker/cli) from 26.1.4+incompatible to 27.0.3+incompatible.
- [Commits](https://github.com/docker/cli/compare/v26.1.4...v27.0.3)

---
updated-dependencies:
- dependency-name: github.com/docker/cli
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* Update all Docker versions to 27

* Swap deprecated methods

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Frederik Ring <frederik.ring@gmail.com>
2024-07-02 21:15:49 +02:00
guangwu
4cb47a4818
fix: close backup file (#419) 2024-05-16 09:35:28 +02:00
e613f6046f Fix issues raised by linter 2024-04-26 17:10:06 +02:00
dependabot[bot]
292d47eb19
Bump github.com/docker/cli from 24.0.9+incompatible to 26.1.0+incompatible (#411)
* Bump github.com/docker/cli

Bumps [github.com/docker/cli](https://github.com/docker/cli) from 24.0.9+incompatible to 26.1.0+incompatible.
- [Commits](https://github.com/docker/cli/compare/v24.0.9...v26.1.0)

---
updated-dependencies:
- dependency-name: github.com/docker/cli
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* Upgrade docker/docker to matching version

* Tidy go.mod

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Frederik Ring <frederik.ring@gmail.com>
2024-04-25 17:33:07 +02:00
f2ef48803c
Print stack trace when encountering unexpected panic (#406) 2024-04-15 15:12:10 +02:00
8b69566291
Result of query for services is used before handling possible error (#405)
* Result of query for services is used before handling possible error

* Return early when a non-replicated service is matched
2024-04-15 15:08:37 +02:00
1af472077c
Update author reference in license statements (#393) 2024-03-15 11:42:22 +01:00
baf34ec1f7
Allow authentication using connection string when targeting Azure Blob Storage (#383)
* Allow authentication using connection string when targeting Azure Blob Storage

* Bail on ambiguous configuration
2024-03-08 20:23:30 +01:00
2d3e79cf5e
Also forward exec output when failing to demultiplex (#379) 2024-03-01 09:18:39 +01:00
5abfe5bb39
Swarm mode check fails on non-standard Info responses (#376)
* Swarm mode check fails on non-standard Info responses

* Add unit test

* Remove balena tests, add note to docs
2024-02-27 21:12:36 +00:00
9b768c71e6
Lines from conf files that are comments should not be passed to shell.Expand (#374) 2024-02-23 17:53:04 +01:00
060a6daa7a Use proper path expansion 2024-02-22 17:42:53 +01:00
02ba9939a2
Revert "Values without a backing env var should not be expanded (#368)" (#371)
This reverts commit 911fc5a223.
2024-02-21 18:43:02 +01:00
911fc5a223
Values without a backing env var should not be expanded (#368)
* Values without a backing env var should not be expanded

* Add unit tests for sourcing behavior

* Replace godotenv with shell lib
2024-02-21 17:44:37 +01:00
52c22a1891 Auto prepend caller when wrapping errors 2024-02-16 20:19:58 +01:00
83fa0aae48 Refactor handling of runtime configuration to prepare for reloading 2024-02-16 20:19:58 +01:00
a01fc3df3f
Conf files should expand env vars (#363) 2024-02-15 12:04:44 +01:00
5c8bc107de
Remove stray log statement (#359) 2024-02-13 19:54:18 +01:00
9a1e885138
Env vars should propagate when using conf.d (#358)
* Extend confd test case to test for env var propagation

* Env vars set in conf.d files are expected to propagate

* Lock needs to be acquired when instantiating script
2024-02-13 15:43:04 +01:00
4c74313222 Periodically collect runtime info when requested 2024-02-12 16:04:12 +01:00
de03d4f704 Docker client expects to be closed after usage in long running program 2024-02-12 16:04:12 +01:00
65626dd3d4 Hoist control for exiting script a level up (#348)
* Hoist control for exiting script a level up

* Do not accidentally nil out errors

* Log when running schedule

* Remove duplicate log line

* Warn on cron schedule that will never run
2024-02-12 16:04:12 +01:00
pixxon
1d45062100 Move cron scheduling inside application (#338)
* Move cron scheduling inside application

* Make envvar a fallback and check for errors

* Panic significantly less

* propagate error out of runBackup

* Add structured logging

* FIx error propagation to exit

* Enable the new scheduler by default

* Review fixes

* Added docs and better error propagation
2024-02-12 16:04:12 +01:00
0f224e4fb8
Document socket-proxy permissions, return early when update failed on scaling down (#343)
* Do not await containers when there was an error on scaling

* Add test case for usage with socket proxy

* Add documentation on required permissions for docker-socket-proxy

* Add full list of used Docker APIs to doc

* CONTAINER_START and CONTAINER_STOP is not needed
2024-02-05 14:27:06 +01:00
63b545787e
Exclusive file lock is released prematurely (#339) 2024-02-01 18:14:18 +01:00
c3daeacecb
Improve Swarm support (#333)
* Query for labeled services as well

* Try scaling down services

* Scale services back up

* Use progress tool from Docker CLI

* In test, label both services

* Clean up error and log messages

* Document scale-up/down approach in docs

* Downgrade Docker CLI to match client

* Document services stats

* Do not rely on PreviousSpec for storing desired replica count

* Log warnings from Docker when updating services

* Check whether container and service labels collide

* Document script behavior on label collision

* Add additional check if all containers have been removed

* Scale services concurrently

* Move docker interaction code into own file

* Factor out code for service updating

* Time out after five minutes of not reaching desired container count

* Inline handling of in-swarm container level restart

* Timer is more suitable for timeout race

* Timeout when scaling down services should be configurable

* Choose better filename

* Reflect changes in naming

* Rename and deprecate BACKUP_STOP_CONTAINER_LABEL

* Improve logging

* Further simplify logging
2024-01-31 12:17:41 +01:00
ed5abd5ba8
Panic handling does not log reason for script being halted (#334) 2024-01-26 20:02:09 +01:00
810c8871ec
Adopt v2 API for openpgp package (#332) 2024-01-23 20:19:42 +01:00
Alexander
64daf7b132
Add method for converting object to JSON in template (#319)
* Add method for converting object to JSON in template

* Split  to ,

* Rename toJson function
2023-12-20 20:44:45 +01:00
0325889ac4
Pruning method logs nonsensical configuration values (#301)
* Pruning method logs nonsensical configuration values

* Adjust test assertion about log output
2023-11-04 12:19:44 +01:00
1d549042fc
Error message on demultiplexing command output receives bad error arg (#295) 2023-10-29 15:43:34 +01:00
MaxJa4
336c5bed71
Replace Gzip with PGzip (#266)
* Replace Gzip with optimized PGzip. Add concurrency option.

* Add shortened timeout for 'dc down' too.

* Add NaturalNumberZero to allow zero.

* Add test for concurrency=0

* Rename to GZIP_PARALLELISM

* Fix block size. Fix compression level. Fix CI.

* Refactor compression writer fetching. Renamed WholeNumber
2023-09-03 16:49:52 +02:00
MaxJa4
43c4961116
Support _FILE based configuration values (#264)
* Replace envconfig with env

* Adjust config options and processing

* Added _FILE variant for all password vars.

* Try pathenvconfig

* Revert everything so far

* Use our fork of envconfig with custom lookup

* Use our fork of envconfig with custom lookup

* Test compose timeout option

* Remove secret resolving and specific _FILE config

* Fix timing issue in swarm tests

* Revert "Test compose timeout option"

This reverts commit ab50b21748, reversing
changes made to 0282514b2b.

Revert "Test compose timeout option"

This reverts commit 0282514b2b.

* Use offen/envconfig v1.5.0

* Add info about _FILE in README

* Value > File. Panic on file error. Panic on duplicate presence.

* Test panic on duplicate vars and panic on file error.
2023-08-30 20:14:24 +02:00
MaxJa4
ad4e2af83f
Exclude specific backends from pruning (#262)
* Skip backends while pruning

* Add pruning test step and silence download log for better readability

* Add test cases for pruning in all backends

Also add -q or --quiet-pull to all tests.

* Add test case for skipping backends while pruning

* Adjusted test logging, generate new test spec file

* Gitignore for temp test file
2023-08-27 19:19:11 +02:00
MaxJa4
5fcc96edf9
Cleanup: Lint warnings and deprecated packages (#263)
* Fix lint warnings and std lib deprecations

* Replace deprecated std lib with maintained drop-in replacement fork

Backwartds compatible with original package and suggested by std lib due to security and stability issues.

* OAuth2 is now a direct dependency due to Dropbox

* Undo change

* Revert "Replace deprecated std lib with maintained drop-in replacement fork"

This reverts commit 2887bd409f.

* Update channel handling

* Add linter for PRs

* Rename CI, fetch all issues, add govet
2023-08-27 18:14:55 +02:00
3d7677f02a
Print context in log field instead of prepending to message (#260)
* Print context in log field instead of prepending to message

* Log messages on pruning do not need a description anymore

* Remove redundant information from logs and errors
2023-08-25 12:44:43 +02:00
MaxJa4
e08a3303bf
Add new storage backend: Dropbox (#103) (#251)
* Add new storage backend: Dropbox (#103)

* Remove duplicate check

* Add concurrency level for parallel upload to dropbox.

* Fixed some instabilites. Changed default concurrency to 6.

* Added some env config vars to readme. WIP

* Wrap errors for storage backend creation.

* Fixed token issue, added OAuth2 including recipe and docs.

* Readme typo fix

* Test for dropbox integration

* Update info and TOC

* Missed a file

* Docker-compose fix

* Fix endpoint connection

* Fix container names

* Fix log fetching

* Fix log fetching (again)

* Print command output to logs

* Addressing comments part 1

* Address comments part 2

* OpenAPI Mock spec path adjusted
* Dropbox FileMetadata reflection refactored
* NaturalNumber type added

* Add OAuth2 mock server for CI testing

* Fix env name of oauth2 endpoint

* Remove hostname

* Add forgotten change to commit...

* Fix oauth2 endpoint

"Worked on my machine"

* Try again

* Try suggested hostname again

* Fix docker internal DNS resolving issues (as suggested by oauth2 mock docs)

* Add docker network, remove hostname

* Network not external

* Last hostname try

* Add more delay, add oauth2 endpoint log

* Temp CI log output of command even when failing

* Try different config and method

* Add custom server-hostname. Rename test folder to accellerate debugging

* Try that fix again

* Adding quotes

* Port fix attempt

* Try localhost

* Try extra hosts

* Change network mode

* Undo some changes

* Use static IP

* Remove specific IP binding

* Change to default net driver

* Fix static IP

* Squash for revert

* Revert "Squash for revert"

This reverts commit e9b617be9a.

* Actual fix for CI testing from #257
2023-08-24 19:33:47 +02:00
MaxJa4
47326c7c59
Fix storage backends not outputting any info logs (#250) 2023-08-20 20:35:25 +02:00
Michal Middleton
67e7288855
Add support for zstd compression (#249)
Co-authored-by: Michal Middleton <jafa81@gmail.com>
2023-08-19 19:20:13 +02:00
67d978f515
Drop logrus dependency, log using slog package from stdlib (#247) 2023-08-10 19:41:03 +02:00
1c6f64e254
Current Docker client breaks in newer Go versions (#241)
* Current Docker client breaks in newer Go versions

* Cater for breaking API changes in Docker client

* Update Docker client

* Unpin Go version used for build

* Tidy sum file
2023-07-25 19:46:57 +02:00
Erwan LE PRADO
5ea9a7ce15
feat: add better handler for part size (#214)
* feat: add better handler for part size


fix: use local file 


fix: try with another path


fix: use bytes 


chore: go back


go back readme


goback


goback


goback

* chore: better handling

* fix: typo readme

* chore: wrong comparaison

* fix: typo
2023-06-02 16:30:02 +02:00
37c01a578c
TaskTemplate.ForceUpdate is a counter (#209) 2023-04-26 08:45:12 +02:00
Diulgher Artiom
7f20036b15
Possibility to use -u (user) option in docker exec (#203)
* Add user option for docker exec

* Add test for user option

* Return test version for image

* remove gitea config file

* refactor tests

* remove comments & fix image name

* add docs

* cleanup

* Update README.md with suggested correction

Co-authored-by: Frederik Ring <frederik.ring@gmail.com>

* fix backup command & bind folder instead of volume

---------

Co-authored-by: tao <generaltao.md@gmail.com>
Co-authored-by: Frederik Ring <frederik.ring@gmail.com>
2023-04-02 11:12:10 +02:00
66ad124ddd any can be used instead of interface{} 2023-03-16 19:48:12 +01:00
2d37e08743
Use go 1.20, join errors using stdlib (#182)
* Use go 1.20, join errors using stdlib

* Use go 1.20 proper
2023-02-02 21:07:25 +01:00