mirror of
https://github.com/offen/docker-volume-backup.git
synced 2024-11-25 06:30:29 +01:00
* Add new storage backend: Dropbox (#103)
* Remove duplicate check
* Add concurrency level for parallel upload to dropbox.
* Fixed some instabilites. Changed default concurrency to 6.
* Added some env config vars to readme. WIP
* Wrap errors for storage backend creation.
* Fixed token issue, added OAuth2 including recipe and docs.
* Readme typo fix
* Test for dropbox integration
* Update info and TOC
* Missed a file
* Docker-compose fix
* Fix endpoint connection
* Fix container names
* Fix log fetching
* Fix log fetching (again)
* Print command output to logs
* Addressing comments part 1
* Address comments part 2
* OpenAPI Mock spec path adjusted
* Dropbox FileMetadata reflection refactored
* NaturalNumber type added
* Add OAuth2 mock server for CI testing
* Fix env name of oauth2 endpoint
* Remove hostname
* Add forgotten change to commit...
* Fix oauth2 endpoint
"Worked on my machine"
* Try again
* Try suggested hostname again
* Fix docker internal DNS resolving issues (as suggested by oauth2 mock docs)
* Add docker network, remove hostname
* Network not external
* Last hostname try
* Add more delay, add oauth2 endpoint log
* Temp CI log output of command even when failing
* Try different config and method
* Add custom server-hostname. Rename test folder to accellerate debugging
* Try that fix again
* Adding quotes
* Port fix attempt
* Try localhost
* Try extra hosts
* Change network mode
* Undo some changes
* Use static IP
* Remove specific IP binding
* Change to default net driver
* Fix static IP
* Squash for revert
* Revert "Squash for revert"
This reverts commit e9b617be9a.
* Actual fix for CI testing from #257
This commit is contained in:
MaxJa4
GitHub
parent
47326c7c59
commit
e08a3303bf
81
README.md
81
README.md
@ -4,10 +4,10 @@
|
||||
|
||||
# docker-volume-backup
|
||||
|
||||
Backup Docker volumes locally or to any S3, WebDAV, Azure Blob Storage or SSH compatible storage.
|
||||
Backup Docker volumes locally or to any S3, WebDAV, Azure Blob Storage, Dropbox or SSH compatible storage.
|
||||
|
||||
The [offen/docker-volume-backup](https://hub.docker.com/r/offen/docker-volume-backup) Docker image can be used as a lightweight (below 15MB) sidecar container to an existing Docker setup.
|
||||
It handles __recurring or one-off backups of Docker volumes__ to a __local directory__, __any S3, WebDAV, Azure Blob Storage or SSH compatible storage (or any combination) and rotates away old backups__ if configured. It also supports __encrypting your backups using GPG__ and __sending notifications for failed backup runs__.
|
||||
It handles __recurring or one-off backups of Docker volumes__ to a __local directory__, __any S3, WebDAV, Azure Blob Storage, Dropbox or SSH compatible storage (or any combination) and rotates away old backups__ if configured. It also supports __encrypting your backups using GPG__ and __sending notifications for failed backup runs__.
|
||||
|
||||
<!-- MarkdownTOC -->
|
||||
|
||||
@ -36,6 +36,7 @@ It handles __recurring or one-off backups of Docker volumes__ to a __local direc
|
||||
- [Define different retention schedules](#define-different-retention-schedules)
|
||||
- [Use special characters in notification URLs](#use-special-characters-in-notification-urls)
|
||||
- [Handle file uploads using third party tools](#handle-file-uploads-using-third-party-tools)
|
||||
- [Setup Dropbox storage backend](#setup-dropbox-storage-backend)
|
||||
- [Recipes](#recipes)
|
||||
- [Backing up to AWS S3](#backing-up-to-aws-s3)
|
||||
- [Backing up to Filebase](#backing-up-to-filebase)
|
||||
@ -44,6 +45,7 @@ It handles __recurring or one-off backups of Docker volumes__ to a __local direc
|
||||
- [Backing up to WebDAV](#backing-up-to-webdav)
|
||||
- [Backing up to SSH](#backing-up-to-ssh)
|
||||
- [Backing up to Azure Blob Storage](#backing-up-to-azure-blob-storage)
|
||||
- [Backing up to Dropbox](#backing-up-to-dropbox)
|
||||
- [Backing up locally](#backing-up-locally)
|
||||
- [Backing up to AWS S3 as well as locally](#backing-up-to-aws-s3-as-well-as-locally)
|
||||
- [Running on a custom cron schedule](#running-on-a-custom-cron-schedule)
|
||||
@ -356,6 +358,26 @@ You can populate below template according to your requirements and use it as you
|
||||
|
||||
# AZURE_STORAGE_ENDPOINT="https://{{ .AccountName }}.blob.core.windows.net/"
|
||||
|
||||
# Absolute remote path in your Dropbox where the backups shall be stored.
|
||||
# Note: Use your app's subpath in Dropbox, if it doesn't have global access.
|
||||
# Consulte the README for further information.
|
||||
|
||||
# DROPBOX_REMOTE_PATH="/my/directory"
|
||||
|
||||
# Number of concurrent chunked uploads for Dropbox.
|
||||
# Values above 6 usually result in no enhancements.
|
||||
|
||||
# DROPBOX_CONCURRENCY_LEVEL="6"
|
||||
|
||||
# App key and app secret from your app created at https://www.dropbox.com/developers/apps/info
|
||||
|
||||
# DROPBOX_APP_KEY=""
|
||||
# DROPBOX_APP_SECRET=""
|
||||
|
||||
# Refresh token to request new short-lived tokens (OAuth2). Consult README to see how to get one.
|
||||
|
||||
# DROPBOX_REFRESH_TOKEN=""
|
||||
|
||||
# In addition to storing backups remotely, you can also keep local copies.
|
||||
# Pass a container-local path to store your backups if needed. You also need to
|
||||
# mount a local folder or Docker volume into that location (`/archive`
|
||||
@ -1020,6 +1042,37 @@ volumes:
|
||||
|
||||
Commands will be invoked with the filepath of the tar archive passed as `COMMAND_RUNTIME_BACKUP_FILEPATH`.
|
||||
|
||||
### Setup Dropbox storage backend
|
||||
|
||||
#### Auth-Setup:
|
||||
|
||||
1. Create a new Dropbox App in the [App Console](https://www.dropbox.com/developers/apps)
|
||||
2. Open your new Dropbox App and set `DROPBOX_APP_KEY` and `DROPBOX_APP_SECRET` in your environment (e.g. docker-compose.yml) accordingly
|
||||
3. Click on `Permissions` in your app and make sure, that the following permissions are cranted (or more):
|
||||
- `files.metadata.write`
|
||||
- `files.metadata.read`
|
||||
- `files.content.write`
|
||||
- `files.content.read`
|
||||
4. Replace APPKEY in `https://www.dropbox.com/oauth2/authorize?client_id=APPKEY&token_access_type=offline&response_type=code` with the app key from step 2
|
||||
5. Visit the URL and confirm the access of your app. This gives you an `auth code` -> save it somewhere!
|
||||
6. Replace AUTHCODE, APPKEY, APPSECRET accordingly and perform the request:
|
||||
```
|
||||
curl https://api.dropbox.com/oauth2/token \
|
||||
-d code=AUTHCODE \
|
||||
-d grant_type=authorization_code \
|
||||
-d client_id=APPKEY \
|
||||
-d client_secret=APPSECRET
|
||||
```
|
||||
7. Execute the request. You will get a JSON formatted reply. Use the value of the `refresh_token` for the last environment variable `DROPBOX_REFRESH_TOKEN`
|
||||
8. You should now have `DROPBOX_APP_KEY`, `DROPBOX_APP_SECRET` and `DROPBOX_REFRESH_TOKEN` set. These don't expire.
|
||||
|
||||
Note: Using the "Generated access token" in the app console is not supported, as it is only very short lived and therefore not suitable for an automatic backup solution. The refresh token handles this automatically - the setup procedure above is only needed once.
|
||||
|
||||
#### Other parameters
|
||||
|
||||
Important: If you chose `App folder` access during the creation of your Dropbox app in step 1 above, you can only write in the app's directory!
|
||||
This means, that `DROPBOX_REMOTE_PATH` must start with e.g. `/Apps/YOUR_APP_NAME` or `/Apps/YOUR_APP_NAME/some_sub_dir`
|
||||
|
||||
## Recipes
|
||||
|
||||
This section lists configuration for some real-world use cases that you can mix and match according to your needs.
|
||||
@ -1187,6 +1240,30 @@ volumes:
|
||||
data:
|
||||
```
|
||||
|
||||
### Backing up to Dropbox
|
||||
|
||||
See [Dropbox Setup](#setup-dropbox-storage-backend) on how to get the appropriate environment values.
|
||||
|
||||
```yml
|
||||
version: '3'
|
||||
|
||||
services:
|
||||
# ... define other services using the `data` volume here
|
||||
backup:
|
||||
image: offen/docker-volume-backup:v2
|
||||
environment:
|
||||
DROPBOX_REFRESH_TOKEN: REFRESH_KEY # replace
|
||||
DROPBOX_APP_KEY: APP_KEY # replace
|
||||
DROPBOX_APP_SECRET: APP_SECRET # replace
|
||||
DROPBOX_REMOTE_PATH: /Apps/my-test-app/some_subdir # replace
|
||||
volumes:
|
||||
- data:/backup/my-app-backup:ro
|
||||
- /var/run/docker.sock:/var/run/docker.sock:ro
|
||||
|
||||
volumes:
|
||||
data:
|
||||
```
|
||||
|
||||
### Backing up locally
|
||||
|
||||
```yml
|
||||
|
||||
@ -10,6 +10,7 @@ import (
|
||||
"io/ioutil"
|
||||
"os"
|
||||
"regexp"
|
||||
"strconv"
|
||||
"time"
|
||||
)
|
||||
|
||||
@ -70,6 +71,13 @@ type Config struct {
|
||||
AzureStorageContainerName string `split_words:"true"`
|
||||
AzureStoragePath string `split_words:"true"`
|
||||
AzureStorageEndpoint string `split_words:"true" default:"https://{{ .AccountName }}.blob.core.windows.net/"`
|
||||
DropboxEndpoint string `split_words:"true" default:"https://api.dropbox.com/"`
|
||||
DropboxOAuth2Endpoint string `envconfig:"DROPBOX_OAUTH2_ENDPOINT" default:"https://api.dropbox.com/"`
|
||||
DropboxRefreshToken string `split_words:"true"`
|
||||
DropboxAppKey string `split_words:"true"`
|
||||
DropboxAppSecret string `split_words:"true"`
|
||||
DropboxRemotePath string `split_words:"true"`
|
||||
DropboxConcurrencyLevel NaturalNumber `split_words:"true" default:"6"`
|
||||
}
|
||||
|
||||
func (c *Config) resolveSecret(envVar string, secretPath string) (string, error) {
|
||||
@ -135,3 +143,21 @@ func (r *RegexpDecoder) Decode(v string) error {
|
||||
*r = RegexpDecoder{Re: re}
|
||||
return nil
|
||||
}
|
||||
|
||||
type NaturalNumber int
|
||||
|
||||
func (n *NaturalNumber) Decode(v string) error {
|
||||
asInt, err := strconv.Atoi(v)
|
||||
if err != nil {
|
||||
return fmt.Errorf("config: error converting %s to int", v)
|
||||
}
|
||||
if asInt <= 0 {
|
||||
return fmt.Errorf("config: expected a natural number, got %d", asInt)
|
||||
}
|
||||
*n = NaturalNumber(asInt)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (n *NaturalNumber) Int() int {
|
||||
return int(*n)
|
||||
}
|
||||
|
||||
@ -19,6 +19,7 @@ import (
|
||||
|
||||
"github.com/offen/docker-volume-backup/internal/storage"
|
||||
"github.com/offen/docker-volume-backup/internal/storage/azure"
|
||||
"github.com/offen/docker-volume-backup/internal/storage/dropbox"
|
||||
"github.com/offen/docker-volume-backup/internal/storage/local"
|
||||
"github.com/offen/docker-volume-backup/internal/storage/s3"
|
||||
"github.com/offen/docker-volume-backup/internal/storage/ssh"
|
||||
@ -75,6 +76,7 @@ func newScript() (*script, error) {
|
||||
"SSH": {},
|
||||
"Local": {},
|
||||
"Azure": {},
|
||||
"Dropbox": {},
|
||||
},
|
||||
},
|
||||
}
|
||||
@ -155,7 +157,7 @@ func newScript() (*script, error) {
|
||||
PartSize: s.c.AwsPartSize,
|
||||
}
|
||||
if s3Backend, err := s3.NewStorageBackend(s3Config, logFunc); err != nil {
|
||||
return nil, err
|
||||
return nil, fmt.Errorf("newScript: error creating s3 storage backend: %w", err)
|
||||
} else {
|
||||
s.storages = append(s.storages, s3Backend)
|
||||
}
|
||||
@ -170,7 +172,7 @@ func newScript() (*script, error) {
|
||||
RemotePath: s.c.WebdavPath,
|
||||
}
|
||||
if webdavBackend, err := webdav.NewStorageBackend(webDavConfig, logFunc); err != nil {
|
||||
return nil, err
|
||||
return nil, fmt.Errorf("newScript: error creating webdav storage backend: %w", err)
|
||||
} else {
|
||||
s.storages = append(s.storages, webdavBackend)
|
||||
}
|
||||
@ -187,7 +189,7 @@ func newScript() (*script, error) {
|
||||
RemotePath: s.c.SSHRemotePath,
|
||||
}
|
||||
if sshBackend, err := ssh.NewStorageBackend(sshConfig, logFunc); err != nil {
|
||||
return nil, err
|
||||
return nil, fmt.Errorf("newScript: error creating ssh storage backend: %w", err)
|
||||
} else {
|
||||
s.storages = append(s.storages, sshBackend)
|
||||
}
|
||||
@ -212,11 +214,28 @@ func newScript() (*script, error) {
|
||||
}
|
||||
azureBackend, err := azure.NewStorageBackend(azureConfig, logFunc)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
return nil, fmt.Errorf("newScript: error creating azure storage backend: %w", err)
|
||||
}
|
||||
s.storages = append(s.storages, azureBackend)
|
||||
}
|
||||
|
||||
if s.c.DropboxRefreshToken != "" && s.c.DropboxAppKey != "" && s.c.DropboxAppSecret != "" {
|
||||
dropboxConfig := dropbox.Config{
|
||||
Endpoint: s.c.DropboxEndpoint,
|
||||
OAuth2Endpoint: s.c.DropboxOAuth2Endpoint,
|
||||
RefreshToken: s.c.DropboxRefreshToken,
|
||||
AppKey: s.c.DropboxAppKey,
|
||||
AppSecret: s.c.DropboxAppSecret,
|
||||
RemotePath: s.c.DropboxRemotePath,
|
||||
ConcurrencyLevel: s.c.DropboxConcurrencyLevel.Int(),
|
||||
}
|
||||
dropboxBackend, err := dropbox.NewStorageBackend(dropboxConfig, logFunc)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("newScript: error creating dropbox storage backend: %w", err)
|
||||
}
|
||||
s.storages = append(s.storages, dropboxBackend)
|
||||
}
|
||||
|
||||
if s.c.EmailNotificationRecipient != "" {
|
||||
emailURL := fmt.Sprintf(
|
||||
"smtp://%s:%s@%s:%d/?from=%s&to=%s",
|
||||
|
||||
8
go.mod
8
go.mod
@ -20,6 +20,13 @@ require (
|
||||
golang.org/x/sync v0.3.0
|
||||
)
|
||||
|
||||
require (
|
||||
github.com/golang/protobuf v1.5.2 // indirect
|
||||
golang.org/x/oauth2 v0.0.0-20221014153046-6fdb5e3db783 // indirect
|
||||
google.golang.org/appengine v1.6.7 // indirect
|
||||
google.golang.org/protobuf v1.28.1 // indirect
|
||||
)
|
||||
|
||||
require (
|
||||
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.6.0 // indirect
|
||||
github.com/Azure/azure-sdk-for-go/sdk/internal v1.3.0 // indirect
|
||||
@ -28,6 +35,7 @@ require (
|
||||
github.com/docker/distribution v2.8.2+incompatible // indirect
|
||||
github.com/docker/go-connections v0.4.0 // indirect
|
||||
github.com/docker/go-units v0.4.0 // indirect
|
||||
github.com/dropbox/dropbox-sdk-go-unofficial/v6 v6.0.5
|
||||
github.com/dustin/go-humanize v1.0.1 // indirect
|
||||
github.com/fatih/color v1.13.0 // indirect
|
||||
github.com/gogo/protobuf v1.3.2 // indirect
|
||||
|
||||
4
go.sum
4
go.sum
@ -257,6 +257,8 @@ github.com/docker/go-connections v0.4.0 h1:El9xVISelRB7BuFusrZozjnkIM5YnzCViNKoh
|
||||
github.com/docker/go-connections v0.4.0/go.mod h1:Gbd7IOopHjR8Iph03tsViu4nIes5XhDvyHbTtUxmeec=
|
||||
github.com/docker/go-units v0.4.0 h1:3uh0PgVws3nIA0Q+MwDC8yjEPf9zjRfZZWXZYDct3Tw=
|
||||
github.com/docker/go-units v0.4.0/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk=
|
||||
github.com/dropbox/dropbox-sdk-go-unofficial/v6 v6.0.5 h1:FT+t0UEDykcor4y3dMVKXIiWJETBpRgERYTGlmMd7HU=
|
||||
github.com/dropbox/dropbox-sdk-go-unofficial/v6 v6.0.5/go.mod h1:rSS3kM9XMzSQ6pw91Qgd6yB5jdt70N4OdtrAf74As5M=
|
||||
github.com/dustin/go-humanize v1.0.0/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk=
|
||||
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
|
||||
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
|
||||
@ -785,6 +787,7 @@ golang.org/x/oauth2 v0.0.0-20220608161450-d0670ef3b1eb/go.mod h1:jaDAt6Dkxork7Lm
|
||||
golang.org/x/oauth2 v0.0.0-20220622183110-fd043fe589d2/go.mod h1:jaDAt6Dkxork7LmZnYtzbRWj0W47D86a3TGe0YHBvmE=
|
||||
golang.org/x/oauth2 v0.0.0-20220822191816-0ebed06d0094/go.mod h1:h4gKUeWbJ4rQPri7E0u6Gs4e9Ri2zaLxzw5DI5XGrYg=
|
||||
golang.org/x/oauth2 v0.0.0-20220909003341-f21342109be1/go.mod h1:h4gKUeWbJ4rQPri7E0u6Gs4e9Ri2zaLxzw5DI5XGrYg=
|
||||
golang.org/x/oauth2 v0.0.0-20221014153046-6fdb5e3db783 h1:nt+Q6cXKz4MosCSpnbMtqiQ8Oz0pxTef2B4Vca2lvfk=
|
||||
golang.org/x/oauth2 v0.0.0-20221014153046-6fdb5e3db783/go.mod h1:h4gKUeWbJ4rQPri7E0u6Gs4e9Ri2zaLxzw5DI5XGrYg=
|
||||
golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
|
||||
golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
|
||||
@ -1046,6 +1049,7 @@ google.golang.org/appengine v1.5.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7
|
||||
google.golang.org/appengine v1.6.1/go.mod h1:i06prIuMbXzDqacNJfV5OdTW448YApPu5ww/cMBSeb0=
|
||||
google.golang.org/appengine v1.6.5/go.mod h1:8WjMMxjGQR8xUklV/ARdw2HLXBOI7O7uCIDZVag1xfc=
|
||||
google.golang.org/appengine v1.6.6/go.mod h1:8WjMMxjGQR8xUklV/ARdw2HLXBOI7O7uCIDZVag1xfc=
|
||||
google.golang.org/appengine v1.6.7 h1:FZR1q0exgwxzPzp/aF+VccGrSfxfPpkBqjIIEq3ru6c=
|
||||
google.golang.org/appengine v1.6.7/go.mod h1:8WjMMxjGQR8xUklV/ARdw2HLXBOI7O7uCIDZVag1xfc=
|
||||
google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc=
|
||||
google.golang.org/genproto v0.0.0-20190307195333-5fe7a883aa19/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
|
||||
|
||||
260
internal/storage/dropbox/dropbox.go
Normal file
260
internal/storage/dropbox/dropbox.go
Normal file
@ -0,0 +1,260 @@
|
||||
package dropbox
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"fmt"
|
||||
"net/url"
|
||||
"os"
|
||||
"path"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/dropbox/dropbox-sdk-go-unofficial/v6/dropbox"
|
||||
"github.com/dropbox/dropbox-sdk-go-unofficial/v6/dropbox/files"
|
||||
"github.com/offen/docker-volume-backup/internal/storage"
|
||||
"golang.org/x/oauth2"
|
||||
)
|
||||
|
||||
type dropboxStorage struct {
|
||||
*storage.StorageBackend
|
||||
client files.Client
|
||||
concurrencyLevel int
|
||||
}
|
||||
|
||||
// Config allows to configure a Dropbox storage backend.
|
||||
type Config struct {
|
||||
Endpoint string
|
||||
OAuth2Endpoint string
|
||||
RefreshToken string
|
||||
AppKey string
|
||||
AppSecret string
|
||||
RemotePath string
|
||||
ConcurrencyLevel int
|
||||
}
|
||||
|
||||
// NewStorageBackend creates and initializes a new Dropbox storage backend.
|
||||
func NewStorageBackend(opts Config, logFunc storage.Log) (storage.Backend, error) {
|
||||
tokenUrl, _ := url.JoinPath(opts.OAuth2Endpoint, "oauth2/token")
|
||||
|
||||
conf := &oauth2.Config{
|
||||
ClientID: opts.AppKey,
|
||||
ClientSecret: opts.AppSecret,
|
||||
Endpoint: oauth2.Endpoint{
|
||||
TokenURL: tokenUrl,
|
||||
},
|
||||
}
|
||||
|
||||
logFunc(storage.LogLevelInfo, "Dropbox", "Fetching fresh access token for Dropbox storage backend.")
|
||||
tkSource := conf.TokenSource(context.Background(), &oauth2.Token{RefreshToken: opts.RefreshToken})
|
||||
token, err := tkSource.Token()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("(*dropboxStorage).NewStorageBackend: Error refreshing token: %w", err)
|
||||
}
|
||||
|
||||
dbxConfig := dropbox.Config{
|
||||
Token: token.AccessToken,
|
||||
}
|
||||
|
||||
if opts.Endpoint != "https://api.dropbox.com/" {
|
||||
dbxConfig.URLGenerator = func(hostType string, namespace string, route string) string {
|
||||
return fmt.Sprintf("%s/%d/%s/%s", opts.Endpoint, 2, namespace, route)
|
||||
}
|
||||
}
|
||||
|
||||
client := files.New(dbxConfig)
|
||||
|
||||
if opts.ConcurrencyLevel < 1 {
|
||||
logFunc(storage.LogLevelWarning, "Dropbox", "Concurrency level must be at least 1! Using 1 instead of %d.", opts.ConcurrencyLevel)
|
||||
opts.ConcurrencyLevel = 1
|
||||
}
|
||||
|
||||
return &dropboxStorage{
|
||||
StorageBackend: &storage.StorageBackend{
|
||||
DestinationPath: opts.RemotePath,
|
||||
Log: logFunc,
|
||||
},
|
||||
client: client,
|
||||
concurrencyLevel: opts.ConcurrencyLevel,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// Name returns the name of the storage backend
|
||||
func (b *dropboxStorage) Name() string {
|
||||
return "Dropbox"
|
||||
}
|
||||
|
||||
// Copy copies the given file to the WebDav storage backend.
|
||||
func (b *dropboxStorage) Copy(file string) error {
|
||||
_, name := path.Split(file)
|
||||
|
||||
folderArg := files.NewCreateFolderArg(b.DestinationPath)
|
||||
if _, err := b.client.CreateFolderV2(folderArg); err != nil {
|
||||
switch err := err.(type) {
|
||||
case files.CreateFolderV2APIError:
|
||||
if err.EndpointError.Path.Tag != files.WriteErrorConflict {
|
||||
return fmt.Errorf("(*dropboxStorage).Copy: Error creating directory '%s' in Dropbox: %w", b.DestinationPath, err)
|
||||
}
|
||||
b.Log(storage.LogLevelInfo, b.Name(), "Destination path '%s' already exists in Dropbox, no new directory required.", b.DestinationPath)
|
||||
default:
|
||||
return fmt.Errorf("(*dropboxStorage).Copy: Error creating directory '%s' in Dropbox: %w", b.DestinationPath, err)
|
||||
}
|
||||
}
|
||||
|
||||
r, err := os.Open(file)
|
||||
if err != nil {
|
||||
return fmt.Errorf("(*dropboxStorage).Copy: Error opening the file to be uploaded: %w", err)
|
||||
}
|
||||
defer r.Close()
|
||||
|
||||
// Start new upload session and get session id
|
||||
|
||||
b.Log(storage.LogLevelInfo, b.Name(), "Starting upload session for backup '%s' to Dropbox at path '%s'.", file, b.DestinationPath)
|
||||
|
||||
var sessionId string
|
||||
uploadSessionStartArg := files.NewUploadSessionStartArg()
|
||||
uploadSessionStartArg.SessionType = &files.UploadSessionType{Tagged: dropbox.Tagged{Tag: files.UploadSessionTypeConcurrent}}
|
||||
if res, err := b.client.UploadSessionStart(uploadSessionStartArg, nil); err != nil {
|
||||
return fmt.Errorf("(*dropboxStorage).Copy: Error starting the upload session: %w", err)
|
||||
} else {
|
||||
sessionId = res.SessionId
|
||||
}
|
||||
|
||||
// Send the file in 148MB chunks (Dropbox API limit is 150MB, concurrent upload requires a multiple of 4MB though)
|
||||
// Last append can be any size <= 150MB with Close=True
|
||||
|
||||
const chunkSize = 148 * 1024 * 1024 // 148MB
|
||||
var offset uint64 = 0
|
||||
var guard = make(chan struct{}, b.concurrencyLevel)
|
||||
var errorChn = make(chan error, b.concurrencyLevel)
|
||||
var EOFChn = make(chan bool, b.concurrencyLevel)
|
||||
var mu sync.Mutex
|
||||
var wg sync.WaitGroup
|
||||
|
||||
loop:
|
||||
for {
|
||||
guard <- struct{}{} // limit concurrency
|
||||
select {
|
||||
case err := <-errorChn: // error from goroutine
|
||||
return err
|
||||
case <-EOFChn: // EOF from goroutine
|
||||
wg.Wait() // wait for all goroutines to finish
|
||||
break loop
|
||||
default:
|
||||
}
|
||||
|
||||
go func() {
|
||||
defer func() {
|
||||
wg.Done()
|
||||
<-guard
|
||||
}()
|
||||
wg.Add(1)
|
||||
chunk := make([]byte, chunkSize)
|
||||
|
||||
mu.Lock() // to preserve offset of chunks
|
||||
|
||||
select {
|
||||
case <-EOFChn:
|
||||
EOFChn <- true // put it back for outer loop
|
||||
mu.Unlock()
|
||||
return // already EOF
|
||||
default:
|
||||
}
|
||||
|
||||
bytesRead, err := r.Read(chunk)
|
||||
if err != nil {
|
||||
errorChn <- fmt.Errorf("(*dropboxStorage).Copy: Error reading the file to be uploaded: %w", err)
|
||||
mu.Unlock()
|
||||
return
|
||||
}
|
||||
chunk = chunk[:bytesRead]
|
||||
|
||||
uploadSessionAppendArg := files.NewUploadSessionAppendArg(
|
||||
files.NewUploadSessionCursor(sessionId, offset),
|
||||
)
|
||||
isEOF := bytesRead < chunkSize
|
||||
uploadSessionAppendArg.Close = isEOF
|
||||
if isEOF {
|
||||
EOFChn <- true
|
||||
}
|
||||
offset += uint64(bytesRead)
|
||||
|
||||
mu.Unlock()
|
||||
|
||||
if err := b.client.UploadSessionAppendV2(uploadSessionAppendArg, bytes.NewReader(chunk)); err != nil {
|
||||
errorChn <- fmt.Errorf("(*dropboxStorage).Copy: Error appending the file to the upload session: %w", err)
|
||||
return
|
||||
}
|
||||
}()
|
||||
}
|
||||
|
||||
// Finish the upload session, commit the file (no new data added)
|
||||
|
||||
_, err = b.client.UploadSessionFinish(
|
||||
files.NewUploadSessionFinishArg(
|
||||
files.NewUploadSessionCursor(sessionId, 0),
|
||||
files.NewCommitInfo(filepath.Join(b.DestinationPath, name)),
|
||||
), nil)
|
||||
if err != nil {
|
||||
return fmt.Errorf("(*dropboxStorage).Copy: Error finishing the upload session: %w", err)
|
||||
}
|
||||
|
||||
b.Log(storage.LogLevelInfo, b.Name(), "Uploaded a copy of backup '%s' to Dropbox at path '%s'.", file, b.DestinationPath)
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// Prune rotates away backups according to the configuration and provided deadline for the Dropbox storage backend.
|
||||
func (b *dropboxStorage) Prune(deadline time.Time, pruningPrefix string) (*storage.PruneStats, error) {
|
||||
var entries []files.IsMetadata
|
||||
res, err := b.client.ListFolder(files.NewListFolderArg(b.DestinationPath))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("(*webDavStorage).Prune: Error looking up candidates from remote storage: %w", err)
|
||||
}
|
||||
entries = append(entries, res.Entries...)
|
||||
|
||||
for res.HasMore {
|
||||
res, err = b.client.ListFolderContinue(files.NewListFolderContinueArg(res.Cursor))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("(*webDavStorage).Prune: Error looking up candidates from remote storage: %w", err)
|
||||
}
|
||||
entries = append(entries, res.Entries...)
|
||||
}
|
||||
|
||||
var matches []*files.FileMetadata
|
||||
var lenCandidates int
|
||||
for _, candidate := range entries {
|
||||
switch candidate := candidate.(type) {
|
||||
case *files.FileMetadata:
|
||||
if !strings.HasPrefix(candidate.Name, pruningPrefix) {
|
||||
continue
|
||||
}
|
||||
lenCandidates++
|
||||
if candidate.ServerModified.Before(deadline) {
|
||||
matches = append(matches, candidate)
|
||||
}
|
||||
default:
|
||||
continue
|
||||
}
|
||||
}
|
||||
|
||||
stats := &storage.PruneStats{
|
||||
Total: uint(lenCandidates),
|
||||
Pruned: uint(len(matches)),
|
||||
}
|
||||
|
||||
if err := b.DoPrune(b.Name(), len(matches), lenCandidates, "Dropbox backup(s)", func() error {
|
||||
for _, match := range matches {
|
||||
if _, err := b.client.DeleteV2(files.NewDeleteArg(filepath.Join(b.DestinationPath, match.Name))); err != nil {
|
||||
return fmt.Errorf("(*dropboxStorage).Prune: Error removing file from Dropbox storage: %w", err)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}); err != nil {
|
||||
return stats, err
|
||||
}
|
||||
|
||||
return stats, nil
|
||||
}
|
||||
57
test/dropbox/docker-compose.yml
Normal file
57
test/dropbox/docker-compose.yml
Normal file
@ -0,0 +1,57 @@
|
||||
version: '3'
|
||||
|
||||
services:
|
||||
openapi_mock:
|
||||
image: muonsoft/openapi-mock
|
||||
environment:
|
||||
OPENAPI_MOCK_USE_EXAMPLES: if_present
|
||||
OPENAPI_MOCK_SPECIFICATION_URL: '/etc/openapi/user_v2.yaml'
|
||||
ports:
|
||||
- 8080:8080
|
||||
volumes:
|
||||
- ./user_v2.yaml:/etc/openapi/user_v2.yaml
|
||||
|
||||
oauth2_mock:
|
||||
image: ghcr.io/navikt/mock-oauth2-server:1.0.0
|
||||
ports:
|
||||
- 8090:8090
|
||||
environment:
|
||||
PORT: 8090
|
||||
JSON_CONFIG_PATH: '/etc/oauth2/config.json'
|
||||
volumes:
|
||||
- ./oauth2_config.json:/etc/oauth2/config.json
|
||||
|
||||
backup:
|
||||
image: offen/docker-volume-backup:${TEST_VERSION:-canary}
|
||||
hostname: hostnametoken
|
||||
depends_on:
|
||||
- openapi_mock
|
||||
- oauth2_mock
|
||||
restart: always
|
||||
environment:
|
||||
BACKUP_FILENAME_EXPAND: 'true'
|
||||
BACKUP_FILENAME: test-$$HOSTNAME.tar.gz
|
||||
BACKUP_CRON_EXPRESSION: 0 0 5 31 2 ?
|
||||
BACKUP_RETENTION_DAYS: ${BACKUP_RETENTION_DAYS:-7}
|
||||
BACKUP_PRUNING_LEEWAY: 5s
|
||||
BACKUP_PRUNING_PREFIX: test
|
||||
DROPBOX_ENDPOINT: http://openapi_mock:8080
|
||||
DROPBOX_OAUTH2_ENDPOINT: http://oauth2_mock:8090
|
||||
DROPBOX_REFRESH_TOKEN: test
|
||||
DROPBOX_APP_KEY: test
|
||||
DROPBOX_APP_SECRET: test
|
||||
DROPBOX_REMOTE_PATH: /test
|
||||
DROPBOX_CONCURRENCY_LEVEL: 6
|
||||
volumes:
|
||||
- app_data:/backup/app_data:ro
|
||||
- /var/run/docker.sock:/var/run/docker.sock
|
||||
|
||||
offen:
|
||||
image: offen/offen:latest
|
||||
labels:
|
||||
- docker-volume-backup.stop-during-backup=true
|
||||
volumes:
|
||||
- app_data:/var/opt/offen
|
||||
|
||||
volumes:
|
||||
app_data:
|
||||
37
test/dropbox/oauth2_config.json
Normal file
37
test/dropbox/oauth2_config.json
Normal file
@ -0,0 +1,37 @@
|
||||
{
|
||||
"interactiveLogin": true,
|
||||
"httpServer": "NettyWrapper",
|
||||
"tokenCallbacks": [
|
||||
{
|
||||
"issuerId": "issuer1",
|
||||
"tokenExpiry": 120,
|
||||
"requestMappings": [
|
||||
{
|
||||
"requestParam": "scope",
|
||||
"match": "scope1",
|
||||
"claims": {
|
||||
"sub": "subByScope",
|
||||
"aud": [
|
||||
"audByScope"
|
||||
]
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"issuerId": "issuer2",
|
||||
"requestMappings": [
|
||||
{
|
||||
"requestParam": "someparam",
|
||||
"match": "somevalue",
|
||||
"claims": {
|
||||
"sub": "subBySomeParam",
|
||||
"aud": [
|
||||
"audBySomeParam"
|
||||
]
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
40
test/dropbox/run.sh
Normal file
40
test/dropbox/run.sh
Normal file
@ -0,0 +1,40 @@
|
||||
#!/bin/sh
|
||||
|
||||
set -e
|
||||
|
||||
cd "$(dirname "$0")"
|
||||
. ../util.sh
|
||||
current_test=$(basename $(pwd))
|
||||
|
||||
docker compose up -d
|
||||
sleep 5
|
||||
|
||||
logs=$(docker compose exec -T backup backup)
|
||||
|
||||
sleep 5
|
||||
|
||||
expect_running_containers "4"
|
||||
|
||||
echo "$logs"
|
||||
if echo "$logs" | grep -q "ERROR"; then
|
||||
fail "Backup failed, errors reported: $dvb_logs"
|
||||
else
|
||||
pass "Backup succeeded, no errors reported."
|
||||
fi
|
||||
|
||||
# The second part of this test checks if backups get deleted when the retention
|
||||
# is set to 0 days (which it should not as it would mean all backups get deleted)
|
||||
# TODO: find out if we can test actual deletion without having to wait for a day
|
||||
BACKUP_RETENTION_DAYS="0" docker compose up -d
|
||||
sleep 5
|
||||
|
||||
logs=$(docker compose exec -T backup backup)
|
||||
|
||||
echo "$logs"
|
||||
if echo "$logs" | grep -q "Refusing to do so, please check your configuration"; then
|
||||
pass "Remote backups have not been deleted."
|
||||
else
|
||||
fail "Remote backups would have been deleted: $dvb_logs"
|
||||
fi
|
||||
|
||||
docker compose down --volumes
|
||||
12753
test/dropbox/user_v2.yaml
Normal file
12753
test/dropbox/user_v2.yaml
Normal file
File diff suppressed because it is too large
Load Diff
Loading…
Reference in New Issue
Block a user