Compare commits

...

10 Commits

Author SHA1 Message Date
Pablo Fernandez
c469f5d9d4 Apply corrections from cubic 2026-07-10 17:40:33 +01:00
Pablo Fernandez
de098add09 Tidy up the cloud storage sections in vmalert, vmbackup, vmrestore and vmbackupmanager
Reference the new guide when possible.
2026-07-10 17:27:04 +01:00
Pablo Fernandez
627eb7c804 Add suggestions from Max 2026-07-10 16:57:45 +01:00
Pablo (Tomas) Fernandez
ed715a89ac Update docs/guides/connecting-vm-components-to-cloud-storage/README.md
Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>
Signed-off-by: Pablo (Tomas) Fernandez <46322567+TomFern@users.noreply.github.com>
2026-06-29 14:10:04 +01:00
Pablo (Tomas) Fernandez
f16bee6973 Update docs/guides/connecting-vm-components-to-cloud-storage/README.md
Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>
Signed-off-by: Pablo (Tomas) Fernandez <46322567+TomFern@users.noreply.github.com>
2026-06-29 14:09:55 +01:00
Pablo Fernandez
c0edf0c983 Apply corrections from cubic 2026-06-29 14:07:06 +01:00
Pablo Fernandez
81262b2fb5 Proofreading pass 2026-06-29 13:45:28 +01:00
Pablo Fernandez
2aec74f23b Add draft for guide to connecting vm components to cloud storage 2026-06-29 11:29:46 +01:00
Pablo Fernandez
d3f86b24a1 Separate vmalert from vmbackup and expand examples 2026-06-22 14:07:44 +01:00
Pablo Fernandez
60de3bad8e Build on a more expanded reference for using credentials
See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/11073
2026-06-12 18:15:30 +01:00
6 changed files with 523 additions and 75 deletions

View File

@@ -0,0 +1,426 @@
Several VictoriaMetrics components can connect to cloud storage to read or write object data.
The following table shows the supported types of storage for each component:
| Component | AWS S3 and S3-compatible | Google Cloud Storage | Azure Blob Storage |
|-----------|----|----------------------|--------------------|
| [vmbackup](https://docs.victoriametrics.com/victoriametrics/vmbackup/) | ✅ | ✅ | ✅ |
| [vmrestore](https://docs.victoriametrics.com/victoriametrics/vmrestore/) | ✅ | ✅ | ✅ |
| [vmbackupmanager](https://docs.victoriametrics.com/victoriametrics/vmbackupmanager/) | ✅ | ✅ | ✅ |
| [vmalert](https://docs.victoriametrics.com/victoriametrics/vmalert/) | ✅ | ✅ | ❌ |
All these components use the same underlying libraries, so the authentication setup is largely the same. The main difference is in the command-line flags:
- vmalert uses `-s3.*` prefixed flags (e.g., `-s3.credsFilePath`)
- backup and restore tools use unprefixed flags (e.g., `-credsFilePath`)
See the [component reference](https://docs.victoriametrics.com/guides/connecting-vm-components-to-cloud-storage/#per-component-flag-reference) for details.
## Obtaining credentials
You need to supply credentials so the component can connect to the cloud storage. The setup differs by provider; the sections below cover AWS S3, S3compatible systems, GCS, and Azure Blob Storage.
### AWS S3
1. In AWS, [create an IAM user](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html) or role with the minimum permissions for the target bucket (see table below).
1. [Create an access key](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html) for that IAM identity.
1. Copy the **Access key ID** and **Secret access key** values. You will use them in the credentials file or environment variables.
| Component | S3 usage | Minimum S3 permissions |
|-----------------|-------------------------------------------------------------|------------------------|
| vmalert (Enterprise) | Reads alerting/recording rules from bucket. | `s3:GetObject`, `s3:ListBucket` on the rules bucket/prefix.|
| vmrestore | Restores backups from S3 buckets | `s3:GetObject`, `s3:ListBucket` on the backup bucket/prefix.|
| vmbackup | Uploads backups to S3 and may delete old backup objects during maintenance.| `s3:PutObject`, `s3:GetObject`, `s3:ListBucket`, `s3:DeleteObject` on the backup bucket/prefix.|
| vmbackupmanager (Enterprise) | Automates backups using vmbackup behavior. | Same as vmbackup. |
### S3-compatible storage (MinIO, Ceph)
Generate access keys using your storage system's admin interface or CLI. The credentials follow the same format as AWS S3.
### Google Cloud Storage
1. Open the Google Cloud Console and go to **IAM & Admin > Service Accounts**.
1. Click **Create service account**, enter a name, and assign a Storage role (for example, Storage Object Admin).
1. Open the service account, go to **Keys**, then click **Add key > Create new key**.
1. Choose **JSON** as the key type and click **Create**.
1. Store the JSON file on the machine running the VictoriaMetrics component.
### Azure Blob Storage
> Azure does not use credential files.
1. Log in to the Azure Portal.
1. Search for and select **Storage accounts**.
1. Click on your specific storage account name. (This is your `AZURE_STORAGE_ACCOUNT_NAME`).
1. In the left menu under **Security + networking**, click **Access keys**.
1. Copy the key value from either **key1** or **key2** (this is your `AZURE_STORAGE_ACCOUNT_KEY`).
1. Define the access keys as environment variables.
```sh
export AZURE_STORAGE_ACCOUNT_NAME=mystorageaccount
export AZURE_STORAGE_ACCOUNT_KEY=myaccountkey
```
## Authenticating with the cloud provider
Provide the credentials as a file or with environment variables, along with the path to the cloud storage bucket. The syntax for the bucket name depends on the cloud provider:
- `s3://`: for AWS S3 and self-hosted S3-compatible storage (MinIO, Ceph)
- `gs://`: Google Cloud Storage
- `azblob://`: Azure Blob Storage
### vmbackup and vmrestore
The following example backups to an AWS S3 bucket using a credentials file:
```sh
vmbackup \
-storageDataPath=/data \
-snapshot.createURL=http://localhost:8428/snapshot/create \
-dst=s3://victoriametrics-backup/backup01 \
-credsFilePath=/etc/credentials
```
In order to restore from the same backup from AWS S3:
```sh
vmrestore \
-src=s3://victoriametrics-backup/backup01 \
-storageDataPath=/data \
-credsFilePath=/etc/credentials
```
> To use non-AWS S3 buckets such as MinIO or Ceph, you must [supply the `-customS3Endpoint` argument](https://docs.victoriametrics.com/guides/connecting-vm-components-to-cloud-storage/#s3-compatible).
Alternatively, you can set the access keys as environment variables instead of using a credential file:
```sh
export AWS_ACCESS_KEY_ID=YOUR_AWS_ACCESS_KEY
export AWS_SECRET_ACCESS_KEY=YOUR_SECRET_AWS_ACCESS_KEY
vmbackup \
-storageDataPath=/data \
-snapshot.createURL=http://localhost:8428/snapshot/create \
-dst=s3://victoriametrics-backup/backup01
vmrestore \
-src=s3://victoriametrics-backup/backup01 \
-storageDataPath=/data
```
Backups on Google Cloud Storage use the `gs://` prefix in the destination:
```sh
vmbackup \
-storageDataPath=/data \
-snapshot.createURL=http://localhost:8428/snapshot/create \
-dst=gs://victoriametrics-backup/backup01 \
-credsFilePath=/etc/credentials
```
You can restore this backup with:
```sh
vmrestore \
-src=gs://victoriametrics-backup/backup01 \
-storageDataPath=/data \
-credsFilePath=/etc/credentials
```
On Google Cloud, you can define the path to the JSON credential file with `GOOGLE_APPLICATION_CREDENTIALS`. For example:
```sh
export GOOGLE_APPLICATION_CREDENTIALS=/etc/credentials
vmbackup \
-storageDataPath=/data \
-snapshot.createURL=http://localhost:8428/snapshot/create \
-dst=gs://victoriametrics-backup/backup01
vmrestore \
-src=gs://victoriametrics-backup/backup01 \
-storageDataPath=/data
```
For Azure Blob Storage, use the `azblob://` prefix and rely on environment variables instead of `-credsFilePath`.
```sh
export AZURE_STORAGE_ACCOUNT_NAME=myaccount
export AZURE_STORAGE_ACCOUNT_KEY=mykey
vmbackup \
-storageDataPath=/data \
-snapshot.createURL=http://localhost:8428/snapshot/create \
-dst=azblob://victoriametrics-backup/backup01
vmrestore \
-src=azblob://victoriametrics-backup/backup01 \
-storageDataPath=/data
```
### vmbackupmanager
> vmbackupmanager only works in the [Enterprise](https://docs.victoriametrics.com/victoriametrics/enterprise/) edition.
To manage backups with vmbackupmanager on AWS S3, add the credentials with the `-credsFilePath` flag:
```sh
vmbackupmanager \
-dst=s3://vmstorage-data/backups \
-credsFilePath=/etc/credentials \
-storageDataPath=/vmstorage-data \
-snapshot.createURL=http://vmstorage:8482/snapshot/create \
-licenseFile=/etc/vm-license
```
Or define the access keys using environment variables:
```sh
export AWS_ACCESS_KEY_ID=YOUR_AWS_ACCESS_KEY
export AWS_SECRET_ACCESS_KEY=YOUR_SECRET_AWS_ACCESS_KEY
vmbackupmanager \
-dst=s3://vmstorage-data/backups \
-storageDataPath=/vmstorage-data \
-snapshot.createURL=http://vmstorage:8482/snapshot/create \
-licenseFile=/etc/vm-license
```
> To use non-AWS S3 buckets, you must [supply the `-customS3Endpoint` argument](#s3-compatible-endpoints).
Automated backups on Google Cloud Storage take the following form:
```sh
vmbackupmanager \
-dst=gs://vmstorage-data/backups \
-credsFilePath=/etc/credentials \
-storageDataPath=/vmstorage-data \
-snapshot.createURL=http://vmstorage:8482/snapshot/create \
-licenseFile=/etc/vm-license
```
As with vmbackup and vmrestore, you can also define the path to the JSON credential file with `GOOGLE_APPLICATION_CREDENTIALS`:
```sh
export GOOGLE_APPLICATION_CREDENTIALS=/etc/credentials
vmbackupmanager \
-dst=gs://vmstorage-data/backups \
-storageDataPath=/vmstorage-data \
-snapshot.createURL=http://vmstorage:8482/snapshot/create \
-licenseFile=/etc/vm-license
```
vmbackupmanager can also use Azure Blob Storage by defining environment variables:
```sh
export AZURE_STORAGE_ACCOUNT_NAME=mystorageaccount
export AZURE_STORAGE_ACCOUNT_KEY=myaccountkey
vmbackupmanager \
-dst=azblob://vmstorage-data/backups \
-storageDataPath=/vmstorage-data \
-snapshot.createURL=http://vmstorage:8482/snapshot/create \
-licenseFile=/etc/vm-license
```
### vmalert
> - vmalert cloud storage command line flags are prefixed with `-s3.` for S3 buckets *and* Google Cloud Storage.
> - Cloud storage only works in the [Enterprise](https://docs.victoriametrics.com/victoriametrics/enterprise/) edition.
Read alerting rules from an S3 bucket. The `-rule` flag accepts a prefix, so it matches all files starting with `alerts_` in the `rules` folder:
```sh
vmalert \
-rule=s3://my-alert-bucket/rules/alerts_ \
-s3.credsFilePath=/etc/vmalert/aws-credentials \
-datasource.url=http://vmselect:8481/select/0/prometheus \
-notifier.url=http://alertmanager:9093 \
-licenseFile=/etc/vm-license
```
Instead of a credential file, you can supply the access keys using environment variables:
```sh
export AWS_ACCESS_KEY_ID=YOUR_AWS_ACCESS_KEY
export AWS_SECRET_ACCESS_KEY=YOUR_SECRET_AWS_ACCESS_KEY
vmalert \
-rule=s3://my-alert-bucket/rules/alerts_ \
-datasource.url=http://vmselect:8481/select/0/prometheus \
-notifier.url=http://alertmanager:9093 \
-licenseFile=/etc/vm-license
```
> To use non-AWS S3 buckets, you must [supply the `-s3.customEndpoint` argument](#s3-compatible-endpoints).
To read rules from Google Cloud Storage:
```sh
vmalert \
-rule=gs://my-alert-bucket/rules/alerts_ \
-s3.credsFilePath=/etc/vmalert/gcp-service-account.json \
-datasource.url=http://vmselect:8481/select/0/prometheus \
-notifier.url=http://alertmanager:9093 \
-licenseFile=/etc/vm-license
```
If you prefer, you can supply the path to the JSON credential file with the `GOOGLE_APPLICATION_CREDENTIALS` environment variable:
```sh
export GOOGLE_APPLICATION_CREDENTIALS=/etc/credentials
vmalert \
-rule=gs://my-alert-bucket/rules/alerts_ \
-datasource.url=http://vmselect:8481/select/0/prometheus \
-notifier.url=http://alertmanager:9093 \
-licenseFile=/etc/vm-license
```
## Credentials files format
The file format depends on the storage provider.
### S3 credentials
The file uses the standard AWS shared credentials format used by the [AWS CLI](https://docs.aws.amazon.com/cli/v1/userguide/cli-configure-files.html) and [AWS SDKs](https://docs.aws.amazon.com/sdkref/latest/guide/file-format.html):
```ini
[default]
aws_access_key_id=YOUR_AWS_ACCESS_KEY
aws_secret_access_key=YOUR_AWS_SECRET_ACCESS_KEY
```
You can define multiple profiles in a single file:
```ini
[default]
aws_access_key_id=DEFAULT_ACCESS_KEY
aws_secret_access_key=DEFAULT_SECRET_KEY
[monitoring]
aws_access_key_id=MONITORING_ACCESS_KEY
aws_secret_access_key=MONITORING_SECRET_KEY
[alerts]
aws_access_key_id=ALERTS_ACCESS_KEY
aws_secret_access_key=ALERTS_SECRET_KEY
```
Use the `-configProfile` flag (or `-s3.configProfile` in vmalert) to select a non-default profile:
```sh
-configProfile=alerts
```
You can separate credentials from other configuration settings. Put credentials in one file:
```ini
[default]
aws_access_key_id=DEFAULT_ACCESS_KEY
aws_secret_access_key=DEFAULT_SECRET_KEY
```
And non-sensitive settings in another:
```ini
[default]
region=us-east-1
```
Then pass both files:
```sh
-configFilePath=/etc/aws-config \
-credsFilePath=/etc/credentials
```
### GCS credentials file format
The file is the JSON key downloaded from Google Cloud Console. Its content looks like this:
```json
{
"type": "service_account",
"project_id": "project-id",
"private_key_id": "key-id",
"private_key": "-----BEGIN PRIVATE KEY-----\nprivate-key\n-----END PRIVATE KEY-----\n",
"client_email": "service-account-email",
"client_id": "client-id",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://accounts.google.com/o/oauth2/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/service-account-email"
}
```
This is the standard service account key format defined by [Google Cloud IAM](https://developers.google.com/workspace/guides/create-credentials).
### Azure Blob Storage
Azure does not support credentials via file. Use environment variables instead.
```sh
export AZURE_STORAGE_ACCOUNT_NAME=mystorageaccount
export AZURE_STORAGE_ACCOUNT_KEY=myaccountkey
```
## Self-hosted and S3-compatible endpoints {#s3-compatible}
For S3-compatible storage such as MinIO or Ceph, set a custom endpoint with the `-customS3Endpoint` flag for vmbackup, vmrestore, and vmbackupmanager. For example:
```sh
vmbackup \
-storageDataPath=/data \
-snapshot.createURL=http://localhost:8428/snapshot/create \
-dst=s3://victoriametrics-backup/backup01 \
-customS3Endpoint=http://minio.example.local:9000
```
On vmalert, use the `-s3.customEndpoint` flag instead:
```sh
vmalert \
-rule=s3://my-alert-bucket/rules/alerts_ \
-s3.customEndpoint=http://minio.example.local:9000 \
-s3.credsFilePath=/etc/vmalert/aws-credentials \
-datasource.url=http://vmselect:8481/select/0/prometheus \
-notifier.url=http://alertmanager:9093 \
-licenseFile=/etc/vm-license
```
### Addressing S3-compatible buckets
When connecting to non-AWS S3-compatible buckets, there is an additional flag you might need to configure:
- `-s3ForcePathStyle`: on vmbackupmanager, vmbackup, and vmrestore.
- `-s3.forcePathStyle`: on vmalert.
The flag changes the expected URL pattern for a bucket.
| Flag value | Address-style | Example | Use with |
|------------|---------------|---------|----------|
| `true` (default) | Path-style | `https://endpoint/bucket/key` | MinIO, Ceph, most S3-compatible storages |
| `false` | Virtual host-style | `https://bucket.endpoint/key` | [Aliyun OSS](https://www.aliyun.com/product/oss) and other endpoints that require it |
> The flag only takes effect when you use a custom endpoint (`-customS3Endpoint` or `-s3.customEndpoint` on vmalert). When connecting to real AWS S3, the SDK handles addressing automatically.
## Per-component flag reference
The table below shows how the same concept maps to different flag names across components.
| Concept | vmalert | vmbackup, vmrestore, and vmbackupmanager |
|---|---|---|---|---|
| Credentials file | `-s3.credsFilePath` | `-credsFilePath` |
| Config file | `-s3.configFilePath` | `-configFilePath` |
| Profile selection | `-s3.configProfile` | `-configProfile` |
| Custom endpoint | `-s3.customEndpoint` | `-customS3Endpoint` |
| Force path style | `-s3.forcePathStyle` | `-s3ForcePathStyle` |
| TLS insecure | N/A | `-s3TLSInsecureSkipVerify` |
| Storage class | N/A | `-s3StorageClass` |

View File

@@ -0,0 +1,12 @@
---
weight: 5
title: Connecting VictoriaMetrics components to cloud storage
menu:
docs:
parent: guides
weight: 5
tags:
- metrics
- guide
---
{{% content "README.md" %}}

View File

@@ -546,7 +546,7 @@ tags at [Docker Hub](https://hub.docker.com/r/victoriametrics/vmalert/tags) and
## Reading rules from object storage
[Enterprise version](https://docs.victoriametrics.com/victoriametrics/enterprise/) of `vmalert` may read alerting and recording rules
The [Enterprise version](https://docs.victoriametrics.com/victoriametrics/enterprise/) of `vmalert` may read alerting and recording rules
from object storage:
* `./bin/vmalert -rule=s3://bucket/dir/alert.rules` would read rules from the given path at S3 bucket
@@ -563,6 +563,48 @@ The following [command-line flags](#flags) can be used for fine-tuning access to
* `-s3.customEndpoint` - custom S3 endpoint for use with S3-compatible storages (e.g. MinIO). S3 is used if not set.
* `-s3.forcePathStyle` - prefixing endpoint with bucket name when set false, true by default.
### S3 (AWS and S3-compatible)
The following example reads rules from an S3 bucket:
```sh
./bin/vmalert \
-rule=s3://my-alert-bucket/rules/alerts_ \
-s3.credsFilePath=/etc/vmalert/aws-credentials \
-datasource.url=http://vmselect:8481/select/0/prometheus \
-notifier.url=http://alertmanager:9093 \
-licenseFile=/etc/vm-license
```
For S3-compatible backends such as [MinIO](https://www.min.io/) or [Ceph](https://ceph.io/), add `-s3.customEndpoint`:
```sh
./bin/vmalert \
-rule=s3://victoriametrics-alert-rules/rules_ \
-s3.credsFilePath=/etc/vmalert/aws-credentials \
-s3.customEndpoint=http://minio.example.local:9000 \
-datasource.url=http://vmselect:8481/select/0/prometheus \
-notifier.url=http://alertmanager:9093 \
-licenseFile=/etc/vm-license
```
See [Connecting VM components to cloud storage](https://docs.victoriametrics.com/guides/connecting-vm-components-to-cloud-storage/) for details on creating IAM users, credentials file format, environment variables, and IAM roles.
### Google Cloud Storage (GCS)
The following example reads rules from a GCS bucket:
```sh
./bin/vmalert \
-rule=gs://my-alert-bucket/rules/alerts_ \
-s3.credsFilePath=/etc/vmalert/gcp-service-account.json \
-datasource.url=http://vmselect:8481/select/0/prometheus \
-notifier.url=http://alertmanager:9093 \
-licenseFile=/etc/vm-license
```
See [Connecting VM components to cloud storage](https://docs.victoriametrics.com/guides/connecting-vm-components-to-cloud-storage/) for details on creating service accounts and downloading JSON keys.
## Topology examples
The following sections are showing how `vmalert` may be used and configured

View File

@@ -204,61 +204,45 @@ See [this article](https://medium.com/@valyala/speeding-up-backups-for-big-time-
### Providing credentials as a file
Obtaining credentials from a file.
See [Connecting VM components to cloud storage](https://docs.victoriametrics.com/guides/connecting-vm-components-to-cloud-storage/) for instructions on obtaining credentials and providing them via credential files, environment variables, cloud provider metadata service, or Kubernetes secrets and IAM roles.
Add flag `-credsFilePath=/etc/credentials` with the following content:
The following examples show the most common authentication patterns for each storage provider.
* for S3 (AWS, MinIO or other S3 compatible storages):
#### S3 (AWS and S3-compatible)
```sh
[default]
aws_access_key_id=theaccesskey
aws_secret_access_key=thesecretaccesskeyvalue
```
```sh
vmbackup \
-storageDataPath=/data \
-snapshot.createURL=http://localhost:8428/snapshot/create \
-dst=s3://victoriametrics-backup/backup01 \
-credsFilePath=/etc/credentials
```
* for GCP cloud storage:
#### Google Cloud Storage (GCS)
```json
{
"type": "service_account",
"project_id": "project-id",
"private_key_id": "key-id",
"private_key": "-----BEGIN PRIVATE KEY-----\nprivate-key\n-----END PRIVATE KEY-----\n",
"client_email": "service-account-email",
"client_id": "client-id",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://accounts.google.com/o/oauth2/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/service-account-email"
}
```
```sh
vmbackup \
-storageDataPath=/data \
-snapshot.createURL=http://localhost:8428/snapshot/create \
-dst=gs://victoriametrics-backup/backup01 \
-credsFilePath=/etc/credentials
```
### Providing credentials via env variables
#### Azure Blob Storage
Obtaining credentials from env variables.
```sh
export AZURE_STORAGE_ACCOUNT_NAME=mystorageaccount
export AZURE_STORAGE_ACCOUNT_KEY=myaccountkey
* For AWS S3 compatible storages set env variable `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`.
Also you can set env variable `AWS_SHARED_CREDENTIALS_FILE` with path to credentials file.
* For GCE cloud storage set env variable `GOOGLE_APPLICATION_CREDENTIALS` with path to credentials file.
* For Azure storage use one of these env variables:
* `AZURE_STORAGE_ACCOUNT_CONNECTION_STRING`: use a connection string (must be either SAS Token or Account/Key)
* `AZURE_STORAGE_ACCOUNT_NAME` and `AZURE_STORAGE_ACCOUNT_KEY`: use a specific account name and key (either primary or secondary)
* `AZURE_USE_DEFAULT_CREDENTIAL` and `AZURE_STORAGE_ACCOUNT_NAME`: use the `DefaultAzureCredential` to allow the Azure library
to search for multiple options (for example, managed identity related variables). Note that if multiple credentials are available,
it is required to specify the `AZURE_CLIENT_ID` to select specific credentials.
The `AZURE_STORAGE_DOMAIN` can be used for optionally overriding the default domain for the Azure storage service.
Please, note that `vmbackup` will use credentials provided by cloud providers metadata service [when applicable](https://docs.victoriametrics.com/victoriametrics/vmbackup/#using-cloud-providers-metadata-service).
### Using cloud providers metadata service
`vmbackup` and `vmbackupmanager` will automatically use cloud providers metadata service in order to obtain credentials if they are running in cloud environment and credentials are not explicitly provided via flags or env variables.
vmbackup \
-storageDataPath=/data \
-snapshot.createURL=http://localhost:8428/snapshot/create \
-dst=azblob://victoriametrics-backup/backup01
```
### Providing credentials in Kubernetes
The simplest way to provide credentials in Kubernetes is to use [Secrets](https://kubernetes.io/docs/concepts/configuration/secret/)
and inject them into the pod as environment variables. For example, the following secret can be used for AWS S3 credentials:
The simplest way to provide credentials in Kubernetes is to use [Secrets](https://kubernetes.io/docs/concepts/configuration/secret/) and inject them into the pod as environment variables. For example, the following secret can be used for AWS S3 credentials:
```yaml
apiVersion: v1
@@ -266,14 +250,13 @@ kind: Secret
metadata:
name: vmbackup-credentials
data:
access_key: key
secret_key: secret
access_key: <base64-encoded-key>
secret_key: <base64-encoded-secret>
```
And then it can be injected into the pod as environment variables:
```yaml
...
env:
- name: AWS_ACCESS_KEY_ID
valueFrom:
@@ -285,7 +268,6 @@ env:
secretKeyRef:
key: secret_key
name: vmbackup-credentials
...
```
A more secure way is to use IAM roles to provide tokens for pods instead of managing credentials manually.

View File

@@ -88,34 +88,21 @@ There are two flags which could help with performance tuning:
### Example of Usage
GCS and cluster version. You need to have a credentials file in json format with following structure:
credentials.json
```json
{
"type": "service_account",
"project_id": "<project>",
"private_key_id": "",
"private_key": "-----BEGIN PRIVATE KEY-----\-----END PRIVATE KEY-----\n",
"client_email": "test@<project>.iam.gserviceaccount.com",
"client_id": "",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://oauth2.googleapis.com/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/test%40<project>.iam.gserviceaccount.com"
}
```
Backup manager launched with the following configuration:
The following command backs up data to a GCS bucket:
```sh
export NODE_IP=192.168.0.10
export VMSTORAGE_ENDPOINT=http://127.0.0.1:8428
./vmbackupmanager -dst=gs://vmstorage-data/$NODE_IP -credsFilePath=credentials.json -storageDataPath=/vmstorage-data -snapshot.createURL=$VMSTORAGE_ENDPOINT/snapshot/create -licenseFile=/path/to/vm-license
./vmbackupmanager \
-dst=gs://vmstorage-data/$NODE_IP \
-credsFilePath=credentials.json \
-storageDataPath=/vmstorage-data \
-snapshot.createURL=$VMSTORAGE_ENDPOINT/snapshot/create \
-licenseFile=/path/to/vm-license
```
See [Connecting VM components to cloud storage](https://docs.victoriametrics.com/guides/connecting-vm-components-to-cloud-storage/) for instructions on obtaining credentials and configuring access to S3, GCS, and Azure Blob Storage.
Expected logs in vmbackupmanager:
```sh
@@ -147,8 +134,7 @@ objects. Typical object storage systems implement server-side copy by creating n
This is very fast and efficient. Unfortunately there are systems such as [S3 Glacier](https://aws.amazon.com/s3/storage-classes/glacier/),
which perform full object copy during server-side copying. This may be slow and expensive.
Please, see [vmbackup docs](https://docs.victoriametrics.com/victoriametrics/vmbackup/#advanced-usage) for more examples of authentication with different
storage types.
See [Connecting VM components to cloud storage](https://docs.victoriametrics.com/guides/connecting-vm-components-to-cloud-storage/) for more examples of authentication with different storage types.
### Backup Retention Policy

View File

@@ -47,13 +47,13 @@ i.e. the end result would be similar to [rsync --delete](https://askubuntu.com/q
## Troubleshooting
* See [how to setup credentials via environment variables](https://docs.victoriametrics.com/victoriametrics/vmbackup/#providing-credentials-via-env-variables).
* See [Connecting VM components to cloud storage](https://docs.victoriametrics.com/guides/connecting-vm-components-to-cloud-storage/) for instructions on configuring credentials.
* If `vmrestore` consumes all the network bandwidth, then set `-maxBytesPerSecond` to the desired value.
* If `vmrestore` has been interrupted due to temporary error, then just restart it with the same args. It will resume the restore process.
## Advanced usage
Please, see [vmbackup docs](https://docs.victoriametrics.com/victoriametrics/vmbackup/#advanced-usage) for examples of authentication
See [Connecting VM components to cloud storage](https://docs.victoriametrics.com/guides/connecting-vm-components-to-cloud-storage/) for examples of authentication
with different storage types.
Run `vmrestore -help` in order to see all the available options: