mirror of
https://github.com/VictoriaMetrics/VictoriaMetrics.git
synced 2026-06-29 21:46:01 +03:00
Compare commits
7 Commits
dashboards
...
docs/file-
| Author | SHA1 | Date | |
|---|---|---|---|
|
ed715a89ac | ||
|
|
f16bee6973 | ||
|
|
c0edf0c983 | ||
|
|
81262b2fb5 | ||
|
|
2aec74f23b | ||
|
|
d3f86b24a1 | ||
|
|
60de3bad8e |
405
docs/guides/connecting-vm-components-to-cloud-storage/README.md
Normal file
405
docs/guides/connecting-vm-components-to-cloud-storage/README.md
Normal file
@@ -0,0 +1,405 @@
|
||||
Several VictoriaMetrics components can connect to cloud storage to read or write object data.
|
||||
|
||||
| Component | AWS S3 and S3-compatible | Google Cloud Storage | Azure Blob Storage |
|
||||
|-----------|----|----------------------|--------------------|
|
||||
| [vmbackup](https://docs.victoriametrics.com/victoriametrics/vmbackup/) | ✅ | ✅ | ✅ |
|
||||
| [vmrestore](https://docs.victoriametrics.com/victoriametrics/vmrestore/) | ✅ | ✅ | ✅ |
|
||||
| [vmbackupmanager](https://docs.victoriametrics.com/victoriametrics/vmbackupmanager/) | ✅ | ✅ | ✅ |
|
||||
| [vmalert](https://docs.victoriametrics.com/victoriametrics/vmalert/) | ✅ | ✅ | ❌ |
|
||||
|
||||
All these components use the same underlying libraries, so the authentication setup is largely the same. The main difference is in flag names:
|
||||
|
||||
- vmalert uses `-s3.*` prefixed flags (e.g., `-s3.credsFilePath`)
|
||||
- backup and restore tools use unprefixed flags (e.g., `-credsFilePath`)
|
||||
|
||||
See the [component reference](#per-component-flag-reference) for details.
|
||||
|
||||
## Obtaining credentials
|
||||
|
||||
You need to supply credentials so the component can connect to the cloud storage. The setup differs by provider; the sections below cover AWS S3, S3‑compatible systems, GCS, and Azure Blob Storage.
|
||||
|
||||
### AWS S3
|
||||
|
||||
1. In AWS, [create an IAM user](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html) or role with permissions to read and write the target bucket.
|
||||
1. [Create an access key](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html) for that IAM identity.
|
||||
1. Copy the **Access key ID** and **Secret access key** values. You will use them in the credentials file or environment variables.
|
||||
|
||||
### S3-compatible storage (MinIO, Ceph)
|
||||
|
||||
Generate access keys using your storage system's admin interface or CLI. The credentials follow the same format as AWS S3.
|
||||
|
||||
### Google Cloud Storage
|
||||
|
||||
1. Open the Google Cloud Console and go to **IAM & Admin > Service Accounts**.
|
||||
1. Click **Create service account**, enter a name, and assign a Storage role (for example, Storage Object Admin).
|
||||
1. Open the service account, go to **Keys**, then click **Add key > Create new key**.
|
||||
1. Choose **JSON** as the key type and click **Create**.
|
||||
1. Store the JSON file on the machine running the VictoriaMetrics component.
|
||||
|
||||
### Azure Blob Storage
|
||||
|
||||
> Azure does not use credential files.
|
||||
|
||||
1. Log in to the Azure Portal.
|
||||
1. Search for and select **Storage accounts**.
|
||||
1. Click on your specific storage account name. (This is your `AZURE_STORAGE_ACCOUNT_NAME`).
|
||||
1. In the left menu under **Security + networking**, click **Access keys**.
|
||||
1. Copy the key value from either **key1** or **key2** (this is your `AZURE_STORAGE_ACCOUNT_KEY`).
|
||||
1. Define the access keys as environment variables.
|
||||
```sh
|
||||
export AZURE_STORAGE_ACCOUNT_NAME=mystorageaccount
|
||||
export AZURE_STORAGE_ACCOUNT_KEY=myaccountkey
|
||||
```
|
||||
|
||||
## Authenticating with the cloud provider
|
||||
|
||||
Provide the credentials as a file or with environment variables, along with the path to the cloud storage bucket. The syntax for the bucket name depends on the cloud provider:
|
||||
|
||||
- `s3://`: for AWS S3 and S3-compatible storage (MinIO, Ceph)
|
||||
- `gs://`: Google Cloud Storage
|
||||
- `azblob://`: Azure Blob Storage
|
||||
|
||||
### vmbackup and vmrestore
|
||||
|
||||
The following example backs up to an AWS S3 bucket using a credentials file:
|
||||
|
||||
```sh
|
||||
vmbackup \
|
||||
-storageDataPath=/data \
|
||||
-snapshot.createURL=http://localhost:8428/snapshot/create \
|
||||
-dst=s3://victoriametrics-backup/backup01 \
|
||||
-credsFilePath=/etc/credentials
|
||||
```
|
||||
|
||||
In order to restore from the same backup from AWS S3:
|
||||
|
||||
```sh
|
||||
vmrestore \
|
||||
-src=s3://victoriametrics-backup/backup01 \
|
||||
-storageDataPath=/data \
|
||||
-credsFilePath=/etc/credentials
|
||||
|
||||
```
|
||||
|
||||
Alternatively, you can set the access keys as environment variables instead of using a credential file:
|
||||
|
||||
```sh
|
||||
export AWS_ACCESS_KEY_ID=YOUR_AWS_ACCESS_KEY
|
||||
export AWS_SECRET_ACCESS_KEY=YOUR_SECRET_AWS_ACCESS_KEY
|
||||
|
||||
vmbackup \
|
||||
-storageDataPath=/data \
|
||||
-snapshot.createURL=http://localhost:8428/snapshot/create \
|
||||
-dst=s3://victoriametrics-backup/backup01
|
||||
|
||||
vmrestore \
|
||||
-src=s3://victoriametrics-backup/backup01 \
|
||||
-storageDataPath=/data
|
||||
```
|
||||
|
||||
> To use non-AWS S3 buckets, you must [supply the `-customS3Endpoint` argument](#s3-compatible-endpoints).
|
||||
|
||||
Backups on Google Cloud Storage use the `gs://` prefix in the destination:
|
||||
|
||||
```sh
|
||||
vmbackup \
|
||||
-storageDataPath=/data \
|
||||
-snapshot.createURL=http://localhost:8428/snapshot/create \
|
||||
-dst=gs://victoriametrics-backup/backup01 \
|
||||
-credsFilePath=/etc/credentials
|
||||
```
|
||||
|
||||
You can restore this backup with:
|
||||
|
||||
```sh
|
||||
vmrestore \
|
||||
-src=gs://victoriametrics-backup/backup01 \
|
||||
-storageDataPath=/data \
|
||||
-credsFilePath=/etc/credentials
|
||||
```
|
||||
|
||||
On Google Cloud, you can define the path to the JSON credential file with `GOOGLE_APPLICATION_CREDENTIALS`. For example:
|
||||
|
||||
```sh
|
||||
export GOOGLE_APPLICATION_CREDENTIALS=/etc/credentials
|
||||
|
||||
vmbackup \
|
||||
-storageDataPath=/data \
|
||||
-snapshot.createURL=http://localhost:8428/snapshot/create \
|
||||
-dst=gs://victoriametrics-backup/backup01
|
||||
|
||||
vmrestore \
|
||||
-src=gs://victoriametrics-backup/backup01 \
|
||||
-storageDataPath=/data
|
||||
```
|
||||
|
||||
For Azure Blob Storage, use the `azblob://` prefix and rely on environment variables instead of `-credsFilePath`.
|
||||
|
||||
```sh
|
||||
export AZURE_STORAGE_ACCOUNT_NAME=myaccount
|
||||
export AZURE_STORAGE_ACCOUNT_KEY=mykey
|
||||
|
||||
vmbackup \
|
||||
-storageDataPath=/data \
|
||||
-snapshot.createURL=http://localhost:8428/snapshot/create \
|
||||
-dst=azblob://victoriametrics-backup/backup01
|
||||
|
||||
vmrestore \
|
||||
-src=azblob://victoriametrics-backup/backup01 \
|
||||
-storageDataPath=/data
|
||||
```
|
||||
|
||||
### vmbackupmanager
|
||||
|
||||
> vmbackupmanager only works in the [Enterprise](https://docs.victoriametrics.com/victoriametrics/enterprise/) edition.
|
||||
|
||||
To manage backups with vmbackupmanager on AWS S3, add the credentials with the `-credsFilePath` flag:
|
||||
|
||||
```sh
|
||||
vmbackupmanager \
|
||||
-dst=s3://vmstorage-data/backups \
|
||||
-credsFilePath=/etc/credentials \
|
||||
-storageDataPath=/vmstorage-data \
|
||||
-snapshot.createURL=http://vmstorage:8482/snapshot/create \
|
||||
-licenseFile=/etc/vm-license
|
||||
```
|
||||
|
||||
Or define the access keys using environment variables:
|
||||
|
||||
```sh
|
||||
export AWS_ACCESS_KEY_ID=YOUR_AWS_ACCESS_KEY
|
||||
export AWS_SECRET_ACCESS_KEY=YOUR_SECRET_AWS_ACCESS_KEY
|
||||
|
||||
vmbackupmanager \
|
||||
-dst=s3://vmstorage-data/backups \
|
||||
-storageDataPath=/vmstorage-data \
|
||||
-snapshot.createURL=http://vmstorage:8482/snapshot/create \
|
||||
-licenseFile=/etc/vm-license
|
||||
```
|
||||
|
||||
> To use non-AWS S3 buckets, you must [supply the `-customS3Endpoint` argument](#s3-compatible-endpoints).
|
||||
|
||||
Automated backups on Google Cloud Storage take the following form:
|
||||
|
||||
```sh
|
||||
vmbackupmanager \
|
||||
-dst=gs://vmstorage-data/backups \
|
||||
-credsFilePath=/etc/credentials \
|
||||
-storageDataPath=/vmstorage-data \
|
||||
-snapshot.createURL=http://vmstorage:8482/snapshot/create \
|
||||
-licenseFile=/etc/vm-license
|
||||
```
|
||||
|
||||
As with vmbackup and vmrestore, you can also define the path to the JSON credential file with `GOOGLE_APPLICATION_CREDENTIALS`:
|
||||
|
||||
```sh
|
||||
export GOOGLE_APPLICATION_CREDENTIALS=/etc/credentials
|
||||
|
||||
vmbackupmanager \
|
||||
-dst=gs://vmstorage-data/backups \
|
||||
-storageDataPath=/vmstorage-data \
|
||||
-snapshot.createURL=http://vmstorage:8482/snapshot/create \
|
||||
-licenseFile=/etc/vm-license
|
||||
```
|
||||
|
||||
vmbackupmanager can also use Azure Blob Storage by defining environment variables:
|
||||
|
||||
```sh
|
||||
export AZURE_STORAGE_ACCOUNT_NAME=mystorageaccount
|
||||
export AZURE_STORAGE_ACCOUNT_KEY=myaccountkey
|
||||
|
||||
vmbackupmanager \
|
||||
-dst=azblob://vmstorage-data/backups \
|
||||
-storageDataPath=/vmstorage-data \
|
||||
-snapshot.createURL=http://vmstorage:8482/snapshot/create \
|
||||
-licenseFile=/etc/vm-license
|
||||
```
|
||||
|
||||
### vmalert
|
||||
|
||||
> - vmalert cloud storage command line flags are prefixed with `-s3.` for S3 buckets *and* Google Cloud Storage.
|
||||
> - Cloud storage only works in the [Enterprise](https://docs.victoriametrics.com/victoriametrics/enterprise/) edition.
|
||||
|
||||
Read alerting rules from an S3 bucket. The `-rule` flag accepts a prefix, so it matches all files starting with `alerts_` in the `rules` folder:
|
||||
|
||||
```sh
|
||||
vmalert \
|
||||
-rule=s3://my-alert-bucket/rules/alerts_ \
|
||||
-s3.credsFilePath=/etc/vmalert/aws-credentials \
|
||||
-datasource.url=http://vmselect:8481/select/0/prometheus \
|
||||
-notifier.url=http://alertmanager:9093
|
||||
```
|
||||
|
||||
Instead of a credential file, you can supply the access keys using environment variables:
|
||||
|
||||
```sh
|
||||
export AWS_ACCESS_KEY_ID=YOUR_AWS_ACCESS_KEY
|
||||
export AWS_SECRET_ACCESS_KEY=YOUR_SECRET_AWS_ACCESS_KEY
|
||||
|
||||
vmalert \
|
||||
-rule=s3://my-alert-bucket/rules/alerts_ \
|
||||
-datasource.url=http://vmselect:8481/select/0/prometheus \
|
||||
-notifier.url=http://alertmanager:9093
|
||||
```
|
||||
|
||||
> To use non-AWS S3 buckets, you must [supply the `-s3.customEndpoint` argument](#s3-compatible-endpoints).
|
||||
|
||||
To read rules from Google Cloud Storage:
|
||||
|
||||
```sh
|
||||
vmalert \
|
||||
-rule=gs://my-alert-bucket/rules/alerts_ \
|
||||
-s3.credsFilePath=/etc/vmalert/gcp-service-account.json \
|
||||
-datasource.url=http://vmselect:8481/select/0/prometheus \
|
||||
-notifier.url=http://alertmanager:9093
|
||||
```
|
||||
|
||||
If you prefer, you can supply the path to the JSON credential file with the `GOOGLE_APPLICATION_CREDENTIALS` environment variable:
|
||||
|
||||
```sh
|
||||
export GOOGLE_APPLICATION_CREDENTIALS=/etc/credentials
|
||||
|
||||
vmalert \
|
||||
-rule=gs://my-alert-bucket/rules/alerts_ \
|
||||
-datasource.url=http://vmselect:8481/select/0/prometheus \
|
||||
-notifier.url=http://alertmanager:9093
|
||||
```
|
||||
|
||||
## Credentials files format
|
||||
|
||||
The file format depends on the storage provider.
|
||||
|
||||
### S3 credentials
|
||||
|
||||
The file uses the standard AWS shared credentials format used by the [AWS CLI](https://docs.aws.amazon.com/cli/v1/userguide/cli-configure-files.html) and [AWS SDKs](https://docs.aws.amazon.com/sdkref/latest/guide/file-format.html):
|
||||
|
||||
```ini
|
||||
[default]
|
||||
aws_access_key_id=YOUR_AWS_ACCESS_KEY
|
||||
aws_secret_access_key=YOUR_AWS_SECRET_ACCESS_KEY
|
||||
```
|
||||
|
||||
You can define multiple profiles in a single file:
|
||||
|
||||
```ini
|
||||
[default]
|
||||
aws_access_key_id=DEFAULT_ACCESS_KEY
|
||||
aws_secret_access_key=DEFAULT_SECRET_KEY
|
||||
|
||||
[monitoring]
|
||||
aws_access_key_id=MONITORING_ACCESS_KEY
|
||||
aws_secret_access_key=MONITORING_SECRET_KEY
|
||||
|
||||
[alerts]
|
||||
aws_access_key_id=ALERTS_ACCESS_KEY
|
||||
aws_secret_access_key=ALERTS_SECRET_KEY
|
||||
```
|
||||
|
||||
Use the `-configProfile` flag (or `-s3.configProfile` in vmalert) to select a non-default profile:
|
||||
|
||||
```sh
|
||||
-configProfile=alerts
|
||||
```
|
||||
|
||||
You can separate credentials from other configuration settings. Put credentials in one file:
|
||||
|
||||
```ini
|
||||
[default]
|
||||
aws_access_key_id=DEFAULT_ACCESS_KEY
|
||||
aws_secret_access_key=DEFAULT_SECRET_KEY
|
||||
```
|
||||
|
||||
And non-sensitive settings in another:
|
||||
|
||||
```ini
|
||||
[default]
|
||||
region=us-east-1
|
||||
```
|
||||
|
||||
Then pass both files:
|
||||
|
||||
```sh
|
||||
-configFilePath=/etc/aws-config \
|
||||
-credsFilePath=/etc/credentials
|
||||
```
|
||||
|
||||
### GCS credentials file format
|
||||
|
||||
The file is the JSON key downloaded from Google Cloud Console. Its content looks like this:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "service_account",
|
||||
"project_id": "project-id",
|
||||
"private_key_id": "key-id",
|
||||
"private_key": "-----BEGIN PRIVATE KEY-----\nprivate-key\n-----END PRIVATE KEY-----\n",
|
||||
"client_email": "service-account-email",
|
||||
"client_id": "client-id",
|
||||
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
|
||||
"token_uri": "https://accounts.google.com/o/oauth2/token",
|
||||
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
|
||||
"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/service-account-email"
|
||||
}
|
||||
```
|
||||
|
||||
This is the standard service account key format defined by [Google Cloud IAM](https://developers.google.com/workspace/guides/create-credentials).
|
||||
|
||||
### Azure Blob Storage
|
||||
|
||||
Azure does not support credentials via file. Use environment variables instead.
|
||||
|
||||
## S3-compatible endpoints
|
||||
|
||||
For S3-compatible storage such as MinIO or Ceph, set a custom endpoint with the `-customS3Endpoint` flag for vmbackup, vmrestore, and vmbackupmanager. For example:
|
||||
|
||||
```sh
|
||||
vmbackup \
|
||||
-storageDataPath=/data \
|
||||
-snapshot.createURL=http://localhost:8428/snapshot/create \
|
||||
-dst=s3://victoriametrics-backup/backup01 \
|
||||
-customS3Endpoint=http://minio.example.local:9000
|
||||
```
|
||||
|
||||
On vmalert, use the `-s3.customEndpoint` flag instead:
|
||||
|
||||
```sh
|
||||
vmalert \
|
||||
-rule=s3://my-alert-bucket/rules/alerts_ \
|
||||
-s3.customEndpoint=http://minio.example.local:9000 \
|
||||
-s3.credsFilePath=/etc/vmalert/aws-credentials \
|
||||
-datasource.url=http://vmselect:8481/select/0/prometheus \
|
||||
-notifier.url=http://alertmanager:9093
|
||||
```
|
||||
|
||||
### Addressing S3-compatible buckets
|
||||
|
||||
When connecting to non-AWS S3-compatible buckets, there is an additional flag you might need to configure:
|
||||
|
||||
- `-s3ForcePathStyle`: on vmbackupmanager, vmbackup, and vmrestore.
|
||||
- `-s3.forcePathStyle`: on vmalert.
|
||||
|
||||
The flag changes the expected URL pattern for a bucket.
|
||||
|
||||
| Flag value | Address-style | Example | Use with |
|
||||
|------------|---------------|---------|----------|
|
||||
| `true` (default) | Path-style | `https://endpoint/bucket/key` | MinIO, Ceph, most S3-compatible storages |
|
||||
| `false` | Virtual host-style | `https://bucket.endpoint/key` | [Aliyun OSS](https://www.aliyun.com/product/oss) and other endpoints that require it |
|
||||
|
||||
> The flag only takes effect when you use a custom endpoint (`-customS3Endpoint` or `-s3.customEndpoint` on vmalert). When connecting to real AWS S3, the SDK handles addressing automatically.
|
||||
|
||||
## Per-component flag reference
|
||||
|
||||
The table below shows how the same concept maps to different flag names across components.
|
||||
|
||||
| Concept | vmalert | vmbackup, vmrestore, and vmbackupmanager |
|
||||
|---|---|---|---|---|
|
||||
| Credentials file | `-s3.credsFilePath` | `-credsFilePath` |
|
||||
| Config file | `-s3.configFilePath` | `-configFilePath` |
|
||||
| Profile selection | `-s3.configProfile` | `-configProfile` |
|
||||
| Custom endpoint | `-s3.customEndpoint` | `-customS3Endpoint` |
|
||||
| Force path style | `-s3.forcePathStyle` | `-s3ForcePathStyle` |
|
||||
| TLS insecure | N/A | `-s3TLSInsecureSkipVerify` |
|
||||
| Storage class | N/A | `-s3StorageClass` |
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,12 @@
|
||||
---
|
||||
weight: 5
|
||||
title: Connecting VictoriaMetrics components to cloud storage
|
||||
menu:
|
||||
docs:
|
||||
parent: guides
|
||||
weight: 5
|
||||
tags:
|
||||
- metrics
|
||||
- guide
|
||||
---
|
||||
{{% content "README.md" %}}
|
||||
@@ -546,7 +546,7 @@ tags at [Docker Hub](https://hub.docker.com/r/victoriametrics/vmalert/tags) and
|
||||
|
||||
## Reading rules from object storage
|
||||
|
||||
[Enterprise version](https://docs.victoriametrics.com/victoriametrics/enterprise/) of `vmalert` may read alerting and recording rules
|
||||
The [Enterprise version](https://docs.victoriametrics.com/victoriametrics/enterprise/) of `vmalert` may read alerting and recording rules
|
||||
from object storage:
|
||||
|
||||
* `./bin/vmalert -rule=s3://bucket/dir/alert.rules` would read rules from the given path at S3 bucket
|
||||
@@ -563,6 +563,133 @@ The following [command-line flags](#flags) can be used for fine-tuning access to
|
||||
* `-s3.customEndpoint` - custom S3 endpoint for use with S3-compatible storages (e.g. MinIO). S3 is used if not set.
|
||||
* `-s3.forcePathStyle` - prefixing endpoint with bucket name when set false, true by default.
|
||||
|
||||
### S3 (AWS and S3-compatible)
|
||||
|
||||
1. In AWS, [create an IAM user](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html) or role with read-only permissions on the target bucket.
|
||||
2. [Create an access key](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html) for that IAM identity and copy the **Access key** and **Secret access key** values
|
||||
3. On the machine running vmalert, create a credentials file with the following content and point `-s3.credsFilePath` to it:
|
||||
|
||||
```ini
|
||||
[default]
|
||||
aws_access_key_id=YOUR_AWS_ACCESS_KEY
|
||||
aws_secret_access_key=YOUR_AWS_SECRET_ACCESS_KEY
|
||||
```
|
||||
|
||||
This format matches the standard shared AWS credentials file used by the [AWS CLI](https://docs.aws.amazon.com/cli/v1/userguide/cli-configure-files.html) and [AWS SDKs](https://docs.aws.amazon.com/sdkref/latest/guide/file-format.html).
|
||||
|
||||
The following example reads the rules from the S3 bucket called `my-alert-bucket`. If our bucket contains `rules/alerts_cpu.yml` and `rules/alerts_memory.yml`, then `-rule=s3://my-alert-bucket/rules/alerts_` matches both files because both keys start with that exact prefix
|
||||
|
||||
```sh
|
||||
./bin/vmalert \
|
||||
-rule=s3://my-alert-bucket/rules/alerts_ \
|
||||
-s3.credsFilePath=/etc/vmalert/aws-credentials \
|
||||
-datasource.url=http://vmselect:8481/select/0/prometheus \
|
||||
-notifier.url=http://alertmanager:9093
|
||||
```
|
||||
|
||||
You define multiple profiles in a single credentials file with different access keys:
|
||||
|
||||
```ini
|
||||
[default]
|
||||
aws_access_key_id=DEFAULT_ACCESS_KEY
|
||||
aws_secret_access_key=DEFAULT_SECRET_KEY
|
||||
|
||||
[monitoring]
|
||||
aws_access_key_id=MONITORING_ACCESS_KEY
|
||||
aws_secret_access_key=MONITORING_SECRET_KEY
|
||||
|
||||
[alerts]
|
||||
aws_access_key_id=ALERTS_ACCESS_KEY
|
||||
aws_secret_access_key=ALERTS_SECRET_KEY
|
||||
```
|
||||
|
||||
Then you can start vmalert with `-s3.configProfile` to use a non-default profile:
|
||||
|
||||
```sh
|
||||
./bin/vmalert \
|
||||
-rule=s3://my-alert-bucket/rules/alerts_ \
|
||||
-s3.credsFilePath=/etc/vmalert/aws-credentials \
|
||||
-s3.configProfile=alerts
|
||||
```
|
||||
|
||||
Additionally, you can separate the credentials file from other configuration files. For example, one file can have credentials:
|
||||
|
||||
|
||||
```ini
|
||||
[default]
|
||||
aws_access_key_id=DEFAULT_ACCESS_KEY
|
||||
aws_secret_access_key=DEFAULT_SECRET_KEY
|
||||
```
|
||||
|
||||
And leave the rest of the non-sensitive settings in a separate configuration file:
|
||||
|
||||
```ini
|
||||
[default]
|
||||
region=us-east-1
|
||||
```
|
||||
|
||||
Then pass both files to vmalert:
|
||||
|
||||
```sh
|
||||
./bin/vmalert \
|
||||
-rule=s3://my-alert-bucket/rules/alerts_ \
|
||||
-s3.credsFilePath=/etc/vmalert/aws-credentials \
|
||||
-s3.configFilePath=/etc/vmalert/aws-config
|
||||
```
|
||||
|
||||
For S3-compatible backends such as [MinIO](https://www.min.io/) or [Ceph](https://ceph.io/), create access keys in the respective
|
||||
system and use the same file format and set a custom endpoint with `-s3.customS3Endpoint`.
|
||||
|
||||
For example:
|
||||
|
||||
```sh
|
||||
./bin/vmalert \
|
||||
-rule=s3://victoriametrics-alert-rules/rules_ \
|
||||
-s3.credsFilePath=/etc/vmalert/aws-credentials \
|
||||
-s3.customEndpoint=http://minio.example.local:9000 \
|
||||
-datasource.url=http://vmselect:8481/select/0/prometheus \
|
||||
-notifier.url=http://alertmanager:9093
|
||||
```
|
||||
|
||||
### Google Cloud Storage (GCS)
|
||||
|
||||
To create a service account and download the credential file, follow these steps:
|
||||
|
||||
1. Open the Google Cloud Console and go to **IAM & Admin → Service Accounts**.
|
||||
2. Click **Create service account**.
|
||||
3. Enter a service account name.
|
||||
4. Assign the role the account needs to access Google Cloud Storage. See [IAM permissions for JSON methods](https://docs.cloud.google.com/storage/docs/access-control/iam-json) for more details.
|
||||
5. Open the service account, go to **Keys**, then click **Add key → Create new key**.
|
||||
6. Choose **JSON** as the key type
|
||||
7. Save the downloaded JSON file on the machine running vmalert and point `-s3.credsFilePath` to it. The file contents look similar to:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "service_account",
|
||||
"project_id": "project-id",
|
||||
"private_key_id": "key-id",
|
||||
"private_key": "-----BEGIN PRIVATE KEY-----\nprivate-key\n-----END PRIVATE KEY-----\n",
|
||||
"client_email": "service-account-email",
|
||||
"client_id": "client-id",
|
||||
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
|
||||
"token_uri": "https://accounts.google.com/o/oauth2/token",
|
||||
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
|
||||
"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/service-account-email"
|
||||
}
|
||||
```
|
||||
|
||||
This JSON is the standard service account key format defined by [Google Cloud IAM](https://developers.google.com/workspace/guides/create-credentials) and is used by Google client libraries and tools.
|
||||
|
||||
The following example shows how to load rules from Google Cloud Storage in vmalert:
|
||||
|
||||
```sh
|
||||
./bin/vmalert \
|
||||
-rule=gs://my-alert-bucket/rules/alerts_ \
|
||||
-s3.credsFilePath=/etc/vmalert/gcp-service-account.json \
|
||||
-datasource.url=http://vmselect:8481/select/0/prometheus \
|
||||
-notifier.url=http://alertmanager:9093
|
||||
```
|
||||
|
||||
## Topology examples
|
||||
|
||||
The following sections are showing how `vmalert` may be used and configured
|
||||
|
||||
@@ -204,38 +204,80 @@ See [this article](https://medium.com/@valyala/speeding-up-backups-for-big-time-
|
||||
|
||||
### Providing credentials as a file
|
||||
|
||||
Obtaining credentials from a file.
|
||||
`vmbackup` and `vmbackupmanager` can load credentials from a file via the `-credsFilePath` flag to access remote S3-compatible buckets and Google Cloud Storage.
|
||||
|
||||
Add flag `-credsFilePath=/etc/credentials` with the following content:
|
||||
To use a credential file, add the flag:
|
||||
|
||||
* for S3 (AWS, MinIO or other S3 compatible storages):
|
||||
```sh
|
||||
-credsFilePath=/etc/credentials
|
||||
```
|
||||
|
||||
```sh
|
||||
[default]
|
||||
aws_access_key_id=theaccesskey
|
||||
aws_secret_access_key=thesecretaccesskeyvalue
|
||||
```
|
||||
The argument should point to a file with one of the formats below, depending on the storage provider.
|
||||
|
||||
* for GCP cloud storage:
|
||||
#### S3 (AWS and S3-compatible)
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "service_account",
|
||||
"project_id": "project-id",
|
||||
"private_key_id": "key-id",
|
||||
"private_key": "-----BEGIN PRIVATE KEY-----\nprivate-key\n-----END PRIVATE KEY-----\n",
|
||||
"client_email": "service-account-email",
|
||||
"client_id": "client-id",
|
||||
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
|
||||
"token_uri": "https://accounts.google.com/o/oauth2/token",
|
||||
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
|
||||
"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/service-account-email"
|
||||
}
|
||||
```
|
||||
1. In AWS, [create an IAM user](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html) or role with permissions to read and write the target bucket.
|
||||
2. [Create an access key](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html) for that IAM identity and copy the **Access key** and **Secret access key** values
|
||||
3. On the machine running `vmbackup`, create a credentials file with the following content and point `-credsFilePath` to it:
|
||||
|
||||
```ini
|
||||
[default]
|
||||
aws_access_key_id=YOUR_AWS_ACCESS_KEY
|
||||
aws_secret_access_key=YOUR_AWS_SECRET_ACCESS_KEY
|
||||
```
|
||||
|
||||
This format matches the standard shared AWS credentials file used by the [AWS CLI](https://docs.aws.amazon.com/cli/v1/userguide/cli-configure-files.html) and [AWS SDKs](https://docs.aws.amazon.com/sdkref/latest/guide/file-format.html).
|
||||
|
||||
For S3-compatible backends such as [MinIO](https://www.min.io/) or [Ceph](https://ceph.io/), create access keys in the respective
|
||||
system and use the same file format and set a custom endpoint with `-customS3Endpoint`.
|
||||
|
||||
For example:
|
||||
|
||||
```sh
|
||||
vmbackup \
|
||||
-storageDataPath=/data \
|
||||
-snapshot.createURL=http://localhost:8428/snapshot/create \
|
||||
-dst=s3://victoriametrics-backup/backup01 \
|
||||
-customS3Endpoint=http://minio.example.local:9000 \
|
||||
-credsFilePath=/etc/credentials
|
||||
```
|
||||
|
||||
#### Google Cloud Storage (GCS)
|
||||
|
||||
To create an IAM user and download the credential file, follow these steps:
|
||||
|
||||
1. Open the Google Cloud Console and go to **IAM & Admin → Service Accounts**.
|
||||
2. Click **Create service account**.
|
||||
3. Enter a service account name.
|
||||
4. Assign the role the account needs to access Google Cloud Storage. See [IAM permissions for JSON methods](https://docs.cloud.google.com/storage/docs/access-control/iam-json) for more details.
|
||||
5. Open the service account, go to **Keys**, then click **Add key → Create new key**.
|
||||
6. Choose **JSON** as the key type
|
||||
7. Save the downloaded JSON file on the machine running `vmbackup` and point `-credsFilePath` to it. The file contents look similar to:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "service_account",
|
||||
"project_id": "project-id",
|
||||
"private_key_id": "key-id",
|
||||
"private_key": "-----BEGIN PRIVATE KEY-----\nprivate-key\n-----END PRIVATE KEY-----\n",
|
||||
"client_email": "service-account-email",
|
||||
"client_id": "client-id",
|
||||
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
|
||||
"token_uri": "https://accounts.google.com/o/oauth2/token",
|
||||
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
|
||||
"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/service-account-email"
|
||||
}
|
||||
```
|
||||
|
||||
This JSON is the standard service account key format defined by [Google Cloud IAM](https://developers.google.com/workspace/guides/create-credentials) and is used by Google client libraries and tools.
|
||||
|
||||
#### Azure Blob Storage
|
||||
|
||||
Azure Blob Storage uses environment variables rather than `-credsFilePath` in `vmbackup`. See [providing credentials via env variables](https://docs.victoriametrics.com/victoriametrics/vmbackup/#providing-credentials-via-env-variables) for details.
|
||||
|
||||
### Providing credentials via env variables
|
||||
|
||||
Obtaining credentials from env variables.
|
||||
Obtaining credentials from environment variables.
|
||||
|
||||
* For AWS S3 compatible storages set env variable `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`.
|
||||
Also you can set env variable `AWS_SHARED_CREDENTIALS_FILE` with path to credentials file.
|
||||
|
||||
Reference in New Issue
Block a user