Compare commits

..

15 Commits

Author SHA1 Message Date
dependabot[bot]
308fefc685 build(deps): bump github/codeql-action/init from 4.36.2 to 4.36.3
Bumps [github/codeql-action/init](https://github.com/github/codeql-action) from 4.36.2 to 4.36.3.
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](8aad20d150...54f647b7e1)

---
updated-dependencies:
- dependency-name: github/codeql-action/init
  dependency-version: 4.36.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-07-23 04:16:33 +00:00
Nikolay
344d3e1911 lib/httputil: add load-balancing http transport
This commit adds http client load-balancing with DNS discovery. For both A and SRV records.
It helps to route HTTP requests evenly for across discovered backends.

Discovered IP addresses are cached locally.

 The main benefit of this feature is to remove intermediate vmauth as a load-balancer between
vmagent (vmalert) and remote targets. Which simplifies components management and
reduces operational overhead.

 By default, it's disable and could be used with `dns+` or `srv+` hostname suffix at url.
For example, `-remoteWrite.url=http://dns+victoria-metrics:8428/api/v1/write`.

Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2388
2026-07-22 18:34:23 +02:00
Andrei Baidarov
8b94ca0e30 lib/persistentqueue: expose fastQueue name in metrics
This commit adds `name` label to the `vm_persistentqueue_*` metrics. It provides more context for queues with human read-able format. Because `path` label contains only on-disk path, which is not really useful.

 For vmagent case it will contain remote.writeURL position at command-line args.

Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7944
2026-07-22 18:28:58 +02:00
f41gh7
a9ae7e1da0 vendor: update golang.org/x/... packages 2026-07-22 18:00:29 +02:00
Max Kotliar
ba59d78f10 docs: update supported LTS versions 2026-07-22 12:56:17 +03:00
Max Kotliar
1c774f9216 app/vmstorage: enhance vmselectAddr flag description (#11270)
The change applies only to the master and must not be ported to the
cluster version

Follow-up on
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/10926
2026-07-21 18:46:37 +03:00
Max Kotliar
7ea9bfa8d6 docs: bump version to v1.148.0
Signed-off-by: Max Kotliar <mkotlyar@victoriametrics.com>
2026-07-21 18:06:37 +03:00
Artem Fetishev
2b95c11095 lib/storage: rewrite TestIndexDB (#11283)
This makes the test suitable for testing cases when global or per-day
indexes are disabled.

Related to https://github.com/VictoriaMetrics/VictoriaMetrics/pull/11196

Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
2026-07-21 15:43:00 +02:00
Rudransh Shrivastava
655ed9e8d1 .github/workflows/vmui.yml: remove eslint annotation action (#11259)
Remove the unmaintained third-party `ataylorme/eslint-annotate-action`
from the vmui workflow in order to reduce supply-chain risk.
Switch `npm run lint` back to human-readable ESLint output, since the
JSON report was used only by the removed action.

The `make vmui-lint` command still fails CI on lint errors. Printing
ESLint errors in the workflow logs should be enough for blocking pull
requests with lint errors.

See the alternatives considered and why this was removed:
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/11259#issuecomment-4981601165

cc: @arkid15r 

Ref:
[ataylorme/eslint-annotate-action](https://github.com/ataylorme/eslint-annotate-action)

Signed-off-by: Rudransh Shrivastava <rudransh@victoriametrics.com>
2026-07-21 15:13:47 +03:00
Pablo (Tomas) Fernandez
4f676b4012 docs: add "Connecting VictoriaMetrics components to cloud storage" guide (#11106)
This PR adds "Connecting VictoriaMetrics components to cloud storage" guide and references it from vmbackup and vmalert pages. 

Fixes: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/11073
PR https://github.com/VictoriaMetrics/VictoriaMetrics/pull/11106
2026-07-21 15:09:35 +03:00
Artem Fetishev
dce9514c65 docs: update flags with actual v1.148.0 binaries
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
2026-07-20 13:10:17 +02:00
Artem Fetishev
25598da441 docs: bump version to v1.148.0
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
2026-07-20 13:06:43 +02:00
Artem Fetishev
6250255c4f deplyoment/docker: bump version to v1.148.0
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
2026-07-20 13:04:19 +02:00
Artem Fetishev
be7eefcf00 docs: forward port LTS v1.122.27 changelog to upstream
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
2026-07-20 13:01:35 +02:00
Artem Fetishev
79eba63498 docs: forward port LTS v1.136.14 changelog to upstream
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
2026-07-20 13:00:55 +02:00
46 changed files with 1517 additions and 283 deletions

View File

@@ -54,7 +54,7 @@ jobs:
restore-keys: go-artifacts-${{ runner.os }}-codeql-analyze-${{ steps.go.outputs.go-version }}-
- name: Initialize CodeQL
uses: github/codeql-action/init@8aad20d150bbac5944a9f9d289da16a4b0d87c1e # v4.36.2
uses: github/codeql-action/init@54f647b7e1bb85c95cddabcd46b0c578ec92bc1a # v4.36.3
with:
languages: go

View File

@@ -26,9 +26,7 @@ jobs:
vmui-checks:
name: VMUI Checks (lint, test, typecheck)
permissions:
checks: write
contents: read
pull-requests: read
runs-on: ubuntu-latest
steps:
- name: Code checkout
@@ -70,12 +68,6 @@ jobs:
env:
VMUI_SKIP_INSTALL: true
- name: Annotate Code Linting Results
uses: ataylorme/eslint-annotate-action@d57a1193d4c59cbfbf3f86c271f42612f9dbd9e9 # 3.0.0
with:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
report-json: app/vmui/packages/vmui/vmui-lint-report.json
- name: Check overall status
run: |
echo "Lint status: ${{ steps.lint.outcome }}"

View File

@@ -151,17 +151,23 @@ func newHTTPClient(argIdx int, remoteWriteURL, sanitizedURL string, fq *persiste
}
tr.Proxy = http.ProxyURL(pu)
}
hc := &http.Client{
Transport: authCfg.NewRoundTripper(tr),
Timeout: sendTimeout.GetOptionalArg(argIdx),
}
rwURL, err := url.Parse(remoteWriteURL)
if err != nil {
logger.Fatalf("BUG: cannot parse already parsed -remoteWrite.url=%q: %s", remoteWriteURL, err)
}
hc.Transport, rwURL = httputil.NewLoadBalancerTransport(hc.Transport, rwURL)
retryMaxIntervalFlag := retryMaxTime
if retryMaxInterval.String() != "" {
retryMaxIntervalFlag = retryMaxInterval
}
c := &client{
sanitizedURL: sanitizedURL,
remoteWriteURL: remoteWriteURL,
remoteWriteURL: rwURL.String(),
authCfg: authCfg,
awsCfg: awsCfg,
fq: fq,

View File

@@ -11,6 +11,7 @@ import (
"github.com/VictoriaMetrics/VictoriaMetrics/app/vmalert/vmalertutil"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/flagutil"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/httputil"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/promauth"
)
@@ -94,6 +95,12 @@ func Init(extraParams url.Values) (QuerierBuilder, error) {
tr.MaxIdleConns = tr.MaxIdleConnsPerHost
}
tr.IdleConnTimeout = *idleConnectionTimeout
hc := &http.Client{Transport: tr}
datasourceURL, err := url.Parse(*addr)
if err != nil {
logger.Fatalf("BUG: cannot parse already parsed -datasource.url=%q: %s", *addr, err)
}
hc.Transport, datasourceURL = httputil.NewLoadBalancerTransport(tr, datasourceURL)
if extraParams == nil {
extraParams = url.Values{}
@@ -120,9 +127,9 @@ func Init(extraParams url.Values) (QuerierBuilder, error) {
}
return &Client{
c: &http.Client{Transport: tr},
c: hc,
authCfg: authCfg,
datasourceURL: strings.TrimSuffix(*addr, "/"),
datasourceURL: strings.TrimSuffix(datasourceURL.String(), "/"),
appendTypePrefix: *appendTypePrefix,
queryStep: *queryStep,
extraParams: extraParams,

View File

@@ -4,12 +4,14 @@ import (
"flag"
"fmt"
"net/http"
"net/url"
"time"
"github.com/VictoriaMetrics/VictoriaMetrics/app/vmalert/datasource"
"github.com/VictoriaMetrics/VictoriaMetrics/app/vmalert/vmalertutil"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/flagutil"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/httputil"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/promauth"
)
@@ -76,7 +78,13 @@ func Init() (datasource.QuerierBuilder, error) {
return nil, fmt.Errorf("failed to create transport for -remoteRead.url=%q: %w", *addr, err)
}
tr.IdleConnTimeout = *idleConnectionTimeout
c := &http.Client{Transport: tr}
rrURL, err := url.Parse(*addr)
if err != nil {
logger.Fatalf("BUG: cannot parse already parsed -remoteRead.url=%q: %s", *addr, err)
}
c.Transport, rrURL = httputil.NewLoadBalancerTransport(tr, rrURL)
endpointParams, err := flagutil.ParseJSONMap(*oauth2EndpointParams)
if err != nil {
return nil, fmt.Errorf("cannot parse JSON for -remoteRead.oauth2.endpointParams=%s: %w", *oauth2EndpointParams, err)
@@ -89,6 +97,5 @@ func Init() (datasource.QuerierBuilder, error) {
if err != nil {
return nil, fmt.Errorf("failed to configure auth: %w", err)
}
c := &http.Client{Transport: tr}
return datasource.NewPrometheusClient(*addr, authCfg, false, c), nil
return datasource.NewPrometheusClient(rrURL.String(), authCfg, false, c), nil
}

View File

@@ -8,6 +8,7 @@ import (
"fmt"
"io"
"net/http"
"net/url"
"path"
"strings"
"sync"
@@ -111,12 +112,18 @@ func NewClient(ctx context.Context, cfg Config) (*Client, error) {
if cfg.Concurrency > 0 {
cc = cfg.Concurrency
}
hc := &http.Client{
Timeout: *sendTimeout,
Transport: cfg.Transport,
}
rwURL, err := url.Parse(cfg.Addr)
if err != nil {
logger.Fatalf("cannot parse already parsed -remoteWrite.url=%q: %s", cfg.Addr, err)
}
hc.Transport, rwURL = httputil.NewLoadBalancerTransport(hc.Transport, rwURL)
c := &Client{
c: &http.Client{
Timeout: *sendTimeout,
Transport: cfg.Transport,
},
addr: strings.TrimSuffix(cfg.Addr, "/"),
c: hc,
addr: strings.TrimSuffix(rwURL.String(), "/"),
authCfg: cfg.AuthCfg,
flushInterval: cfg.FlushInterval,
maxBatchSize: cfg.MaxBatchSize,

View File

@@ -35,7 +35,10 @@ var (
"By default, or when set to 0, -maxBackfillAge equals to -retentionPeriod, e.g. it is unlimited within the configured retention. "+
"This can be useful for limiting ingestion of historical samples, for example, when older data has been moved to another storage tier. "+
"See https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#retention")
vmselectAddr = flag.String("vmselectAddr", "", "TCP address to accept connections from vmselect services")
vmselectAddr = flag.String("vmselectAddr", "", "TCP address to listen for incoming connections from vmselect. "+
"When set, the node will be able to accept cluster-native vmselect RPC requests as if it were vmstorage. "+
"The tenant ID assigned to this node's data is controlled by -accountID and -projectID flags. "+
"See https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#multi-tenancy")
vmselectDisableRPCCompression = flag.Bool("rpc.disableCompression", false, "Whether to disable compression of the data sent from vmstorage to vmselect. "+
"This reduces CPU usage at the cost of higher network bandwidth usage")
snapshotAuthKey = flagutil.NewPassword("snapshotAuthKey", "authKey, which must be passed in query string to /snapshot* pages. It overrides -httpAuth.*")

View File

@@ -9,7 +9,7 @@
"start": "vite",
"start:playground": "cross-env PLAYGROUND=true npm run start",
"build": "vite build",
"lint": "eslint --output-file vmui-lint-report.json --format json 'src/**/*.{ts,tsx}'",
"lint": "eslint --format stylish 'src/**/*.{ts,tsx}'",
"lint:local": "eslint --ext .ts,.tsx -f stylish src",
"lint:fix": "eslint 'src/**/*.{ts,tsx}' --fix",
"copy-metricsql-docs": "cp ../../../../docs/MetricsQL.md src/assets/MetricsQL.md || true",

View File

@@ -3,7 +3,7 @@ services:
# It scrapes targets defined in --promscrape.config
# And forward them to --remoteWrite.url
vmagent:
image: victoriametrics/vmagent:v1.147.0
image: victoriametrics/vmagent:v1.148.0
depends_on:
- "vmauth"
ports:
@@ -42,14 +42,14 @@ services:
# vmstorage shards. Each shard receives 1/N of all metrics sent to vminserts,
# where N is number of vmstorages (2 in this case).
vmstorage-1:
image: victoriametrics/vmstorage:v1.147.0-cluster
image: victoriametrics/vmstorage:v1.148.0-cluster
volumes:
- strgdata-1:/storage
command:
- "--storageDataPath=/storage"
restart: always
vmstorage-2:
image: victoriametrics/vmstorage:v1.147.0-cluster
image: victoriametrics/vmstorage:v1.148.0-cluster
volumes:
- strgdata-2:/storage
command:
@@ -59,7 +59,7 @@ services:
# vminsert is ingestion frontend. It receives metrics pushed by vmagent,
# pre-process them and distributes across configured vmstorage shards.
vminsert-1:
image: victoriametrics/vminsert:v1.147.0-cluster
image: victoriametrics/vminsert:v1.148.0-cluster
depends_on:
- "vmstorage-1"
- "vmstorage-2"
@@ -68,7 +68,7 @@ services:
- "--storageNode=vmstorage-2:8400"
restart: always
vminsert-2:
image: victoriametrics/vminsert:v1.147.0-cluster
image: victoriametrics/vminsert:v1.148.0-cluster
depends_on:
- "vmstorage-1"
- "vmstorage-2"
@@ -80,7 +80,7 @@ services:
# vmselect is a query fronted. It serves read queries in MetricsQL or PromQL.
# vmselect collects results from configured `--storageNode` shards.
vmselect-1:
image: victoriametrics/vmselect:v1.147.0-cluster
image: victoriametrics/vmselect:v1.148.0-cluster
depends_on:
- "vmstorage-1"
- "vmstorage-2"
@@ -90,7 +90,7 @@ services:
- "--vmalert.proxyURL=http://vmalert:8880"
restart: always
vmselect-2:
image: victoriametrics/vmselect:v1.147.0-cluster
image: victoriametrics/vmselect:v1.148.0-cluster
depends_on:
- "vmstorage-1"
- "vmstorage-2"
@@ -105,7 +105,7 @@ services:
# read requests from Grafana, vmui, vmalert among vmselects.
# It can be used as an authentication proxy.
vmauth:
image: victoriametrics/vmauth:v1.147.0
image: victoriametrics/vmauth:v1.148.0
depends_on:
- "vmselect-1"
- "vmselect-2"
@@ -119,7 +119,7 @@ services:
# vmalert executes alerting and recording rules
vmalert:
image: victoriametrics/vmalert:v1.147.0
image: victoriametrics/vmalert:v1.148.0
depends_on:
- "vmauth"
ports:

View File

@@ -3,7 +3,7 @@ services:
# It scrapes targets defined in --promscrape.config
# And forward them to --remoteWrite.url
vmagent:
image: victoriametrics/vmagent:v1.147.0
image: victoriametrics/vmagent:v1.148.0
depends_on:
- "victoriametrics"
ports:
@@ -18,7 +18,7 @@ services:
# VictoriaMetrics instance, a single process responsible for
# storing metrics and serve read requests.
victoriametrics:
image: victoriametrics/victoria-metrics:v1.147.0
image: victoriametrics/victoria-metrics:v1.148.0
ports:
- 8428:8428
- 8089:8089
@@ -59,7 +59,7 @@ services:
# vmalert executes alerting and recording rules
vmalert:
image: victoriametrics/vmalert:v1.147.0
image: victoriametrics/vmalert:v1.148.0
depends_on:
- "victoriametrics"
- "alertmanager"

View File

@@ -10,7 +10,7 @@ groups:
concurrency: 2
rules:
- alert: PersistentQueueIsDroppingData
expr: sum(increase(vm_persistentqueue_bytes_dropped_total[5m])) without (path) > 0
expr: sum(increase(vm_persistentqueue_bytes_dropped_total[5m])) without (path,name) > 0
for: 10m
labels:
severity: critical
@@ -201,4 +201,4 @@ groups:
annotations:
summary: "Persistent Queue (url {{ $labels.url }}) of {{ $labels.instance }} (job:{{ $labels.job }}) will run out of space in 4 hours."
description: "RemoteWrite destination ({{ $labels.url }}) is unavailable or unable to receive data in a timely manner, so the persistent queue size is growing.
Once the available space is exhausted, some samples will be discarded and cause incident. Please check the health of remoteWrite destination ({{ $labels.url }})."
Once the available space is exhausted, some samples will be discarded and cause incident. Please check the health of remoteWrite destination ({{ $labels.url }})."

View File

@@ -1,6 +1,6 @@
services:
vmagent:
image: victoriametrics/vmagent:v1.147.0
image: victoriametrics/vmagent:v1.148.0
depends_on:
- "victoriametrics"
ports:
@@ -14,7 +14,7 @@ services:
restart: always
victoriametrics:
image: victoriametrics/victoria-metrics:v1.147.0
image: victoriametrics/victoria-metrics:v1.148.0
ports:
- 8428:8428
volumes:
@@ -40,7 +40,7 @@ services:
restart: always
vmalert:
image: victoriametrics/vmalert:v1.147.0
image: victoriametrics/vmalert:v1.148.0
depends_on:
- "victoriametrics"
ports:

View File

@@ -10,9 +10,9 @@ sitemap:
- To use *vmanomaly*, part of the enterprise package, a license key is required. Obtain your key [here](https://victoriametrics.com/products/enterprise/trial/) for this tutorial or for enterprise use.
- In the tutorial, we'll be using the following VictoriaMetrics components:
- [VictoriaMetrics Single-Node](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/) (v1.147.0)
- [vmalert](https://docs.victoriametrics.com/victoriametrics/vmalert/) (v1.147.0)
- [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/) (v1.147.0)
- [VictoriaMetrics Single-Node](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/) (v1.148.0)
- [vmalert](https://docs.victoriametrics.com/victoriametrics/vmalert/) (v1.148.0)
- [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/) (v1.148.0)
- [Grafana](https://grafana.com/) (v12.2.0)
- [Docker](https://docs.docker.com/get-docker/) and [Docker Compose](https://docs.docker.com/compose/)
- [Node exporter](https://github.com/prometheus/node_exporter#node-exporter) (v1.9.1) and [Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/) (v0.28.1)
@@ -323,7 +323,7 @@ Let's wrap it all up together into the `docker-compose.yml` file.
services:
vmagent:
container_name: vmagent
image: victoriametrics/vmagent:v1.147.0
image: victoriametrics/vmagent:v1.148.0
depends_on:
- "victoriametrics"
ports:
@@ -340,7 +340,7 @@ services:
victoriametrics:
container_name: victoriametrics
image: victoriametrics/victoria-metrics:v1.147.0
image: victoriametrics/victoria-metrics:v1.148.0
ports:
- 8428:8428
volumes:
@@ -373,7 +373,7 @@ services:
vmalert:
container_name: vmalert
image: victoriametrics/vmalert:v1.147.0
image: victoriametrics/vmalert:v1.148.0
depends_on:
- "victoriametrics"
ports:

View File

@@ -0,0 +1,426 @@
Several VictoriaMetrics components can connect to cloud storage to read or write object data.
The following table shows the supported types of storage for each component:
| Component | AWS S3 and S3-compatible | Google Cloud Storage | Azure Blob Storage |
|-----------|----|----------------------|--------------------|
| [vmbackup](https://docs.victoriametrics.com/victoriametrics/vmbackup/) | ✅ | ✅ | ✅ |
| [vmrestore](https://docs.victoriametrics.com/victoriametrics/vmrestore/) | ✅ | ✅ | ✅ |
| [vmbackupmanager](https://docs.victoriametrics.com/victoriametrics/vmbackupmanager/) | ✅ | ✅ | ✅ |
| [vmalert](https://docs.victoriametrics.com/victoriametrics/vmalert/) | ✅ | ✅ | ❌ |
All these components use the same underlying libraries, so the authentication setup is largely the same. The main difference is in the command-line flags:
- vmalert uses `-s3.*` prefixed flags (e.g., `-s3.credsFilePath`)
- backup and restore tools use unprefixed flags (e.g., `-credsFilePath`)
See the [component reference](https://docs.victoriametrics.com/guides/connecting-vm-components-to-cloud-storage/#per-component-flag-reference) for details.
## Obtaining credentials
You need to supply credentials so the component can connect to the cloud storage. The setup differs by provider; the sections below cover AWS S3, S3compatible systems, GCS, and Azure Blob Storage.
### AWS S3
1. In AWS, [create an IAM user](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html) or role with the minimum permissions for the target bucket (see table below).
1. [Create an access key](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html) for that IAM identity.
1. Copy the **Access key ID** and **Secret access key** values. You will use them in the credentials file or environment variables.
| Component | S3 usage | Minimum S3 permissions |
|-----------------|-------------------------------------------------------------|------------------------|
| vmalert (Enterprise) | Reads alerting/recording rules from bucket. | `s3:GetObject`, `s3:ListBucket` on the rules bucket/prefix.|
| vmrestore | Restores backups from S3 buckets | `s3:GetObject`, `s3:ListBucket` on the backup bucket/prefix.|
| vmbackup | Uploads backups to S3 and may delete old backup objects during maintenance.| `s3:PutObject`, `s3:GetObject`, `s3:ListBucket`, `s3:DeleteObject` on the backup bucket/prefix.|
| vmbackupmanager (Enterprise) | Automates backups using vmbackup behavior. | Same as vmbackup. |
### S3-compatible storage (MinIO, Ceph)
Generate access keys using your storage system's admin interface or CLI. The credentials follow the same format as AWS S3.
### Google Cloud Storage
1. Open the Google Cloud Console and go to **IAM & Admin > Service Accounts**.
1. Click **Create service account**, enter a name, and assign a Storage role (for example, Storage Object Admin).
1. Open the service account, go to **Keys**, then click **Add key > Create new key**.
1. Choose **JSON** as the key type and click **Create**.
1. Store the JSON file on the machine running the VictoriaMetrics component.
### Azure Blob Storage
> Azure does not use credential files.
1. Log in to the Azure Portal.
1. Search for and select **Storage accounts**.
1. Click on your specific storage account name. (This is your `AZURE_STORAGE_ACCOUNT_NAME`).
1. In the left menu under **Security + networking**, click **Access keys**.
1. Copy the key value from either **key1** or **key2** (this is your `AZURE_STORAGE_ACCOUNT_KEY`).
1. Define the access keys as environment variables.
```sh
export AZURE_STORAGE_ACCOUNT_NAME=mystorageaccount
export AZURE_STORAGE_ACCOUNT_KEY=myaccountkey
```
## Authenticating with the cloud provider
Provide the credentials as a file or with environment variables, along with the path to the cloud storage bucket. The syntax for the bucket name depends on the cloud provider:
- `s3://`: for AWS S3 and self-hosted S3-compatible storage (MinIO, Ceph)
- `gs://`: Google Cloud Storage
- `azblob://`: Azure Blob Storage
### vmbackup and vmrestore
The following example backups to an AWS S3 bucket using a credentials file:
```sh
vmbackup \
-storageDataPath=/data \
-snapshot.createURL=http://localhost:8428/snapshot/create \
-dst=s3://victoriametrics-backup/backup01 \
-credsFilePath=/etc/credentials
```
In order to restore from the same backup from AWS S3:
```sh
vmrestore \
-src=s3://victoriametrics-backup/backup01 \
-storageDataPath=/data \
-credsFilePath=/etc/credentials
```
> To use non-AWS S3 buckets such as MinIO or Ceph, you must [supply the `-customS3Endpoint` argument](https://docs.victoriametrics.com/guides/connecting-vm-components-to-cloud-storage/#s3-compatible).
Alternatively, you can set the access keys as environment variables instead of using a credential file:
```sh
export AWS_ACCESS_KEY_ID=YOUR_AWS_ACCESS_KEY
export AWS_SECRET_ACCESS_KEY=YOUR_SECRET_AWS_ACCESS_KEY
vmbackup \
-storageDataPath=/data \
-snapshot.createURL=http://localhost:8428/snapshot/create \
-dst=s3://victoriametrics-backup/backup01
vmrestore \
-src=s3://victoriametrics-backup/backup01 \
-storageDataPath=/data
```
Backups on Google Cloud Storage use the `gs://` prefix in the destination:
```sh
vmbackup \
-storageDataPath=/data \
-snapshot.createURL=http://localhost:8428/snapshot/create \
-dst=gs://victoriametrics-backup/backup01 \
-credsFilePath=/etc/credentials
```
You can restore this backup with:
```sh
vmrestore \
-src=gs://victoriametrics-backup/backup01 \
-storageDataPath=/data \
-credsFilePath=/etc/credentials
```
On Google Cloud, you can define the path to the JSON credential file with `GOOGLE_APPLICATION_CREDENTIALS`. For example:
```sh
export GOOGLE_APPLICATION_CREDENTIALS=/etc/credentials
vmbackup \
-storageDataPath=/data \
-snapshot.createURL=http://localhost:8428/snapshot/create \
-dst=gs://victoriametrics-backup/backup01
vmrestore \
-src=gs://victoriametrics-backup/backup01 \
-storageDataPath=/data
```
For Azure Blob Storage, use the `azblob://` prefix and rely on environment variables instead of `-credsFilePath`.
```sh
export AZURE_STORAGE_ACCOUNT_NAME=myaccount
export AZURE_STORAGE_ACCOUNT_KEY=mykey
vmbackup \
-storageDataPath=/data \
-snapshot.createURL=http://localhost:8428/snapshot/create \
-dst=azblob://victoriametrics-backup/backup01
vmrestore \
-src=azblob://victoriametrics-backup/backup01 \
-storageDataPath=/data
```
### vmbackupmanager
> vmbackupmanager only works in the [Enterprise](https://docs.victoriametrics.com/victoriametrics/enterprise/) edition.
To manage backups with vmbackupmanager on AWS S3, add the credentials with the `-credsFilePath` flag:
```sh
vmbackupmanager \
-dst=s3://vmstorage-data/backups \
-credsFilePath=/etc/credentials \
-storageDataPath=/vmstorage-data \
-snapshot.createURL=http://vmstorage:8482/snapshot/create \
-licenseFile=/etc/vm-license
```
Or define the access keys using environment variables:
```sh
export AWS_ACCESS_KEY_ID=YOUR_AWS_ACCESS_KEY
export AWS_SECRET_ACCESS_KEY=YOUR_SECRET_AWS_ACCESS_KEY
vmbackupmanager \
-dst=s3://vmstorage-data/backups \
-storageDataPath=/vmstorage-data \
-snapshot.createURL=http://vmstorage:8482/snapshot/create \
-licenseFile=/etc/vm-license
```
> To use non-AWS S3 buckets, you must [supply the `-customS3Endpoint` argument](#s3-compatible).
Automated backups on Google Cloud Storage take the following form:
```sh
vmbackupmanager \
-dst=gs://vmstorage-data/backups \
-credsFilePath=/etc/credentials \
-storageDataPath=/vmstorage-data \
-snapshot.createURL=http://vmstorage:8482/snapshot/create \
-licenseFile=/etc/vm-license
```
As with vmbackup and vmrestore, you can also define the path to the JSON credential file with `GOOGLE_APPLICATION_CREDENTIALS`:
```sh
export GOOGLE_APPLICATION_CREDENTIALS=/etc/credentials
vmbackupmanager \
-dst=gs://vmstorage-data/backups \
-storageDataPath=/vmstorage-data \
-snapshot.createURL=http://vmstorage:8482/snapshot/create \
-licenseFile=/etc/vm-license
```
vmbackupmanager can also use Azure Blob Storage by defining environment variables:
```sh
export AZURE_STORAGE_ACCOUNT_NAME=mystorageaccount
export AZURE_STORAGE_ACCOUNT_KEY=myaccountkey
vmbackupmanager \
-dst=azblob://vmstorage-data/backups \
-storageDataPath=/vmstorage-data \
-snapshot.createURL=http://vmstorage:8482/snapshot/create \
-licenseFile=/etc/vm-license
```
### vmalert
> - vmalert cloud storage command line flags are prefixed with `-s3.` for S3 buckets *and* Google Cloud Storage.
> - Cloud storage only works in the [Enterprise](https://docs.victoriametrics.com/victoriametrics/enterprise/) edition.
Read alerting rules from an S3 bucket. The `-rule` flag accepts a prefix, so it matches all files starting with `alerts_` in the `rules` folder:
```sh
vmalert \
-rule=s3://my-alert-bucket/rules/alerts_ \
-s3.credsFilePath=/etc/vmalert/aws-credentials \
-datasource.url=http://vmselect:8481/select/0/prometheus \
-notifier.url=http://alertmanager:9093 \
-licenseFile=/etc/vm-license
```
Instead of a credential file, you can supply the access keys using environment variables:
```sh
export AWS_ACCESS_KEY_ID=YOUR_AWS_ACCESS_KEY
export AWS_SECRET_ACCESS_KEY=YOUR_SECRET_AWS_ACCESS_KEY
vmalert \
-rule=s3://my-alert-bucket/rules/alerts_ \
-datasource.url=http://vmselect:8481/select/0/prometheus \
-notifier.url=http://alertmanager:9093 \
-licenseFile=/etc/vm-license
```
> To use non-AWS S3 buckets, you must [supply the `-s3.customEndpoint` argument](#s3-compatible).
To read rules from Google Cloud Storage:
```sh
vmalert \
-rule=gs://my-alert-bucket/rules/alerts_ \
-s3.credsFilePath=/etc/vmalert/gcp-service-account.json \
-datasource.url=http://vmselect:8481/select/0/prometheus \
-notifier.url=http://alertmanager:9093 \
-licenseFile=/etc/vm-license
```
If you prefer, you can supply the path to the JSON credential file with the `GOOGLE_APPLICATION_CREDENTIALS` environment variable:
```sh
export GOOGLE_APPLICATION_CREDENTIALS=/etc/credentials
vmalert \
-rule=gs://my-alert-bucket/rules/alerts_ \
-datasource.url=http://vmselect:8481/select/0/prometheus \
-notifier.url=http://alertmanager:9093 \
-licenseFile=/etc/vm-license
```
## Credentials files format
The file format depends on the storage provider.
### S3 credentials
The file uses the standard AWS shared credentials format used by the [AWS CLI](https://docs.aws.amazon.com/cli/v1/userguide/cli-configure-files.html) and [AWS SDKs](https://docs.aws.amazon.com/sdkref/latest/guide/file-format.html):
```ini
[default]
aws_access_key_id=YOUR_AWS_ACCESS_KEY
aws_secret_access_key=YOUR_AWS_SECRET_ACCESS_KEY
```
You can define multiple profiles in a single file:
```ini
[default]
aws_access_key_id=DEFAULT_ACCESS_KEY
aws_secret_access_key=DEFAULT_SECRET_KEY
[monitoring]
aws_access_key_id=MONITORING_ACCESS_KEY
aws_secret_access_key=MONITORING_SECRET_KEY
[alerts]
aws_access_key_id=ALERTS_ACCESS_KEY
aws_secret_access_key=ALERTS_SECRET_KEY
```
Use the `-configProfile` flag (or `-s3.configProfile` in vmalert) to select a non-default profile:
```sh
-configProfile=alerts
```
You can separate credentials from other configuration settings. Put credentials in one file:
```ini
[default]
aws_access_key_id=DEFAULT_ACCESS_KEY
aws_secret_access_key=DEFAULT_SECRET_KEY
```
And non-sensitive settings in another:
```ini
[default]
region=us-east-1
```
Then pass both files:
```sh
-configFilePath=/etc/aws-config \
-credsFilePath=/etc/credentials
```
### GCS credentials file format
The file is the JSON key downloaded from Google Cloud Console. Its content looks like this:
```json
{
"type": "service_account",
"project_id": "project-id",
"private_key_id": "key-id",
"private_key": "-----BEGIN PRIVATE KEY-----\nprivate-key\n-----END PRIVATE KEY-----\n",
"client_email": "service-account-email",
"client_id": "client-id",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://accounts.google.com/o/oauth2/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/service-account-email"
}
```
This is the standard service account key format defined by [Google Cloud IAM](https://developers.google.com/workspace/guides/create-credentials).
### Azure Blob Storage
Azure does not support credentials via file. Use environment variables instead.
```sh
export AZURE_STORAGE_ACCOUNT_NAME=mystorageaccount
export AZURE_STORAGE_ACCOUNT_KEY=myaccountkey
```
## Self-hosted and S3-compatible endpoints
For S3-compatible storage such as MinIO or Ceph, set a custom endpoint with the `-customS3Endpoint` flag for vmbackup, vmrestore, and vmbackupmanager. For example:
```sh
vmbackup \
-storageDataPath=/data \
-snapshot.createURL=http://localhost:8428/snapshot/create \
-dst=s3://victoriametrics-backup/backup01 \
-customS3Endpoint=http://minio.example.local:9000
```
On vmalert, use the `-s3.customEndpoint` flag instead:
```sh
vmalert \
-rule=s3://my-alert-bucket/rules/alerts_ \
-s3.customEndpoint=http://minio.example.local:9000 \
-s3.credsFilePath=/etc/vmalert/aws-credentials \
-datasource.url=http://vmselect:8481/select/0/prometheus \
-notifier.url=http://alertmanager:9093 \
-licenseFile=/etc/vm-license
```
### Addressing S3-compatible buckets {#s3-compatible}
When connecting to non-AWS S3-compatible buckets, there is an additional flag you might need to configure:
- `-s3ForcePathStyle`: on vmbackupmanager, vmbackup, and vmrestore.
- `-s3.forcePathStyle`: on vmalert.
The flag changes the expected URL pattern for a bucket.
| Flag value | Address-style | Example | Use with |
|------------|---------------|---------|----------|
| `true` (default) | Path-style | `https://endpoint/bucket/key` | MinIO, Ceph, most S3-compatible storages |
| `false` | Virtual host-style | `https://bucket.endpoint/key` | [Aliyun OSS](https://www.aliyun.com/product/oss) and other endpoints that require it |
> The flag only takes effect when you use a custom endpoint (`-customS3Endpoint` or `-s3.customEndpoint` on vmalert). When connecting to real AWS S3, the SDK handles addressing automatically.
## Per-component flag reference
The table below shows how the same concept maps to different flag names across components.
| Concept | vmalert | vmbackup, vmrestore, and vmbackupmanager |
|---|---|---|---|---|
| Credentials file | `-s3.credsFilePath` | `-credsFilePath` |
| Config file | `-s3.configFilePath` | `-configFilePath` |
| Profile selection | `-s3.configProfile` | `-configProfile` |
| Custom endpoint | `-s3.customEndpoint` | `-customS3Endpoint` |
| Force path style | `-s3.forcePathStyle` | `-s3ForcePathStyle` |
| TLS insecure | N/A | `-s3TLSInsecureSkipVerify` |
| Storage class | N/A | `-s3StorageClass` |

View File

@@ -0,0 +1,12 @@
---
weight: 5
title: Connecting VictoriaMetrics components to cloud storage
menu:
docs:
parent: guides
weight: 5
tags:
- metrics
- guide
---
{{% content "README.md" %}}

View File

@@ -240,23 +240,23 @@ vmagent will write data into VictoriaMetrics single-node and cluster (with tenan
# compose.yaml
services:
vmsingle:
image: victoriametrics/victoria-metrics:v1.147.0
image: victoriametrics/victoria-metrics:v1.148.0
vmstorage:
image: victoriametrics/vmstorage:v1.147.0-cluster
image: victoriametrics/vmstorage:v1.148.0-cluster
vminsert:
image: victoriametrics/vminsert:v1.147.0-cluster
image: victoriametrics/vminsert:v1.148.0-cluster
command:
- -storageNode=vmstorage:8400
vmselect:
image: victoriametrics/vmselect:v1.147.0-cluster
image: victoriametrics/vmselect:v1.148.0-cluster
command:
- -storageNode=vmstorage:8401
vmagent:
image: victoriametrics/vmagent:v1.147.0
image: victoriametrics/vmagent:v1.148.0
volumes:
- ./scrape.yaml:/etc/vmagent/config.yaml
command:
@@ -308,7 +308,7 @@ Now add the vmauth service to `compose.yaml`:
# compose.yaml
services:
vmauth:
image: docker.io/victoriametrics/vmauth:v1.147.0
image: docker.io/victoriametrics/vmauth:v1.148.0
ports:
- 8427:8427
volumes:

View File

@@ -155,15 +155,15 @@ These services will store and query the metrics scraped by vmagent.
# compose.yaml
services:
vmstorage:
image: victoriametrics/vmstorage:v1.147.0-cluster
image: victoriametrics/vmstorage:v1.148.0-cluster
vminsert:
image: victoriametrics/vminsert:v1.147.0-cluster
image: victoriametrics/vminsert:v1.148.0-cluster
command:
- -storageNode=vmstorage:8400
vmselect:
image: victoriametrics/vmselect:v1.147.0-cluster
image: victoriametrics/vmselect:v1.148.0-cluster
command:
- -storageNode=vmstorage:8401
ports:
@@ -196,7 +196,7 @@ Add the vmauth service to `compose.yaml`:
# compose.yaml
services:
vmauth:
image: victoriametrics/vmauth:v1.147.0-enterprise
image: victoriametrics/vmauth:v1.148.0-enterprise
ports:
- 8427:8427
volumes:
@@ -251,7 +251,7 @@ Add the vmagent service to `compose.yaml` with OAuth2 configuration:
# compose.yaml
services:
vmagent:
image: victoriametrics/vmagent:v1.147.0
image: victoriametrics/vmagent:v1.148.0
volumes:
- ./scrape.yaml:/etc/vmagent/config.yaml
command:

View File

@@ -107,7 +107,7 @@ The final piece is the Docker Compose file. This ties all the services together
# compose.yml
services:
victoriametrics:
image: victoriametrics/victoria-metrics:v1.147.0
image: victoriametrics/victoria-metrics:v1.148.0
command:
- "--storageDataPath=/victoria-metrics-data"
- "--selfScrapeInterval=10s"
@@ -128,7 +128,7 @@ services:
- ./alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro
vmalert:
image: victoriametrics/vmalert:v1.147.0
image: victoriametrics/vmalert:v1.148.0
depends_on:
- victoriametrics
- alertmanager

View File

@@ -27,5 +27,5 @@ to [the latest available releases](https://docs.victoriametrics.com/victoriametr
## Currently supported LTS release lines
- v1.136.x - the latest one is [v1.136.4 LTS release](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.136.4)
- v1.122.x - the latest one is [v1.122.19 LTS release](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.122.19)
- v1.148.x - the latest one is [v1.148.0 LTS release](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.148.0)
- v1.136.x - the latest one is [v1.136.14 LTS release](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.136.14)

View File

@@ -53,8 +53,8 @@ and unpack it. It contains a single `victoria-metrics-prod` binary.
For example, on Linux with `amd64` architecture:
```sh
wget https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.147.0/victoria-metrics-linux-amd64-v1.147.0.tar.gz
tar xzf victoria-metrics-linux-amd64-v1.147.0.tar.gz
wget https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.148.0/victoria-metrics-linux-amd64-v1.148.0.tar.gz
tar xzf victoria-metrics-linux-amd64-v1.148.0.tar.gz
```
The binary is self-contained and requires no installation - it is ready to run as is.
@@ -229,9 +229,9 @@ Download the newest available [VictoriaMetrics release](https://docs.victoriamet
from [DockerHub](https://hub.docker.com/r/victoriametrics/victoria-metrics) or [Quay](https://quay.io/repository/victoriametrics/victoria-metrics?tab=tags):
```sh
docker pull victoriametrics/victoria-metrics:v1.147.0
docker pull victoriametrics/victoria-metrics:v1.148.0
docker run -it --rm -v `pwd`/victoria-metrics-data:/victoria-metrics-data -p 8428:8428 \
victoriametrics/victoria-metrics:v1.147.0 --selfScrapeInterval=5s -storageDataPath=victoria-metrics-data
victoriametrics/victoria-metrics:v1.148.0 --selfScrapeInterval=5s -storageDataPath=victoria-metrics-data
```
_For Enterprise images, see [this link](https://docs.victoriametrics.com/victoriametrics/enterprise/#docker-images)._

View File

@@ -26,9 +26,11 @@ See also [LTS releases](https://docs.victoriametrics.com/victoriametrics/lts-rel
## tip
* FEATURE: [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/): add `name` label identifying the corresponding `-remoteWrite.url` target to the `vm_persistentqueue_*` metrics exposed by persistent queue. See [#7944](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7944). Thanks to @tIGO for contribution.
## [v1.148.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.148.0)
Release candidate
Released at 2026-07-20
* SECURITY: upgrade Go builder from Go1.26.4 to Go1.26.5. See [the list of issues addressed in Go1.26.5](https://github.com/golang/go/issues?q=milestone%3AGo1.26.5%20label%3ACherryPickApproved).
@@ -36,6 +38,7 @@ Release candidate
* FEATURE: [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/) and [vmsingle](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/): support scraping metrics over Unix domain sockets. The socket path can be configured via the `__unix_socket__` target label. See [#11156](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/11156). Thanks to @vinyas-bharadwaj for contribution.
* FEATURE: [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/) and [vmsingle](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/): Improve background discovery performance for [http_sd](https://docs.victoriametrics.com/victoriametrics/sd_configs/#http_sd_configs) discovery. See [#8838](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8838).
* FEATURE: [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/) and [vmsingle](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/): allow overriding `max_scrape_size` on a per-target basis via the `__max_scrape_size__` label during target relabeling. See [#11188](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/11188).
* FEATURE: [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/) and [vmalert](https://docs.victoriametrics.com/victoriametrics/vmalert/): add client side least-loaded load-balancing with `DNS` discovery. See [#2388](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2388) and these [vmagent DNS URLs](https://docs.victoriametrics.com/victoriametrics/vmagent/#dns-urls), [vmalert DNS URLs](https://docs.victoriametrics.com/victoriametrics/vmalert/#dns-urls).
* FEATURE: [vmstorage](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/) and [vmsingle](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/): add `-maxBackfillAge` command-line flag for limiting ingestion of samples with historical timestamps, for example, when older data has been moved between storage tiers (nvme/hdd, hot/cold). See [#11199](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/11199). Thanks to @AshwinRamaniPsg for contribution.
* FEATURE: [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/): automatically preload relabeling rules configured via `-remoteWrite.relabelConfig` and `-remoteWrite.urlRelabelConfig` in the [metrics relabel debug UI](https://docs.victoriametrics.com/victoriametrics/relabeling/#relabel-debugging). See [#9918](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/9918).
@@ -347,6 +350,27 @@ It enables back `Discovered targets` debug UI by default.
* BUGFIX: `vmstorage` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/): properly apply `extra_filters[]` filter when querying `vm_account_id` or `vm_project_id` labels via [multitenant](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#multitenancy) request for `/api/v1/label/…/values` API. Before, `extra_filters` was ignored. See [#10503](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/10503).
* BUGFIX: [vmsingle](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/) and `vmselect` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/): revert the use of rollup result cache for [instant queries](https://docs.victoriametrics.com/keyConcepts.html#instant-query) that contain [`rate`](https://docs.victoriametrics.com/MetricsQL.html#rate) function with a lookbehind window larger than `-search.minWindowForInstantRollupOptimization`. The cache usage was removed since [v1.132.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.132.0). See [#10098](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/10098#issuecomment-3895011084) for more details.
## [v1.136.14](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.136.14)
Released at 2026-07-17
**v1.136.x is a line of [LTS releases](https://docs.victoriametrics.com/victoriametrics/lts-releases/). It contains important up-to-date bugfixes for [VictoriaMetrics enterprise](https://docs.victoriametrics.com/victoriametrics/enterprise/).
All these fixes are also included in [the latest community release](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest).
The v1.136.x line will be supported for at least 12 months since [v1.136.0](https://docs.victoriametrics.com/victoriametrics/changelog/#v11360) release**
* SECURITY: upgrade Go builder from Go1.26.4 to Go1.26.5. See [the list of issues addressed in Go1.26.5](https://github.com/golang/go/issues?q=milestone%3AGo1.26.5%20label%3ACherryPickApproved).
* BUGFIX: [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/): flush pending persistent queue data to chunk file before updating the metadata. This prevents the metadata writer offset from getting ahead of the chunk file size and avoids losing the persistent queue after an unclean shutdown. See [#11192](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/11192).
* BUGFIX: [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/): fix a possible data race when processing OpenTelemetry metadata. See [#11238](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/11238). Thanks to @nevgeny for contribution.
* BUGFIX: [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/): atomically write persistent queue metainfo to prevent possible file corruption on ungraceful shutdown. See [#11192](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/11192).
* BUGFIX: [vmbackup](https://docs.victoriametrics.com/victoriametrics/vmbackup/) and [vmbackupmanager](https://docs.victoriametrics.com/victoriametrics/vmbackupmanager/): retry S3 requests failing with `HTTP 429` status code or `TooManyRequests` error code. Previously such requests were not retried, so a short burst of rate limiting would fail the whole backup. See [#11218](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/11218). Thanks to @gautamrizwani for contribution.
* BUGFIX: `vmselect` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/): properly apply limit to metrics metadata response. See [#11139](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/11139).
* BUGFIX: `vminsert` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/): Now drops metadata blocks when communicating with vmstorage nodes over the legacy RPC protocol. To avoid this limitation, upgrade `vmstorage` to a version that supports the new RPC protocol (>= [v1.137.0](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/docs/victoriametrics/changelog/CHANGELOG.md#v11370)). See [#11146](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/11146).
* BUGFIX: [vmui](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#vmui): keep only one header navigation dropdown (`Explore`, `Tools`) open at a time. Previously, hovering across two dropdowns could briefly leave both open due to the close delay. See [#11224](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/11224). Thanks to @antedotee for contribution.
* BUGFIX: [vmui](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#vmui): preserve newline formatting in alert and rule annotations on the Alerting page. See [#11171](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/11171).
* BUGFIX: [vmui](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#vmui): hide `Total metric names` stats on [Cardinality Explorer](https://docs.victoriametrics.com/victoriametrics/#cardinality-explorer) page when user selects a specific metric or label to focus. See [#11154](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/11154) for details. Thanks to @lghuy05 for the contribution.
* BUGFIX: [vmauth](https://docs.victoriametrics.com/victoriametrics/vmauth/): return `408 Request Timeout` instead of `400 Bad Request` when the request body isn't received within `-maxQueueDuration`. This prevents `vmagent` from incorrectly downgrading the remote write protocol and dropping data when `vmauth` is used as a proxy for a remote write endpoint. See [#11272](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/11272).
## [v1.136.13](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.136.13)
Released at 2026-07-03
@@ -745,6 +769,18 @@ See changes [here](https://docs.victoriametrics.com/victoriametrics/changelog/ch
See changes [here](https://docs.victoriametrics.com/victoriametrics/changelog/changelog_2025/#v11230)
## [v1.122.27](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.122.27)
Released at 2026-07-17
**v1.122.x is a line of [LTS releases](https://docs.victoriametrics.com/victoriametrics/lts-releases/). It contains important up-to-date bugfixes for [VictoriaMetrics enterprise](https://docs.victoriametrics.com/victoriametrics/enterprise/).
All these fixes are also included in [the latest community release](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest).
The v1.122.x line will be supported for at least 12 months since [v1.122.0](https://docs.victoriametrics.com/victoriametrics/changelog/#v11220) release**
* BUGFIX: [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/): flush pending persistent queue data to chunk file before updating the metadata. This prevents the metadata writer offset from getting ahead of the chunk file size and avoids losing the persistent queue after an unclean shutdown. See [#11192](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/11192).
* BUGFIX: [vmbackup](https://docs.victoriametrics.com/victoriametrics/vmbackup/) and [vmbackupmanager](https://docs.victoriametrics.com/victoriametrics/vmbackupmanager/): retry S3 requests failing with `HTTP 429` status code or `TooManyRequests` error code. Previously such requests were not retried, so a short burst of rate limiting would fail the whole backup. See [#11218](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/11218). Thanks to @gautamrizwani for contribution.
* BUGFIX: [vmui](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#vmui): hide `Total metric names` stats on [Cardinality Explorer](https://docs.victoriametrics.com/victoriametrics/#cardinality-explorer) page when user selects a specific metric or label to focus. See [#11154](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/11154) for details. Thanks to @lghuy05 for the contribution.
## [v1.122.26](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.122.26)
Released at 2026-07-03

View File

@@ -121,7 +121,7 @@ It is allowed to run Enterprise components in [cases listed here](https://docs.v
Binary releases of Enterprise components are available at [the releases page for VictoriaMetrics](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest),
[the releases page for VictoriaLogs](https://github.com/VictoriaMetrics/VictoriaLogs/releases/latest)
and [the releases page for VictoriaTraces](https://github.com/VictoriaMetrics/VictoriaTraces/releases/latest).
Enterprise binaries and packages have `enterprise` suffix in their names. For example, `victoria-metrics-linux-amd64-v1.147.0-enterprise.tar.gz`.
Enterprise binaries and packages have `enterprise` suffix in their names. For example, `victoria-metrics-linux-amd64-v1.148.0-enterprise.tar.gz`.
In order to run binary release of Enterprise component, please download the `*-enterprise.tar.gz` archive for your OS and architecture
from the corresponding releases page and unpack it. Then run the unpacked binary.
@@ -139,8 +139,8 @@ For example, the following command runs VictoriaMetrics Enterprise binary with t
obtained at [this page](https://victoriametrics.com/products/enterprise/trial/):
```sh
wget https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.147.0/victoria-metrics-linux-amd64-v1.147.0-enterprise.tar.gz
tar -xzf victoria-metrics-linux-amd64-v1.147.0-enterprise.tar.gz
wget https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.148.0/victoria-metrics-linux-amd64-v1.148.0-enterprise.tar.gz
tar -xzf victoria-metrics-linux-amd64-v1.148.0-enterprise.tar.gz
./victoria-metrics-prod -license=BASE64_ENCODED_LICENSE_KEY
```
@@ -155,7 +155,7 @@ Alternatively, VictoriaMetrics Enterprise license can be stored in the file and
It is allowed to run Enterprise components in [cases listed here](https://docs.victoriametrics.com/victoriametrics/enterprise/#valid-cases-for-victoriametrics-enterprise).
Docker images for Enterprise components are available at [VictoriaMetrics Docker Hub](https://hub.docker.com/u/victoriametrics) and [VictoriaMetrics Quay](https://quay.io/organization/victoriametrics).
Enterprise docker images have `enterprise` suffix in their names. For example, `victoriametrics/victoria-metrics:v1.147.0-enterprise`.
Enterprise docker images have `enterprise` suffix in their names. For example, `victoriametrics/victoria-metrics:v1.148.0-enterprise`.
In order to run Docker image of VictoriaMetrics Enterprise component, it is required to provide the license key via the command-line
flag as described in the [binary-releases](https://docs.victoriametrics.com/victoriametrics/enterprise/#binary-releases) section.
@@ -165,13 +165,13 @@ Enterprise license key can be obtained at [this page](https://victoriametrics.co
For example, the following command runs VictoriaMetrics Enterprise Docker image with the specified license key:
```sh
docker run --name=victoria-metrics victoriametrics/victoria-metrics:v1.147.0-enterprise -license=BASE64_ENCODED_LICENSE_KEY
docker run --name=victoria-metrics victoriametrics/victoria-metrics:v1.148.0-enterprise -license=BASE64_ENCODED_LICENSE_KEY
```
Alternatively, the license code can be stored in the file and then referred via `-licenseFile` command-line flag:
```sh
docker run --name=victoria-metrics -v /vm-license:/vm-license victoriametrics/victoria-metrics:v1.147.0-enterprise -licenseFile=/path/to/vm-license
docker run --name=victoria-metrics -v /vm-license:/vm-license victoriametrics/victoria-metrics:v1.148.0-enterprise -licenseFile=/path/to/vm-license
```
Example docker-compose configuration:
@@ -181,7 +181,7 @@ version: "3.5"
services:
victoriametrics:
container_name: victoriametrics
image: victoriametrics/victoria-metrics:v1.147.0
image: victoriametrics/victoria-metrics:v1.148.0
ports:
- 8428:8428
volumes:
@@ -213,7 +213,7 @@ is used to provide the license key in plain-text:
```yaml
server:
image:
tag: v1.147.0-enterprise
tag: v1.148.0-enterprise
license:
key: {BASE64_ENCODED_LICENSE_KEY}
@@ -224,7 +224,7 @@ In order to provide the license key via existing secret, the following values fi
```yaml
server:
image:
tag: v1.147.0-enterprise
tag: v1.148.0-enterprise
license:
secret:
@@ -274,7 +274,7 @@ spec:
license:
key: {BASE64_ENCODED_LICENSE_KEY}
image:
tag: v1.147.0-enterprise
tag: v1.148.0-enterprise
```
In order to provide the license key via an existing secret, the following custom resource is used:
@@ -291,7 +291,7 @@ spec:
name: vm-license
key: license
image:
tag: v1.147.0-enterprise
tag: v1.148.0-enterprise
```
Example secret with license key:
@@ -342,7 +342,7 @@ Builds are available for amd64 and arm64 architectures.
Example archive:
`victoria-metrics-linux-amd64-v1.147.0-enterprise.tar.gz`
`victoria-metrics-linux-amd64-v1.148.0-enterprise.tar.gz`
Includes:
@@ -351,7 +351,7 @@ Includes:
Example Docker image:
`victoriametrics/victoria-metrics:v1.147.0-enterprise-fips` uses the FIPS-compatible binary and based on `scratch` image.
`victoriametrics/victoria-metrics:v1.148.0-enterprise-fips` uses the FIPS-compatible binary and based on `scratch` image.
## What Happens to Licensed Components When a License Expires

View File

@@ -35,8 +35,8 @@ scrape_configs:
After you created the `scrape.yaml` file, download and unpack [single-node VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/) to the same directory:
```sh
wget https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.147.0/victoria-metrics-linux-amd64-v1.147.0.tar.gz
tar xzf victoria-metrics-linux-amd64-v1.147.0.tar.gz
wget https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.148.0/victoria-metrics-linux-amd64-v1.148.0.tar.gz
tar xzf victoria-metrics-linux-amd64-v1.148.0.tar.gz
```
Then start VictoriaMetrics and instruct it to scrape targets defined in `scrape.yaml` and save scraped metrics
@@ -150,8 +150,8 @@ Then start [single-node VictoriaMetrics](https://docs.victoriametrics.com/victor
```yaml
# Download and unpack single-node VictoriaMetrics
wget https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.147.0/victoria-metrics-linux-amd64-v1.147.0.tar.gz
tar xzf victoria-metrics-linux-amd64-v1.147.0.tar.gz
wget https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.148.0/victoria-metrics-linux-amd64-v1.148.0.tar.gz
tar xzf victoria-metrics-linux-amd64-v1.148.0.tar.gz
# Run single-node VictoriaMetrics with the given scrape.yaml
./victoria-metrics-prod -promscrape.config=scrape.yaml

View File

@@ -188,6 +188,9 @@ See the docs at https://docs.victoriametrics.com/victoriametrics/
Timezone to use for timestamps in logs. Timezone must be a valid IANA Time Zone. For example: America/New_York, Europe/Berlin, Etc/GMT+3 or Local (default "UTC")
-loggerWarnsPerSecondLimit int
Per-second limit on the number of WARN messages. If more than the given number of warns are emitted per second, then the remaining warns are suppressed. Zero values disable the rate limit
-maxBackfillAge value
The maximum allowed age for the ingested samples with historical timestamps. Samples with timestamps older than now-maxBackfillAge are rejected during data ingestion. By default, or when set to 0, -maxBackfillAge equals to -retentionPeriod, e.g. it is unlimited within the configured retention. This can be useful for limiting ingestion of historical samples, for example, when older data has been moved to another storage tier. See https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#retention
The following optional suffixes are supported: s (second), h (hour), d (day), w (week), M (month), y (year). If suffix isn't set, then the duration is counted in months (default 0)
-maxConcurrentInserts int
The maximum number of concurrent insert requests. Set higher value when clients send data over slow networks. Default value depends on the number of available CPU cores. It should work fine in most cases since it minimizes resource usage. See also -insert.maxQueueDuration (default 2*cgroup.AvailableCPUs())
-maxIngestionRate int

View File

@@ -466,6 +466,43 @@ and `-remoteWrite.streamAggr.config`:
There is also the `-promscrape.configCheckInterval` command-line flag, which can be used to automatically reload configs from the updated `-promscrape.config` file.
## DNS URLs
If `vmagent` encounters URLs with the `dns+` prefix in the hostname (such as `http://dns+some-addr:8428/some/path`), it resolves `some-addr` into IP addresses
via [DNS A records](https://datatracker.ietf.org/doc/html/rfc1035#section-3.4.1). The port from the original URL is appended to each discovered IP address.
Each discovered IP address is used for least-loaded balancing of write requests.
DNS URLs are supported in the following places:
* In `-remoteWrite.url` command-line flag. For example, if `victoria-metrics` [DNS A Record](https://datatracker.ietf.org/doc/html/rfc1035#section-3.4.1) record contains
`192.168.1.15` IP address, then `-remoteWrite.url=http://dns+victoria-metrics:8428/api/v1/write` is automatically resolved into
`-remoteWrite.url=http://192.168.1.15:8428/api/v1/write`.
DNS URLs are useful when client-side HTTP load balancing is needed. A good example
is a [Kubernetes headless Service](https://kubernetes.io/docs/concepts/services-networking/service/#headless-services),
which returns multiple IP addresses for a single hostname.
### DNS URLs and HTTPS
When a `dns+` URL uses the `https` scheme, `vmagent` connects to the discovered
IP addresses directly. This affects [TLS](https://en.wikipedia.org/wiki/Transport_Layer_Security)
in two ways:
* No [SNI](https://en.wikipedia.org/wiki/Server_Name_Indication) is sent in the TLS handshake,
since the connection target is an IP address rather than a hostname.
* The server certificate is verified against the IP address, so the verification fails
unless the certificate contains the corresponding
[IP SAN](https://en.wikipedia.org/wiki/Subject_Alternative_Name) entries.
To use `dns+` URLs with HTTPS, pass the original hostname via the `-remoteWrite.tlsServerName`
command-line flag. It is used both as SNI and as the name the server certificate
is verified against:
```sh
-remoteWrite.url=https://dns+victoria-metrics:8428/api/v1/write
-remoteWrite.tlsServerName=victoria-metrics
```
## SRV URLs
If `vmagent` encounters URLs with `srv+` prefix in hostname (such as `http://srv+some-addr/some/path`), then it resolves `some-addr` [DNS SRV](https://en.wikipedia.org/wiki/SRV_record)
@@ -476,7 +513,7 @@ SRV URLs are supported in the following places:
* In `-remoteWrite.url` command-line flag. For example, if `victoria-metrics` [DNS SRV](https://en.wikipedia.org/wiki/SRV_record) record contains
`victoria-metrics-host:8428` TCP address, then `-remoteWrite.url=http://srv+victoria-metrics/api/v1/write` is automatically resolved into
`-remoteWrite.url=http://victoria-metrics-host:8428/api/v1/write`. If the DNS SRV record is resolved into multiple TCP addresses, then `vmagent`
uses a randomly chosen address for each connection it establishes to the remote storage.
performs per request least-loaded load-balancing.
* In scrape target addresses aka `__address__` label. See [these docs](https://docs.victoriametrics.com/victoriametrics/relabeling/#how-to-modify-scrape-urls-in-targets) for details.

View File

@@ -546,7 +546,7 @@ tags at [Docker Hub](https://hub.docker.com/r/victoriametrics/vmalert/tags) and
## Reading rules from object storage
[Enterprise version](https://docs.victoriametrics.com/victoriametrics/enterprise/) of `vmalert` may read alerting and recording rules
The [Enterprise version](https://docs.victoriametrics.com/victoriametrics/enterprise/) of `vmalert` may read alerting and recording rules
from object storage:
* `./bin/vmalert -rule=s3://bucket/dir/alert.rules` would read rules from the given path at S3 bucket
@@ -563,6 +563,48 @@ The following [command-line flags](#flags) can be used for fine-tuning access to
* `-s3.customEndpoint` - custom S3 endpoint for use with S3-compatible storages (e.g. MinIO). S3 is used if not set.
* `-s3.forcePathStyle` - prefixing endpoint with bucket name when set false, true by default.
### S3 (AWS and S3-compatible)
The following example reads rules from an S3 bucket:
```sh
./bin/vmalert \
-rule=s3://my-alert-bucket/rules/alerts_ \
-s3.credsFilePath=/etc/vmalert/aws-credentials \
-datasource.url=http://vmselect:8481/select/0/prometheus \
-notifier.url=http://alertmanager:9093 \
-licenseFile=/etc/vm-license
```
For S3-compatible backends such as [MinIO](https://www.min.io/) or [Ceph](https://ceph.io/), add `-s3.customEndpoint`:
```sh
./bin/vmalert \
-rule=s3://victoriametrics-alert-rules/rules_ \
-s3.credsFilePath=/etc/vmalert/aws-credentials \
-s3.customEndpoint=http://minio.example.local:9000 \
-datasource.url=http://vmselect:8481/select/0/prometheus \
-notifier.url=http://alertmanager:9093 \
-licenseFile=/etc/vm-license
```
See [Connecting VM components to cloud storage](https://docs.victoriametrics.com/guides/connecting-vm-components-to-cloud-storage/) for details on creating IAM users, credentials file format, environment variables, and IAM roles.
### Google Cloud Storage (GCS)
The following example reads rules from a GCS bucket:
```sh
./bin/vmalert \
-rule=gs://my-alert-bucket/rules/alerts_ \
-s3.credsFilePath=/etc/vmalert/gcp-service-account.json \
-datasource.url=http://vmselect:8481/select/0/prometheus \
-notifier.url=http://alertmanager:9093 \
-licenseFile=/etc/vm-license
```
See [Connecting VM components to cloud storage](https://docs.victoriametrics.com/guides/connecting-vm-components-to-cloud-storage/) for details on creating service accounts and downloading JSON keys.
## Topology examples
The following sections are showing how `vmalert` may be used and configured
@@ -1470,6 +1512,59 @@ alert_relabel_configs:
The configuration file can be [hot-reloaded](#hot-config-reload).
## DNS URLs
If `vmalert` encounters URLs with the `dns+` prefix in the hostname (such as `http://dns+some-addr:8428/some/path`), it resolves `some-addr` into IP addresses via DNS A/AAAA records.
The port from the original URL is appended to each discovered IP address.
Each discovered IP address is used for least-loaded balancing of write requests.
DNS URLs are supported in the following places:
* In `-remoteWrite.url`, `-remoteRead.url` and `-datasource.url` command-line flags. For example, if `victoria-metrics` [DNS A Record](https://datatracker.ietf.org/doc/html/rfc1035#section-3.4.1) record contains
`192.168.1.15` IP address, then `-remoteWrite.url=http://dns+victoria-metrics:8428` is automatically resolved into
`-remoteWrite.url=http://192.168.1.15:8428`.
DNS URLs are useful when client-side HTTP load balancing is needed. A good example
is a [Kubernetes headless Service](https://kubernetes.io/docs/concepts/services-networking/service/#headless-services),
which returns multiple IP addresses for a single hostname.
### DNS URLs and HTTPS
When a `dns+` URL uses the `https` scheme, `vmalert` connects to the discovered
IP addresses directly. No [SNI](https://en.wikipedia.org/wiki/Server_Name_Indication)
is sent in the TLS handshake, and the server certificate is verified against the IP address,
which fails unless the certificate contains the corresponding
[IP SAN](https://en.wikipedia.org/wiki/Subject_Alternative_Name) entries.
To use `dns+` URLs with HTTPS, pass the original hostname via the corresponding
`tlsServerName` command-line flag - `-datasource.tlsServerName`, `-remoteRead.tlsServerName`
or `-remoteWrite.tlsServerName`. It is used both as SNI and as the name the server
certificate is verified against:
```sh
-datasource.url=https://dns+victoria-metrics:8428
-datasource.tlsServerName=victoria-metrics
```
Alternatively, issue server certificates with IP SAN entries for every backend IP address.
Avoid `tlsInsecureSkipVerify` flags for working around this, since they disable
server certificate verification completely.
## SRV URLs
If `vmalert` encounters URLs with `srv+` prefix in hostname (such as `http://srv+some-addr/some/path`), then it resolves `some-addr` [DNS SRV](https://en.wikipedia.org/wiki/SRV_record)
record into TCP address with hostname and TCP port, and then use the resulting URL when it needs to connect to it.
SRV URLs are supported in the following places:
* In `-remoteWrite.url`, `-remoteRead.url` and `-datasource.url` command-line flags. For example, if `victoria-metrics` [DNS SRV](https://en.wikipedia.org/wiki/SRV_record) record contains
`victoria-metrics-host:8085`, then `-remoteWrite.url=http://srv+victoria-metrics:8428` is automatically resolved into
`-remoteWrite.url=http://victoria-metrics-host:8085`. If the DNS SRV record is resolved into multiple TCP addresses, then `vmalert`
performs per request round-robin load-balancing.
SRV URLs are useful when HTTP services run on different TCP ports or when their TCP ports can change over time (for instance, after a restart).
## Contributing
`vmalert` is mostly designed and built by VictoriaMetrics community.

View File

@@ -204,61 +204,45 @@ See [this article](https://medium.com/@valyala/speeding-up-backups-for-big-time-
### Providing credentials as a file
Obtaining credentials from a file.
See [Connecting VM components to cloud storage](https://docs.victoriametrics.com/guides/connecting-vm-components-to-cloud-storage/) for instructions on obtaining credentials and providing them via credential files, environment variables, cloud provider metadata service, or Kubernetes secrets and IAM roles.
Add flag `-credsFilePath=/etc/credentials` with the following content:
The following examples show the most common authentication patterns for each storage provider.
* for S3 (AWS, MinIO or other S3 compatible storages):
#### S3 (AWS and S3-compatible)
```sh
[default]
aws_access_key_id=theaccesskey
aws_secret_access_key=thesecretaccesskeyvalue
```
```sh
vmbackup \
-storageDataPath=/data \
-snapshot.createURL=http://localhost:8428/snapshot/create \
-dst=s3://victoriametrics-backup/backup01 \
-credsFilePath=/etc/credentials
```
* for GCP cloud storage:
#### Google Cloud Storage (GCS)
```json
{
"type": "service_account",
"project_id": "project-id",
"private_key_id": "key-id",
"private_key": "-----BEGIN PRIVATE KEY-----\nprivate-key\n-----END PRIVATE KEY-----\n",
"client_email": "service-account-email",
"client_id": "client-id",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://accounts.google.com/o/oauth2/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/service-account-email"
}
```
```sh
vmbackup \
-storageDataPath=/data \
-snapshot.createURL=http://localhost:8428/snapshot/create \
-dst=gs://victoriametrics-backup/backup01 \
-credsFilePath=/etc/credentials
```
### Providing credentials via env variables
#### Azure Blob Storage
Obtaining credentials from env variables.
```sh
export AZURE_STORAGE_ACCOUNT_NAME=mystorageaccount
export AZURE_STORAGE_ACCOUNT_KEY=myaccountkey
* For AWS S3 compatible storages set env variable `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`.
Also you can set env variable `AWS_SHARED_CREDENTIALS_FILE` with path to credentials file.
* For GCE cloud storage set env variable `GOOGLE_APPLICATION_CREDENTIALS` with path to credentials file.
* For Azure storage use one of these env variables:
* `AZURE_STORAGE_ACCOUNT_CONNECTION_STRING`: use a connection string (must be either SAS Token or Account/Key)
* `AZURE_STORAGE_ACCOUNT_NAME` and `AZURE_STORAGE_ACCOUNT_KEY`: use a specific account name and key (either primary or secondary)
* `AZURE_USE_DEFAULT_CREDENTIAL` and `AZURE_STORAGE_ACCOUNT_NAME`: use the `DefaultAzureCredential` to allow the Azure library
to search for multiple options (for example, managed identity related variables). Note that if multiple credentials are available,
it is required to specify the `AZURE_CLIENT_ID` to select specific credentials.
The `AZURE_STORAGE_DOMAIN` can be used for optionally overriding the default domain for the Azure storage service.
Please, note that `vmbackup` will use credentials provided by cloud providers metadata service [when applicable](https://docs.victoriametrics.com/victoriametrics/vmbackup/#using-cloud-providers-metadata-service).
### Using cloud providers metadata service
`vmbackup` and `vmbackupmanager` will automatically use cloud providers metadata service in order to obtain credentials if they are running in cloud environment and credentials are not explicitly provided via flags or env variables.
vmbackup \
-storageDataPath=/data \
-snapshot.createURL=http://localhost:8428/snapshot/create \
-dst=azblob://victoriametrics-backup/backup01
```
### Providing credentials in Kubernetes
The simplest way to provide credentials in Kubernetes is to use [Secrets](https://kubernetes.io/docs/concepts/configuration/secret/)
and inject them into the pod as environment variables. For example, the following secret can be used for AWS S3 credentials:
The simplest way to provide credentials in Kubernetes is to use [Secrets](https://kubernetes.io/docs/concepts/configuration/secret/) and inject them into the pod as environment variables. For example, the following secret can be used for AWS S3 credentials:
```yaml
apiVersion: v1
@@ -266,14 +250,13 @@ kind: Secret
metadata:
name: vmbackup-credentials
data:
access_key: key
secret_key: secret
access_key: <base64-encoded-key>
secret_key: <base64-encoded-secret>
```
And then it can be injected into the pod as environment variables:
```yaml
...
env:
- name: AWS_ACCESS_KEY_ID
valueFrom:
@@ -285,7 +268,6 @@ env:
secretKeyRef:
key: secret_key
name: vmbackup-credentials
...
```
A more secure way is to use IAM roles to provide tokens for pods instead of managing credentials manually.

View File

@@ -88,34 +88,21 @@ There are two flags which could help with performance tuning:
### Example of Usage
GCS and cluster version. You need to have a credentials file in json format with following structure:
credentials.json
```json
{
"type": "service_account",
"project_id": "<project>",
"private_key_id": "",
"private_key": "-----BEGIN PRIVATE KEY-----\-----END PRIVATE KEY-----\n",
"client_email": "test@<project>.iam.gserviceaccount.com",
"client_id": "",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://oauth2.googleapis.com/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/test%40<project>.iam.gserviceaccount.com"
}
```
Backup manager launched with the following configuration:
The following command backs up data to a GCS bucket:
```sh
export NODE_IP=192.168.0.10
export VMSTORAGE_ENDPOINT=http://127.0.0.1:8428
./vmbackupmanager -dst=gs://vmstorage-data/$NODE_IP -credsFilePath=credentials.json -storageDataPath=/vmstorage-data -snapshot.createURL=$VMSTORAGE_ENDPOINT/snapshot/create -licenseFile=/path/to/vm-license
./vmbackupmanager \
-dst=gs://vmstorage-data/$NODE_IP \
-credsFilePath=credentials.json \
-storageDataPath=/vmstorage-data \
-snapshot.createURL=$VMSTORAGE_ENDPOINT/snapshot/create \
-licenseFile=/path/to/vm-license
```
See [Connecting VM components to cloud storage](https://docs.victoriametrics.com/guides/connecting-vm-components-to-cloud-storage/) for instructions on obtaining credentials and configuring access to S3, GCS, and Azure Blob Storage.
Expected logs in vmbackupmanager:
```sh
@@ -147,8 +134,7 @@ objects. Typical object storage systems implement server-side copy by creating n
This is very fast and efficient. Unfortunately there are systems such as [S3 Glacier](https://aws.amazon.com/s3/storage-classes/glacier/),
which perform full object copy during server-side copying. This may be slow and expensive.
Please, see [vmbackup docs](https://docs.victoriametrics.com/victoriametrics/vmbackup/#advanced-usage) for more examples of authentication with different
storage types.
See [Connecting VM components to cloud storage](https://docs.victoriametrics.com/guides/connecting-vm-components-to-cloud-storage/) for more examples of authentication with different storage types.
### Backup Retention Policy

View File

@@ -34,9 +34,9 @@ vmctl command-line tool is available as:
Download and unpack vmctl:
```sh
wget https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.147.0/vmutils-darwin-arm64-v1.147.0.tar.gz
wget https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.148.0/vmutils-darwin-arm64-v1.148.0.tar.gz
tar xzf vmutils-darwin-arm64-v1.147.0.tar.gz
tar xzf vmutils-darwin-arm64-v1.148.0.tar.gz
```
Once binary is unpacked, see the full list of supported modes by running the following command:

View File

@@ -47,13 +47,13 @@ i.e. the end result would be similar to [rsync --delete](https://askubuntu.com/q
## Troubleshooting
* See [how to setup credentials via environment variables](https://docs.victoriametrics.com/victoriametrics/vmbackup/#providing-credentials-via-env-variables).
* See [Connecting VM components to cloud storage](https://docs.victoriametrics.com/guides/connecting-vm-components-to-cloud-storage/) for instructions on configuring credentials.
* If `vmrestore` consumes all the network bandwidth, then set `-maxBytesPerSecond` to the desired value.
* If `vmrestore` has been interrupted due to temporary error, then just restart it with the same args. It will resume the restore process.
## Advanced usage
Please, see [vmbackup docs](https://docs.victoriametrics.com/victoriametrics/vmbackup/#advanced-usage) for examples of authentication
See [Connecting VM components to cloud storage](https://docs.victoriametrics.com/guides/connecting-vm-components-to-cloud-storage/) for examples of authentication
with different storage types.
Run `vmrestore -help` in order to see all the available options:

View File

@@ -41,6 +41,8 @@ See the docs at https://docs.victoriametrics.com/victoriametrics/cluster-victori
Flag value can be read from the given http/https url when using -deleteAuthKey=http://host/path or -deleteAuthKey=https://host/path
-denyQueryTracing
Whether to disable the ability to trace queries. See https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#query-tracing
-enableMetadata
Whether to enable metadata processing for metrics scraped from targets, received via VictoriaMetrics remote write, Prometheus remote write v1 or OpenTelemetry protocol. See also remoteWrite.maxMetadataPerBlock (default true)
-enableMultitenancyViaHeaders
Enables multitenancy via HTTP headers. See https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#multitenancy-via-headers
-enableTCP6
@@ -103,6 +105,8 @@ See the docs at https://docs.victoriametrics.com/victoriametrics/cluster-victori
Whether to use proxy protocol for connections accepted at the given -httpListenAddr . See https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt . With enabled proxy protocol http server cannot serve regular /metrics endpoint. Use -pushmetrics.url for metrics pushing
Supports array of values separated by comma or specified via multiple flags.
Empty values are set to false.
-insert.maxQueueDuration duration
The maximum duration to wait in the queue when -maxConcurrentInserts concurrent insert requests are executed (default 1m0s)
-internStringCacheExpireDuration duration
The expiry duration for caches for interned strings. See https://en.wikipedia.org/wiki/String_interning . See also -internStringMaxLen and -internStringDisableCache (default 6m0s)
-internStringDisableCache
@@ -127,6 +131,8 @@ See the docs at https://docs.victoriametrics.com/victoriametrics/cluster-victori
Timezone to use for timestamps in logs. Timezone must be a valid IANA Time Zone. For example: America/New_York, Europe/Berlin, Etc/GMT+3 or Local (default "UTC")
-loggerWarnsPerSecondLimit int
Per-second limit on the number of WARN messages. If more than the given number of warns are emitted per second, then the remaining warns are suppressed. Zero values disable the rate limit
-maxConcurrentInserts int
The maximum number of concurrent insert requests. Set higher value when clients send data over slow networks. Default value depends on the number of available CPU cores. It should work fine in most cases since it minimizes resource usage. See also -insert.maxQueueDuration (default 24)
-memory.allowedBytes size
Allowed size of system memory VictoriaMetrics caches may occupy. This option overrides -memory.allowedPercent if set to a non-zero value. Too low a value may increase the cache miss rate usually resulting in higher CPU and disk IO usage. Too high a value may evict too much data from the OS page cache resulting in higher disk IO usage. The process may behave unexpectedly if this flag is set too small (e.g., 1 byte).
Supports the following optional suffixes for size values: KB, MB, GB, TB, KiB, MiB, GiB, TiB (default 0)
@@ -148,6 +154,119 @@ See the docs at https://docs.victoriametrics.com/victoriametrics/cluster-victori
Flag value can be read from the given http/https url when using -pprofAuthKey=http://host/path or -pprofAuthKey=https://host/path
-prevCacheRemovalPercent float
Items in the previous caches are removed when the percent of requests it serves becomes lower than this value. Higher values reduce memory usage at the cost of higher CPU usage. See also -cacheExpireDuration (default 0.1)
-promscrape.azureSDCheckInterval duration
Interval for checking for changes in Azure. This works only if azure_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/victoriametrics/sd_configs/#azure_sd_configs for details (default 1m0s)
-promscrape.cluster.memberLabel string
If non-empty, then the label with this name and the -promscrape.cluster.memberNum value is added to all the scraped metrics. See https://docs.victoriametrics.com/victoriametrics/vmagent/#scraping-big-number-of-targets for more info
-promscrape.cluster.memberNum string
The number of vmagent instance in the cluster of scrapers. It must be a unique value in the range 0 ... promscrape.cluster.membersCount-1 across scrapers in the cluster. Can be specified as pod name of Kubernetes StatefulSet - pod-name-Num, where Num is a numeric part of pod name. See also -promscrape.cluster.memberLabel . See https://docs.victoriametrics.com/victoriametrics/vmagent/#scraping-big-number-of-targets for more info (default "0")
-promscrape.cluster.memberURLTemplate string
An optional template for URL to access vmagent instance with the given -promscrape.cluster.memberNum value. Every %d occurrence in the template is substituted with -promscrape.cluster.memberNum at urls to vmagent instances responsible for scraping the given target at /service-discovery page. For example -promscrape.cluster.memberURLTemplate='http://vmagent-%d:8429/targets'. See https://docs.victoriametrics.com/victoriametrics/vmagent/#scraping-big-number-of-targets for more details
-promscrape.cluster.membersCount int
The number of members in a cluster of scrapers. Each member must have a unique -promscrape.cluster.memberNum in the range 0 ... promscrape.cluster.membersCount-1 . Each member then scrapes roughly 1/N of all the targets. By default, cluster scraping is disabled, i.e. a single scraper scrapes all the targets. See https://docs.victoriametrics.com/victoriametrics/vmagent/#scraping-big-number-of-targets for more info (default 1)
-promscrape.cluster.name string
Optional name of the cluster. If multiple vmagent clusters scrape the same targets, then each cluster must have unique name in order to properly de-duplicate samples received from these clusters. See https://docs.victoriametrics.com/victoriametrics/vmagent/#scraping-big-number-of-targets for more info
-promscrape.cluster.replicationFactor int
The number of members in the cluster, which scrape the same targets. If the replication factor is greater than 1, then the deduplication must be enabled at remote storage side. See https://docs.victoriametrics.com/victoriametrics/vmagent/#scraping-big-number-of-targets for more info (default 1)
-promscrape.cluster.shardByLabels array
Optional list of target labels, which will be used for sharding targets among cluster members if -promscrape.cluster.membersCount is greater than 1. If none of the specified labels are found in a target, then all the target labels will be used for sharding. See https://docs.victoriametrics.com/victoriametrics/vmagent/#scraping-big-number-of-targets for more info
Supports an array of values separated by comma or specified via multiple flags.
Each array item can contain comma inside single-quoted or double-quoted string, {}, [] and () braces.
-promscrape.config string
Optional path to Prometheus config file with 'scrape_configs' section containing targets to scrape. The path can point to local file and to http url. See https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#how-to-scrape-prometheus-exporters-such-as-node-exporter for details
-promscrape.config.dryRun
Checks -promscrape.config file for errors and unsupported fields and then exits. Returns non-zero exit code on parsing errors and emits these errors to stderr. See also -promscrape.config.strictParse command-line flag. Pass -loggerLevel=ERROR if you don't need to see info messages in the output.
-promscrape.config.strictParse
Whether to deny unsupported fields in -promscrape.config . Set to false in order to silently skip unsupported fields (default true)
-promscrape.configCheckInterval duration
Interval for checking for changes in -promscrape.config file. By default, the checking is disabled. See how to reload -promscrape.config file at https://docs.victoriametrics.com/victoriametrics/vmagent/#configuration-update
-promscrape.consul.waitTime duration
Wait time used by Consul service discovery. Default value is used if not set
-promscrape.consulSDCheckInterval duration
Interval for checking for changes in Consul. This works only if consul_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/victoriametrics/sd_configs/#consul_sd_configs for details (default 30s)
-promscrape.consulagentSDCheckInterval duration
Interval for checking for changes in Consul Agent. This works only if consulagent_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/victoriametrics/sd_configs/#consulagent_sd_configs for details (default 30s)
-promscrape.digitaloceanSDCheckInterval duration
Interval for checking for changes in digital ocean. This works only if digitalocean_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/victoriametrics/sd_configs/#digitalocean_sd_configs for details (default 1m0s)
-promscrape.disableCompression
Whether to disable sending 'Accept-Encoding: gzip' request headers to all the scrape targets. This may reduce CPU usage on scrape targets at the cost of higher network bandwidth utilization. It is possible to set 'disable_compression: true' individually per each 'scrape_config' section in '-promscrape.config' for fine-grained control
-promscrape.disableKeepAlive
Whether to disable HTTP keep-alive connections when scraping all the targets. This may be useful when targets has no support for HTTP keep-alive connection. It is possible to set 'disable_keepalive: true' individually per each 'scrape_config' section in '-promscrape.config' for fine-grained control. Note that disabling HTTP keep-alive may increase load on both vmagent and scrape targets
-promscrape.discovery.concurrency int
The maximum number of concurrent requests to Prometheus autodiscovery API (Consul, Kubernetes, etc.) (default 100)
-promscrape.discovery.concurrentWaitTime duration
The maximum duration for waiting to perform API requests if more than -promscrape.discovery.concurrency requests are simultaneously performed (default 1m0s)
-promscrape.dnsSDCheckInterval duration
Interval for checking for changes in dns. This works only if dns_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/victoriametrics/sd_configs/#dns_sd_configs for details (default 30s)
-promscrape.dockerSDCheckInterval duration
Interval for checking for changes in docker. This works only if docker_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/victoriametrics/sd_configs/#docker_sd_configs for details (default 30s)
-promscrape.dockerswarmSDCheckInterval duration
Interval for checking for changes in dockerswarm. This works only if dockerswarm_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/victoriametrics/sd_configs/#dockerswarm_sd_configs for details (default 30s)
-promscrape.dropOriginalLabels
Whether to drop original labels for scrape targets at /targets and /api/v1/targets pages. This may be needed for reducing memory usage when original labels for big number of scrape targets occupy big amounts of memory. Note that this reduces debuggability for improper per-target relabeling configs
-promscrape.ec2SDCheckInterval duration
Interval for checking for changes in ec2. This works only if ec2_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/victoriametrics/sd_configs/#ec2_sd_configs for details (default 1m0s)
-promscrape.eurekaSDCheckInterval duration
Interval for checking for changes in eureka. This works only if eureka_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/victoriametrics/sd_configs/#eureka_sd_configs for details (default 30s)
-promscrape.fileSDCheckInterval duration
Interval for checking for changes in 'file_sd_config'. See https://docs.victoriametrics.com/victoriametrics/sd_configs/#file_sd_configs for details (default 1m0s)
-promscrape.gceSDCheckInterval duration
Interval for checking for changes in gce. This works only if gce_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/victoriametrics/sd_configs/#gce_sd_configs for details (default 1m0s)
-promscrape.hetznerSDCheckInterval duration
Interval for checking for changes in Hetzner API. This works only if hetzner_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/victoriametrics/sd_configs/#hetzner_sd_configs for details (default 1m0s)
-promscrape.httpSDCheckInterval duration
Interval for checking for changes in http endpoint service discovery. This works only if http_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/victoriametrics/sd_configs/#http_sd_configs for details (default 1m0s)
-promscrape.kubernetes.apiServerTimeout duration
How frequently to reload the full state from Kubernetes API server (default 30m0s)
-promscrape.kubernetes.attachNamespaceMetadataAll
Whether to set attach_metadata.namespace=true for all the kubernetes_sd_configs at -promscrape.config . It is possible to set attach_metadata.namespace=false individually per each kubernetes_sd_configs . See https://docs.victoriametrics.com/victoriametrics/sd_configs/#kubernetes_sd_configs
-promscrape.kubernetes.attachNodeMetadataAll
Whether to set attach_metadata.node=true for all the kubernetes_sd_configs at -promscrape.config . It is possible to set attach_metadata.node=false individually per each kubernetes_sd_configs . See https://docs.victoriametrics.com/victoriametrics/sd_configs/#kubernetes_sd_configs
-promscrape.kubernetes.useHTTP2Client
Whether to use HTTP/2 client for connection to Kubernetes API server. This may reduce amount of concurrent connections to API server when watching for a big number of Kubernetes objects.
-promscrape.kubernetesSDCheckInterval duration
Interval for checking for changes in Kubernetes API server. This works only if kubernetes_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/victoriametrics/sd_configs/#kubernetes_sd_configs for details (default 30s)
-promscrape.kumaSDCheckInterval duration
Interval for checking for changes in kuma service discovery. This works only if kuma_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/victoriametrics/sd_configs/#kuma_sd_configs for details (default 30s)
-promscrape.marathonSDCheckInterval duration
Interval for checking for changes in Marathon REST API. This works only if marathon_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/victoriametrics/sd_configs/#marathon_sd_configs for details (default 30s)
-promscrape.maxDroppedTargets int
The maximum number of droppedTargets to show at /api/v1/targets page. Increase this value if your setup drops more scrape targets during relabeling and you need investigating labels for all the dropped targets. Note that the increased number of tracked dropped targets may result in increased memory usage (default 10000)
-promscrape.maxResponseHeadersSize size
The maximum size of http response headers from Prometheus scrape targets
Supports the following optional suffixes for size values: KB, MB, GB, TB, KiB, MiB, GiB, TiB (default 4096)
-promscrape.maxScrapeSize size
The maximum size of uncompressed scrape response in bytes to process from Prometheus targets. Bigger uncompressed responses are rejected. See also max_scrape_size option at https://docs.victoriametrics.com/victoriametrics/sd_configs/#scrape_configs
Supports the following optional suffixes for size values: KB, MB, GB, TB, KiB, MiB, GiB, TiB (default 16777216)
-promscrape.minResponseSizeForStreamParse size
The minimum target response size for automatic switching to stream parsing mode, which can reduce memory usage. See https://docs.victoriametrics.com/victoriametrics/vmagent/#stream-parsing-mode
Supports the following optional suffixes for size values: KB, MB, GB, TB, KiB, MiB, GiB, TiB (default 1000000)
-promscrape.noStaleMarkers
Whether to disable sending Prometheus stale markers for metrics when scrape target disappears. This option may reduce memory usage if stale markers aren't needed for your setup. This option also disables populating the scrape_series_added metric. See https://prometheus.io/docs/concepts/jobs_instances/#automatically-generated-labels-and-time-series
-promscrape.nomad.waitTime duration
Wait time used by Nomad service discovery. Default value is used if not set
-promscrape.nomadSDCheckInterval duration
Interval for checking for changes in Nomad. This works only if nomad_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/victoriametrics/sd_configs/#nomad_sd_configs for details (default 30s)
-promscrape.openstackSDCheckInterval duration
Interval for checking for changes in openstack API server. This works only if openstack_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/victoriametrics/sd_configs/#openstack_sd_configs for details (default 30s)
-promscrape.ovhcloudSDCheckInterval duration
Interval for checking for changes in OVH Cloud API. This works only if ovhcloud_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/victoriametrics/sd_configs/#ovhcloud_sd_configs for details (default 30s)
-promscrape.puppetdbSDCheckInterval duration
Interval for checking for changes in PuppetDB API. This works only if puppetdb_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/victoriametrics/sd_configs/#puppetdb_sd_configs for details (default 30s)
-promscrape.seriesLimitPerTarget int
Optional limit on the number of unique time series a single scrape target can expose. See https://docs.victoriametrics.com/victoriametrics/vmagent/#cardinality-limiter for more info
-promscrape.streamParse
Whether to enable stream parsing for metrics obtained from scrape targets. This may be useful for reducing memory usage when millions of metrics are exposed per each scrape target. It is possible to set 'stream_parse: true' individually per each 'scrape_config' section in '-promscrape.config' for fine-grained control
-promscrape.suppressDuplicateScrapeTargetErrors
Whether to suppress 'duplicate scrape target' errors; see https://docs.victoriametrics.com/victoriametrics/vmagent/#troubleshooting for details
-promscrape.suppressScrapeErrors
Whether to suppress scrape errors logging. The last error for each target is always available at '/targets' page even if scrape errors logging is suppressed. See also -promscrape.suppressScrapeErrorsDelay
-promscrape.suppressScrapeErrorsDelay duration
The delay for suppressing repeated scrape errors logging per each scrape targets. This may be used for reducing the number of log lines related to scrape errors. See also -promscrape.suppressScrapeErrors
-promscrape.vultrSDCheckInterval duration
Interval for checking for changes in Vultr. This works only if vultr_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/victoriametrics/sd_configs/#vultr_sd_configs for details (default 30s)
-promscrape.yandexcloudSDCheckInterval duration
Interval for checking for changes in Yandex Cloud API. This works only if yandexcloud_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/victoriametrics/sd_configs/#yandexcloud_sd_configs for details (default 30s)
-pushmetrics.disableCompression
Whether to disable request body compression when pushing metrics to every -pushmetrics.url
-pushmetrics.extraLabel array

View File

@@ -130,6 +130,9 @@ See the docs at https://docs.victoriametrics.com/victoriametrics/cluster-victori
Timezone to use for timestamps in logs. Timezone must be a valid IANA Time Zone. For example: America/New_York, Europe/Berlin, Etc/GMT+3 or Local (default "UTC")
-loggerWarnsPerSecondLimit int
Per-second limit on the number of WARN messages. If more than the given number of warns are emitted per second, then the remaining warns are suppressed. Zero values disable the rate limit
-maxBackfillAge value
The maximum allowed age for the ingested samples with historical timestamps. Samples with timestamps older than now-maxBackfillAge are rejected during data ingestion. By default, or when set to 0, -maxBackfillAge equals to -retentionPeriod, e.g. it is unlimited within the configured retention. This can be useful for limiting ingestion of historical samples, for example, when older data has been moved to another storage tier. See https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#retention
The following optional suffixes are supported: s (second), h (hour), d (day), w (week), M (month), y (year). If suffix isn't set, then the duration is counted in months (default 0)
-maxConcurrentInserts int
The maximum number of concurrent insert requests. Set higher value when clients send data over slow networks. Default value depends on the number of available CPU cores. It should work fine in most cases since it minimizes resource usage. See also -insert.maxQueueDuration (default 2*cgroup.AvailableCPUs())
-memory.allowedBytes size

20
go.mod
View File

@@ -34,7 +34,7 @@ require (
github.com/valyala/gozstd v1.24.0
github.com/valyala/histogram v1.2.0
github.com/valyala/quicktemplate v1.8.0
golang.org/x/net v0.56.0
golang.org/x/net v0.57.0
golang.org/x/oauth2 v0.36.0
golang.org/x/sys v0.47.0
google.golang.org/api v0.284.0
@@ -130,6 +130,7 @@ require (
github.com/valyala/bytebufferpool v1.0.0 // indirect
github.com/x448/float16 v0.8.4 // indirect
github.com/xrash/smetrics v0.0.0-20250705151800-55b8f293f342 // indirect
github.com/yuin/goldmark v1.8.4 // indirect
go.opentelemetry.io/auto/sdk v1.2.1 // indirect
go.opentelemetry.io/collector/component v1.60.0 // indirect
go.opentelemetry.io/collector/confmap v1.60.0 // indirect
@@ -155,12 +156,19 @@ require (
go.uber.org/zap v1.28.0 // indirect
go.yaml.in/yaml/v2 v2.4.4 // indirect
go.yaml.in/yaml/v3 v3.0.4 // indirect
golang.org/x/crypto v0.53.0 // indirect
golang.org/x/exp v0.0.0-20260611194520-c48552f49976 // indirect
golang.org/x/sync v0.21.0 // indirect
golang.org/x/term v0.44.0 // indirect
golang.org/x/text v0.38.0 // indirect
golang.org/x/crypto v0.54.0 // indirect
golang.org/x/exp v0.0.0-20260718201538-764159d718ef // indirect
golang.org/x/mod v0.38.0 // indirect
golang.org/x/sync v0.22.0 // indirect
golang.org/x/telemetry v0.0.0-20260717140457-bdb89881bb75 // indirect
golang.org/x/term v0.45.0 // indirect
golang.org/x/text v0.40.0 // indirect
golang.org/x/time v0.15.0 // indirect
golang.org/x/tools v0.48.0 // indirect
golang.org/x/tools/go/expect v0.1.1-deprecated // indirect
golang.org/x/tools/go/packages/packagestest v0.1.1-deprecated // indirect
golang.org/x/tools/godoc v0.1.0-deprecated // indirect
golang.org/x/xerrors v0.0.0-20240903120638-7835f813f4da // indirect
google.golang.org/genproto v0.0.0-20260615183401-62b3387ff324 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20260615183401-62b3387ff324 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20260615183401-62b3387ff324 // indirect

29
go.sum
View File

@@ -438,6 +438,8 @@ github.com/xrash/smetrics v0.0.0-20250705151800-55b8f293f342 h1:FnBeRrxr7OU4VvAz
github.com/xrash/smetrics v0.0.0-20250705151800-55b8f293f342/go.mod h1:Ohn+xnUBiLI6FVj/9LpzZWtj1/D6lUovWYBkxHVV3aM=
github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
github.com/yuin/goldmark v1.8.4 h1:oat/nd3U6NeQqFEL3xpEJq7d7c86NI+DbSNGAs4xnjA=
github.com/yuin/goldmark v1.8.4/go.mod h1:ip/1k0VRfGynBgxOz0yCqHrbZXhcjxyuS66Brc7iBKg=
go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64=
go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y=
go.opentelemetry.io/collector/component v1.60.0 h1:LpIjHMn7OOjUsFR84ROc2kqPbP1xnKyDCGi7ZVqEaKU=
@@ -521,18 +523,26 @@ golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8U
golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/crypto v0.53.0 h1:QZ4Muo8THX6CizN2vPPd5fBGHyogrdK9fG4wLPFUsto=
golang.org/x/crypto v0.53.0/go.mod h1:DNLU434OwVakk9PzuwV8w62mAJpRJL3vsgcfp4Qnsio=
golang.org/x/crypto v0.54.0 h1:YLIA59K4fiNzHzjnZt2tUJQjQtUWfWbeHBqKtk3eScw=
golang.org/x/crypto v0.54.0/go.mod h1:KWL8ny2AZdGR2cWmzeHrp2azQPGogOv+HeQaVEXC2dk=
golang.org/x/exp v0.0.0-20260611194520-c48552f49976 h1:X8Hz2ImujgbmetVuW+w2YkyZChE3cBpZi2P158rTG9M=
golang.org/x/exp v0.0.0-20260611194520-c48552f49976/go.mod h1:vnf4pv9iKZXY58sQE1L86zmNWJ4159e1RkcWiLCkeEY=
golang.org/x/exp v0.0.0-20260718201538-764159d718ef h1:LkZ48HFgy/TvhTI0bcWkjgFkgLyKUwcTbDjS0DUjw+A=
golang.org/x/exp v0.0.0-20260718201538-764159d718ef/go.mod h1:EdfpwwqSu+0Li0mzskwHU6FWDV3t9Q+RZDo3QMUtL3Q=
golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.37.0 h1:vF1DjpVEshcIqoEaauuHebaLk1O1forxjxBaVn884JQ=
golang.org/x/mod v0.37.0/go.mod h1:m8S8VeM9r4dzDwjrKO0a1sZP3YjeMamRRlD+fmR2Q/0=
golang.org/x/mod v0.38.0 h1:MECBjubtXD7yj4HrhIUcywNaGeNVUdfVnxmPajOk4yk=
golang.org/x/mod v0.38.0/go.mod h1:V6Xz0pq8TQ3dGqVQ1FVHuelZpAL0uNhSkk9ogYP3c40=
golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
golang.org/x/net v0.56.0 h1:Rw8j/hFzGvJUZwNBXnAtf5sVDVt+65SK2C7IxCxZt5o=
golang.org/x/net v0.56.0/go.mod h1:D3Ku6r+V6JROoZK144D2XfMHFcMq/0zSfLelVTCFKec=
golang.org/x/net v0.57.0 h1:K5+3DljvIuDG9/Jv9rvyMywYNFCQ9RSUY6OOTTkT+tE=
golang.org/x/net v0.57.0/go.mod h1:KpXc8iv+r3XplLAG/f7Jsf9RPszJzdR0f58q9vGOuEU=
golang.org/x/oauth2 v0.36.0 h1:peZ/1z27fi9hUOFCAZaHyrpWG5lwe0RJEEEeH0ThlIs=
golang.org/x/oauth2 v0.36.0/go.mod h1:YDBUJMTkDnJS+A4BP4eZBjCqtokkg1hODuPjwiGPO7Q=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
@@ -540,6 +550,8 @@ golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJ
golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.21.0 h1:HLII4xRRTtCRkxYp4HNFF0Js/Og6q2i++KXbg0gHCwM=
golang.org/x/sync v0.21.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
golang.org/x/sync v0.22.0 h1:SZjpbeLmrCk4xhRSZFNZW5gFUeCeFgjekvI/+gfScek=
golang.org/x/sync v0.22.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
@@ -548,12 +560,18 @@ golang.org/x/sys v0.46.0 h1:noSf2Fq6F8DBgS+LysIkx7rIExoNHJsxOAtPp4rthXw=
golang.org/x/sys v0.46.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs=
golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
golang.org/x/telemetry v0.0.0-20260717140457-bdb89881bb75 h1:I9ygRooEYoVHV0SRNOSr/KVjTf5EeJ52BuNkVjsP2GU=
golang.org/x/telemetry v0.0.0-20260717140457-bdb89881bb75/go.mod h1:LV7u5Oco+Z/g6XI7PqN+EUUUGGkEcmB1uj2ceI0fOVg=
golang.org/x/term v0.44.0 h1:0rLvDRCtNj0gZkyIXhCyOb2OAzEhLVqc4B+hrsBhrmc=
golang.org/x/term v0.44.0/go.mod h1:7ze4MdzUzLXpSAoFP1H0bOI9aXDqveSvatT5vKcFh2Y=
golang.org/x/term v0.45.0 h1:NwWyBmoJCbfTHpxrWoZ9C6/VxOf7ic219I8xZZFdrf0=
golang.org/x/term v0.45.0/go.mod h1:9aqxs0blBcrm/n0L9QW0aRVD+ktan8ssZromtqJC43w=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.38.0 h1:sXmwo9DwP3OK9EZ7PqAdaooSGozfl/3a6/xJcbzPRhE=
golang.org/x/text v0.38.0/go.mod h1:YXZt3QhHUKYT53r2lLKFIVi6Ao1jdzrTR/KQ09qyxF4=
golang.org/x/text v0.40.0 h1:Ub2Z6/xjgF1WrYQz2nuITOEegKFtiIy+rieRJ5lHZKs=
golang.org/x/text v0.40.0/go.mod h1:hpnzDAfGV753zIKo+wk3u1bVKCGPbrnF7+7LBF/UHVY=
golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U=
golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
@@ -562,10 +580,21 @@ golang.org/x/tools v0.0.0-20200619180055-7c47624df98f/go.mod h1:EkVYQZoAsY45+roY
golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA=
golang.org/x/tools v0.46.0 h1:7jTurBkPZu4moS/Uy4OQT1M+QBlsj3wejyZwsT8Z7rk=
golang.org/x/tools v0.46.0/go.mod h1:FrD85F8l+NWL+9XWBSyVSHO6Ne4jutsfIFba7AWQ5Ys=
golang.org/x/tools v0.47.0 h1:7Kn5x/d1svx/PzryTsqeoZN4TZwqeH5pGWjefhLi/1Q=
golang.org/x/tools v0.48.0 h1:3+hClM1aLL5mjMKm5ovokw9epgRXPuu2tILgismM6RE=
golang.org/x/tools v0.48.0/go.mod h1:08xX0orndb/F7jJxGDicx061tyd5pcMto75YMAXr6lk=
golang.org/x/tools/go/expect v0.1.1-deprecated h1:jpBZDwmgPhXsKZC6WhL20P4b/wmnpsEAGHaNy0n/rJM=
golang.org/x/tools/go/expect v0.1.1-deprecated/go.mod h1:eihoPOH+FgIqa3FpoTwguz/bVUSGBlGQU67vpBeOrBY=
golang.org/x/tools/go/packages/packagestest v0.1.1-deprecated h1:1h2MnaIAIXISqTFKdENegdpAgUXz6NrPEsbIeWaBRvM=
golang.org/x/tools/go/packages/packagestest v0.1.1-deprecated/go.mod h1:RVAQXBGNv1ib0J382/DPCRS/BPnsGebyM1Gj5VSDpG8=
golang.org/x/tools/godoc v0.1.0-deprecated h1:o+aZ1BOj6Hsx/GBdJO/s815sqftjSnrZZwyYTHODvtk=
golang.org/x/tools/godoc v0.1.0-deprecated/go.mod h1:qM63CriJ961IHWmnWa9CjZnBndniPt4a3CK0PVB9bIg=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20240903120638-7835f813f4da h1:noIWHXmPHxILtqtCOPIhSt0ABwskkZKjD3bXGnZGpNY=
golang.org/x/xerrors v0.0.0-20240903120638-7835f813f4da/go.mod h1:NDW/Ps6MPRej6fsCIbMTohpP40sJ/P/vI1MoTEGwX90=
gonum.org/v1/gonum v0.17.0 h1:VbpOemQlsSMrYmn7T2OUvQ4dqxQXU+ouZFQsZOx50z4=
gonum.org/v1/gonum v0.17.0/go.mod h1:El3tOrEuMpv2UdMrbNlKEh9vd86bmQ6vqIcDwxEOc1E=
google.golang.org/api v0.284.0 h1:i+cKTgeQRcRySkP7QTl5PDO7/pAm8EcMFIUMlNbk4Vc=

View File

@@ -0,0 +1,334 @@
package httputil
import (
"context"
"fmt"
"io"
"net"
"net/http"
"net/netip"
"net/url"
"sort"
"strconv"
"strings"
"sync/atomic"
"time"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/fasttime"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/netutil"
)
const (
brokenBackendTimeout = 5 * time.Second
backendDiscoveryInterval = 10 * time.Second
backendDiscoveryTimeout = 10 * time.Second
)
// NewLoadBalancerTransport returns new RoundTripper that performs least-loaded HTTP requests loadbalancing
// based on discovered backends for the given url host
// and update url with load-balancing prefix
//
// It returns origin transport and url if load-balancing is not needed for given url
func NewLoadBalancerTransport(origin http.RoundTripper, originURL *url.URL) (http.RoundTripper, *url.URL) {
modifiedURL := *originURL
var discoverFunc func(context.Context, string, string) ([]*backend, error)
switch {
case strings.HasPrefix(originURL.Host, "dns+"):
modifiedURL.Host = modifiedURL.Host[4:]
discoverFunc = discoverDNSBackends
case strings.HasPrefix(originURL.Host, "srv+"):
modifiedURL.Host = modifiedURL.Host[4:]
discoverFunc = discoverSRVBackends
default:
return origin, originURL
}
host, port, err := net.SplitHostPort(modifiedURL.Host)
if err != nil {
host = modifiedURL.Host
port = "80"
if modifiedURL.Scheme == "https" {
port = "443"
}
}
t := &loadbalancerTransport{
tr: origin,
host: host,
port: port,
discoverFunc: discoverFunc,
}
t.discoverBackends()
return t, &modifiedURL
}
type loadbalancerTransport struct {
tr http.RoundTripper
host string
port string
discoverFunc func(context.Context, string, string) ([]*backend, error)
nextDiscoveryDeadline atomic.Uint64
discovering atomic.Bool
dbs atomic.Pointer[discoveredBackends]
}
type discoveredBackends struct {
backends []*backend
// n is an atomic counter, which is used for balancing load among available backends.
n atomic.Uint32
}
// getLeastLoadedBackend returns least loaded backend
// caller must release backend with backend.put() method
func (dbs *discoveredBackends) getLeastLoadedBackend() *backend {
firstB := dbs.backends[0]
if len(dbs.backends) == 1 {
firstB.get()
return firstB
}
// Slow path - select other backends.
n := dbs.n.Add(1) - 1
for i := range uint32(len(dbs.backends)) {
idx := (n + i) % uint32(len(dbs.backends))
bu := dbs.backends[idx]
if bu.isBroken() {
continue
}
// The Load() in front of CompareAndSwap() avoids CAS overhead for items with values bigger than 0.
if bu.concurrentRequests.Load() == 0 && bu.concurrentRequests.CompareAndSwap(0, 1) {
dbs.n.CompareAndSwap(n+1, idx+1)
// There is no need in the call b.get(), because we already incremented b.concurrentRequests above.
return bu
}
}
// Slow path - return the backend with the minimum number of concurrently executed requests.
bMinIdx := n % uint32(len(dbs.backends))
minRequests := dbs.backends[bMinIdx].concurrentRequests.Load()
for i := uint32(1); i < uint32(len(dbs.backends)); i++ {
idx := (n + i) % uint32(len(dbs.backends))
bu := dbs.backends[idx]
if bu.isBroken() {
continue
}
reqs := bu.concurrentRequests.Load()
if reqs < minRequests || dbs.backends[bMinIdx].isBroken() {
bMinIdx = idx
minRequests = reqs
}
}
bMin := dbs.backends[bMinIdx]
if bMin.isBroken() {
// If all backends are broken, then returns the first backend.
firstB.get()
return firstB
}
bMin.get()
dbs.n.CompareAndSwap(n+1, bMinIdx+1)
return bMin
}
type backend struct {
addr string
concurrentRequests atomic.Int32
brokenDeadline atomic.Uint64
}
func (b *backend) get() {
b.concurrentRequests.Add(1)
}
func (b *backend) put() {
b.concurrentRequests.Add(-1)
}
func (b *backend) isBroken() bool {
bd := b.brokenDeadline.Load()
if bd == 0 {
return false
}
ct := fasttime.UnixTimestamp()
return ct < bd
}
// RoundTrip implements http.RoundTripper interface
func (lb *loadbalancerTransport) RoundTrip(r *http.Request) (*http.Response, error) {
dbs := lb.getBackends()
if dbs == nil || len(dbs.backends) == 0 {
return nil, fmt.Errorf("no backends found for hostname=%q", lb.host)
}
maxRetries := len(dbs.backends)
var lastErr error
for range maxRetries {
b := dbs.getLeastLoadedBackend()
resp, err := lb.doRequest(r, b)
if err != nil {
ct := fasttime.UnixTimestamp()
brokenDeadline := ct + uint64(brokenBackendTimeout.Seconds())
b.brokenDeadline.Store(brokenDeadline)
if !netutil.IsTrivialNetworkError(err) {
return nil, err
}
// perform the same check for retry as http.Request.isReplayable does
canRetry := r.Body == nil || r.Body == http.NoBody || r.GetBody != nil
if !canRetry {
return nil, err
}
lastErr = err
continue
}
return resp, nil
}
return nil, fmt.Errorf("all backends are unavailable: %w", lastErr)
}
func (lb *loadbalancerTransport) doRequest(r *http.Request, b *backend) (*http.Response, error) {
r2 := r.Clone(r.Context())
if r.GetBody != nil {
body, err := r.GetBody()
if err != nil {
b.put()
return nil, err
}
r2.Body = body
}
r2.URL.Host = b.addr
if r2.Host == "" {
r2.Host = r.URL.Host
}
resp, err := lb.tr.RoundTrip(r2)
if err != nil {
b.put()
return nil, err
}
// wrap response body with readCloser that releases backend after Close call
// it's needed to properly account loaded backends at getLeastLoadedBackends
resp.Body = newReleaseReadCloser(resp.Body, b)
return resp, nil
}
func (lb *loadbalancerTransport) getBackends() *discoveredBackends {
ct := fasttime.UnixTimestamp()
deadline := lb.nextDiscoveryDeadline.Load()
if ct < deadline || !lb.discovering.CompareAndSwap(false, true) {
return lb.dbs.Load()
}
lb.discoverBackends()
return lb.dbs.Load()
}
func (lb *loadbalancerTransport) discoverBackends() {
ctx, cancel := context.WithTimeout(context.Background(), backendDiscoveryTimeout)
defer func() {
cancel()
ct := fasttime.UnixTimestamp()
nextDeadline := ct + uint64(backendDiscoveryInterval.Seconds())
lb.nextDiscoveryDeadline.Store(nextDeadline)
lb.discovering.Store(false)
}()
backends, err := lb.discoverFunc(ctx, lb.host, lb.port)
if err != nil {
logger.Errorf("cannot discover backends: %s, retry in %s", err, backendDiscoveryInterval)
return
}
dbs := &discoveredBackends{
backends: backends,
}
sort.Slice(dbs.backends, func(i, j int) bool {
return dbs.backends[i].addr < dbs.backends[j].addr
})
prevBackends := lb.dbs.Load()
if areBackendsEqual(prevBackends, dbs) {
return
}
lb.dbs.Store(dbs)
}
func discoverDNSBackends(ctx context.Context, host, port string) ([]*backend, error) {
addrs, err := netutil.Resolver.LookupIPAddr(ctx, host)
if err != nil {
return nil, fmt.Errorf("failed to lookupIPAddr for host: %q: %w", host, err)
}
backends := make([]*backend, 0, len(addrs))
for _, addr := range addrs {
if !netutil.TCP6Enabled() {
ip, ok := netip.AddrFromSlice(addr.IP)
if !ok {
logger.Panicf("BUG: cannot build netip Addr from slice addr: %q", addr.IP.String())
}
if !ip.Unmap().Is4() {
continue
}
}
ip := addr.IP.String()
if len(port) > 0 {
ip = net.JoinHostPort(ip, port)
}
backends = append(backends, &backend{addr: ip})
}
return backends, nil
}
func discoverSRVBackends(ctx context.Context, host, port string) ([]*backend, error) {
_, addrs, err := netutil.Resolver.LookupSRV(ctx, "", "", host)
if err != nil {
return nil, fmt.Errorf("failed to LookupSRV records for host: %q: %w", host, err)
}
backends := make([]*backend, 0, len(addrs))
for _, addr := range addrs {
hostPort := port
if addr.Port > 0 {
hostPort = strconv.FormatUint(uint64(addr.Port), 10)
}
hostAddr := net.JoinHostPort(addr.Target, hostPort)
backends = append(backends, &backend{addr: hostAddr})
}
return backends, nil
}
func newReleaseReadCloser(responseBody io.ReadCloser, b *backend) io.ReadCloser {
return &releaseReadCloser{
ReadCloser: responseBody,
b: b,
}
}
type releaseReadCloser struct {
io.ReadCloser
b *backend
released atomic.Bool
}
func (rrc *releaseReadCloser) Close() error {
if rrc.released.CompareAndSwap(false, true) {
// Close method could be called multiple times
// and it must produce idempotent result
rrc.b.put()
}
return rrc.ReadCloser.Close()
}
func areBackendsEqual(left, right *discoveredBackends) bool {
if left == nil || right == nil {
return left == right
}
if len(left.backends) != len(right.backends) {
return false
}
for idx, leftB := range left.backends {
rightB := right.backends[idx]
if leftB.addr != rightB.addr {
return false
}
}
return true
}

View File

@@ -0,0 +1,127 @@
package httputil
import (
"context"
"fmt"
"net"
"net/http"
"net/netip"
"net/url"
"sync"
"testing"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/netutil"
)
type testRemoteServer struct {
mu sync.Mutex
requestsPerHost map[string]int
totalRequests int
firstError error
}
func (trs *testRemoteServer) RoundTrip(r *http.Request) (*http.Response, error) {
trs.mu.Lock()
if trs.firstError != nil && trs.totalRequests == 0 {
err := trs.firstError
trs.firstError = nil
trs.totalRequests++
trs.mu.Unlock()
return nil, err
}
trs.totalRequests++
if trs.requestsPerHost == nil {
trs.requestsPerHost = make(map[string]int)
}
trs.requestsPerHost[r.URL.Host]++
trs.mu.Unlock()
return &http.Response{StatusCode: http.StatusOK, Body: http.NoBody}, nil
}
type testDNSResolver struct {
ips []net.IPAddr
}
func (tdr *testDNSResolver) LookupSRV(_ context.Context, _, _, name string) (cname string, addrs []*net.SRV, err error) {
return "", nil, fmt.Errorf("unexpected LookupMX call for name=%q", name)
}
func (tdr *testDNSResolver) LookupIPAddr(ctx context.Context, host string) ([]net.IPAddr, error) {
return tdr.ips, nil
}
func (tdr *testDNSResolver) LookupMX(_ context.Context, name string) ([]*net.MX, error) {
return nil, fmt.Errorf("unexpected LookupMX call for name=%q", name)
}
func TestLoadbalancerTransport(t *testing.T) {
f := func(discoveredIPs []string, trs *testRemoteServer) {
t.Helper()
parsedIPs := make([]net.IPAddr, 0, len(discoveredIPs))
for _, dIP := range discoveredIPs {
pIP, err := netip.ParseAddr(dIP)
if err != nil {
t.Fatalf("cannot parse IP=%q: %s", dIP, err)
}
parsedIPs = append(parsedIPs, net.IPAddr{IP: pIP.AsSlice()})
}
tdr := &testDNSResolver{ips: parsedIPs}
originResolver := netutil.Resolver
defer func() { netutil.Resolver = originResolver }()
netutil.Resolver = tdr
requestURL, err := url.Parse("http://dns+vmsingle.example.com:8429/api/v1/write")
if err != nil {
t.Fatalf("cannot parse url: %s", err)
}
lbt, requestURL := NewLoadBalancerTransport(trs, requestURL)
if len(discoveredIPs) == 0 {
r, err := http.NewRequest(http.MethodGet, requestURL.String(), nil)
if err != nil {
t.Fatalf("cannot create http request: %s", err)
}
_, err = lbt.RoundTrip(r)
if err == nil {
t.Fatalf("expected no backends found error")
}
return
}
expectedRequestsPerHost := 2
for range len(discoveredIPs) * expectedRequestsPerHost {
r, err := http.NewRequest(http.MethodGet, requestURL.String(), nil)
if err != nil {
t.Fatalf("cannot create http request: %s", err)
}
resp, err := lbt.RoundTrip(r)
if err != nil {
t.Fatalf("unexpected error: %s", err)
}
resp.Body.Close()
}
requestsPerHost := trs.requestsPerHost
for _, dIP := range discoveredIPs {
expectedHostPort := net.JoinHostPort(dIP, "8429")
gotRequestsPerHost, ok := requestsPerHost[expectedHostPort]
if !ok {
t.Fatalf("not found expected backend request for: %q", expectedHostPort)
}
if gotRequestsPerHost != expectedRequestsPerHost {
t.Fatalf("unexpected requests per host:%q %d:%d (-;+)", expectedHostPort, expectedRequestsPerHost, gotRequestsPerHost)
}
}
}
trs := testRemoteServer{}
f([]string{"1.1.1.1"}, &trs)
trs = testRemoteServer{}
f([]string{"1.1.1.1", "2.2.2.2", "5.5.5.5"}, &trs)
// empty backends, expecting error
trs = testRemoteServer{}
f([]string{}, &trs)
}

View File

@@ -95,14 +95,14 @@ func mustOpenFastQueue(path, name string, opts OpenFastQueueOpts) *FastQueue {
}
fq.cond.L = &fq.mu
fq.lastInmemoryBlockReadTime = fasttime.UnixTimestamp()
_ = metrics.GetOrCreateGauge(fmt.Sprintf(`vm_persistentqueue_bytes_pending{path=%q}`, path), func() float64 {
_ = metrics.GetOrCreateGauge(fmt.Sprintf(`vm_persistentqueue_bytes_pending{path=%q, name=%q}`, path, name), func() float64 {
fq.mu.Lock()
n := fq.pq.GetPendingBytes()
fq.mu.Unlock()
return float64(n)
})
_ = metrics.GetOrCreateGauge(fmt.Sprintf(`vm_persistentqueue_free_disk_space_bytes{path=%q}`, path), func() float64 {
_ = metrics.GetOrCreateGauge(fmt.Sprintf(`vm_persistentqueue_free_disk_space_bytes{path=%q, name=%q}`, path, name), func() float64 {
freeSpaceBytes := fs.MustGetFreeSpace(path)
// Limited by disk space if remoteWrite.maxDiskUsagePerURL wasn't set
if maxPendingBytes == 0 {

View File

@@ -156,12 +156,12 @@ func tryOpeningQueue(path, name string, chunkFileSize, maxBlockSize, maxPendingB
q.dir = path
q.name = name
q.blocksDropped = metrics.GetOrCreateCounter(fmt.Sprintf(`vm_persistentqueue_blocks_dropped_total{path=%q}`, path))
q.bytesDropped = metrics.GetOrCreateCounter(fmt.Sprintf(`vm_persistentqueue_bytes_dropped_total{path=%q}`, path))
q.blocksWritten = metrics.GetOrCreateCounter(fmt.Sprintf(`vm_persistentqueue_blocks_written_total{path=%q}`, path))
q.bytesWritten = metrics.GetOrCreateCounter(fmt.Sprintf(`vm_persistentqueue_bytes_written_total{path=%q}`, path))
q.blocksRead = metrics.GetOrCreateCounter(fmt.Sprintf(`vm_persistentqueue_blocks_read_total{path=%q}`, path))
q.bytesRead = metrics.GetOrCreateCounter(fmt.Sprintf(`vm_persistentqueue_bytes_read_total{path=%q}`, path))
q.blocksDropped = metrics.GetOrCreateCounter(fmt.Sprintf(`vm_persistentqueue_blocks_dropped_total{path=%q, name=%q}`, path, name))
q.bytesDropped = metrics.GetOrCreateCounter(fmt.Sprintf(`vm_persistentqueue_bytes_dropped_total{path=%q, name=%q}`, path, name))
q.blocksWritten = metrics.GetOrCreateCounter(fmt.Sprintf(`vm_persistentqueue_blocks_written_total{path=%q, name=%q}`, path, name))
q.bytesWritten = metrics.GetOrCreateCounter(fmt.Sprintf(`vm_persistentqueue_bytes_written_total{path=%q, name=%q}`, path, name))
q.blocksRead = metrics.GetOrCreateCounter(fmt.Sprintf(`vm_persistentqueue_blocks_read_total{path=%q, name=%q}`, path, name))
q.bytesRead = metrics.GetOrCreateCounter(fmt.Sprintf(`vm_persistentqueue_bytes_read_total{path=%q, name=%q}`, path, name))
cleanOnError := func() {
if q.reader != nil {

View File

@@ -485,78 +485,65 @@ func TestIndexDBOpenClose(t *testing.T) {
}
func TestIndexDB(t *testing.T) {
const metricGroups = 10
timestamp := time.Now().UnixMilli()
defer testRemoveAll(t)
t.Run("serial", func(t *testing.T) {
const path = "TestIndexDB-serial"
s := MustOpenStorage(path, OpenOptions{})
ptw := s.tb.MustGetPartition(timestamp)
db := ptw.pt.idb
mns, tsids, err := testIndexDBGetOrCreateTSIDByName(db, metricGroups, timestamp)
if err != nil {
t.Fatalf("unexpected error: %s", err)
}
if err := testIndexDBCheckTSIDByName(db, mns, tsids, timestamp, false); err != nil {
t.Fatalf("unexpected error: %s", err)
}
// Re-open the storage and verify it works as expected.
s.tb.PutPartition(ptw)
s.MustClose()
s = MustOpenStorage(path, OpenOptions{})
ptw = s.tb.MustGetPartition(timestamp)
db = ptw.pt.idb
if err := testIndexDBCheckTSIDByName(db, mns, tsids, timestamp, false); err != nil {
t.Fatalf("unexpected error: %s", err)
}
s.tb.PutPartition(ptw)
s.MustClose()
fs.MustRemoveDir(path)
})
t.Run("concurrent", func(t *testing.T) {
const path = "TestIndexDB-concurrent"
s := MustOpenStorage(path, OpenOptions{})
f := func(t *testing.T, concurrency int, disablePerDayIndex bool) {
const metricGroups = 10
now := time.Now().UTC()
timestamp := now.UnixMilli()
date := uint64(timestamp / msecPerDay)
searchTR := TimeRange{
MinTimestamp: time.Date(now.Year(), now.Month(), now.Day(), 0, 0, 0, 0, time.UTC).UnixMilli(),
MaxTimestamp: time.Date(now.Year(), now.Month(), now.Day(), 23, 59, 59, 999_999_999, time.UTC).UnixMilli(),
}
if disablePerDayIndex {
searchTR = globalIndexTimeRange
}
s := MustOpenStorage(t.Name(), OpenOptions{
DisablePerDayIndex: disablePerDayIndex,
})
defer s.MustClose()
ptw := s.tb.MustGetPartition(timestamp)
defer s.tb.PutPartition(ptw)
db := ptw.pt.idb
ch := make(chan error, 3)
for range cap(ch) {
go func() {
mns, tsid, err := testIndexDBGetOrCreateTSIDByName(db, metricGroups, timestamp)
var wg sync.WaitGroup
errs := make([]error, concurrency)
isConcurrent := concurrency > 1
for i := range concurrency {
wg.Go(func() {
mns, tsids, err := testIndexDBGetOrCreateTSIDByName(db, metricGroups, date)
if err != nil {
ch <- err
errs[i] = fmt.Errorf("testIndexDBGetOrCreateTSIDByName failed unexpectedly: %w", err)
return
}
if err := testIndexDBCheckTSIDByName(db, mns, tsid, timestamp, true); err != nil {
ch <- err
return
if err := testIndexDBCheckTSIDByName(db, mns, tsids, date, searchTR, isConcurrent); err != nil {
errs[i] = fmt.Errorf("testIndexDBCheckTSIDByName failed unexpectedly: %w", err)
}
ch <- nil
}()
})
}
deadlineCh := time.After(30 * time.Second)
for range cap(ch) {
select {
case err := <-ch:
if err != nil {
t.Fatalf("unexpected error: %s", err)
}
case <-deadlineCh:
t.Fatalf("timeout")
wg.Wait()
for i, err := range errs {
if err != nil {
t.Errorf("[worker %d] %s", i, err)
}
}
}
s.tb.PutPartition(ptw)
s.MustClose()
fs.MustRemoveDir(path)
})
for _, concurrency := range []int{1, 4} {
for _, disablePerDayIndex := range []bool{false, true} {
name := fmt.Sprintf("concurrency=%d/disablePerDayIndex=%t", concurrency, disablePerDayIndex)
t.Run(name, func(t *testing.T) {
f(t, concurrency, disablePerDayIndex)
// Repeat the same test on non-empty reopened storage.
f(t, concurrency, disablePerDayIndex)
})
}
}
}
func testIndexDBGetOrCreateTSIDByName(db *indexDB, metricGroups int, timestamp int64) ([]MetricName, []TSID, error) {
func testIndexDBGetOrCreateTSIDByName(db *indexDB, metricGroups int, date uint64) ([]MetricName, []TSID, error) {
r := rand.New(rand.NewSource(1))
// Create tsids.
var mns []MetricName
@@ -564,8 +551,6 @@ func testIndexDBGetOrCreateTSIDByName(db *indexDB, metricGroups int, timestamp i
is := db.getIndexSearch(noDeadline)
date := uint64(timestamp) / msecPerDay
var metricNameBuf []byte
for i := range 401 {
var mn MetricName
@@ -602,7 +587,7 @@ func testIndexDBGetOrCreateTSIDByName(db *indexDB, metricGroups int, timestamp i
return mns, tsids, nil
}
func testIndexDBCheckTSIDByName(db *indexDB, mns []MetricName, tsids []TSID, timestamp int64, isConcurrent bool) error {
func testIndexDBCheckTSIDByName(db *indexDB, mns []MetricName, tsids []TSID, date uint64, tr TimeRange, isConcurrent bool) error {
timeseriesCounters := make(map[uint64]bool)
var tsidLocal TSID
var metricNameCopy []byte
@@ -618,7 +603,7 @@ func testIndexDBCheckTSIDByName(db *indexDB, mns []MetricName, tsids []TSID, tim
metricName := mn.Marshal(nil)
is := db.getIndexSearch(noDeadline)
if !is.getTSIDByMetricName(&tsidLocal, metricName, uint64(timestamp)/msecPerDay) {
if !is.getTSIDByMetricName(&tsidLocal, metricName, date) {
return fmt.Errorf("cannot obtain tsid #%d for mn %s", i, mn)
}
db.putIndexSearch(is)
@@ -652,7 +637,7 @@ func testIndexDBCheckTSIDByName(db *indexDB, mns []MetricName, tsids []TSID, tim
}
// Test SearchLabelValues
lvs, err := db.SearchLabelValues(nil, "__name__", nil, TimeRange{}, 1e5, 1e9, noDeadline)
lvs, err := db.SearchLabelValues(nil, "__name__", nil, tr, 1e5, 1e9, noDeadline)
if err != nil {
return fmt.Errorf("error in SearchLabelValues(labelName=%q): %w", "__name__", err)
}
@@ -661,7 +646,7 @@ func testIndexDBCheckTSIDByName(db *indexDB, mns []MetricName, tsids []TSID, tim
}
for i := range mn.Tags {
tag := &mn.Tags[i]
lvs, err := db.SearchLabelValues(nil, string(tag.Key), nil, TimeRange{}, 1e5, 1e9, noDeadline)
lvs, err := db.SearchLabelValues(nil, string(tag.Key), nil, tr, 1e5, 1e9, noDeadline)
if err != nil {
return fmt.Errorf("error in SearchLabelValues(labelName=%q): %w", tag.Key, err)
}
@@ -672,10 +657,10 @@ func testIndexDBCheckTSIDByName(db *indexDB, mns []MetricName, tsids []TSID, tim
}
}
// Test SearchLabelNames (empty filters, global time range)
lns, err := db.SearchLabelNames(nil, nil, TimeRange{}, 1e5, 1e9, noDeadline)
// Test SearchLabelNames (empty filter)
lns, err := db.SearchLabelNames(nil, nil, tr, 1e5, 1e9, noDeadline)
if err != nil {
return fmt.Errorf("error in SearchLabelNames(empty filter, global time range): %w", err)
return fmt.Errorf("error in SearchLabelNames(empty filter): %w", err)
}
if _, ok := lns["__name__"]; !ok {
return fmt.Errorf("cannot find __name__ in %q", lns)
@@ -700,10 +685,6 @@ func testIndexDBCheckTSIDByName(db *indexDB, mns []MetricName, tsids []TSID, tim
}
// Try tag filters.
tr := TimeRange{
MinTimestamp: timestamp - msecPerDay,
MaxTimestamp: timestamp + msecPerDay,
}
for i := range mns {
mn := &mns[i]
tsid := &tsids[i]

View File

@@ -55,7 +55,7 @@ type transportConfig struct {
// Registered is called by net/http.Transport.RegisterProtocol,
// to let us know that it understands the registration mechanism we're using.
func (t transportConfig) Registered(t1 *http.Transport) {
t.t.t1 = t1
t.t.lazyt1 = t1
}
func (t transportConfig) DisableCompression() bool {
@@ -145,29 +145,30 @@ func (t transportConfig) DialFromContext(ctx context.Context, network, address s
type transportInternal struct {
initOnce sync.Once
t1 *http.Transport
lazyt1 *http.Transport
}
func (t *Transport) init() {
func (t *Transport) init() *http.Transport {
t.initOnce.Do(func() {
if t.t1 != nil {
if t.lazyt1 != nil {
return
}
t1 := &http.Transport{}
t.configure(t1)
})
return t.lazyt1
}
func (t *Transport) configure(t1 *http.Transport) {
t1.RegisterProtocol("http/2", transportConfig{t})
// tr2.t1 is set by transportConfig.Registered.
if t.t1 != t1 {
// tr2.lazyt1 is set by transportConfig.Registered.
if t.lazyt1 != t1 {
panic("http2: net/http does not support this version of x/net/http2")
}
}
func (t *Transport) roundTripOpt(req *http.Request, opt RoundTripOpt) (*http.Response, error) {
t.init()
t1 := t.init()
if req.URL.Scheme == "http" && !t.AllowHTTP {
return nil, errors.New("http2: unencrypted HTTP/2 not enabled")
@@ -188,22 +189,23 @@ func (t *Transport) roundTripOpt(req *http.Request, opt RoundTripOpt) (*http.Res
ctx := context.WithValue(req.Context(), http2TransportContextKey{}, t)
req = req.WithContext(ctx)
return t.t1.RoundTrip(req)
return t1.RoundTrip(req)
}
func (t *Transport) closeIdleConnections() {
t.init()
t.t1.CloseIdleConnections()
t1 := t.init()
t1.CloseIdleConnections()
}
func (t *Transport) newUserClientConn(c net.Conn) (*ClientConn, error) {
t1 := t.init()
// http.Transport's NewClientConn doesn't provide a supported way to create
// a connection from a net.Conn. (This might be useful to add in the future?)
// We're going to craftily sneak one in via the context key, with the
// scheme of "http/2" telling NewClientConn to look for it.
ctx := context.WithValue(context.Background(), netConnContextKey{}, c)
nhcc, err := t.t1.NewClientConn(ctx, "http/2", "")
nhcc, err := t1.NewClientConn(ctx, "http/2", "")
if err != nil {
return nil, err
}

View File

@@ -400,7 +400,11 @@ func (p *Profile) process(s string, toASCII bool) (string, error) {
// Spec says keep the old label.
continue
}
if unicode16 && err == nil && len(u) > 0 && isASCII(u) {
if err == nil && len(u) > 0 && isASCII(u) {
// UTS 43 pre-revision 33 doesn't classify a xn-- label
// which contains only ASCII characters as an error,
// but that's a specification bug and a security issue.
// Always return an error in this case.
err = punyError(enc)
}
isBidi = isBidi || bidirule.DirectionString(u) != bidi.LeftToRight

View File

@@ -24,7 +24,7 @@ func NewWeighted(n int64) *Weighted {
}
// Weighted provides a way to bound concurrent access to a resource.
// The callers can request access with a given weight.
// The callers can request access with a given non-negative weight.
type Weighted struct {
size int64
cur int64
@@ -32,10 +32,13 @@ type Weighted struct {
waiters list.List
}
// Acquire acquires the semaphore with a weight of n, blocking until resources
// Acquire acquires the semaphore with a non-negative weight of n, blocking until resources
// are available or ctx is done. On success, returns nil. On failure, returns
// ctx.Err() and leaves the semaphore unchanged.
func (s *Weighted) Acquire(ctx context.Context, n int64) error {
if n < 0 {
panic("semaphore: n < 0")
}
done := ctx.Done()
s.mu.Lock()
@@ -106,9 +109,12 @@ func (s *Weighted) Acquire(ctx context.Context, n int64) error {
}
}
// TryAcquire acquires the semaphore with a weight of n without blocking.
// TryAcquire acquires the semaphore with a non-negative weight of n without blocking.
// On success, returns true. On failure, returns false and leaves the semaphore unchanged.
func (s *Weighted) TryAcquire(n int64) bool {
if n < 0 {
panic("semaphore: n < 0")
}
s.mu.Lock()
success := s.size-s.cur >= n && s.waiters.Len() == 0
if success {
@@ -118,8 +124,11 @@ func (s *Weighted) TryAcquire(n int64) bool {
return success
}
// Release releases the semaphore with a weight of n.
// Release releases the semaphore with a non-negative weight of n.
func (s *Weighted) Release(n int64) {
if n < 0 {
panic("semaphore: n < 0")
}
s.mu.Lock()
s.cur -= n
if s.cur < 0 {

View File

@@ -121,8 +121,12 @@ func (p Properties) BoundaryAfter() bool {
//
// When all 6 bits are zero, the character is inert, meaning it is never
// influenced by normalization.
//
// We set flags to 0x80 (high bit 7 unused in quick check data) to indicate an invalid rune.
type qcInfo uint8
func (p Properties) isInvalid() bool { return p.flags == 0x80 }
func (p Properties) isYesC() bool { return p.flags&0x10 == 0 }
func (p Properties) isYesD() bool { return p.flags&0x4 == 0 }
@@ -247,6 +251,9 @@ func (f Form) PropertiesString(s string) Properties {
// to a Properties. See the comment at the top of the file
// for more information on the format.
func compInfo(v uint16, sz int) Properties {
if sz == 0 {
return Properties{flags: 0x80, size: 1}
}
if v == 0 {
return Properties{size: uint8(sz)}
} else if v >= 0x8000 {
@@ -254,7 +261,7 @@ func compInfo(v uint16, sz int) Properties {
size: uint8(sz),
ccc: uint8(v),
tccc: uint8(v),
flags: qcInfo(v >> 8),
flags: qcInfo(v>>8) & 0x3f,
}
if p.ccc > 0 || p.combinesBackward() {
p.nLead = uint8(p.flags & 0x3)

View File

@@ -376,16 +376,12 @@ func nextComposed(i *Iter) []byte {
goto doNorm
}
prevCC = i.info.tccc
sz := int(i.info.size)
if sz == 0 {
sz = 1 // illegal rune: copy byte-by-byte
}
p := outp + sz
p := outp + int(i.info.size)
if p > len(i.buf) {
break
}
outp = p
i.p += sz
i.p += int(i.info.size)
if i.p >= i.rb.nsrc {
i.setDone()
break

View File

@@ -148,7 +148,7 @@ func (f Form) IsNormalString(s string) bool {
// patched buffer and whether the decomposition is still in progress.
func patchTail(rb *reorderBuffer) bool {
info, p := lastRuneStart(&rb.f, rb.out)
if p == -1 || info.size == 0 {
if p == -1 || info.isInvalid() {
return true
}
end := p + int(info.size)
@@ -225,7 +225,7 @@ func doAppend(rb *reorderBuffer, out []byte, p int) []byte {
}
fd := &rb.f
if doMerge {
var info Properties
info := Properties{flags: 0x80, size: 1} // invalid rune
if p < n {
info = fd.info(src, p)
if !info.BoundaryBefore() || info.nLeadingNonStarters() > 0 {
@@ -235,7 +235,7 @@ func doAppend(rb *reorderBuffer, out []byte, p int) []byte {
p = decomposeSegment(rb, p, true)
}
}
if info.size == 0 {
if info.isInvalid() {
rb.doFlush()
// Append incomplete UTF-8 encoding.
return src.appendSlice(rb.out, p, n)
@@ -314,7 +314,7 @@ func (f *formInfo) quickSpan(src input, i, end int, atEOF bool) (n int, ok bool)
continue
}
info := f.info(src, i)
if info.size == 0 {
if info.isInvalid() {
if atEOF {
// include incomplete runes
return n, true
@@ -379,7 +379,7 @@ func (f Form) firstBoundary(src input, nsrc int) int {
// CGJ insertion points correctly. Luckily it doesn't have to.
for {
info := fd.info(src, i)
if info.size == 0 {
if info.isInvalid() {
return -1
}
if s := ss.next(info); s != ssSuccess {
@@ -424,7 +424,7 @@ func (f Form) nextBoundary(src input, nsrc int, atEOF bool) int {
}
fd := formTable[f]
info := fd.info(src, 0)
if info.size == 0 {
if info.isInvalid() {
if atEOF {
return 1
}
@@ -435,7 +435,7 @@ func (f Form) nextBoundary(src input, nsrc int, atEOF bool) int {
for i := int(info.size); i < nsrc; i += int(info.size) {
info = fd.info(src, i)
if info.size == 0 {
if info.isInvalid() {
if atEOF {
return i
}
@@ -465,7 +465,7 @@ func lastBoundary(fd *formInfo, b []byte) int {
if p == -1 {
return -1
}
if info.size == 0 { // ends with incomplete rune
if info.isInvalid() { // ends with incomplete rune
if p == 0 { // starts with incomplete rune
return -1
}
@@ -504,7 +504,7 @@ func lastBoundary(fd *formInfo, b []byte) int {
func decomposeSegment(rb *reorderBuffer, sp int, atEOF bool) int {
// Force one character to be consumed.
info := rb.f.info(rb.src, sp)
if info.size == 0 {
if info.isInvalid() {
return 0
}
if s := rb.ss.next(info); s == ssStarter {
@@ -528,7 +528,7 @@ func decomposeSegment(rb *reorderBuffer, sp int, atEOF bool) int {
break
}
info = rb.f.info(rb.src, sp)
if info.size == 0 {
if info.isInvalid() {
if !atEOF {
return int(iShortSrc)
}

28
vendor/modules.txt vendored
View File

@@ -721,6 +721,8 @@ github.com/x448/float16
# github.com/xrash/smetrics v0.0.0-20250705151800-55b8f293f342
## explicit; go 1.15
github.com/xrash/smetrics
# github.com/yuin/goldmark v1.8.4
## explicit; go 1.22
# go.opentelemetry.io/auto/sdk v1.2.1
## explicit; go 1.24.0
go.opentelemetry.io/auto/sdk
@@ -858,7 +860,7 @@ go.yaml.in/yaml/v2
# go.yaml.in/yaml/v3 v3.0.4
## explicit; go 1.16
go.yaml.in/yaml/v3
# golang.org/x/crypto v0.53.0
# golang.org/x/crypto v0.54.0
## explicit; go 1.25.0
golang.org/x/crypto/chacha20
golang.org/x/crypto/chacha20poly1305
@@ -869,10 +871,12 @@ golang.org/x/crypto/internal/alias
golang.org/x/crypto/internal/poly1305
golang.org/x/crypto/pkcs12
golang.org/x/crypto/pkcs12/internal/rc2
# golang.org/x/exp v0.0.0-20260611194520-c48552f49976
# golang.org/x/exp v0.0.0-20260718201538-764159d718ef
## explicit; go 1.25.0
golang.org/x/exp/constraints
# golang.org/x/net v0.56.0
# golang.org/x/mod v0.38.0
## explicit; go 1.25.0
# golang.org/x/net v0.57.0
## explicit; go 1.25.0
golang.org/x/net/http/httpguts
golang.org/x/net/http/httpproxy
@@ -898,7 +902,7 @@ golang.org/x/oauth2/google/internal/stsexchange
golang.org/x/oauth2/internal
golang.org/x/oauth2/jws
golang.org/x/oauth2/jwt
# golang.org/x/sync v0.21.0
# golang.org/x/sync v0.22.0
## explicit; go 1.25.0
golang.org/x/sync/errgroup
golang.org/x/sync/semaphore
@@ -909,10 +913,12 @@ golang.org/x/sys/plan9
golang.org/x/sys/unix
golang.org/x/sys/windows
golang.org/x/sys/windows/registry
# golang.org/x/term v0.44.0
# golang.org/x/telemetry v0.0.0-20260717140457-bdb89881bb75
## explicit; go 1.25.0
# golang.org/x/term v0.45.0
## explicit; go 1.25.0
golang.org/x/term
# golang.org/x/text v0.38.0
# golang.org/x/text v0.40.0
## explicit; go 1.25.0
golang.org/x/text/secure/bidirule
golang.org/x/text/transform
@@ -921,6 +927,16 @@ golang.org/x/text/unicode/norm
# golang.org/x/time v0.15.0
## explicit; go 1.25.0
golang.org/x/time/rate
# golang.org/x/tools v0.48.0
## explicit; go 1.25.0
# golang.org/x/tools/go/expect v0.1.1-deprecated
## explicit; go 1.23.0
# golang.org/x/tools/go/packages/packagestest v0.1.1-deprecated
## explicit; go 1.23.0
# golang.org/x/tools/godoc v0.1.0-deprecated
## explicit; go 1.24.0
# golang.org/x/xerrors v0.0.0-20240903120638-7835f813f4da
## explicit; go 1.18
# google.golang.org/api v0.284.0
## explicit; go 1.25.8
google.golang.org/api/googleapi