Compare commits

..

30 Commits

Author SHA1 Message Date
Max Kotliar
679646a3b3 docs: upd vmestimator docs 2026-06-26 19:01:40 +03:00
Max Kotliar
bb3c038e2f docs: follow-up on 29af0809; fix vmestimator image oversize 2026-06-26 18:48:46 +03:00
Max Kotliar
8f32b6648f docs: follow-up on 29af0809
Attempt to fix image oversize in vmestimator doc
2026-06-26 18:36:12 +03:00
Max Kotliar
df5f11623f docs: publish vmestimator docs
I tried to adopt publish-docs CI job. But it's not possible to add a
content to docs/victoriametrics. CI job from vmestimator overwrite the
whole content.

That's why I'll be copy pasting doc updates from vmestimator repo to VM
manually.
2026-06-26 18:23:29 +03:00
Roman Khavronenko
6c8a41f5ed docs: add missing mentions of new SA output sum_samples_total (#11178)
Follow-up after
16422b2d14

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Signed-off-by: Roman Khavronenko <hagen1778@gmail.com>
Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>
2026-06-26 16:49:43 +02:00
Roman Khavronenko
e749a6ce8d docs: update security recommendations (#11136)
- move SBOM page down as general security recommendations must be
mentioned first
- separate httplib security-related flags to a separate section, so it
can be cross-referenced by products that use this lib
- mention that /metrics might require to be protected

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Signed-off-by: Pablo (Tomas) Fernandez <46322567+TomFern@users.noreply.github.com>
Co-authored-by: Pablo Fernandez <46322567+TomFern@users.noreply.github.com>
Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>
2026-06-26 16:49:14 +02:00
Hui Wang
615176ad55 doc: correct sharding configuration guidance for stream aggregation (#11167)
Rewrite the guidance for `-remoteWrite.shardByURL.labels` and
`-remoteWrite.shardByURL.ignoreLabels` from the perspective of the
aggregation config, rather than the other way around, since users
typically define aggregation rules first and then configure sharding.

And the previous statements are wrong.
For example, I have 3 raw series
```
{env=1, pod=1, cluster=1}
{env=1, pod=2, cluster=1}
{env=2, pod=1, cluster=2}
```

If I have `by: [env]`, then `-remoteWrite.shardByURL.labels` cannot be
set to `env,pod`, because that could route `{env=1, pod=1, cluster=1}`
and `{env=1, pod=2, cluster=1}` to different aggregators, causing each
aggregator to produce its own partial result for `env=1`.
In this case, `-remoteWrite.shardByURL.labels` can only use labels that
are a subset of `by` as `env`.

If I have `without: [env, pod]`, then
`-remoteWrite.shardByURL.ignoreLabels` cannot be set to just `env`,
because that would still allow `{env=1, pod=1, cluster=1}` and `{env=1,
pod=2, cluster=1}` to be routed to different aggregators, causing each
aggregator to produce its own partial result for `cluster=1`.
In this case, `-remoteWrite.shardByURL.ignoreLabels` must include all
labels listed in `without`as `env,pod`.
2026-06-26 16:48:43 +02:00
Cuong Le
3aec167f00 docs/victoriametrics: add an AI policy section to the contributing guide (#11170)
This PR adds a short `AI policy` section to the contributing guide.

We've received quite a few raw AI-generated PRs on VictoriaLogs
recently, so we'd like to clarify our view on AI usage. In short:

1. You can use AI.
2. You are responsible for what you submit, AI or not.
3. Raw AI slop will be closed without review.
2026-06-26 16:48:31 +02:00
Roman Khavronenko
6f633e5654 docs: update vmagent's multitenancy doc based on user's feedback (#11173)
* mention actual api path for opentlemetry in HowToPush docs for
consistency with other protocols;
* update vmagent=>vminsert remoteWrite.url path to prometheus-rw, as it
is the only one that is supported in vmagent/vminsert communication;
* add links to diagrams, so users could easier get to the corresponding
docs;
* add some visual notes to increase awarness of missconfigs

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Pablo Fernandez <46322567+TomFern@users.noreply.github.com>
2026-06-26 16:48:09 +02:00
Fred Navruzov
50a827256a docs/vmanomaly: fill in missing args and links (post v1.29.7 update) (#11165)
Addition of missing links/args and slight refactor of changelog notes
for clarity (post v1.29.7 update)

Follow-up on e30e8be1f4
2026-06-25 10:27:18 +03:00
Fred Navruzov
e30e8be1f4 docs/vmanomaly: update to v1.29.7 (#11163)
Update anomaly detection docs to align with `v1.29.7` release

---------

Signed-off-by: Fred Navruzov <fred-navruzov@users.noreply.github.com>
2026-06-24 21:22:58 +03:00
f41gh7
24ac567a9f fixes tests after dce8193c16 2026-06-24 17:13:04 +02:00
Aliaksandr Valialkin
dce8193c16 vendor: update github.com/VictoriaMetrics/VictoriaLogs from v1.121.1-0.20260616132739-c901a1e31cb3 to v1.51.1-0.20260624061259-dc94972a8708 2026-06-24 09:40:23 +02:00
Immanuel Tikhonov
e196479fb2 apptest: fix invalid go test command in README (#1484)
apptest/README.md points to `go test ./app/apptest`, but that dir is not
in this repo. Fresh clone, instant fail, kinda rough.

This switches it to `go test ./apptest/...` and adds a short note that
`go test ./...` needs the test binaries built first.

How to repro:
1. Fresh clone the repo.
2. Run `go test ./app/apptest`.
3. See `stat .../app/apptest: directory not found`.

---------

Signed-off-by: immanuwell <pchpr.00@list.ru>
Signed-off-by: Immanuel Tikhonov <122638311+immanuwell@users.noreply.github.com>
Co-authored-by: Phuong Le <39565248+func25@users.noreply.github.com>
2026-06-24 08:09:16 +02:00
Hui Wang
1c774564a2 stream aggregation: improve rate_xx results with out of order samples (#11140)
Previously, there could be an unexpected increase in `rate` if an
out-of-order sample was ingested after the previous flush.
Before:
<img width="1503" height="835" alt="image"
src="https://github.com/user-attachments/assets/3fb05f3e-51af-4c5c-989a-ec3da089fc23"
/>
After:
<img width="2546" height="924" alt="image"
src="https://github.com/user-attachments/assets/db8d8b0e-bbc0-4927-947a-713fc1fb4c5f"
/>

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2026-06-23 12:46:50 +02:00
Max Kotliar
e1c554d4a6 docs/changelog: follow-up on 3419328f
Follow-up on
3419328f1c
2026-06-23 11:48:23 +03:00
Zhu Jiekun
3419328f1c app/vmselect: propagate cache reset operation to selectNode (#11118)
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/11112
PR https://github.com/VictoriaMetrics/VictoriaMetrics/pull/11118

Propagate cache reset operation to `-selectNode` when `/internal/resetRollupResultCache` is called. Previously, the propagation only happened when `delete_series` API was called.

To avoid an infinite loop, the propagation happens only when `propagate=1` query arg is set.

Note that if `-search.resetCacheAuthKey` is configured. It will be used as `authKey` query arg while propagating requests to other select instances.

---------

Signed-off-by: Zhu Jiekun <jiekun@victoriametrics.com>
Co-authored-by: Max Kotliar <mkotlyar@victoriametrics.com>
2026-06-23 11:17:02 +03:00
Max Kotliar
e841e45877 dashboards: fix typo "Sey major ..." -> "See major ..."
The credit goes to @marco-m who proposed the change in
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/11126.
2026-06-23 11:01:20 +03:00
Max Kotliar
d53d8849e7 lib/promrelabel: hide relabel debug cross-link when no target id is present (#11143)
The cross-link between `/metric-relabel-debug` and `/target-relabel-debug` pages has always been rendered, regardless of whether a target `id` query param was present or not. The target ID is responsible for preloading either metrics or the target relabeling config.

The commit hides the cross-link when no target ID is present. Without the ID, pages are essentially the same, so there is no need to link them. It should reduce user confusion.
2026-06-23 10:53:44 +03:00
Max Kotliar
d3641394d9 app/vmctl: fix edit link in vmctl's README 2026-06-22 18:01:23 +03:00
Pablo (Tomas) Fernandez
53a8f4bd47 docs: add docs-check-links target to docs Makefile (#11121)
We have a `check-links` target in the vmdocs repository. This is a good
start but it only detects broken links after changes in docs are merged
into master in this repository.

It would be nice to have a way to check the documentation in this
repository (before creating a PR for instance). This PR introduces a
`docs-check-links` target to the docs/Makefile. This lets us see if we
have any new broken links in the current branch/PR before merging into
main.

Pros:
- We can preview changes in docs before merging to master
- No new tools/big changes required, only add a new target to the
Makefile
- Once we fix all links, we could add a GH Actions check

Cons:
- You need access to the vmdocs repo for this target to work
- Running the check in GH Actions could be potentially complex because
it needs to clone vmdocs, build the container, etc

If this seems like a good idea, I could add the same check to the other
repos (VL, VT, etc).

PR https://github.com/VictoriaMetrics/VictoriaMetrics/pull/11121
2026-06-22 16:10:30 +03:00
Max Kotliar
2b256952c9 go.mod: update metricsql pacakge
v0.87.2 reverts range() function support (PR:
https://github.com/VictoriaMetrics/metricsql/pull/76)

Reason given in
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/11028#issuecomment-4728207220
2026-06-22 16:06:51 +03:00
Aliaksandr Valialkin
12086e75de docs/victoriametrics/goals.md: typo fix: replace the dot with a comma 2026-06-22 13:54:40 +02:00
f41gh7
d426575622 docs: update flags with actual v1.146.0 binaries
Signed-off-by: f41gh7 <nik@victoriametrics.com>
2026-06-22 13:12:38 +02:00
f41gh7
a76b1ce0e3 docs: bump version to v1.146.0
Signed-off-by: f41gh7 <nik@victoriametrics.com>
2026-06-22 13:10:24 +02:00
f41gh7
5f49fb7f31 deplyoment/docker: bump version to v1.146.0
Signed-off-by: f41gh7 <nik@victoriametrics.com>
2026-06-22 13:09:55 +02:00
f41gh7
80d1104fca update changelog
Signed-off-by: f41gh7 <nik@victoriametrics.com>
2026-06-22 13:08:23 +02:00
f41gh7
ae59c2624c docs: forward port LTS v1.122.25 changelog to upstream
Signed-off-by: f41gh7 <nik@victoriametrics.com>
2026-06-22 13:06:02 +02:00
f41gh7
4661f69d9f docs: forward port LTS v1.136.12 changelog to upstream
Signed-off-by: f41gh7 <nik@victoriametrics.com>
2026-06-22 13:05:43 +02:00
f41gh7
4d9901fbf4 docs/changelog: cut release v1.146.0
Signed-off-by: f41gh7 <nik@victoriametrics.com>
2026-06-19 14:19:09 +02:00
108 changed files with 1942 additions and 1046 deletions

View File

@@ -457,12 +457,10 @@ func TestSetIntervalAsTimeFilter(t *testing.T) {
f(`* | count()`, "vlogs", true)
f(`error OR _time:5m | count()`, "vlogs", true)
f(`(_time: 5m AND error) OR (_time: 5m AND warn) | count()`, "vlogs", true)
f(`* | error OR _time:5m | count()`, "vlogs", true)
f(`_time:5m | count()`, "vlogs", false)
f(`_time:2023-04-25T22:45:59Z | count()`, "vlogs", false)
f(`error AND _time:5m | count()`, "vlogs", false)
f(`* | error AND _time:5m | count()`, "vlogs", false)
}
func TestRecordingRuleExec_Partial(t *testing.T) {

View File

@@ -1,3 +1,3 @@
See vmctl docs [here](https://docs.victoriametrics.com/victoriametrics/vmctl/).
vmctl docs can be edited at [docs/vmctl.md](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/docs/victoriametrics/vmctl.md).
vmctl docs can be edited at [docs/vmctl.md](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/docs/victoriametrics/vmctl/vmctl.md).

View File

@@ -4842,137 +4842,6 @@ func TestExecSuccess(t *testing.T) {
resultExpected := []netstorage.Result{}
f(q, resultExpected)
})
// buckets that are consecutively empty at left and right ends will not be preserved.
t.Run(`buckets_limit(trim_zero_preserve_empty_when_limit_not_reached)`, func(t *testing.T) {
t.Parallel()
q := `sort(buckets_limit(3, (
alias(label_set(36, "le", "+Inf"), "metric"),
alias(label_set(36, "le", "25"), "metric"),
alias(label_set(36, "le", "21"), "metric"),
alias(label_set(36, "le", "19"), "metric"),
alias(label_set(36, "le", "18"), "metric"),
alias(label_set(36, "le", "17"), "metric"),
alias(label_set(36, "le", "16"), "metric"),
alias(label_set(27, "le", "12"), "metric"),
alias(label_set(14, "le", "9"), "metric"),
alias(label_set(0, "le", "6"), "metric"),
alias(label_set(0, "le", "1"), "metric"),
)))`
r1 := netstorage.Result{
MetricName: metricNameExpected,
Values: []float64{14, 14, 14, 14, 14, 14},
Timestamps: timestampsExpected,
}
r1.MetricName.MetricGroup = []byte("metric")
r1.MetricName.Tags = []storage.Tag{
{
Key: []byte("le"),
Value: []byte("9"),
},
}
r2 := netstorage.Result{
MetricName: metricNameExpected,
Values: []float64{27, 27, 27, 27, 27, 27},
Timestamps: timestampsExpected,
}
r2.MetricName.MetricGroup = []byte("metric")
r2.MetricName.Tags = []storage.Tag{
{
Key: []byte("le"),
Value: []byte("12"),
},
}
r3 := netstorage.Result{
MetricName: metricNameExpected,
Values: []float64{36, 36, 36, 36, 36, 36},
Timestamps: timestampsExpected,
}
r3.MetricName.MetricGroup = []byte("metric")
r3.MetricName.Tags = []storage.Tag{
{
Key: []byte("le"),
Value: []byte("16"),
},
}
resultExpected := []netstorage.Result{r1, r2, r3}
f(q, resultExpected)
})
// the number of non-empty bucket doesn't reach the given "limit", so some empty buckets will be preserved
t.Run(`buckets_limit(trim_zero)`, func(t *testing.T) {
t.Parallel()
q := `sort(buckets_limit(5, (
alias(label_set(36, "le", "18"), "metric"),
alias(label_set(36, "le", "17"), "metric"),
alias(label_set(36, "le", "16"), "metric"),
alias(label_set(27, "le", "12"), "metric"),
alias(label_set(14, "le", "9"), "metric"),
alias(label_set(0, "le", "6"), "metric"),
alias(label_set(0, "le", "1"), "metric"),
)))`
r1 := netstorage.Result{
MetricName: metricNameExpected,
Values: []float64{0, 0, 0, 0, 0, 0},
Timestamps: timestampsExpected,
}
r1.MetricName.MetricGroup = []byte("metric")
r1.MetricName.Tags = []storage.Tag{
{
Key: []byte("le"),
Value: []byte("6"),
},
}
r2 := netstorage.Result{
MetricName: metricNameExpected,
Values: []float64{14, 14, 14, 14, 14, 14},
Timestamps: timestampsExpected,
}
r2.MetricName.MetricGroup = []byte("metric")
r2.MetricName.Tags = []storage.Tag{
{
Key: []byte("le"),
Value: []byte("9"),
},
}
r3 := netstorage.Result{
MetricName: metricNameExpected,
Values: []float64{27, 27, 27, 27, 27, 27},
Timestamps: timestampsExpected,
}
r3.MetricName.MetricGroup = []byte("metric")
r3.MetricName.Tags = []storage.Tag{
{
Key: []byte("le"),
Value: []byte("12"),
},
}
r4 := netstorage.Result{
MetricName: metricNameExpected,
Values: []float64{36, 36, 36, 36, 36, 36},
Timestamps: timestampsExpected,
}
r4.MetricName.MetricGroup = []byte("metric")
r4.MetricName.Tags = []storage.Tag{
{
Key: []byte("le"),
Value: []byte("16"),
},
}
r5 := netstorage.Result{
MetricName: metricNameExpected,
Values: []float64{36, 36, 36, 36, 36, 36},
Timestamps: timestampsExpected,
}
r5.MetricName.MetricGroup = []byte("metric")
r5.MetricName.Tags = []storage.Tag{
{
Key: []byte("le"),
Value: []byte("17"),
},
}
resultExpected := []netstorage.Result{r1, r2, r3, r4, r5}
f(q, resultExpected)
})
t.Run(`buckets_limit(unused)`, func(t *testing.T) {
t.Parallel()
q := `sort(buckets_limit(5, (
@@ -6359,6 +6228,50 @@ func TestExecSuccess(t *testing.T) {
resultExpected := []netstorage.Result{r1, r2, r3, r4, r5, r6, r7}
f(q, resultExpected)
})
t.Run(`sum(histogram_over_time) by (vmrange)`, func(t *testing.T) {
t.Parallel()
q := `sort_by_label(
buckets_limit(
3,
sum(histogram_over_time(alias(label_set(rand(0)*1.3+1.1, "foo", "bar"), "xxx")[200s:5s])) by (vmrange)
), "le"
)`
r1 := netstorage.Result{
MetricName: metricNameExpected,
Values: []float64{40, 40, 40, 40, 40, 40},
Timestamps: timestampsExpected,
}
r1.MetricName.Tags = []storage.Tag{
{
Key: []byte("le"),
Value: []byte("+Inf"),
},
}
r2 := netstorage.Result{
MetricName: metricNameExpected,
Values: []float64{0, 0, 0, 0, 0, 0},
Timestamps: timestampsExpected,
}
r2.MetricName.Tags = []storage.Tag{
{
Key: []byte("le"),
Value: []byte("1.000e+00"),
},
}
r3 := netstorage.Result{
MetricName: metricNameExpected,
Values: []float64{40, 40, 40, 40, 40, 40},
Timestamps: timestampsExpected,
}
r3.MetricName.Tags = []storage.Tag{
{
Key: []byte("le"),
Value: []byte("2.448e+00"),
},
}
resultExpected := []netstorage.Result{r1, r2, r3}
f(q, resultExpected)
})
t.Run(`sum(histogram_over_time)`, func(t *testing.T) {
t.Parallel()
q := `sum(histogram_over_time(alias(label_set(rand(0)*1.3+1.1, "foo", "bar"), "xxx")[200s:5s]))`

View File

@@ -461,41 +461,6 @@ func transformBucketsLimit(tfa *transformFuncArg) ([]*timeseries, error) {
prevValue = value
}
}
// Remove buckets that are consecutively empty at left and right ends to obtain more accurate max and min values.
// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/10417.
epsilon := 1e-9
l := 0
r := len(leGroup) - 1
trimLeft := true
for r-l+1 > limit {
leftHits := math.Abs(leGroup[l].hits)
rightHits := math.Abs(leGroup[r].hits)
leftEmpty := !math.IsNaN(leftHits) && leftHits < epsilon
rightEmpty := !math.IsNaN(rightHits) && rightHits < epsilon
if !leftEmpty && !rightEmpty {
break
}
if trimLeft {
if leftHits < epsilon {
l++
}
// switch the trim pointer to the right side if needed
if rightHits < epsilon {
trimLeft = false
}
} else {
if rightHits < epsilon {
r--
}
// switch the trim pointer to the left side if needed
if leftHits < epsilon {
trimLeft = true
}
}
}
leGroup = leGroup[l : r+1]
for len(leGroup) > limit {
// Preserve the first and the last bucket for better accuracy for min and max values
xxMinIdx := 1

View File

@@ -4,7 +4,7 @@ The `apptest` package contains the integration tests for the VictoriaMetrics
applications (such as vmstorage, vminsert, and vmselect).
An integration test aims at verifying the behavior of an application as a whole,
as apposed to a unit test that verifies the behavior of a building block of an
as opposed to a unit test that verifies the behavior of a building block of an
application.
To achieve that an integration test starts an application in a separate process
@@ -19,10 +19,10 @@ work together as a system.
The package provides a collection of helpers to start applications and make
queries to them:
- `app.go` - contains the generic code for staring an application and should
- `app.go` - contains the generic code for starting an application and should
not be used by integration tests directly.
- `{vmstorage,vminsert,etc}.go` - build on top of `app.go` and provide the
code for staring a specific application.
code for starting a specific application.
- `client.go` - provides helper functions for sending HTTP requests to
applications.
@@ -36,7 +36,7 @@ the application binary files to be built and put into the `bin` directory. The
build rule used for running integration tests, `make apptest`,
accounts for that, it builds all application binaries before running the tests.
But if you want to run the tests without `make`, i.e. by executing
`go test ./app/apptest`, you will need to build the binaries first (for example,
`go test ./apptest/tests`, you will need to build the binaries first (for example,
by executing `make all`).
Not all binaries can be built from `master` branch, cluster binaries can be built

View File

@@ -2083,7 +2083,7 @@
"type": "prometheus",
"uid": "$ds"
},
"description": "Shows memory pressure based on [Pressure Stall Information](https://docs.kernel.org/accounting/psi.html).\n\n**Lower is better.**\n\nPressure is measured as amount of time within 1sec time window the process was:\n- waiting: at least one thread was blocked on memory.\n- stalled: every thread was blocked on memory (severe pressure).\n\nElevated memory pressure can slowdown the process performance by utilizing more disk IO. Consider increasing amount of available RAM limit or decreasing the load on the process.\n\nSeу major page faults rate panel in Troubleshooting section if this metric continued to be high.",
"description": "Shows memory pressure based on [Pressure Stall Information](https://docs.kernel.org/accounting/psi.html).\n\n**Lower is better.**\n\nPressure is measured as amount of time within 1sec time window the process was:\n- waiting: at least one thread was blocked on memory.\n- stalled: every thread was blocked on memory (severe pressure).\n\nElevated memory pressure can slowdown the process performance by utilizing more disk IO. Consider increasing amount of available RAM limit or decreasing the load on the process.\n\nSee major page faults rate panel in Troubleshooting section if this metric continued to be high.",
"fieldConfig": {
"defaults": {
"color": {

View File

@@ -2388,7 +2388,7 @@
"type": "prometheus",
"uid": "$ds"
},
"description": "Shows memory pressure based on [Pressure Stall Information](https://docs.kernel.org/accounting/psi.html).\n\n**Lower is better.**\n\nPressure is measured as amount of time within 1sec time window the process was:\n- waiting: at least one thread was blocked on memory.\n- stalled: every thread was blocked on memory (severe pressure).\n\nElevated memory pressure can slowdown the process performance by utilizing more disk IO. Consider increasing amount of available RAM limit or decreasing the load on the process.\n\nSeу major page faults rate panel in Troubleshooting section if this metric continued to be high.",
"description": "Shows memory pressure based on [Pressure Stall Information](https://docs.kernel.org/accounting/psi.html).\n\n**Lower is better.**\n\nPressure is measured as amount of time within 1sec time window the process was:\n- waiting: at least one thread was blocked on memory.\n- stalled: every thread was blocked on memory (severe pressure).\n\nElevated memory pressure can slowdown the process performance by utilizing more disk IO. Consider increasing amount of available RAM limit or decreasing the load on the process.\n\nSee major page faults rate panel in Troubleshooting section if this metric continued to be high.",
"fieldConfig": {
"defaults": {
"color": {

View File

@@ -2084,7 +2084,7 @@
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Shows memory pressure based on [Pressure Stall Information](https://docs.kernel.org/accounting/psi.html).\n\n**Lower is better.**\n\nPressure is measured as amount of time within 1sec time window the process was:\n- waiting: at least one thread was blocked on memory.\n- stalled: every thread was blocked on memory (severe pressure).\n\nElevated memory pressure can slowdown the process performance by utilizing more disk IO. Consider increasing amount of available RAM limit or decreasing the load on the process.\n\nSeу major page faults rate panel in Troubleshooting section if this metric continued to be high.",
"description": "Shows memory pressure based on [Pressure Stall Information](https://docs.kernel.org/accounting/psi.html).\n\n**Lower is better.**\n\nPressure is measured as amount of time within 1sec time window the process was:\n- waiting: at least one thread was blocked on memory.\n- stalled: every thread was blocked on memory (severe pressure).\n\nElevated memory pressure can slowdown the process performance by utilizing more disk IO. Consider increasing amount of available RAM limit or decreasing the load on the process.\n\nSee major page faults rate panel in Troubleshooting section if this metric continued to be high.",
"fieldConfig": {
"defaults": {
"color": {

View File

@@ -2389,7 +2389,7 @@
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Shows memory pressure based on [Pressure Stall Information](https://docs.kernel.org/accounting/psi.html).\n\n**Lower is better.**\n\nPressure is measured as amount of time within 1sec time window the process was:\n- waiting: at least one thread was blocked on memory.\n- stalled: every thread was blocked on memory (severe pressure).\n\nElevated memory pressure can slowdown the process performance by utilizing more disk IO. Consider increasing amount of available RAM limit or decreasing the load on the process.\n\nSeу major page faults rate panel in Troubleshooting section if this metric continued to be high.",
"description": "Shows memory pressure based on [Pressure Stall Information](https://docs.kernel.org/accounting/psi.html).\n\n**Lower is better.**\n\nPressure is measured as amount of time within 1sec time window the process was:\n- waiting: at least one thread was blocked on memory.\n- stalled: every thread was blocked on memory (severe pressure).\n\nElevated memory pressure can slowdown the process performance by utilizing more disk IO. Consider increasing amount of available RAM limit or decreasing the load on the process.\n\nSee major page faults rate panel in Troubleshooting section if this metric continued to be high.",
"fieldConfig": {
"defaults": {
"color": {

View File

@@ -2165,7 +2165,7 @@
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Shows memory pressure based on [Pressure Stall Information](https://docs.kernel.org/accounting/psi.html).\n\n**Lower is better.**\n\nPressure is measured as amount of time within 1sec time window the process was:\n- waiting: at least one thread was blocked on memory.\n- stalled: every thread was blocked on memory (severe pressure).\n\nElevated memory pressure can slowdown the process performance by utilizing more disk IO. Consider increasing amount of available RAM limit or decreasing the load on the process.\n\nSeу major page faults rate panel in Troubleshooting section if this metric continued to be high.",
"description": "Shows memory pressure based on [Pressure Stall Information](https://docs.kernel.org/accounting/psi.html).\n\n**Lower is better.**\n\nPressure is measured as amount of time within 1sec time window the process was:\n- waiting: at least one thread was blocked on memory.\n- stalled: every thread was blocked on memory (severe pressure).\n\nElevated memory pressure can slowdown the process performance by utilizing more disk IO. Consider increasing amount of available RAM limit or decreasing the load on the process.\n\nSee major page faults rate panel in Troubleshooting section if this metric continued to be high.",
"fieldConfig": {
"defaults": {
"color": {

View File

@@ -1840,7 +1840,7 @@
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"description": "Shows memory pressure based on [Pressure Stall Information](https://docs.kernel.org/accounting/psi.html).\n\n**Lower is better.**\n\nPressure is measured as amount of time within 1sec time window the process was:\n- waiting: at least one thread was blocked on memory.\n- stalled: every thread was blocked on memory (severe pressure).\n\nElevated memory pressure can slowdown the process performance by utilizing more disk IO. Consider increasing amount of available RAM limit or decreasing the load on the process.\n\nSeу major page faults rate panel in Troubleshooting section if this metric continued to be high.",
"description": "Shows memory pressure based on [Pressure Stall Information](https://docs.kernel.org/accounting/psi.html).\n\n**Lower is better.**\n\nPressure is measured as amount of time within 1sec time window the process was:\n- waiting: at least one thread was blocked on memory.\n- stalled: every thread was blocked on memory (severe pressure).\n\nElevated memory pressure can slowdown the process performance by utilizing more disk IO. Consider increasing amount of available RAM limit or decreasing the load on the process.\n\nSee major page faults rate panel in Troubleshooting section if this metric continued to be high.",
"fieldConfig": {
"defaults": {
"color": {

View File

@@ -2164,7 +2164,7 @@
"type": "prometheus",
"uid": "$ds"
},
"description": "Shows memory pressure based on [Pressure Stall Information](https://docs.kernel.org/accounting/psi.html).\n\n**Lower is better.**\n\nPressure is measured as amount of time within 1sec time window the process was:\n- waiting: at least one thread was blocked on memory.\n- stalled: every thread was blocked on memory (severe pressure).\n\nElevated memory pressure can slowdown the process performance by utilizing more disk IO. Consider increasing amount of available RAM limit or decreasing the load on the process.\n\nSeу major page faults rate panel in Troubleshooting section if this metric continued to be high.",
"description": "Shows memory pressure based on [Pressure Stall Information](https://docs.kernel.org/accounting/psi.html).\n\n**Lower is better.**\n\nPressure is measured as amount of time within 1sec time window the process was:\n- waiting: at least one thread was blocked on memory.\n- stalled: every thread was blocked on memory (severe pressure).\n\nElevated memory pressure can slowdown the process performance by utilizing more disk IO. Consider increasing amount of available RAM limit or decreasing the load on the process.\n\nSee major page faults rate panel in Troubleshooting section if this metric continued to be high.",
"fieldConfig": {
"defaults": {
"color": {

View File

@@ -1839,7 +1839,7 @@
"type": "prometheus",
"uid": "$ds"
},
"description": "Shows memory pressure based on [Pressure Stall Information](https://docs.kernel.org/accounting/psi.html).\n\n**Lower is better.**\n\nPressure is measured as amount of time within 1sec time window the process was:\n- waiting: at least one thread was blocked on memory.\n- stalled: every thread was blocked on memory (severe pressure).\n\nElevated memory pressure can slowdown the process performance by utilizing more disk IO. Consider increasing amount of available RAM limit or decreasing the load on the process.\n\nSeу major page faults rate panel in Troubleshooting section if this metric continued to be high.",
"description": "Shows memory pressure based on [Pressure Stall Information](https://docs.kernel.org/accounting/psi.html).\n\n**Lower is better.**\n\nPressure is measured as amount of time within 1sec time window the process was:\n- waiting: at least one thread was blocked on memory.\n- stalled: every thread was blocked on memory (severe pressure).\n\nElevated memory pressure can slowdown the process performance by utilizing more disk IO. Consider increasing amount of available RAM limit or decreasing the load on the process.\n\nSee major page faults rate panel in Troubleshooting section if this metric continued to be high.",
"fieldConfig": {
"defaults": {
"color": {

View File

@@ -3,7 +3,7 @@ services:
# It scrapes targets defined in --promscrape.config
# And forward them to --remoteWrite.url
vmagent:
image: victoriametrics/vmagent:v1.145.0
image: victoriametrics/vmagent:v1.146.0
depends_on:
- "vmauth"
ports:
@@ -42,14 +42,14 @@ services:
# vmstorage shards. Each shard receives 1/N of all metrics sent to vminserts,
# where N is number of vmstorages (2 in this case).
vmstorage-1:
image: victoriametrics/vmstorage:v1.145.0-cluster
image: victoriametrics/vmstorage:v1.146.0-cluster
volumes:
- strgdata-1:/storage
command:
- "--storageDataPath=/storage"
restart: always
vmstorage-2:
image: victoriametrics/vmstorage:v1.145.0-cluster
image: victoriametrics/vmstorage:v1.146.0-cluster
volumes:
- strgdata-2:/storage
command:
@@ -59,7 +59,7 @@ services:
# vminsert is ingestion frontend. It receives metrics pushed by vmagent,
# pre-process them and distributes across configured vmstorage shards.
vminsert-1:
image: victoriametrics/vminsert:v1.145.0-cluster
image: victoriametrics/vminsert:v1.146.0-cluster
depends_on:
- "vmstorage-1"
- "vmstorage-2"
@@ -68,7 +68,7 @@ services:
- "--storageNode=vmstorage-2:8400"
restart: always
vminsert-2:
image: victoriametrics/vminsert:v1.145.0-cluster
image: victoriametrics/vminsert:v1.146.0-cluster
depends_on:
- "vmstorage-1"
- "vmstorage-2"
@@ -80,7 +80,7 @@ services:
# vmselect is a query fronted. It serves read queries in MetricsQL or PromQL.
# vmselect collects results from configured `--storageNode` shards.
vmselect-1:
image: victoriametrics/vmselect:v1.145.0-cluster
image: victoriametrics/vmselect:v1.146.0-cluster
depends_on:
- "vmstorage-1"
- "vmstorage-2"
@@ -90,7 +90,7 @@ services:
- "--vmalert.proxyURL=http://vmalert:8880"
restart: always
vmselect-2:
image: victoriametrics/vmselect:v1.145.0-cluster
image: victoriametrics/vmselect:v1.146.0-cluster
depends_on:
- "vmstorage-1"
- "vmstorage-2"
@@ -105,7 +105,7 @@ services:
# read requests from Grafana, vmui, vmalert among vmselects.
# It can be used as an authentication proxy.
vmauth:
image: victoriametrics/vmauth:v1.145.0
image: victoriametrics/vmauth:v1.146.0
depends_on:
- "vmselect-1"
- "vmselect-2"
@@ -119,7 +119,7 @@ services:
# vmalert executes alerting and recording rules
vmalert:
image: victoriametrics/vmalert:v1.145.0
image: victoriametrics/vmalert:v1.146.0
depends_on:
- "vmauth"
ports:

View File

@@ -3,7 +3,7 @@ services:
# It scrapes targets defined in --promscrape.config
# And forward them to --remoteWrite.url
vmagent:
image: victoriametrics/vmagent:v1.145.0
image: victoriametrics/vmagent:v1.146.0
depends_on:
- "victoriametrics"
ports:
@@ -18,7 +18,7 @@ services:
# VictoriaMetrics instance, a single process responsible for
# storing metrics and serve read requests.
victoriametrics:
image: victoriametrics/victoria-metrics:v1.145.0
image: victoriametrics/victoria-metrics:v1.146.0
ports:
- 8428:8428
- 8089:8089
@@ -59,7 +59,7 @@ services:
# vmalert executes alerting and recording rules
vmalert:
image: victoriametrics/vmalert:v1.145.0
image: victoriametrics/vmalert:v1.146.0
depends_on:
- "victoriametrics"
- "alertmanager"

View File

@@ -1,6 +1,6 @@
services:
vmagent:
image: victoriametrics/vmagent:v1.145.0
image: victoriametrics/vmagent:v1.146.0
depends_on:
- "victoriametrics"
ports:
@@ -14,7 +14,7 @@ services:
restart: always
victoriametrics:
image: victoriametrics/victoria-metrics:v1.145.0
image: victoriametrics/victoria-metrics:v1.146.0
ports:
- 8428:8428
volumes:
@@ -40,7 +40,7 @@ services:
restart: always
vmalert:
image: victoriametrics/vmalert:v1.145.0
image: victoriametrics/vmalert:v1.146.0
depends_on:
- "victoriametrics"
ports:
@@ -59,7 +59,7 @@ services:
- '--external.alert.source=explore?orgId=1&left=["now-1h","now","VictoriaMetrics",{"expr": },{"mode":"Metrics"},{"ui":[true,true,true,"none"]}]'
restart: always
vmanomaly:
image: victoriametrics/vmanomaly:v1.29.6
image: victoriametrics/vmanomaly:v1.29.7
depends_on:
- "victoriametrics"
ports:

View File

@@ -32,6 +32,17 @@ docs-image:
--platform $(DOCKER_PLATFORM) \
vmdocs
docs-check-links: docs-image
rm -rf vmdocs/public
docker run \
--rm \
--platform $(DOCKER_PLATFORM) \
-v ./vmdocs:/opt/docs \
$(shell for d in ./docs/*/; do printf ' -v %s:/opt/docs/content/%s' "$${d}" "$$(basename $${d})"; done) \
--entrypoint /bin/sh \
vmdocs-docker-package \
-c "yarn install && hugo --minify && yarn run check-links"
docs-debug: docs docs-image
docker run \
--rm \

View File

@@ -14,6 +14,17 @@ aliases:
---
Please find the changelog for VictoriaMetrics Anomaly Detection below.
## v1.29.7
Released: 2026-06-25
- UI: updated [vmanomaly UI](https://docs.victoriametrics.com/anomaly-detection/ui/) from [v1.7.1](https://docs.victoriametrics.com/anomaly-detection/ui/#v171) to [v1.7.2](https://docs.victoriametrics.com/anomaly-detection/ui/#v172), see respective [release notes](https://docs.victoriametrics.com/anomaly-detection/ui/#v172) for details. Notable mentions include `api/v1/server/model` endpoint for accessing production models config and queries from UI, manually or through [AI assistant](https://docs.victoriametrics.com/anomaly-detection/ui/#ai-assistance).
- IMPROVEMENT: Increased high-cardinality inference scaling by optionally scattering periodic infer jobs to reduce contention on shared resources (e.g. datasource, CPU, RAM) when `settings.n_workers > 1` and `scheduler.infer_every` is smaller than the total time to fetch and process all queries. This is controlled by new `scatter_infer_jobs` boolean argument of [Periodic Scheduler](https://docs.victoriametrics.com/anomaly-detection/components/scheduler/#parameters-1) (default: `false`).
- IMPROVEMENT: Optimized internal batching for reader post-fetch series processing, exposing reader processing queue depth (`vmanomaly_reader_processing_tasks_queued` [metric](https://docs.victoriametrics.com/anomaly-detection/components/monitoring/#reader-behaviour-metrics)), and clarifying inference skip logs after data fetch timeouts. See `series_processing_batch_size` argument of [VmReader](https://docs.victoriametrics.com/anomaly-detection/components/reader/#vm-reader) and [VLogsReader](https://docs.victoriametrics.com/anomaly-detection/components/reader/#victorialogs-reader) for details.
- IMPROVEMENT: Refined `VmReader` and `VLogsReader` logging after datasource request failures by suppressing the follow-up generic "No data" or "No unseen data" warning for failed fetches. Failed requests now keep the original datasource error while empty successful responses still emit the no-data warning.
## v1.29.6
Released: 2026-06-17

View File

@@ -423,7 +423,7 @@ services:
# ...
vmanomaly:
container_name: vmanomaly
image: victoriametrics/vmanomaly:v1.29.6
image: victoriametrics/vmanomaly:v1.29.7
# ...
restart: always
volumes:
@@ -641,7 +641,7 @@ options:
Heres an example of using the config splitter to divide configurations based on the `extra_filters` argument from the reader section:
```sh
docker pull victoriametrics/vmanomaly:v1.29.6 && docker image tag victoriametrics/vmanomaly:v1.29.6 vmanomaly
docker pull victoriametrics/vmanomaly:v1.29.7 && docker image tag victoriametrics/vmanomaly:v1.29.7 vmanomaly
```
```sh

View File

@@ -45,7 +45,7 @@ There are 2 types of compatibility to consider when migrating in stateful mode:
| Group start | Group end | Compatibility | Notes |
|---------|--------- |------------|-------|
| [v1.29.1](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1291) | [v1.29.6](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1296) | Fully Compatible | - |
| [v1.29.1](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1291) | [v1.29.7](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1297) | Fully Compatible | - |
| [v1.28.7](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1287) | [v1.29.0](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1290) | Partially compatible* | Dumped models of class [prophet](https://docs.victoriametrics.com/anomaly-detection/components/models/#prophet) and [seasonal quantile](https://docs.victoriametrics.com/anomaly-detection/components/models/#online-seasonal-quantile) have problems with loading to [v1.29.0](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1290) due to dropped `pytz` library. **Upgrading directly from v1.28.7 to [v1.29.1](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1291) with a fix is suggested** |
| [v1.26.0](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1262) | [v1.28.7](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1287) | Fully Compatible | [v1.28.0](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1280) introduced [rolling](https://docs.victoriametrics.com/anomaly-detection/components/models/#rolling-models) model class drop in favor of [online](https://docs.victoriametrics.com/anomaly-detection/components/models/#online-models) models (`rolling_quantile` and `std` models), however, it does not impact compatibility, as artifacts were not produced by default for rolling models. Also, offline `mad` and `zscore` models are redirecting to their respective online counterparts since [v1.28.4](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1284). |
| [v1.25.3](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1253) | [v1.26.0](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1270) | Partially Compatible* | [v1.25.3](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1253) introduced `forecast_at` argument for base [univariate](https://docs.victoriametrics.com/anomaly-detection/components/models/#univariate-models) and `Prophet` [models](https://docs.victoriametrics.com/anomaly-detection/components/models/#prophet), however, itself remains backward-reversible from newer states like [v1.26.2](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1262), [v1.27.0](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1270). (All models except `isolation_forest_multivariate` class will be dropped) |

View File

@@ -132,7 +132,7 @@ Below are the steps to get `vmanomaly` up and running inside a Docker container:
1. Pull Docker image:
```sh
docker pull victoriametrics/vmanomaly:v1.29.6
docker pull victoriametrics/vmanomaly:v1.29.7
```
2. Create the license file with your license key.
@@ -152,7 +152,7 @@ docker run -it \
-v ./license:/license \
-v ./config.yaml:/config.yaml \
-p 8490:8490 \
victoriametrics/vmanomaly:v1.29.6 \
victoriametrics/vmanomaly:v1.29.7 \
/config.yaml \
--licenseFile=/license \
--loggerLevel=INFO \
@@ -169,7 +169,7 @@ docker run -it \
-e VMANOMALY_DATA_DUMPS_DIR=/tmp/vmanomaly/data \
-e VMANOMALY_MODEL_DUMPS_DIR=/tmp/vmanomaly/models \
-p 8490:8490 \
victoriametrics/vmanomaly:v1.29.6 \
victoriametrics/vmanomaly:v1.29.7 \
/config.yaml \
--licenseFile=/license \
--loggerLevel=INFO \
@@ -182,7 +182,7 @@ services:
# ...
vmanomaly:
container_name: vmanomaly
image: victoriametrics/vmanomaly:v1.29.6
image: victoriametrics/vmanomaly:v1.29.7
# ...
restart: always
volumes:
@@ -267,6 +267,7 @@ schedulers:
# https://docs.victoriametrics.com/anomaly-detection/components/scheduler/#periodic-scheduler
class: 'periodic'
infer_every: '5m'
scatter_infer_jobs: true
fit_every: '1d'
fit_window: '4w'
@@ -298,6 +299,7 @@ reader:
datasource_url: "https://play.victoriametrics.com/" # [YOUR_DATASOURCE_URL]
tenant_id: '0:0'
sampling_period: "5m"
series_processing_batch_size: 8 # number of time series to process together while preparing data for fit or infer stages
queries:
# define your queries with MetricsQL - https://docs.victoriametrics.com/victoriametrics/metricsql/
cpu_user:
@@ -413,11 +415,13 @@ For optimal service behavior, consider the following tweaks when configuring `vm
- Configure the **inference frequency** in the [scheduler](https://docs.victoriametrics.com/anomaly-detection/components/scheduler/) section of the configuration file.
- Ensure that `infer_every` aligns with your **minimum required alerting frequency**.
- For example, if receiving **alerts every 15 minutes** is sufficient (when `anomaly_score > 1`), set `infer_every` to match `reader.sampling_period` or override it per query via `reader.queries.query_xxx.step` for an optimal setup.
- Set `scheduler.scatter_infer_jobs` {{% available_from "v1.29.7" anomaly %}} [arg](https://docs.victoriametrics.com/anomaly-detection/components/scheduler/#parameters-1) to `true` to allow for equal distribution of inference jobs across `infer_every` intervals, which can further enhance parallel processing efficiency and reduce resource contention when `reader.queries` contains a large number of queries.
**Reader**:
- Setup the datasource to read data from in the [reader](https://docs.victoriametrics.com/anomaly-detection/components/reader/) section. Include tenant ID if using a [cluster version of VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/) (`multitenant` value {{% available_from "v1.16.2" anomaly %}} can be also used here).
- Define queries for input data using [MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/) under `reader.queries` section. Note, it's possible to override reader-level arguments at query level for increased flexibility, e.g. specifying per-query [timezone](https://docs.victoriametrics.com/anomaly-detection/faq/#handling-timezones) or [sampling period](https://docs.victoriametrics.com/anomaly-detection/components/reader/#config-parameters).
- For longer `fit_window` intervals in scheduler, consider splitting queries into smaller time ranges to avoid excessive memory usage, timeouts and hitting server-side constraints, so they can be queried separately and reconstructed on `vmanomaly` side. Please refer to this [example](https://docs.victoriametrics.com/anomaly-detection/faq/#handling-large-queries-in-vmanomaly) for more details.
- Set `reader.series_processing_batch_size` {{% available_from "v1.29.7" anomaly %}} [arg](https://docs.victoriametrics.com/anomaly-detection/components/reader/#config-parameters) to a reasonable value (4-16, default is 8) to balance between memory usage and processing speed when preparing data for fit or infer stages.
> If applicable - consider [`VLogsReader`](https://docs.victoriametrics.com/anomaly-detection/components/reader/#victorialogs-reader) {{% available_from "v1.26.0" anomaly %}} to perform anomaly detection on **log-derived metrics**. This is particularly useful for scenarios where log data needs to be analyzed for unusual patterns or behaviors, such as error rates or request latencies.

View File

@@ -315,7 +315,7 @@ docker run -it --rm \
-e VMANOMALY_MCP_SERVER_URL=http://mcp-vmanomaly:8081/mcp \
-p 8080:8080 \
-p 8490:8490 \
victoriametrics/vmanomaly:v1.29.6 \
victoriametrics/vmanomaly:v1.29.7 \
vmanomaly_config.yaml
```
@@ -640,6 +640,21 @@ If the **results** look good and the **model configuration should be deployed in
## Changelog
### v1.7.2
Released: 2026-06-25
vmanomaly version: [v1.29.7](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1297)
- FEATURE: Added controls for selecting server-configured scheduled models (drop-down inside [model wizard](#model-panel)) and browsing scheduled queries from the running vmanomaly instance ("Queries" button, "scheduled queries" tab).
- IMPROVEMENT: Surfaced datasource fetch failures from ad-hoc VMUI raw queries as query-level errors instead of returning a successful empty result that triggers a generic "No match" warning. Now the user can see the actual error message from the datasource (e.g. "unauthorized", "not found", etc.) and take appropriate action.
- BUGFIX: Fixed [UI/query-server](#settings-panel) handling of VictoriaMetrics datasource URLs that already include `/select/multitenant/prometheus`. Such URLs are now recognized as cluster datasource URLs, preserving the multitenant path when proxying VMUI requests and allowing `server.use_reader_connection_settings` to reuse [configured reader credentials for authenticated datasources](#authentication).
- BUGFIX: Fixed [settings](#settings-panel) inputs for server and datasource URLs so editing, deleting, or pasting text is no longer immediately reverted to the previous value before applying changes.
- BUGFIX: Fixed [model wizard](#model-panel) settings for [`IsolationForestModel`](https://docs.victoriametrics.com/anomaly-detection/components/models/#isolation-forest-multivariate) `contamination`, allowing decimal float values such as `0.1` or `0,1` to be typed or pasted without being collapsed to `0`, while preserving the `"auto"` value.
### v1.7.1
Released: 2026-06-11

View File

@@ -49,6 +49,7 @@ schedulers:
periodic_online: # alias
class: 'periodic' # scheduler class
infer_every: "30s" # how often to produce anomaly scores for new data
scatter_infer_jobs: true # distribute infer jobs evenly across the infer interval to reduce synchronized bursts
fit_every: "365d" # how often to re-fit the models, for online models used effectively once, then they are updated with new data and won't require re-fit
fit_window: "3d" # how much historical data to use for fit stage
start_from: "00:00" # start from specified time, i.e. 00:00 given timezone and do daily fits as `fit_every` is 1 day
@@ -56,6 +57,7 @@ schedulers:
periodic_offline_1w:
class: 'periodic'
infer_every: "15m"
scatter_infer_jobs: true
fit_every: "24h"
fit_window: "14d"
# if no start_from is specified, jobs will start immediately after service starts
@@ -135,6 +137,7 @@ server:
port: 8490
path_prefix: '/vmanomaly' # optional path prefix for all HTTP routes
max_concurrent_tasks: 4 # maximum number of concurrent anomaly detection tasks processed by backend
use_reader_connection_settings: True # if True, use reader's datasource_url and credentials for UI requests to datasource
uvicorn_config: # optional Uvicorn server configuration
log_level: 'warning'
```

View File

@@ -1265,7 +1265,7 @@ monitoring:
Let's pull the docker image for `vmanomaly`:
```sh
docker pull victoriametrics/vmanomaly:v1.29.6
docker pull victoriametrics/vmanomaly:v1.29.7
```
Now we can run the docker container putting as volumes both config and model file:
@@ -1279,7 +1279,7 @@ docker run -it \
-v $(PWD)/license:/license \
-v $(PWD)/custom_model.py:/vmanomaly/model/custom.py \
-v $(PWD)/custom.yaml:/config.yaml \
victoriametrics/vmanomaly:v1.29.6 /config.yaml \
victoriametrics/vmanomaly:v1.29.7 /config.yaml \
--licenseFile=/license
--watch
```

View File

@@ -458,6 +458,21 @@ Label names [description](#labelnames)
<td>The total number of datapoints received from VictoriaMetrics for the `query_key` query within the specified scheduler `scheduler_alias`, in the `vmanomaly` service running in `preset` mode.</td>
<td>
`url`, `query_key`, `scheduler_alias`, `preset`
</td>
</tr>
<tr>
<td>
<span style="white-space: nowrap;">`vmanomaly_reader_processing_tasks_queued`</span>
</td>
<td>
`Gauge`
</td>
<td>The total number of queued processing tasks {{% available_from "v1.29.7" anomaly %}} (timeseries batches of size `series_processing_batch_size`) for the `query_key` query within the specified scheduler `scheduler_alias`, in the `vmanomaly` service running in `preset` mode. If continuously >0, it may lead to skipped infer runs due to resource contention and timeouts.</td>
<td>
`url`, `query_key`, `scheduler_alias`, `preset`
</td>
</tr>

View File

@@ -421,7 +421,20 @@ Optional argument{{% available_from "v1.18.1" anomaly %}} allows defining **vali
`60s`
</td>
<td>
Optional argument{{% available_from "v1.25.3" anomaly %}} allows specifying a time offset for all queries in `queries`. Defaults to `0s` (0) if not set and can be overridden on a [per-query basis](#per-query-parameters).
Optional argument {{% available_from "v1.25.3" anomaly %}}, allows specifying a time offset for all queries in `queries`. Defaults to `0s` (0) if not set and can be overridden on a [per-query basis](#per-query-parameters).
</td>
</tr>
<tr>
<td>
<span style="white-space: nowrap;">`series_processing_batch_size`</span>
</td>
<td>
`8`
</td>
<td>
Optional argument {{% available_from "v1.29.7" anomaly %}}, allows specifying the number of time series to process together while preparing data for fit or infer stages. Defaults to `8`. Suggested values are 4-16 for high-cardinality queries.
</td>
</tr>
</tbody>
@@ -450,6 +463,7 @@ reader:
sampling_period: '1m'
query_from_last_seen_timestamp: True # false by default
latency_offset: '1ms'
series_processing_batch_size: 8
```
### MetricsQL Playground
@@ -879,6 +893,19 @@ If a path to a CA bundle file (like `ca.crt`), it will verify the certificate us
(Optional) Password for authentication. If set, it will be used to authenticate the request.
</td>
</tr>
<tr>
<td>
<span style="white-space: nowrap;">`series_processing_batch_size`</span>
</td>
<td>
`8`
</td>
<td>
Optional argument {{% available_from "v1.29.7" anomaly %}}, allows specifying the number of time series to process together while preparing data for fit or infer stages. Defaults to `8`. Suggested values are 4-16 for high-cardinality queries.
</td>
</tr>
</tbody>
</table>
@@ -897,6 +924,7 @@ reader:
# tenant_id: '0:0' # for cluster version only
sampling_period: '1m'
max_points_per_query: 10000
series_processing_batch_size: 8
data_range: [0, 'inf'] # reader-level
offset: '0s' # reader-level
timeout: '30s'

View File

@@ -74,40 +74,7 @@ options={`"scheduler.periodic.PeriodicScheduler"`, `"scheduler.oneoff.OneoffSche
### Parameters
For periodic scheduler parameters are defined as differences in times, expressed in difference units, e.g. days, hours, minutes, seconds.
Examples: `"50s"`, `"4m"`, `"3h"`, `"2d"`, `"1w"`.
<table class="params">
<thead>
<tr>
<th></th>
<th>Time granularity</th>
</tr>
</thead>
<tbody>
<tr>
<td>s</td>
<td>seconds</td>
</tr>
<tr>
<td>m</td>
<td>minutes</td>
</tr>
<tr>
<td>h</td>
<td>hours</td>
</tr>
<tr>
<td>d</td>
<td>days</td>
</tr>
<tr>
<td>w</td>
<td>weeks</td>
</tr>
</tbody>
</table>
For periodic scheduler parameters are defined as differences in times, expressed in difference units, e.g. days, hours, minutes, seconds. Time granularity is defined by the last characters of a string. Examples: `"50s"` (seconds), `"4m"` (minutes), `"3h"` (hours), `"2d"` (days), `"1w"` (weeks).
<table class="params">
<thead>
@@ -188,6 +155,21 @@ Specifies when to initiate the first `fit_every` call. Accepts either an ISO 860
Defines the local timezone for the `start_from` parameter, if specified. Defaults to `UTC` if no timezone is provided.
</td>
</tr>
<tr>
<td>
<span style="white-space: nowrap;">`scatter_infer_jobs`{{% available_from "v1.29.7" anomaly %}}</span>
</td>
<td>bool, <span style="white-space: nowrap;">Optional</span></td>
<td>
`true` or `false`
</td>
<td>
If `true`, distribute infer jobs and their dependent data-fetch jobs evenly across the infer interval. This reduces synchronized read and inference bursts for high-scale configurations. Defaults to `false`. Useful when `settings.n_workers > 1`, `reader.queries` cardinality is high, and `scheduler.infer_every` is small.
</td>
</tr>
</tbody>
</table>
@@ -200,6 +182,7 @@ schedulers:
# (or class: "scheduler.periodic.PeriodicScheduler" for versions before v1.13.0, without class alias support)
fit_window: "14d"
infer_every: "1m"
scatter_infer_jobs: true # Distribute infer jobs evenly across the infer interval to reduce synchronized bursts.
fit_every: "1h"
start_from: "20:00" # If launched before 20:00 (local Kyiv time), the first run starts today at 20:00. Otherwise, it starts tomorrow at 20:00.
tz: "Europe/Kyiv" # Defaults to 'UTC' if not specified.

View File

@@ -10,9 +10,9 @@ sitemap:
- To use *vmanomaly*, part of the enterprise package, a license key is required. Obtain your key [here](https://victoriametrics.com/products/enterprise/trial/) for this tutorial or for enterprise use.
- In the tutorial, we'll be using the following VictoriaMetrics components:
- [VictoriaMetrics Single-Node](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/) (v1.145.0)
- [vmalert](https://docs.victoriametrics.com/victoriametrics/vmalert/) (v1.145.0)
- [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/) (v1.145.0)
- [VictoriaMetrics Single-Node](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/) (v1.146.0)
- [vmalert](https://docs.victoriametrics.com/victoriametrics/vmalert/) (v1.146.0)
- [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/) (v1.146.0)
- [Grafana](https://grafana.com/) (v12.2.0)
- [Docker](https://docs.docker.com/get-docker/) and [Docker Compose](https://docs.docker.com/compose/)
- [Node exporter](https://github.com/prometheus/node_exporter#node-exporter) (v1.9.1) and [Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/) (v0.28.1)
@@ -323,7 +323,7 @@ Let's wrap it all up together into the `docker-compose.yml` file.
services:
vmagent:
container_name: vmagent
image: victoriametrics/vmagent:v1.145.0
image: victoriametrics/vmagent:v1.146.0
depends_on:
- "victoriametrics"
ports:
@@ -340,7 +340,7 @@ services:
victoriametrics:
container_name: victoriametrics
image: victoriametrics/victoria-metrics:v1.145.0
image: victoriametrics/victoria-metrics:v1.146.0
ports:
- 8428:8428
volumes:
@@ -373,7 +373,7 @@ services:
vmalert:
container_name: vmalert
image: victoriametrics/vmalert:v1.145.0
image: victoriametrics/vmalert:v1.146.0
depends_on:
- "victoriametrics"
ports:
@@ -395,7 +395,7 @@ services:
restart: always
vmanomaly:
container_name: vmanomaly
image: victoriametrics/vmanomaly:v1.29.6
image: victoriametrics/vmanomaly:v1.29.7
depends_on:
- "victoriametrics"
ports:

View File

@@ -240,23 +240,23 @@ vmagent will write data into VictoriaMetrics single-node and cluster (with tenan
# compose.yaml
services:
vmsingle:
image: victoriametrics/victoria-metrics:v1.145.0
image: victoriametrics/victoria-metrics:v1.146.0
vmstorage:
image: victoriametrics/vmstorage:v1.145.0-cluster
image: victoriametrics/vmstorage:v1.146.0-cluster
vminsert:
image: victoriametrics/vminsert:v1.145.0-cluster
image: victoriametrics/vminsert:v1.146.0-cluster
command:
- -storageNode=vmstorage:8400
vmselect:
image: victoriametrics/vmselect:v1.145.0-cluster
image: victoriametrics/vmselect:v1.146.0-cluster
command:
- -storageNode=vmstorage:8401
vmagent:
image: victoriametrics/vmagent:v1.145.0
image: victoriametrics/vmagent:v1.146.0
volumes:
- ./scrape.yaml:/etc/vmagent/config.yaml
command:
@@ -308,7 +308,7 @@ Now add the vmauth service to `compose.yaml`:
# compose.yaml
services:
vmauth:
image: docker.io/victoriametrics/vmauth:v1.145.0
image: docker.io/victoriametrics/vmauth:v1.146.0
ports:
- 8427:8427
volumes:

View File

@@ -155,15 +155,15 @@ These services will store and query the metrics scraped by vmagent.
# compose.yaml
services:
vmstorage:
image: victoriametrics/vmstorage:v1.145.0-cluster
image: victoriametrics/vmstorage:v1.146.0-cluster
vminsert:
image: victoriametrics/vminsert:v1.145.0-cluster
image: victoriametrics/vminsert:v1.146.0-cluster
command:
- -storageNode=vmstorage:8400
vmselect:
image: victoriametrics/vmselect:v1.145.0-cluster
image: victoriametrics/vmselect:v1.146.0-cluster
command:
- -storageNode=vmstorage:8401
ports:
@@ -196,7 +196,7 @@ Add the vmauth service to `compose.yaml`:
# compose.yaml
services:
vmauth:
image: victoriametrics/vmauth:v1.145.0-enterprise
image: victoriametrics/vmauth:v1.146.0-enterprise
ports:
- 8427:8427
volumes:
@@ -251,7 +251,7 @@ Add the vmagent service to `compose.yaml` with OAuth2 configuration:
# compose.yaml
services:
vmagent:
image: victoriametrics/vmagent:v1.145.0
image: victoriametrics/vmagent:v1.146.0
volumes:
- ./scrape.yaml:/etc/vmagent/config.yaml
command:

View File

@@ -107,7 +107,7 @@ The final piece is the Docker Compose file. This ties all the services together
# compose.yml
services:
victoriametrics:
image: victoriametrics/victoria-metrics:v1.145.0
image: victoriametrics/victoria-metrics:v1.146.0
command:
- "--storageDataPath=/victoria-metrics-data"
- "--selfScrapeInterval=10s"
@@ -128,7 +128,7 @@ services:
- ./alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro
vmalert:
image: victoriametrics/vmalert:v1.145.0
image: victoriametrics/vmalert:v1.146.0
depends_on:
- victoriametrics
- alertmanager

View File

@@ -85,6 +85,21 @@ Pull requests requirements:
See a good example of a [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6487).
## AI policy
You are free to use any AI tools when working on a contribution, on code,
documentation, issues, or anything else. You do not need to disclose whether or
how you used them.
With or without the help of AI, you are responsible for the changes you submit.
Take the effort to understand the code base and every change in your pull request,
and clean up any AI slop before sending it. Do not use AI to automate your
responses to maintainers.
We review contributions on their quality, regardless of how they were produced. A
pull request or issue that looks like unreviewed AI output, with low-quality or
broken changes, may be closed without a detailed review or triage.
## Merging Pull Request
The person who merges the Pull Request is responsible for satisfying the requirements below:

View File

@@ -1229,8 +1229,8 @@ Metric names are stripped from the resulting series. Add [keep_metric_names](#ke
`buckets_limit(limit, buckets)` is a [transform function](#transform-functions), which limits the number
of [histogram buckets](https://valyala.medium.com/improving-histogram-usability-for-prometheus-and-grafana-bc7e5df0e350) to the given `limit`.
If the given `limit` is greater than 0 and less than 3, the effective limit will be raised to 3 and the result will contain 3 buckets.
If the given `limit` is 0 or negative, no buckets will be returned.
The result will preserve the first and the last bucket to improve accuracy for min and max values.
So, if the `limit` is greater than 0 and less than 3, the function will still return 3 buckets: the first bucket, the last bucket, and a selected bucket.
See also [prometheus_buckets](#prometheus_buckets) and [histogram_quantile](#histogram_quantile).

View File

@@ -61,9 +61,9 @@ Download the newest available [VictoriaMetrics release](https://docs.victoriamet
from [DockerHub](https://hub.docker.com/r/victoriametrics/victoria-metrics) or [Quay](https://quay.io/repository/victoriametrics/victoria-metrics?tab=tags):
```sh
docker pull victoriametrics/victoria-metrics:v1.145.0
docker pull victoriametrics/victoria-metrics:v1.146.0
docker run -it --rm -v `pwd`/victoria-metrics-data:/victoria-metrics-data -p 8428:8428 \
victoriametrics/victoria-metrics:v1.145.0 --selfScrapeInterval=5s -storageDataPath=victoria-metrics-data
victoriametrics/victoria-metrics:v1.146.0 --selfScrapeInterval=5s -storageDataPath=victoria-metrics-data
```
_For Enterprise images, see [this link](https://docs.victoriametrics.com/victoriametrics/enterprise/#docker-images)._

View File

@@ -1712,64 +1712,63 @@ The following versions of VictoriaMetrics receive regular security fixes:
| [LTS releases](https://docs.victoriametrics.com/victoriametrics/lts-releases/) | ✅ |
| other releases | ❌ |
### Software Bill of Materials (SBOM)
Every VictoriaMetrics container{{% available_from "v1.137.0" %}} image published to
[Docker Hub](https://hub.docker.com/u/victoriametrics) and [Quay.io](https://quay.io/organization/victoriametrics) include an [SPDX](https://spdx.dev/) SBOM attestation generated automatically by BuildKit during `docker buildx build`.
To inspect the SBOM for an image:
```sh
docker buildx imagetools inspect \
docker.io/victoriametrics/victoria-metrics:latest \
--format "{{ json .SBOM }}"
```
To scan an image using its SBOM attestation with [Trivy](https://github.com/aquasecurity/trivy):
```sh
trivy image --sbom-sources oci \
docker.io/victoriametrics/victoria-metrics:latest
```
### Reporting a Vulnerability
Please report any security issues to <security@victoriametrics.com>
### CVE handling policy
**Source code:** Go dependencies are scanned by [govulncheck](https://pkg.go.dev/golang.org/x/vuln/cmd/govulncheck) in CI.
All vulnerabilities must be fixed before the next scheduled release and backported to [LTS releases](https://docs.victoriametrics.com/victoriametrics/lts-releases/).
**Docker images:** CVE findings in the [Alpine](https://security.alpinelinux.org/) base image pose minimal risk since VictoriaMetrics binaries are statically compiled with no OS dependencies.
When detected, only the Alpine base tag is updated.
Releases proceed as planned even if upstream fixes are not yet available.
For maximum security, hardened [scratch](https://hub.docker.com/_/scratch)-based images are also provided.
All images are continuously scanned by Docker Hub and verified before release using [grype](https://github.com/anchore/grype).
### General security recommendations:
* All the VictoriaMetrics components must run in protected private networks without direct access from untrusted networks such as Internet.
* All VictoriaMetrics components must run in protected private networks without direct access from untrusted networks such as the Internet.
The exception is [vmauth](https://docs.victoriametrics.com/victoriametrics/vmauth/) and [vmgateway](https://docs.victoriametrics.com/victoriametrics/vmgateway/),
which are intended for serving public requests and performing authorization with [TLS termination](https://en.wikipedia.org/wiki/TLS_termination_proxy).
* All the requests from untrusted networks to VictoriaMetrics components must go through auth proxy such as [vmauth](https://docs.victoriametrics.com/victoriametrics/vmauth/)
* All the requests from untrusted networks to VictoriaMetrics components must go through an auth proxy, such as [vmauth](https://docs.victoriametrics.com/victoriametrics/vmauth/)
or [vmgateway](https://docs.victoriametrics.com/victoriametrics/vmgateway/). The proxy must be set up with proper authentication and authorization.
* Prefer using lists of allowed API endpoints, while disallowing access to other endpoints when configuring [vmauth](https://docs.victoriametrics.com/victoriametrics/vmauth/)
in front of VictoriaMetrics components.
* Set reasonable [`Strict-Transport-Security`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Strict-Transport-Security) header value to all the components to mitigate [MitM attacks](https://en.wikipedia.org/wiki/Man-in-the-middle_attack), for example: `max-age=31536000; includeSubDomains`. See `-http.header.hsts` flag.
* Set a reasonable [`Strict-Transport-Security`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Strict-Transport-Security) header value on all the components to mitigate [MitM attacks](https://en.wikipedia.org/wiki/Man-in-the-middle_attack), for example: `max-age=31536000; includeSubDomains`. See `-http.header.hsts` flag.
* Set reasonable [`Content-Security-Policy`](https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP) header value to mitigate [XSS attacks](https://en.wikipedia.org/wiki/Cross-site_scripting). See `-http.header.csp` flag.
* Set reasonable [`X-Frame-Options`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Frame-Options) header value to mitigate [clickjacking attacks](https://en.wikipedia.org/wiki/Clickjacking), for example `DENY`. See `-http.header.frameOptions` flag.
VictoriaMetrics provides the following security-related command-line flags:
The following security-related command-line flags are available for all components with HTTP API:
* `-tls`, `-tlsCertFile` and `-tlsKeyFile` for switching from HTTP to HTTPS at `-httpListenAddr` (TCP port 8428 is listened by default).
* `-tls`, `-tlsCertFile` and `-tlsKeyFile` for switching from HTTP to HTTPS at `-httpListenAddr`.
[Enterprise version of VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics/enterprise/) supports automatic issuing of TLS certificates.
See [these docs](#automatic-issuing-of-tls-certificates).
* `-mtls` and `-mtlsCAFile` for enabling [mTLS](https://en.wikipedia.org/wiki/Mutual_authentication) for requests to `-httpListenAddr`. See [these docs](#mtls-protection).
* `-httpAuth.username` and `-httpAuth.password` for protecting all the HTTP endpoints
with [HTTP Basic Authentication](https://en.wikipedia.org/wiki/Basic_access_authentication).
* `-http.header.hsts`, `-http.header.csp`, and `-http.header.frameOptions` for serving `Strict-Transport-Security`, `Content-Security-Policy`
and `X-Frame-Options` HTTP response headers.
### Protecting service endpoints
All VictoriaMetrics components expose internal metrics in Prometheus exposition format at the `/metrics` page for [#Monitoring](https://docs.victoriametrics.com/victoriametrics/#monitoring).
Consider limiting access to the `/metrics` page to trusted networks only.
The following service endpoints may require protection:
* `-deleteAuthKey` for protecting the `/api/v1/admin/tsdb/delete_series` endpoint. See [how to delete time series](#how-to-delete-time-series).
* `-snapshotAuthKey` for protecting the `/snapshot*` endpoints. See [how to work with snapshots](#how-to-work-with-snapshots).
* `-forceFlushAuthKey` for protecting the `/internal/force_flush` endpoint. See [force flush docs](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#forced-flush).
* `-forceMergeAuthKey` for protecting the `/internal/force_merge` endpoint. See [force merge docs](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#forced-merge).
* `-search.resetCacheAuthKey` for protecting the `/internal/resetRollupResultCache` endpoint. See [backfilling](#backfilling) for more details.
* `-reloadAuthKey` for protecting the `/-/reload` endpoint, which is used for force reloading of [`-promscrape.config`](#how-to-scrape-prometheus-exporters-such-as-node-exporter).
* `-reloadAuthKey` for protecting the `/-/reload` endpoint, which is used to force reload the [`-promscrape.config`](#how-to-scrape-prometheus-exporters-such-as-node-exporter).
* `-configAuthKey` for protecting the `/config` endpoint, since it may contain sensitive information such as passwords.
* `-flagsAuthKey` for protecting the `/flags` endpoint.
* `-pprofAuthKey` for protecting the `/debug/pprof/*` endpoints, which can be used for [profiling](#profiling).
* `-metricNamesStatsResetAuthKey` for protecting the `/api/v1/admin/status/metric_names_stats/reset` endpoint, used for [Metric Names Tracker](#track-ingested-metrics-usage).
* `-denyQueryTracing` for disallowing [query tracing](#query-tracing).
* `-http.header.hsts`, `-http.header.csp`, and `-http.header.frameOptions` for serving `Strict-Transport-Security`, `Content-Security-Policy`
and `X-Frame-Options` HTTP response headers.
Explicitly set internal network interface for TCP and UDP ports for data ingestion with Graphite and OpenTSDB formats.
For example, substitute `-graphiteListenAddr=:2003` with `-graphiteListenAddr=<internal_iface_ip>:2003`. This protects from unexpected requests from untrusted network interfaces.
@@ -1777,17 +1776,6 @@ For example, substitute `-graphiteListenAddr=:2003` with `-graphiteListenAddr=<i
See also [security recommendation for VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#security)
and [the general security page at VictoriaMetrics website](https://victoriametrics.com/security/).
### CVE handling policy
**Source code:** Go dependencies are scanned by [govulncheck](https://pkg.go.dev/golang.org/x/vuln/cmd/govulncheck) in CI.
All vulnerabilities must be fixed before next scheduled release and backported to [LTS releases](https://docs.victoriametrics.com/victoriametrics/lts-releases/).
**Docker images:** CVE findings in [Alpine](https://security.alpinelinux.org/) base image pose minimal risk since VictoriaMetrics binaries are statically compiled with no OS dependencies.
When detected, only the Alpine base tag is updated.
Releases proceed as planned even if upstream fixes are not yet available.
For maximum security, hardened [scratch](https://hub.docker.com/_/scratch)-based images are also provided.
All images are continuously scanned by Docker Hub and verified before release using [grype](https://github.com/anchore/grype).
### mTLS protection
By default `VictoriaMetrics` accepts http requests at `8428` port (this port can be changed via `-httpListenAddr` command-line flags).
@@ -1817,19 +1805,39 @@ This functionality can be evaluated for free according to [these docs](https://d
See also [security recommendations](#security).
### Software Bill of Materials (SBOM)
Every VictoriaMetrics container{{% available_from "v1.137.0" %}} image published to
[Docker Hub](https://hub.docker.com/u/victoriametrics) and [Quay.io](https://quay.io/organization/victoriametrics) include an [SPDX](https://spdx.dev/) SBOM attestation generated automatically by BuildKit during `docker buildx build`.
To inspect the SBOM for an image:
```sh
docker buildx imagetools inspect \
docker.io/victoriametrics/victoria-metrics:latest \
--format "{{ json .SBOM }}"
```
To scan an image using its SBOM attestation with [Trivy](https://github.com/aquasecurity/trivy):
```sh
trivy image --sbom-sources oci \
docker.io/victoriametrics/victoria-metrics:latest
```
## Tuning
* No need in tuning for VictoriaMetrics - it uses reasonable defaults for command-line flags,
* No need to tune for VictoriaMetrics - it uses reasonable defaults for command-line flags,
which are automatically adjusted for the available CPU and RAM resources.
* No need in tuning for Operating System - VictoriaMetrics is optimized for default OS settings.
* No need to tune for Operating System - VictoriaMetrics is optimized for default OS settings.
The only option is increasing the limit on [the number of open files in the OS](https://medium.com/@muhammadtriwibowo/set-permanently-ulimit-n-open-files-in-ubuntu-4d61064429a).
The recommendation is not specific for VictoriaMetrics only but also for any service which handles many HTTP connections and stores data on disk.
* VictoriaMetrics is a write-heavy application and its performance depends on disk performance. So be careful with other
The recommendation is not specific to VictoriaMetrics only, but also for any service that handles many HTTP connections and stores data on disk.
* VictoriaMetrics is a write-heavy application, and its performance depends on disk performance. So be careful with other
applications or utilities (like [fstrim](https://manpages.ubuntu.com/manpages/lunar/en/man8/fstrim.8.html))
which could [exhaust disk resources](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1521).
* The recommended filesystem is `ext4`, the recommended persistent storage is [persistent HDD-based disk on GCP](https://cloud.google.com/compute/docs/disks/#pdspecs),
since it is protected from hardware failures via internal replication and it can be [resized on the fly](https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd).
If you plan to store more than 1TB of data on `ext4` partition, then the following options are recommended to pass to `mkfs.ext4`:
since it is protected from hardware failures via internal replication, and it can be [resized on the fly](https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd).
If you plan to store more than 1TB of data on an `ext4` partition, then the following options are recommended to pass to `mkfs.ext4`:
```sh
mkfs.ext4 ... -O 64bit,huge_file,extent -T huge

View File

@@ -26,28 +26,34 @@ See also [LTS releases](https://docs.victoriametrics.com/victoriametrics/lts-rel
## tip
* BUGFIX: `vmselect` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/): propagate cache reset operation to `selectNode` when `/internal/resetRollupResultCache` is called. Previously, the propagation only happened when the `delete_series` API was called. See [#11112](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/11112).
* BUGFIX: [stream aggregation](https://docs.victoriametrics.com/victoriametrics/stream-aggregation/): fix possible unexpected increases in `rate_avg` and `rate_sum` if an out-of-order sample is ingested after the previous flush. See [#11140](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/11140).
## [v1.146.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.146.0)
Released at 2026-06-22
* FEATURE: all VictoriaMetrics components: add `-http.header.disableServerHostname` command-line flag for disabling the `X-Server-Hostname` HTTP response header. See [#11067](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/11067). Thanks to @zasdaym for contribution.
* FEATURE: [vmsingle](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/) and `vmselect` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/): log calls to [/api/v1/admin/tsdb/delete_series](https://docs.victoriametrics.com/victoriametrics/url-examples/#apiv1admintsdbdelete_series) API handler. This should help to identify events of metrics deletion from the database. See [#11104](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/11104).
* FEATURE: [vmctl](https://docs.victoriametrics.com/victoriametrics/vmctl/): add `-vm-headers` and `-vm-bearer-token` flags for authenticating requests to the VictoriaMetrics import destination. The flags are available in `opentsdb`, `influx`, `remote-read`, `prometheus`, `mimir`, and `thanos` vmctl sub-commands. See [#8897](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8897).
* FEATURE: [vmui](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#vmui): add the `last` value to graph legend statistics. See [#10759](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/10759).
* FEATURE: [stream aggregation](https://docs.victoriametrics.com/victoriametrics/stream-aggregation/): expose `vm_streamaggr_dedup_dropped_samples_total` to allow tracking dropped old samples during [deduplication](https://docs.victoriametrics.com/victoriametrics/stream-aggregation/#deduplication).
* FEATURE: [stream aggregation](https://docs.victoriametrics.com/victoriametrics/stream-aggregation/): use the aggregation rule interval as the default [staleness_interval](https://docs.victoriametrics.com/victoriametrics/stream-aggregation/#staleness) instead of `2*interval`, to reduce spikes when there are gaps between received samples. See [#11102](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/11102).
* FEATURE: [stream aggregation](https://docs.victoriametrics.com/victoriametrics/stream-aggregation/): add new aggregation output [sum_samples_total](https://docs.victoriametrics.com/victoriametrics/stream-aggregation/configuration/#sum_samples_total) for summing input delta values into a cumulative counter. See issues [#11002](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/11002) and [#4843](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4843).
* FEATURE: [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/): add a new flag `-remoteWrite.inmemoryQueues` to prioritize recently ingested data over historical data stored at file-based [persistent queue](https://docs.victoriametrics.com/victoriametrics/vmagent/#on-disk-persistence-and-data-processing-order). See [#8833](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8833)
* FEATURE: [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/): add `-promscrape.cluster.shardByLabels` command-line flag for selecting target labels used for sharding scrape targets among `vmagent` instances in cluster mode. See [#11044](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/11044).
* FEATURE: [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/): add `-promscrape.cluster.shardByLabels` command-line flag for selecting target labels used for sharding scrape targets among `vmagent` instances in cluster mode. See [#11044](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/11044).
* FEATURE: [vmctl](https://docs.victoriametrics.com/victoriametrics/vmctl/): add `-vm-headers` and `-vm-bearer-token` flags for authenticating requests to the VictoriaMetrics import destination. The flags are available in `opentsdb`, `influx`, `remote-read`, `prometheus`, `mimir`, and `thanos` vmctl sub-commands. See [#8897](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8897).
* FEATURE: [vmsingle](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/) and `vmselect` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/): log calls to [/api/v1/admin/tsdb/delete_series](https://docs.victoriametrics.com/victoriametrics/url-examples/#apiv1admintsdbdelete_series) API handler. This should help to identify events of metrics deletion from the database. See [#11104](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/11104).
* FEATURE: [vmui](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#vmui): add the `last` value to graph legend statistics. See [#10759](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/10759).
* FEATURE: [MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/): improve the selection algorithm of [buckets_limit](https://docs.victoriametrics.com/victoriametrics/metricsql/#buckets_limit) to remove consecutive empty buckets at the beginning and end to obtain more accurate min and max values. See [#10417](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/10417).
* BUGFIX: [enterprise](https://docs.victoriametrics.com/enterprise/) [vmsingle](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/) and `vmstorage` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/): properly expose metric `vm_retention_filters_partitions_scheduled_rows`. See [#11138](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/11138)
* BUGFIX: [stream aggregation](https://docs.victoriametrics.com/victoriametrics/stream-aggregation/): fix issue with producing aggregated samples with identical timestamps between flushes. See [#10808](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/10808).
* BUGFIX: [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/): fix potential corruption of remote-write metadata `Unit` values. See [#11120](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/11120). Thanks for @fxrlv for the contribution.
* BUGFIX: [vmalert](https://docs.victoriametrics.com/victoriametrics/vmalert/),[vmauth](https://docs.victoriametrics.com/victoriametrics/vmauth/),[vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/) and [vmsingle](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/): fix rare unbounded shutdown delay when config reload takes longer than `-configCheckInterval`. See [#11107](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/11107). Thanks to @PleasingFungus for contribution.
* BUGFIX: [vmbackup](https://docs.victoriametrics.com/vmbackup/), [vmbackupmanager](https://docs.victoriametrics.com/victoriametrics/vmbackupmanager/): do not fail backup list if directory is absent while using `fs://` destination to align with other protocols. See [6c3c548d](https://github.com/VictoriaMetrics/VictoriaMetrics/commit/6c3c548ddb0385b749e731f52276f130e2a4e4a8).
* BUGFIX: [vmctl](https://docs.victoriametrics.com/victoriametrics/vmctl/): push metrics to configured `-pushmetrics.url` on shutdown when migration fails. Previously, metrics were not pushed if vmctl exited with an error. See [#11081](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/11081). Thanks to @zasdaym for contribution.
* BUGFIX: [vmrestore](https://docs.victoriametrics.com/victoriametrics/vmrestore/): disallow restoring parts outside the configured `-storageDataPath` directory. See [710c920d](https://github.com/VictoriaMetrics/VictoriaMetrics/commit/710c920d6083327042a309e449fae4383617d817).
* BUGFIX: `vmselect` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/): correctly apply long tenant filters. Previously, such filters could be truncated, causing tenants to be matched incorrectly. See [#11096](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/11096). Thanks for @fxrlv for the contribution.
* BUGFIX: `vmselect` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/): fix corrupted metrics metadata when a response contains multiple rows. See [#11115](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/11115). Thanks for @fxrlv for the contribution.
* BUGFIX: [vmbackup](https://docs.victoriametrics.com/vmbackup/), [vmbackupmanager](https://docs.victoriametrics.com/victoriametrics/vmbackupmanager/): do not fail backup list if directory is absent while using `fs://` destination to align with other protocols. See [6c3c548](https://github.com/VictoriaMetrics/VictoriaMetrics/commit/6c3c548ddb0385b749e731f52276f130e2a4e4a8)
* BUGFIX: `vmselect` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/): don't cache empty responses for tenant IDs discovery during [multitenant queries](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#multitenant-reads). This problem was visible during integration tests when multitenant queries were executed before the first ingestion happened. See [#10982](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/10982)
* BUGFIX: [vmsingle](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/) and `vmselect` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/): properly escape `metricFamilyName` at metrics metadata response. See [#11129](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/11129). Thanks for @fxrlv for the contribution.
* BUGFIX: [vmsingle](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/) and `vmstorage` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/): prevent more cases of panic during directory deletion on `NFS`-based mounts. See [#11060](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/11060).
@@ -290,6 +296,25 @@ It enables back `Discovered targets` debug UI by default.
* BUGFIX: `vmstorage` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/): properly apply `extra_filters[]` filter when querying `vm_account_id` or `vm_project_id` labels via [multitenant](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#multitenancy) request for `/api/v1/label/…/values` API. Before, `extra_filters` was ignored. See [#10503](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/10503).
* BUGFIX: [vmsingle](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/) and `vmselect` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/): revert the use of rollup result cache for [instant queries](https://docs.victoriametrics.com/keyConcepts.html#instant-query) that contain [`rate`](https://docs.victoriametrics.com/MetricsQL.html#rate) function with a lookbehind window larger than `-search.minWindowForInstantRollupOptimization`. The cache usage was removed since [v1.132.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.132.0). See [#10098](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/10098#issuecomment-3895011084) for more details.
## [v1.136.12](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.136.12)
Released at 2026-06-19
**v1.136.x is a line of [LTS releases](https://docs.victoriametrics.com/victoriametrics/lts-releases/). It contains important up-to-date bugfixes for [VictoriaMetrics enterprise](https://docs.victoriametrics.com/victoriametrics/enterprise/).
All these fixes are also included in [the latest community release](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest).
The v1.136.x line will be supported for at least 12 months since [v1.136.0](https://docs.victoriametrics.com/victoriametrics/changelog/#v11360) release**
* BUGFIX: [stream aggregation](https://docs.victoriametrics.com/victoriametrics/stream-aggregation/): fix issue with producing aggregated samples with identical timestamps between flushes. See [#10808](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/10808).
* BUGFIX: [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/): fix potential corruption of remote-write metadata `Unit` values. See [#11120](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/11120). Thanks for @fxrlv for the contribution.
* BUGFIX: [vmalert](https://docs.victoriametrics.com/victoriametrics/vmalert/),[vmauth](https://docs.victoriametrics.com/victoriametrics/vmauth/),[vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/) and [vmsingle](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/): fix rare unbounded shutdown delay when config reload takes longer than `-configCheckInterval`. See [#11107](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/11107). Thanks to @PleasingFungus for contribution.
* BUGFIX: [vmbackup](https://docs.victoriametrics.com/vmbackup/), [vmbackupmanager](https://docs.victoriametrics.com/victoriametrics/vmbackupmanager/): do not fail backup list if directory is absent while using `fs://` destination to align with other protocols. See [6c3c548d](https://github.com/VictoriaMetrics/VictoriaMetrics/commit/6c3c548ddb0385b749e731f52276f130e2a4e4a8)
* BUGFIX: [vmctl](https://docs.victoriametrics.com/victoriametrics/vmctl/): push metrics to configured `-pushmetrics.url` on shutdown when migration fails. Previously, metrics were not pushed if vmctl exited with an error. See [#11081](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/11081). Thanks to @zasdaym for contribution.
* BUGFIX: [vmrestore](https://docs.victoriametrics.com/victoriametrics/vmrestore/): disallow restoring parts outside the configured `-storageDataPath` directory. See [710c920d](https://github.com/VictoriaMetrics/VictoriaMetrics/commit/710c920d6083327042a309e449fae4383617d817).
* BUGFIX: `vmselect` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/): fix corrupted metrics metadata when a response contains multiple rows. See [#11115](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/11115). Thanks for @fxrlv for the contribution.
* BUGFIX: [vmsingle](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/) and `vmselect` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/): properly escape `metricFamilyName` at metrics metadata response. See [#11129](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/11129). Thanks for @fxrlv for the contribution.
* BUGFIX: `vmselect` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/): correctly apply long tenant filters. Previously, such filters could be truncated, causing tenants to be matched incorrectly. See [#11096](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/11096). Thanks for @fxrlv for the contribution.
* BUGFIX: [vmsingle](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/) and `vmstorage` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/): prevent more cases of panic during directory deletion on `NFS`-based mounts. See [#11060](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/11060).
## [v1.136.11](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.136.11)
Released at 2026-06-05
@@ -651,6 +676,20 @@ See changes [here](https://docs.victoriametrics.com/victoriametrics/changelog/ch
See changes [here](https://docs.victoriametrics.com/victoriametrics/changelog/changelog_2025/#v11230)
## [v1.122.25](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.122.25)
Released at 2026-06-19
**v1.122.x is a line of [LTS releases](https://docs.victoriametrics.com/victoriametrics/lts-releases/). It contains important up-to-date bugfixes for [VictoriaMetrics enterprise](https://docs.victoriametrics.com/victoriametrics/enterprise/).
All these fixes are also included in [the latest community release](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest).
The v1.122.x line will be supported for at least 12 months since [v1.122.0](https://docs.victoriametrics.com/victoriametrics/changelog/#v11220) release**
* BUGFIX: [stream aggregation](https://docs.victoriametrics.com/victoriametrics/stream-aggregation/): fix issue with producing aggregated samples with identical timestamps between flushes. See [#10808](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/10808).
* BUGFIX: [vmalert](https://docs.victoriametrics.com/victoriametrics/vmalert/),[vmauth](https://docs.victoriametrics.com/victoriametrics/vmauth/),[vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/) and [vmsingle](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/): fix rare unbounded shutdown delay when config reload takes longer than `-configCheckInterval`. See [#11107](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/11107). Thanks to @PleasingFungus for contribution.
* BUGFIX: `vmselect` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/): fix corrupted metrics metadata when a response contains multiple rows. See [#11115](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/11115). Thanks for @fxrlv for the contribution.
* BUGFIX: [vmbackup](https://docs.victoriametrics.com/vmbackup/), [vmbackupmanager](https://docs.victoriametrics.com/victoriametrics/vmbackupmanager/): do not fail backup list if directory is absent while using `fs://` destination to align with other protocols. See [6c3c548d](https://github.com/VictoriaMetrics/VictoriaMetrics/commit/6c3c548ddb0385b749e731f52276f130e2a4e4a8)
* BUGFIX: [vmrestore](https://docs.victoriametrics.com/victoriametrics/vmrestore/): disallow restoring parts outside the configured `-storageDataPath` directory. See [710c920d](https://github.com/VictoriaMetrics/VictoriaMetrics/commit/710c920d6083327042a309e449fae4383617d817).
## [v1.122.24](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.122.24)
Released at 2026-06-05

View File

@@ -121,7 +121,7 @@ It is allowed to run Enterprise components in [cases listed here](https://docs.v
Binary releases of Enterprise components are available at [the releases page for VictoriaMetrics](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest),
[the releases page for VictoriaLogs](https://github.com/VictoriaMetrics/VictoriaLogs/releases/latest)
and [the releases page for VictoriaTraces](https://github.com/VictoriaMetrics/VictoriaTraces/releases/latest).
Enterprise binaries and packages have `enterprise` suffix in their names. For example, `victoria-metrics-linux-amd64-v1.145.0-enterprise.tar.gz`.
Enterprise binaries and packages have `enterprise` suffix in their names. For example, `victoria-metrics-linux-amd64-v1.146.0-enterprise.tar.gz`.
In order to run binary release of Enterprise component, please download the `*-enterprise.tar.gz` archive for your OS and architecture
from the corresponding releases page and unpack it. Then run the unpacked binary.
@@ -139,8 +139,8 @@ For example, the following command runs VictoriaMetrics Enterprise binary with t
obtained at [this page](https://victoriametrics.com/products/enterprise/trial/):
```sh
wget https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.145.0/victoria-metrics-linux-amd64-v1.145.0-enterprise.tar.gz
tar -xzf victoria-metrics-linux-amd64-v1.145.0-enterprise.tar.gz
wget https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.146.0/victoria-metrics-linux-amd64-v1.146.0-enterprise.tar.gz
tar -xzf victoria-metrics-linux-amd64-v1.146.0-enterprise.tar.gz
./victoria-metrics-prod -license=BASE64_ENCODED_LICENSE_KEY
```
@@ -155,7 +155,7 @@ Alternatively, VictoriaMetrics Enterprise license can be stored in the file and
It is allowed to run Enterprise components in [cases listed here](https://docs.victoriametrics.com/victoriametrics/enterprise/#valid-cases-for-victoriametrics-enterprise).
Docker images for Enterprise components are available at [VictoriaMetrics Docker Hub](https://hub.docker.com/u/victoriametrics) and [VictoriaMetrics Quay](https://quay.io/organization/victoriametrics).
Enterprise docker images have `enterprise` suffix in their names. For example, `victoriametrics/victoria-metrics:v1.145.0-enterprise`.
Enterprise docker images have `enterprise` suffix in their names. For example, `victoriametrics/victoria-metrics:v1.146.0-enterprise`.
In order to run Docker image of VictoriaMetrics Enterprise component, it is required to provide the license key via the command-line
flag as described in the [binary-releases](https://docs.victoriametrics.com/victoriametrics/enterprise/#binary-releases) section.
@@ -165,13 +165,13 @@ Enterprise license key can be obtained at [this page](https://victoriametrics.co
For example, the following command runs VictoriaMetrics Enterprise Docker image with the specified license key:
```sh
docker run --name=victoria-metrics victoriametrics/victoria-metrics:v1.145.0-enterprise -license=BASE64_ENCODED_LICENSE_KEY
docker run --name=victoria-metrics victoriametrics/victoria-metrics:v1.146.0-enterprise -license=BASE64_ENCODED_LICENSE_KEY
```
Alternatively, the license code can be stored in the file and then referred via `-licenseFile` command-line flag:
```sh
docker run --name=victoria-metrics -v /vm-license:/vm-license victoriametrics/victoria-metrics:v1.145.0-enterprise -licenseFile=/path/to/vm-license
docker run --name=victoria-metrics -v /vm-license:/vm-license victoriametrics/victoria-metrics:v1.146.0-enterprise -licenseFile=/path/to/vm-license
```
Example docker-compose configuration:
@@ -181,7 +181,7 @@ version: "3.5"
services:
victoriametrics:
container_name: victoriametrics
image: victoriametrics/victoria-metrics:v1.145.0
image: victoriametrics/victoria-metrics:v1.146.0
ports:
- 8428:8428
volumes:
@@ -213,7 +213,7 @@ is used to provide the license key in plain-text:
```yaml
server:
image:
tag: v1.145.0-enterprise
tag: v1.146.0-enterprise
license:
key: {BASE64_ENCODED_LICENSE_KEY}
@@ -224,7 +224,7 @@ In order to provide the license key via existing secret, the following values fi
```yaml
server:
image:
tag: v1.145.0-enterprise
tag: v1.146.0-enterprise
license:
secret:
@@ -274,7 +274,7 @@ spec:
license:
key: {BASE64_ENCODED_LICENSE_KEY}
image:
tag: v1.145.0-enterprise
tag: v1.146.0-enterprise
```
In order to provide the license key via an existing secret, the following custom resource is used:
@@ -291,7 +291,7 @@ spec:
name: vm-license
key: license
image:
tag: v1.145.0-enterprise
tag: v1.146.0-enterprise
```
Example secret with license key:
@@ -342,7 +342,7 @@ Builds are available for amd64 and arm64 architectures.
Example archive:
`victoria-metrics-linux-amd64-v1.145.0-enterprise.tar.gz`
`victoria-metrics-linux-amd64-v1.146.0-enterprise.tar.gz`
Includes:
@@ -351,7 +351,7 @@ Includes:
Example Docker image:
`victoriametrics/victoria-metrics:v1.145.0-enterprise-fips` uses the FIPS-compatible binary and based on `scratch` image.
`victoriametrics/victoria-metrics:v1.146.0-enterprise-fips` uses the FIPS-compatible binary and based on `scratch` image.
## What Happens to Licensed Components When a License Expires

View File

@@ -16,7 +16,7 @@ aliases:
1. The main goal - **to help users and [clients](https://docs.victoriametrics.com/victoriametrics/enterprise/) using VictoriaMetrics products in the most efficient way**.
1. Fixing bugs in the essential functionality of VictoriaMetrics components. Small usability bugs are usually the most annoying,
so they **must be fixed first**. Bugs, which affect a small number of users at some rare edge cases, can be fixed later.
1. Improving [public docs for VictoriaMetrics products](https://docs.victoriametrics.com).
1. Improving [public docs for VictoriaMetrics products](https://docs.victoriametrics.com),
so users could find answers to their questions via Google or any other AI-powered web search without the need
to ask these questions at our [support channels](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#community-and-contributions).
1. Simplifying usage of VictoriaMetrics products without breaking backwards compatibility, so users could regularly

View File

@@ -35,8 +35,8 @@ scrape_configs:
After you created the `scrape.yaml` file, download and unpack [single-node VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/) to the same directory:
```sh
wget https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.145.0/victoria-metrics-linux-amd64-v1.145.0.tar.gz
tar xzf victoria-metrics-linux-amd64-v1.145.0.tar.gz
wget https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.146.0/victoria-metrics-linux-amd64-v1.146.0.tar.gz
tar xzf victoria-metrics-linux-amd64-v1.146.0.tar.gz
```
Then start VictoriaMetrics and instruct it to scrape targets defined in `scrape.yaml` and save scraped metrics
@@ -150,8 +150,8 @@ Then start [single-node VictoriaMetrics](https://docs.victoriametrics.com/victor
```yaml
# Download and unpack single-node VictoriaMetrics
wget https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.145.0/victoria-metrics-linux-amd64-v1.145.0.tar.gz
tar xzf victoria-metrics-linux-amd64-v1.145.0.tar.gz
wget https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.146.0/victoria-metrics-linux-amd64-v1.146.0.tar.gz
tar xzf victoria-metrics-linux-amd64-v1.146.0.tar.gz
# Run single-node VictoriaMetrics with the given scrape.yaml
./victoria-metrics-prod -promscrape.config=scrape.yaml

View File

@@ -624,13 +624,12 @@ command line flags. See how to [shard data across remote write destinations](htt
The following requirements must be met for sharded aggregation to work correctly:
- All sharding vmagents should have the same deterministic sharding configuration.
- The sharding configuration must align with the `by` and `without` lists:
- Labels listed in `by` setting should be a subset of shard's routing key `-remoteWrite.shardByURL.labels`.
With `-remoteWrite.shardByURL.labels=env,job` aggregator's `by` should include `by: env`, `by: job` or both: `by: [env, job]`.
This makes sure that all the samples for the same `env` and `job` are aggregated together and produce the complete output.
- Labels listed in `without` setting should be a superset of shard's routing key `--remoteWrite.shardByURL.ignoreLabels`.
With `-remoteWrite.shardByURL.ignoreLabels=env,job` aggegator's `without` should include at least both labels `without: [env,job]`.
This makes sure that `requests_total{env=test, job=foo}` and `requests_total{env=prod, job=foo}` are routed to the same aggregator
and are aggregated together. See also [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5938#issuecomment-2018470324).
- Labels configured in `-remoteWrite.shardByURL.labels` must be a subset of the labels listed in `by`.
For example, if the aggregation config specifies `by: [env, job]`, then `-remoteWrite.shardByURL.labels` may include `env`, `job`, or both.
This ensures that all samples contributing to the same aggregation result are routed to the same aggregator instance and aggregated together to produce a complete output.
- Labels configured in `-remoteWrite.shardByURL.ignoreLabels` must be a superset of the labels listed in `without`.
For example, if the aggregation config specifies `without: [env, pod]`, then `-remoteWrite.shardByURL.ignoreLabels` must include at least `env` and `pod`.
This ensures that labels removed during aggregation are not used for shard routing.
- Aggregating vmagents should not produce collisions: the aggregation output should be unique across all the sharded agents.
For example, `requests_total:5m_without_env_pod_total` produced by both `vmagent-aggr-1` and `vmagent-aggr-2` will collide
unless they have labels uniquely identifying them. These labels should be either preserved during sharding and aggregation config,

View File

@@ -222,6 +222,7 @@ Below are aggregation functions that can be put in the `outputs` list at [stream
* [stddev](#stddev)
* [stdvar](#stdvar)
* [sum_samples](#sum_samples)
* [sum_samples_total](#sum_samples_total)
* [total](#total)
* [total_prometheus](#total_prometheus)
* [unique_samples](#unique_samples)
@@ -505,10 +506,11 @@ See also:
- [count_samples](#count_samples)
- [count_series](#count_series)
- [sum_samples_total](#sum_samples_total)
### `sum_samples_total`
`sum_samples_total` sums input delta values into a cumulative [counter](https://docs.victoriametrics.com/victoriametrics/keyconcepts/index.html#counter) and outputs the result at the given `interval`.
`sum_samples_total` {{% available_from "v1.146.0" %}}. sums input delta values into a cumulative [counter](https://docs.victoriametrics.com/victoriametrics/keyconcepts/index.html#counter) and outputs the result at the given `interval`.
`sum_samples_total` makes sense only for aggregating delta values from clients such as [StatsD counter](https://github.com/statsd/statsd/blob/master/docs/metric_types.md#counting).
The results of `sum_samples_total` is roughly equal to the following [MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/) query:
@@ -559,6 +561,7 @@ See also:
- [total_prometheus](#total_prometheus)
- [increase](#increase)
- [increase_prometheus](#increase_prometheus)
- [sum_samples_total](#sum_samples_total)
- [rate_sum](#rate_sum)
- [rate_avg](#rate_avg)
@@ -588,6 +591,7 @@ See also:
- [total](#total)
- [increase](#increase)
- [increase_prometheus](#increase_prometheus)
- [sum_samples_total](#sum_samples_total)
- [rate_sum](#rate_sum)
- [rate_avg](#rate_avg)

View File

@@ -622,11 +622,13 @@ curl -Is http://localhost:8428/internal/resetRollupResultCache
Cluster version of VictoriaMetrics:
```sh
curl -Is http://<vmselect>:8481/internal/resetRollupResultCache
curl -Is http://<vmselect>:8481/internal/resetRollupResultCache?propagate=1
```
vmselect will propagate this call to the rest of the vmselects listed in its `-selectNode` cmd-line flag. If this
flag isn't set, then cache need to be purged from each vmselect individually.
vmselect will propagate this call to the rest of the vmselects listed in its `-selectNode` cmd-line flag when `propagate=1` argument is set.
If this flag or the `propagate` argument isn't set, then cache need to be purged from each vmselect individually.
If `-search.resetCacheAuthKey` is set, it will be attached to the propagation request as query argument.
### TCP and UDP

View File

@@ -275,6 +275,10 @@ See the docs at https://docs.victoriametrics.com/victoriametrics/
Optional name of the cluster. If multiple vmagent clusters scrape the same targets, then each cluster must have unique name in order to properly de-duplicate samples received from these clusters. See https://docs.victoriametrics.com/victoriametrics/vmagent/#scraping-big-number-of-targets for more info
-promscrape.cluster.replicationFactor int
The number of members in the cluster, which scrape the same targets. If the replication factor is greater than 1, then the deduplication must be enabled at remote storage side. See https://docs.victoriametrics.com/victoriametrics/vmagent/#scraping-big-number-of-targets for more info (default 1)
-promscrape.cluster.shardByLabels array
Optional list of target labels, which will be used for sharding targets among cluster members if -promscrape.cluster.membersCount is greater than 1. If none of the specified labels are found in a target, then all the target labels will be used for sharding. See https://docs.victoriametrics.com/victoriametrics/vmagent/#scraping-big-number-of-targets for more info
Supports an array of values separated by comma or specified via multiple flags.
Each array item can contain comma inside single-quoted or double-quoted string, {}, [] and () braces.
-promscrape.config string
Optional path to Prometheus config file with 'scrape_configs' section containing targets to scrape. The path can point to local file and to http url. See https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#how-to-scrape-prometheus-exporters-such-as-node-exporter for details
-promscrape.config.dryRun
@@ -486,13 +490,13 @@ See the docs at https://docs.victoriametrics.com/victoriametrics/
-search.maxTSDBStatusTopNSeries int
The maximum value of 'topN' argument that can be passed to /api/v1/status/tsdb API. This option allows limiting memory usage. See https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#tsdb-stats (default 1000)
-search.maxTagKeys int
The maximum number of tag keys returned from /api/v1/labels . See also -search.maxLabelsAPISeries and -search.maxLabelsAPIDuration (default 100000)
The maximum number of tag keys returned per search. See also -search.maxLabelsAPISeries and -search.maxLabelsAPIDuration (default 100000)
-search.maxTagValueSuffixesPerSearch int
The maximum number of tag value suffixes returned from /metrics/find (default 100000)
-search.maxTagValues int
The maximum number of tag values returned from /api/v1/label/<label_name>/values . See also -search.maxLabelsAPISeries and -search.maxLabelsAPIDuration (default 100000)
The maximum number of tag values returned per search. See also -search.maxLabelsAPISeries and -search.maxLabelsAPIDuration (default 100000)
-search.maxUniqueTimeseries int
The maximum number of unique time series, which can be selected during /api/v1/query and /api/v1/query_range queries. This option allows limiting memory usage. When set to zero, the limit is automatically calculated based on -search.maxConcurrentRequests (inversely proportional) and memory available to the process (proportional).
The maximum number of unique time series, which can be scanned during every query. This allows protecting against heavy queries, which select unexpectedly high number of series. When set to zero, the limit is automatically calculated based on -search.maxConcurrentRequests (inversely proportional) and memory available to the process (proportional). See also -search.max* command-line flags at vmselect
-search.maxWorkersPerQuery int
The maximum number of CPU cores a single query can use. The default value should work good for most cases. The flag can be set to lower values for improving performance of big number of concurrently executed queries. The flag can be set to bigger values for improving performance of heavy queries, which scan big number of time series (>10K) and/or big number of samples (>100M). There is no sense in setting this flag to values bigger than the number of CPU cores available on the system (default netstorage.defaultMaxWorkersPerQuery())
-search.minStalenessInterval duration

View File

@@ -309,11 +309,11 @@ Scraping has additional settings that can be applied before samples are pushed t
`vmagent` supports [the same set of push-based data ingestion protocols as VictoriaMetrics does](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#how-to-import-time-series-data)
in addition to the pull-based Prometheus-compatible targets' scraping:
* DataDog "submit metrics" API. See [these docs](https://docs.victoriametrics.com/victoriametrics/integrations/datadog/).
* Datadog "submit metrics" API. See [these docs](https://docs.victoriametrics.com/victoriametrics/integrations/datadog/).
* InfluxDB line protocol via `http://<vmagent>:8429/write`. See [these docs](https://docs.victoriametrics.com/victoriametrics/integrations/influxdb/).
* Graphite plaintext protocol if the `-graphiteListenAddr` command-line flag is set. See [these docs](https://docs.victoriametrics.com/victoriametrics/integrations/graphite/#ingesting).
* OpenTelemetry HTTP API. See [these docs](https://docs.victoriametrics.com/victoriametrics/integrations/opentelemetry/).
* NewRelic API. See [these docs](https://docs.victoriametrics.com/victoriametrics/integrations/newrelic/#sending-data-from-agent).
* OpenTelemetry HTTP API via `http://<vmagent>:8429/opentelemetry/v1/metrics`. See [these docs](https://docs.victoriametrics.com/victoriametrics/integrations/opentelemetry/).
* New Relic API. See [these docs](https://docs.victoriametrics.com/victoriametrics/integrations/newrelic/#sending-data-from-agent).
* OpenTSDB telnet and http protocols if `-opentsdbListenAddr` command-line flag is set. See [these docs](https://docs.victoriametrics.com/victoriametrics/integrations/opentsdb/).
* Zabbix Connector streaming protocol. See [these docs](https://docs.victoriametrics.com/victoriametrics/integrations/zabbixconnector/#send-data-from-zabbix-connector).
* Prometheus remote write protocol via `http://<vmagent>:8429/api/v1/write`.
@@ -481,29 +481,38 @@ by specifying `-remoteWrite.forcePromProto` command-line flag for the correspond
## Multitenancy
By default, `vmagent` collects the data without [tenant](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#multitenancy) identifiers
and routes it to the remote storage specified via `-remoteWrite.url` command-line flag. The `-remoteWrite.url` can point to `/insert/<tenant_id>/prometheus/api/v1/write` path
at `vminsert` according to [these docs](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#url-format).
and routes it to the remote storage specified via `-remoteWrite.url` command-line flag. Point `-remoteWrite.url` to vminsert's `/insert/<tenant_id>/prometheus/api/v1/write` path
according to [these docs](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#url-format).
> Note: the single-node version of VictoriaMetrics doesn't support multitenancy.
```mermaid
flowchart LR
A["requests_total{instance=foo}"] --> |/api/v1/write| V[vmagent]
B["requests_total{instance=bar}"] <--> |scrape| V
V --> |"/insert/#60;tenant_id#62;/#60;suffix#62;"| C[vminsert]
A["requests_total{instance=foo}"] --> |<a href="https://docs.victoriametrics.com/victoriametrics/vmagent/#how-to-push-data-to-vmagent">push</a>| V[vmagent]
B["requests_total{instance=bar}"] <--> |<a href="https://docs.victoriametrics.com/victoriametrics/vmagent/#how-to-collect-metrics-in-prometheus-format">pull</a>| V
V --> |"/insert/#60;tenant_id#62;/prometheus/api/v1/write"| C[vminsert]
```
In this case, all the metrics written to `/insert/tenant_id/prometheus/api/v1/write` will belong to the specified `<tenant_id>` tenant.
In this case, all the metrics written to `/insert/<tenant_id>/prometheus/api/v1/write` will belong to the specified `<tenant_id>` tenant.
### Multitenancy via labels
vmagent can write data to multiple distinct tenants if `-remoteWrite.url` points to [multitenant URL at VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#multitenancy-via-labels)
and tenant is specified via [multitenancy labels](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#multitenancy-via-labels):
vmagent can write data to **multiple distinct tenants** if `-remoteWrite.url` points to the [multitenant URL in the VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#multitenancy-via-labels)
and the tenant is specified via [multitenancy labels](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#multitenancy-via-labels):
```mermaid
flowchart LR
A["requests_total{instance=foo, vm_account_id=0}"] --> |/api/v1/write| V[vmagent]
B["requests_total{instance=bar, vm_account_id=1}"] <--> |scrape| V
V --> |"/insert/multitenant/#60;suffix#62;"| C[vminsert]
A["requests_total{instance=foo, vm_account_id=0}"] --> |<a href="https://docs.victoriametrics.com/victoriametrics/vmagent/#how-to-push-data-to-vmagent">push</a>| V[vmagent]
B["requests_total{instance=bar, vm_account_id=1}"] <--> |<a href="https://docs.victoriametrics.com/victoriametrics/vmagent/#how-to-collect-metrics-in-prometheus-format">pull</a>| V
V --> |"/insert/multitenant/prometheus/api/v1/write"| C[vminsert]
```
`<tenant_id>` is extracted from the `vm_account_id` and `vm_project_id` labels.
> A single payload pulled from or pushed to vmagent may contain time series belonging to multiple tenants.
When vminsert receives the data on the `/insert/multitenant` path, it extracts `<tenant_id>` from the `vm_account_id` and `vm_project_id` labels for
each distinct time series.
> If `vm_account_id` or `vm_project_id` labels are missing or invalid, then the corresponding accountID and projectID are set to 0.
The `vm_account_id` and `vm_project_id` labels can be specified via [relabeling](https://docs.victoriametrics.com/victoriametrics/relabeling/) before sending the metrics to `-remoteWrite.url`.
For example, the following relabeling rule instructs sending metrics to `<account_id>:0` tenant defined in the `prometheus.io/account_id` annotation of Kubernetes pod deployment:
@@ -516,11 +525,12 @@ scrape_configs:
target_label: vm_account_id
```
vmagent can get tenant identifier from `__tenant_id__` label at target discovery phase.
It implicitly converts `__tenant_id__` label into `vm_account_id` and `vm_project_id` labels and attaches
it to the scraped metrics and metrics metadata.
vmagent can get the tenant identifier from the `__tenant_id__` label during the target discovery phase.
It implicitly converts the `__tenant_id__` label into `vm_account_id` and `vm_project_id` labels and attaches
them to the scraped metrics and metrics metadata.
For example, the following relabeling rule instructs sending metrics to the `10:5` tenant defined in the `prometheus.io/tenant_id: 10:5` annotation of the Kubernetes pod deployment:
For example, the following relabeling rule instructs sending metrics to the `10:5` tenant defined in the `prometheus.io/tenant_id: 10:5`
annotation of the Kubernetes pod deployment:
```yaml
scrape_configs:
@@ -531,47 +541,54 @@ scrape_configs:
target_label: __tenant_id__
```
vmagent can [enforce adding labels](https://docs.victoriametrics.com/victoriametrics/vmagent/#adding-labels-to-metrics) to all scraped
or forwarded metrics.
vmagent can also [enforce adding labels](https://docs.victoriametrics.com/victoriametrics/vmagent/#adding-labels-to-metrics)
on all scraped or forwarded metrics.
### Multitenancy via path
vmagent can write data to multiple distinct tenants if `-remoteWrite.url` points to [multitenant URL at VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#multitenancy-via-labels),
tenant is specified in the [write path](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#url-format), and `-enableMultitenantHandlers` command-line flag is set:
vmagent can write data to multiple distinct tenants if:
* its `-remoteWrite.url` points to the [VictoriaMetrics cluster multitenant URL](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#multitenancy-via-labels)
* its `-enableMultitenantHandlers` command-line flag is set
* clients ingest data into vmagent with the tenant specified in the [write path](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#url-format)
```mermaid
flowchart LR
A["requests_total{instance=foo}"] --> |/insert/0/#60;suffix#62;| V[vmagent]
B["requests_total{instance=bar}"] --> |/insert/1/#60;suffix#62;| V
V --> |"/insert/multitenant/#60;suffix#62;"| C[vminsert]
A["requests_total{instance=foo}"] --> |<a href="https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#url-format">/insert/0/#60;suffix#62;</a>| V[vmagent]
B["requests_total{instance=bar}"] --> |<a href="https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#url-format">/insert/1/#60;suffix#62;</a>| V
V --> |"/insert/multitenant/prometheus/api/v1/write"| C[vminsert]
```
In this configuration, vmagent accepts writes via the same multitenant endpoints (`/insert/<accountID>/<suffix>`) [as vminsert does](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#url-format).
For all received data, vmagent will automatically convert tenant identifiers from the URL to `vm_account_id` and `vm_project_id` labels and set tenant info in metadata.
For all the received data, vmagent will automatically convert tenant identifiers in the URL path to `vm_account_id` and `vm_project_id` labels, and set tenant information in metadata.
These tenant labels are added before applying [relabeling](https://docs.victoriametrics.com/victoriametrics/relabeling/) specified via `-remoteWrite.relabelConfig`
and `-remoteWrite.urlRelabelConfig` command-line flags.
These tenant labels are added before applying [relabeling](https://docs.victoriametrics.com/victoriametrics/relabeling/)
specified via `-remoteWrite.relabelConfig` and `-remoteWrite.urlRelabelConfig` command-line flags.
### Multitenancy via headers
vmagent can write data to multiple distinct tenants if `-remoteWrite.url` points to [multitenant URL at VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#multitenancy-via-labels),
tenant is specified [via headers](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#multitenancy-via-headers) {{% available_from "v1.143.0" %}}, both `-enableMultitenantHandlers` and `-enableMultitenancyViaHeaders` command-line flags are set:
vmagent can write data to multiple distinct tenants if:
* its `-remoteWrite.url` points to the [VictoriaMetrics cluster multitenant URL](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#multitenancy-via-labels)
* its `-enableMultitenantHandlers` and `-enableMultitenancyViaHeaders` command-line flags are both set
* clients ingest data into vmagent with the tenants specified [via headers](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#multitenancy-via-headers) {{% available_from "v1.143.0" %}}
```mermaid
flowchart LR
A["requests_total{instance=foo}"] --> |/insert/#60;suffix#62; <br>--header AccountID: 0| V[vmagent]
B["requests_total{instance=bar}"] --> |/insert/#60;suffix#62; <br>--header AccountID: 1| V
V --> |"/insert/multitenant/#60;suffix#62;"| C[vminsert]
A["requests_total{instance=foo}"] --> |<a href="https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#multitenancy-via-headers">/insert/#60;suffix#62;</a> <br>--header AccountID: 0| V[vmagent]
B["requests_total{instance=bar}"] --> |<a href="https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#multitenancy-via-headers">/insert/#60;suffix#62;</a> <br>--header AccountID: 1| V
V --> |"/insert/multitenant/prometheus/api/v1/write"| C[vminsert]
```
In this configuration, vmagent accepts writes via the same simplified multitenant endpoints (`/insert/<suffix>`) [as vminsert does](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#url-format).
The tenant information is extracted from the `AccountID` and `ProjectID` HTTP headers, which are expected to be included in all incoming requests. If headers are missing, then the tenant is set to `0:0` as the default.
The tenant information is extracted from the `AccountID` and `ProjectID` HTTP headers, which are expected to be included in all incoming requests.
For all received data, vmagent will automatically convert tenant identifiers from the headers to `vm_account_id` and `vm_project_id` labels and set tenant info in metadata.
These tenant labels are added before applying [relabeling](https://docs.victoriametrics.com/victoriametrics/relabeling/) specified via `-remoteWrite.relabelConfig`
> If headers are missing, then the tenant is set to `0:0` by default.
For all the received data, vmagent will automatically convert tenant identifiers in the headers to `vm_account_id` and `vm_project_id` labels, and set tenant info in metadata.
These tenant labels are added before applying [relabeling](https://docs.victoriametrics.com/victoriametrics/relabeling/) specified via the `-remoteWrite.relabelConfig`
and `-remoteWrite.urlRelabelConfig` command-line flags.
vmauth can [enforce adding headers](https://docs.victoriametrics.com/victoriametrics/vmauth/#modifying-http-headers) to all
forwarded requests via `headers` param in the config file.
forwarded requests via the `headers` parameter in the config file.
## Adding labels to metrics

View File

@@ -240,6 +240,10 @@ See the docs at https://docs.victoriametrics.com/victoriametrics/vmagent/ .
Optional name of the cluster. If multiple vmagent clusters scrape the same targets, then each cluster must have unique name in order to properly de-duplicate samples received from these clusters. See https://docs.victoriametrics.com/victoriametrics/vmagent/#scraping-big-number-of-targets for more info
-promscrape.cluster.replicationFactor int
The number of members in the cluster, which scrape the same targets. If the replication factor is greater than 1, then the deduplication must be enabled at remote storage side. See https://docs.victoriametrics.com/victoriametrics/vmagent/#scraping-big-number-of-targets for more info (default 1)
-promscrape.cluster.shardByLabels array
Optional list of target labels, which will be used for sharding targets among cluster members if -promscrape.cluster.membersCount is greater than 1. If none of the specified labels are found in a target, then all the target labels will be used for sharding. See https://docs.victoriametrics.com/victoriametrics/vmagent/#scraping-big-number-of-targets for more info
Supports an array of values separated by comma or specified via multiple flags.
Each array item can contain comma inside single-quoted or double-quoted string, {}, [] and () braces.
-promscrape.config string
Optional path to Prometheus config file with 'scrape_configs' section containing targets to scrape. The path can point to local file and to http url. See https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#how-to-scrape-prometheus-exporters-such-as-node-exporter for details
-promscrape.config.dryRun
@@ -435,6 +439,10 @@ See the docs at https://docs.victoriametrics.com/victoriametrics/vmagent/ .
Optional HTTP headers to send with each request to the corresponding -remoteWrite.url. For example, -remoteWrite.headers='My-Auth:foobar' would send 'My-Auth: foobar' HTTP header with every request to the corresponding -remoteWrite.url. Multiple headers must be delimited by '^^': -remoteWrite.headers='header1:value1^^header2:value2'
Supports an array of values separated by comma or specified via multiple flags.
Each array item can contain comma inside single-quoted or double-quoted string, {}, [] and () braces.
-remoteWrite.inmemoryQueues array
The number of additional workers per each -remoteWrite.url, which send only recently ingested data from the in-memory queue, while the file-based queue at -remoteWrite.tmpDataPath is drained by workers configured via -remoteWrite.queues. This reduces delivery lag for fresh samples when the file-based queue contains a backlog accumulated during remote storage outages. (default 0)
Supports array of values separated by comma or specified via multiple flags.
Empty values are set to default value.
-remoteWrite.keepDanglingQueues
Keep persistent queues contents at -remoteWrite.tmpDataPath in case there are no matching -remoteWrite.url. Useful when -remoteWrite.url is changed temporarily and persistent queue files will be needed later on.
-remoteWrite.label array

View File

@@ -34,9 +34,9 @@ vmctl command-line tool is available as:
Download and unpack vmctl:
```sh
wget https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.145.0/vmutils-darwin-arm64-v1.145.0.tar.gz
wget https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.146.0/vmutils-darwin-arm64-v1.146.0.tar.gz
tar xzf vmutils-darwin-arm64-v1.145.0.tar.gz
tar xzf vmutils-darwin-arm64-v1.146.0.tar.gz
```
Once binary is unpacked, see the full list of supported modes by running the following command:

View File

@@ -46,9 +46,13 @@ OPTIONS:
Should be the same as --httpListenAddr value for single-node version or vminsert component.
When importing into the clustered version do not forget to set additionally --vm-account-id flag.
Please note, that vmctl performs initial readiness check for the given address by checking /health endpoint. (default: "http://localhost:8428")
--vm-user value VictoriaMetrics username for basic auth [$VM_USERNAME]
--vm-password value VictoriaMetrics password for basic auth [$VM_PASSWORD]
--vm-account-id value AccountID is an arbitrary 32-bit integer identifying namespace for data ingestion (aka tenant).
--vm-user value VictoriaMetrics username for basic auth [$VM_USERNAME]
--vm-password value VictoriaMetrics password for basic auth [$VM_PASSWORD]
--vm-headers value Optional HTTP headers to send with each request to the corresponding destination address.
For example, --vm-headers='My-Auth:foobar' would send 'My-Auth: foobar' HTTP header with every request to the corresponding destination address.
Multiple headers must be delimited by '^^': --vm-headers='header1:value1^^header2:value2'
--vm-bearer-token value Optional bearer auth token to use for the corresponding --vm-addr
--vm-account-id value AccountID is an arbitrary 32-bit integer identifying namespace for data ingestion (aka tenant).
AccountID is required when importing into the clustered version of VictoriaMetrics.
It is possible to set it as accountID:projectID, where projectID is also arbitrary 32-bit integer.
If projectID isn't set, then it equals to 0

View File

@@ -42,9 +42,13 @@ OPTIONS:
Should be the same as --httpListenAddr value for single-node version or vminsert component.
When importing into the clustered version do not forget to set additionally --vm-account-id flag.
Please note, that vmctl performs initial readiness check for the given address by checking /health endpoint. (default: "http://localhost:8428")
--vm-user value VictoriaMetrics username for basic auth [$VM_USERNAME]
--vm-password value VictoriaMetrics password for basic auth [$VM_PASSWORD]
--vm-account-id value AccountID is an arbitrary 32-bit integer identifying namespace for data ingestion (aka tenant).
--vm-user value VictoriaMetrics username for basic auth [$VM_USERNAME]
--vm-password value VictoriaMetrics password for basic auth [$VM_PASSWORD]
--vm-headers value Optional HTTP headers to send with each request to the corresponding destination address.
For example, --vm-headers='My-Auth:foobar' would send 'My-Auth: foobar' HTTP header with every request to the corresponding destination address.
Multiple headers must be delimited by '^^': --vm-headers='header1:value1^^header2:value2'
--vm-bearer-token value Optional bearer auth token to use for the corresponding --vm-addr
--vm-account-id value AccountID is an arbitrary 32-bit integer identifying namespace for data ingestion (aka tenant).
AccountID is required when importing into the clustered version of VictoriaMetrics.
It is possible to set it as accountID:projectID, where projectID is also arbitrary 32-bit integer.
If projectID isn't set, then it equals to 0

View File

@@ -41,9 +41,13 @@ OPTIONS:
Should be the same as --httpListenAddr value for single-node version or vminsert component.
When importing into the clustered version do not forget to set additionally --vm-account-id flag.
Please note, that vmctl performs initial readiness check for the given address by checking /health endpoint. (default: "http://localhost:8428")
--vm-user value VictoriaMetrics username for basic auth [$VM_USERNAME]
--vm-password value VictoriaMetrics password for basic auth [$VM_PASSWORD]
--vm-account-id value AccountID is an arbitrary 32-bit integer identifying namespace for data ingestion (aka tenant).
--vm-user value VictoriaMetrics username for basic auth [$VM_USERNAME]
--vm-password value VictoriaMetrics password for basic auth [$VM_PASSWORD]
--vm-headers value Optional HTTP headers to send with each request to the corresponding destination address.
For example, --vm-headers='My-Auth:foobar' would send 'My-Auth: foobar' HTTP header with every request to the corresponding destination address.
Multiple headers must be delimited by '^^': --vm-headers='header1:value1^^header2:value2'
--vm-bearer-token value Optional bearer auth token to use for the corresponding --vm-addr
--vm-account-id value AccountID is an arbitrary 32-bit integer identifying namespace for data ingestion (aka tenant).
AccountID is required when importing into the clustered version of VictoriaMetrics.
It is possible to set it as accountID:projectID, where projectID is also arbitrary 32-bit integer.
If projectID isn't set, then it equals to 0

View File

@@ -34,9 +34,13 @@ OPTIONS:
Should be the same as --httpListenAddr value for single-node version or vminsert component.
When importing into the clustered version do not forget to set additionally --vm-account-id flag.
Please note, that vmctl performs initial readiness check for the given address by checking /health endpoint. (default: "http://localhost:8428")
--vm-user value VictoriaMetrics username for basic auth [$VM_USERNAME]
--vm-password value VictoriaMetrics password for basic auth [$VM_PASSWORD]
--vm-account-id value AccountID is an arbitrary 32-bit integer identifying namespace for data ingestion (aka tenant).
--vm-user value VictoriaMetrics username for basic auth [$VM_USERNAME]
--vm-password value VictoriaMetrics password for basic auth [$VM_PASSWORD]
--vm-headers value Optional HTTP headers to send with each request to the corresponding destination address.
For example, --vm-headers='My-Auth:foobar' would send 'My-Auth: foobar' HTTP header with every request to the corresponding destination address.
Multiple headers must be delimited by '^^': --vm-headers='header1:value1^^header2:value2'
--vm-bearer-token value Optional bearer auth token to use for the corresponding --vm-addr
--vm-account-id value AccountID is an arbitrary 32-bit integer identifying namespace for data ingestion (aka tenant).
AccountID is required when importing into the clustered version of VictoriaMetrics.
It is possible to set it as accountID:projectID, where projectID is also arbitrary 32-bit integer.
If projectID isn't set, then it equals to 0

View File

@@ -49,9 +49,13 @@ OPTIONS:
Should be the same as --httpListenAddr value for single-node version or vminsert component.
When importing into the clustered version do not forget to set additionally --vm-account-id flag.
Please note, that vmctl performs initial readiness check for the given address by checking /health endpoint. (default: "http://localhost:8428")
--vm-user value VictoriaMetrics username for basic auth [$VM_USERNAME]
--vm-password value VictoriaMetrics password for basic auth [$VM_PASSWORD]
--vm-account-id value AccountID is an arbitrary 32-bit integer identifying namespace for data ingestion (aka tenant).
--vm-user value VictoriaMetrics username for basic auth [$VM_USERNAME]
--vm-password value VictoriaMetrics password for basic auth [$VM_PASSWORD]
--vm-headers value Optional HTTP headers to send with each request to the corresponding destination address.
For example, --vm-headers='My-Auth:foobar' would send 'My-Auth: foobar' HTTP header with every request to the corresponding destination address.
Multiple headers must be delimited by '^^': --vm-headers='header1:value1^^header2:value2'
--vm-bearer-token value Optional bearer auth token to use for the corresponding --vm-addr
--vm-account-id value AccountID is an arbitrary 32-bit integer identifying namespace for data ingestion (aka tenant).
AccountID is required when importing into the clustered version of VictoriaMetrics.
It is possible to set it as accountID:projectID, where projectID is also arbitrary 32-bit integer.
If projectID isn't set, then it equals to 0

View File

@@ -34,9 +34,13 @@ OPTIONS:
Should be the same as --httpListenAddr value for single-node version or vminsert component.
When importing into the clustered version do not forget to set additionally --vm-account-id flag.
Please note, that vmctl performs initial readiness check for the given address by checking /health endpoint. (default: "http://localhost:8428")
--vm-user value VictoriaMetrics username for basic auth [$VM_USERNAME]
--vm-password value VictoriaMetrics password for basic auth [$VM_PASSWORD]
--vm-account-id value AccountID is an arbitrary 32-bit integer identifying namespace for data ingestion (aka tenant).
--vm-user value VictoriaMetrics username for basic auth [$VM_USERNAME]
--vm-password value VictoriaMetrics password for basic auth [$VM_PASSWORD]
--vm-headers value Optional HTTP headers to send with each request to the corresponding destination address.
For example, --vm-headers='My-Auth:foobar' would send 'My-Auth: foobar' HTTP header with every request to the corresponding destination address.
Multiple headers must be delimited by '^^': --vm-headers='header1:value1^^header2:value2'
--vm-bearer-token value Optional bearer auth token to use for the corresponding --vm-addr
--vm-account-id value AccountID is an arbitrary 32-bit integer identifying namespace for data ingestion (aka tenant).
AccountID is required when importing into the clustered version of VictoriaMetrics.
It is possible to set it as accountID:projectID, where projectID is also arbitrary 32-bit integer.
If projectID isn't set, then it equals to 0

View File

@@ -0,0 +1,530 @@
---
weight: 13
menu:
docs:
parent: victoriametrics
weight: 13
title: vmestimator
tags:
- metrics
- cardinality
aliases:
- /vmestimator.html
- /vmestimator/index.html
- /vmestimator/
---
`vmestimator` measures metrics cardinality across arbitrary label dimensions and exposes the results as metrics.
## Why measure ?
Consider a setup where metrics are scraped from dozens of Prometheus targets.
One day, a team deploys a new version of their service with a `trace_id` or `user_id` label.
Overnight, that job's cardinality explodes from 500 to 500,000 time series.
Suddenly, VictoriaMetrics consumes 100x more memory and disk.
Ingestion slows down, storage struggles to keep up, and in the worst case becomes unavailable.
By the time someone gets paged, the damage is already done: indexes are bloated, caches are oversized, and observability across the entire system is affected.
`vmestimator` continuously tracks cardinality and exposes the estimation results as [metrics](https://github.com/VictoriaMetrics/vmestimator/blob/main/README.md#cardinality-metrics).
This allows alerting on cardinality spikes within minutes and identifying the offending job directly from the alert.
Instead of discovering the problem after it impacts the infrastructure, it becomes possible to react before it turns into an outage.
Per-job cardinality tracking is the most actionable use case, but its not the only one (see [use cases](https://github.com/VictoriaMetrics/vmestimator/#use-cases)).
`vmestimator` can measure cardinality across arbitrary label dimensions,
enabling use cases such as per-tenant usage analysis, long-term trend tracking, and capacity planning.
## Design
We recommend deploying `vmestimator` close to the metrics source, ideally alongside `vmagent` instances that scrape targets.
Each `vmagent` mirrors all ingested metrics into the estimator.
To reduce overhead, persistent queueing and metadata ingestion can be disabled for the estimator remote write path.
It is safe to send metrics from multiple independent `vmagent` instances into a single `vmestimator`.
Run vmestimator (see [configuration](https://github.com/VictoriaMetrics/vmestimator#configuration)):
```bash
/path/to/vmestimator -config=streams.yaml # -httpListenAddr=:8490
```
Run vmagent:
```bash
/path/to/vmagent \
-remoteWrite.url=http://127.0.0.1:8428/api/v1/write \
-remoteWrite.url=http://127.0.0.1:8490/cardinality/api/v1/write \
-remoteWrite.disableOnDiskQueue=false,true \
-remoteWrite.disableMetadata=false,true
```
The next step is to expose cardinality estimates as metrics.
For this, `vmagent` should scrape the estimator `/metrics` endpoint and forward those metrics to a `vmsingle` instance (or another VictoriaMetrics storage).
<img style="min-width:0;width: 100%" src="https://github.com/user-attachments/assets/e52d9210-b6f9-457b-8d8f-1d6ff6ba1416" />
This setup is straightforward and introduces minimal overhead.
The main drawback is that cardinality data shares the same storage with production metrics.
If that storage becomes unavailable, the visibility into cardinality is lost precisely when it may be most needed.
To mitigate this, we recommend running a separate `vmsingle` instance dedicated to scraping and storing VictoriaMetrics-related monitoring signals only.
This pattern is commonly referred to as a monitoring-of-monitoring (MoM) setup.
In this architecture, `vmestimator` metrics are isolated from production observability storage,
ensuring cardinality visibility remains available even during incidents affecting the primary monitoring system.
The resulting topology looks like this:
<img style="min-width:0;width: 100%" src="https://github.com/user-attachments/assets/e2ca4a69-e931-47a1-9d91-99749382d4a9" />
## Install
To quickly try VictoriaMetrics, just download the VictoriaMetrics docker image from [Docker Hub](https://hub.docker.com/r/victoriametrics/vmestimator) or [Quay](https://quay.io/repository/victoriametrics/vmestimator) and start it with the desired [command-line flags](https://github.com/VictoriaMetrics/vmestimator#command-line-flags).
## Configuration
To run vmestimator a `streams.yaml` config has to be provided:
```bash
/path/to/vmestimator -config=streams.yaml # -httpListenAddr=:8490
```
Config reference:
```yaml
streams:
-
# The measurement window: how long unique series are retained before the HLL sketch resets.
# Increases are always reflected immediately. Interval only controls how fast the estimate
# drops after previously seen series disappear.
#
# Running two streams with different intervals (e.g. 5m and 1h) lets you derive churn rate
# by comparing their estimates. See Use Cases -> Churn Rate
#
# default: 5m
interval: '5m'
# Label names used to split the cardinality estimate into per-combination groups.
# Each distinct combination of values for these labels gets its own estimate metric.
# Omit entirely for a single global estimate across all series.
# Examples:
# - ["job"]
# - ["__name__"]
# - ["vm_account_id","vm_project_id"]
#
# default: none (single global estimate)
group_by: ['job']
# Maximum number of distinct groups (HLL sketches) to track.
# Once the limit is reached, excess groups are counted in a single shared "rejected" sketch
# rather than getting their own entry. Acts as a memory cap and a safeguard against OOM
# when the group_by label values grow unboundedly.
# Memory upper bound per stream:
# group_limit * 2^hll_precision bytes.
#
# default: 10000
group_limit: 10000
# Number of shards used to reduce lock contention during parallel ingestion.
# Slightly increases memory for global streams (no group_by); negligible otherwise.
# Leave at the default unless you have profiled lock contention or have a specific reason to change it.
#
# default: min(64, 2*availableCPUs)
buckets: 64
# HyperLogLog precision p, in range [4..18].
# Determines the number of registers m = 2^p and the relative error 1.04 / sqrt(m):
# p=14 → m=16 384, error ~0.81%, memory ~16 KB per sketch (default, suits most cases)
# p=18 → m=262 144, error ~0.20%, memory ~256 KB per sketch (billing-grade accuracy)
# p=10 → m=1 024, error ~3.25%, memory ~1 KB per sketch (thousands of groups, memory-tight)
# See more in https://research.google.com/pubs/archive/40671.pdf
#
# default: 14
hll_precision: 14
# Whether to use the sparse HyperLogLog representation for low-cardinality groups.
# Sparse mode uses far less memory until a group's cardinality reaches ~2^(p-1),
# at which point it automatically promotes to the dense representation.
# See more in # See more in https://research.google.com/pubs/archive/40671.pdf
#
# default: true
hll_sparse: true
# Static labels attached to every output metric produced by this stream entry.
# Useful when multiple vmestimator instances feed the same storage and you need
# to distinguish their estimates in dashboards and alerts.
labels:
env: 'production'
region: 'eu-central-1'
```
## Cardinality Metrics
Cardinality estimates are exposed as the `cardinality_estimate` metric.
All metrics include `interval`, `group_by_keys`, `group_by_values`, and any static labels defined in the stream config.
For global estimates (no `group_by` configured), `group_by_keys` is `__global__` and `group_by_values` is omitted:
```
cardinality_estimate{interval="1h0m0s",group_by_keys="__global__"} 142300
```
For grouped estimates, one summary line shows the total number of distinct groups `group_by_keys="__group__"`, followed by one line per distinct label value combination.
Each per-group line also includes individual `by_{key}="{val}"` labels:
```
cardinality_estimate{interval="5m0s",group_by_keys="__group__",group_by_values="instance,job"} 2
cardinality_estimate{interval="5m0s",group_by_keys="instance,job",group_by_values="host1:9090,prometheus",by_instance="host1:9090",by_job="prometheus"} 312
cardinality_estimate{interval="5m0s",group_by_keys="instance,job",group_by_values="host2:9100,node",by_instance="host2:9100",by_job="node"} 87
```
Note: the total distinct group count in the summary line may exceed the number of per-group lines when `group_limit` is reached
and excess groups are counted in a single shared "rejected" sketch rather than getting their own entry.
By default, cardinality estimates are merged with the estimator's operational metrics and exposed at `/metrics`.
This is controlled by the `-cardinalityMetrics.exposeAt` flag:
- `-cardinalityMetrics.exposeAt=/metrics` (default): cardinality metrics merged with operational metrics at `/metrics`
- `-cardinalityMetrics.exposeAt=/cardinality/metrics`: cardinality metrics exposed at separate path
- `-cardinalityMetrics.exposeAt=`: cardinality metrics not exposed via HTTP
Computing cardinality estimates is expensive, so results are cached.
Cache duration is controlled by `-cardinalityMetrics.cacheTTL` (default: `30s`).
Set to `0` to disable caching entirely.
## Use Cases
### Basic
Global cardinality:
```yaml
# streams.yaml
- interval: '5m'
```
Per metric name cardinality:
```yaml
# streams.yaml
- interval: '5m'
group_by: ['__name__']
```
Per job label cardinality:
```yaml
# streams.yaml
- interval: '5m'
group_by: ['job']
```
Per tenant cardinality:
```yaml
# streams.yaml
- interval: '5m'
group_by: ['vm_account_id', 'vm_project_id']
```
### Churn rate calculation
[Churn rate](https://valyala.medium.com/prometheus-storage-technical-terms-for-humans-4ab4de6c3d48#churn-rate) measures how quickly time series are created and disappear.
[High churn](https://docs.victoriametrics.com/victoriametrics/faq/#what-is-high-churn-rate) means many series appear briefly and are replaced by new ones.
This puts pressure on storage, because each new series must be indexed regardless of how short its lifetime is.
To measure churn, configure two streams with the same `group_by` but different intervals. A short one (`5m`) and a long one (`1h`):
```yaml
# streams.yaml
- interval: '5m'
group_by: ['job']
- interval: '1h'
group_by: ['job']
```
The short interval (`5m`) captures the currently active series.
The long interval (`1h`) retains all series seen over the past hour.
When churn is low, both estimates are roughly equal.
When churn is high, the `1h` estimate grows significantly larger than the `5m` estimate, because the long window accumulates series that have already disappeared.
The following query computes the churn ratio per job:
```
max(cardinality_estimate{group_by_keys="job",interval="1h0m0s"}) without (job)
/
(max(cardinality_estimate{group_by_keys="job",interval="5m0s"}) without (job) * 12)
```
A result near `0` means the series set is stable. The same series were active throughout the entire hour.
A result near `1` means complete churn. Entirely different series appeared each 5-minute window.
Values in between indicate the fraction of maximum possible churn that is occurring.
This helps identify jobs that create the most indexing pressure on storage, even when their current active cardinality appears moderate.
## Alternative solutions
### PromQL
Cardinality can be estimated with PromQL.
Global cardinality:
```
count({__name__=~".*"})
```
Top ten metric names by cardinality:
```
topk(10, count({__name__=~".*"}) by (__name__))
```
Top ten jobs by cardinality:
```
topk(10, count({__name__=~".*"}) by (job))
```
This approach works for small setups but does not scale well, because these queries scan the entire time series set.
Most critically, if the storage is overloaded or unavailable, these queries could not be executed.
### Cardinality explorer
VictoriaMetrics includes a built-in [cardinality explorer](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#cardinality-explorer).
It provides per-metric detail beyond raw series counts: query frequency, last access time, day-over-day change, and share of total cardinality.
It is well suited for in-depth, ad-hoc investigation.
For example, finding metrics that are high-cardinality but rarely queried,
so they can be [dropped via relabeling](https://docs.victoriametrics.com/victoriametrics/relabeling/#how-to-drop-metrics-during-scrape) or reduce cardinality with [stream aggregation](https://docs.victoriametrics.com/victoriametrics/stream-aggregation/).
Both tools serve different purposes and work well together.
Use `vmestimator` for continuous monitoring, alerting, and cross-cluster cardinality tracking.
Use the cardinality explorer when you need to drill into a specific metric or label and understand what is driving its cardinality.
## Cluster
`vmestimator` supports a clustered deployment for high availability or when CPU on a single instance becomes a limiting factor.
Instances are split into two roles: **storage nodes** accept Prometheus remote write and maintain local HyperLogLog sketches; **selector nodes** query all storage nodes, merge their sketches, and expose a unified cardinality estimate. Cardinality estimate results should be scraped from selector nodes.
<img style="min-width:0;width: 100%" src="https://github.com/user-attachments/assets/846e5f77-378a-44dc-a4c8-2a1c64eca9d8" />
**Storage nodes:**
```
vmestimator -config=streams.yaml -httpListenAddr=:8491 -cardinalityMetrics.exposeAt=/cardinality/metrics
vmestimator -config=streams.yaml -httpListenAddr=:8492 -cardinalityMetrics.exposeAt=/cardinality/metrics
vmestimator -config=streams.yaml -httpListenAddr=:8493 -cardinalityMetrics.exposeAt=/cardinality/metrics
```
**Selector nodes:**
```
vmestimator -storageNode=http://vmestimator-storage-1:8491 \
-storageNode=http://vmestimator-storage-2:8492 \
-storageNode=http://vmestimator-storage-3:8493 \
-httpListenAddr=:8490
```
Setting `-cardinalityMetrics.exposeAt=/cardinality/metrics` on storage nodes keeps per-node estimates off the default `/metrics` path. The `/metrics` endpoint then returns only operational metrics, while `/cardinality/metrics` exposes the node's local estimate — useful for inspecting or debugging a specific node.
A selector with `-storageNode` flags and no `-config` runs without local estimators and only merges remote data.
When multiple selector nodes are scraped, each returns a fully merged estimate.
Deduplicate at query time to avoid overcounting:
```
max(cardinality_estimate) without (job)
```
## Operational metrics
When grouping is enabled, vmestimator exposes per-bucket operational metrics at `/metrics`:
- `vmestimator_estimator_group_size{group_by_keys, bucket}` — number of active groups in this bucket after the last rotation
- `vmestimator_estimator_group_rejected_size{group_by_keys}` — estimated number of distinct group values rejected since the last rotation because `group_limit` was reached
- `vmestimator_estimator_group_limit{group_by_keys, bucket}` — configured `group_limit` for this bucket
## Dashboards
Two Grafana dashboards are available in the [dashboards](https://github.com/VictoriaMetrics/vmestimator/tree/main/dashboards) directory:
- `vmestimator.json` — application health: CPU, memory, ingestion rates, concurrent inserts, and group key saturation.
- `cardinality-explorer.json` — cardinality analysis: global estimates, per-group-key series counts, and top-10 highest-cardinality label value combinations.
<img style="min-width:0;width: 100%" src="https://github.com/user-attachments/assets/2bd6a930-1eb5-40ef-8006-8196c1c12397" />
## How to build from sources
It is recommended to use the [docker images](https://hub.docker.com/r/victoriametrics/vmestimator).
### Development build
1. [Install Go](https://golang.org/doc/install).
1. Run `make vmestimator` from the root folder of [the repository](https://github.com/VictoriaMetrics/vmestimator).
It builds `vmestimator` binary and places it into the `bin` folder.
### Production build
1. [Install docker](https://docs.docker.com/install/).
1. Run `make vmestimator-prod` from the root folder of [the repository](https://github.com/VictoriaMetrics/vmestimator).
It builds `vmestimator-prod` binary and puts it into the `bin` folder.
### Building docker images
Run `make package-vmestimator`. It builds `victoriametrics/vmestimator:<PKG_TAG>` docker image locally.
`<PKG_TAG>` is auto-generated image tag, which depends on source code in the repository.
The `<PKG_TAG>` may be manually set via `PKG_TAG=foobar make package-vmestimator`.
The base docker image is [alpine](https://hub.docker.com/_/alpine) but it is possible to use any other base image by setting it via `<ROOT_IMAGE>` environment variable.
For example, the following command builds the image on top of [scratch](https://hub.docker.com/_/scratch) image:
```sh
ROOT_IMAGE=scratch make package-vmrestore
```
You can build and publish to your own registry and namespace:
```
DOCKER_REGISTRIES=ghcr.io DOCKER_NAMESPACE=foo make publish-vmagent
```
## Command-line flags
Run `vmestimate -help` in order to see all the available options:
```
Usage of ./bin/vmestimator:
-cardinalityMetrics.cacheTTL duration
Duration for caching cardinality metrics response (default 30s)
-cardinalityMetrics.exposeAt string
HTTP path for exposing cardinality metrics. If set to the default /metrics, cardinality metrics are merged with regular metrics and exposed together. If set to a different path, only cardinality metrics are exposed at that endpoint. If set to an empty value, cardinality metrics are not exposed via HTTP at all. (default "/metrics")
-config string
Path to YAML configuration file
-enableTCP6
Whether to enable IPv6 for listening and dialing. By default, only IPv4 TCP and UDP are used
-envflag.enable
Whether to enable reading flags from environment variables in addition to the command line. Command line flag values have priority over values from environment vars. Flags are read only from the command line if this flag isn't set. See https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#environment-variables for more details
-envflag.prefix string
Prefix for environment variables if -envflag.enable is set
-filestream.disableFadvise
Whether to disable fadvise() syscall when reading large data files. The fadvise() syscall prevents from eviction of recently accessed data from OS page cache during background merges and backups. In some rare cases it is better to disable the syscall if it uses too much CPU
-flagsAuthKey value
Auth key for /flags endpoint. It must be passed via authKey query arg. It overrides -httpAuth.*
Flag value can be read from the given file when using -flagsAuthKey=file:///abs/path/to/file or -flagsAuthKey=file://./relative/path/to/file.
Flag value can be read from the given http/https url when using -flagsAuthKey=http://host/path or -flagsAuthKey=https://host/path
-fs.maxConcurrency int
The maximum number of concurrent goroutines to work with files; smaller values may help reducing Go scheduling latency on systems with small number of CPU cores; higher values may help reducing data ingestion latency on systems with high-latency storage such as NFS or Ceph (default 160)
-http.connTimeout duration
Incoming connections to -httpListenAddr are closed after the configured timeout. This may help evenly spreading load among a cluster of services behind TCP-level load balancer. Zero value disables closing of incoming connections (default 2m0s)
-http.disableCORS
Disable CORS for all origins (*)
-http.disableKeepAlive
Whether to disable HTTP keep-alive for incoming connections at -httpListenAddr
-http.disableResponseCompression
Disable compression of HTTP responses to save CPU resources. By default, compression is enabled to save network bandwidth
-http.header.csp string
Value for 'Content-Security-Policy' header, recommended: "default-src 'self'"
-http.header.frameOptions string
Value for 'X-Frame-Options' header
-http.header.hsts string
Value for 'Strict-Transport-Security' header, recommended: 'max-age=31536000; includeSubDomains'
-http.idleConnTimeout duration
Timeout for incoming idle http connections (default 1m0s)
-http.maxGracefulShutdownDuration duration
The maximum duration for a graceful shutdown of the HTTP server. A highly loaded server may require increased value for a graceful shutdown (default 7s)
-http.pathPrefix string
An optional prefix to add to all the paths handled by http server. For example, if '-http.pathPrefix=/foo/bar' is set, then all the http requests will be handled on '/foo/bar/*' paths. This may be useful for proxied requests. See https://www.robustperception.io/using-external-urls-and-proxies-with-prometheus
-http.shutdownDelay duration
Optional delay before http server shutdown. During this delay, the server returns non-OK responses from /health page, so load balancers can route new requests to other servers
-httpAuth.password value
Password for HTTP server's Basic Auth. The authentication is disabled if -httpAuth.username is empty
Flag value can be read from the given file when using -httpAuth.password=file:///abs/path/to/file or -httpAuth.password=file://./relative/path/to/file.
Flag value can be read from the given http/https url when using -httpAuth.password=http://host/path or -httpAuth.password=https://host/path
-httpAuth.username string
Username for HTTP server's Basic Auth. The authentication is disabled if empty. See also -httpAuth.password
-httpListenAddr array
TCP address to listen for incoming HTTP requests
Supports an array of values separated by comma or specified via multiple flags.
Each array item can contain comma inside single-quoted or double-quoted string, {}, [] and () braces.
-insert.maxQueueDuration duration
The maximum duration to wait in the queue when -maxConcurrentInserts concurrent insert requests are executed (default 1m0s)
-internStringCacheExpireDuration duration
The expiry duration for caches for interned strings. See https://en.wikipedia.org/wiki/String_interning . See also -internStringMaxLen and -internStringDisableCache (default 6m0s)
-internStringDisableCache
Whether to disable caches for interned strings. This may reduce memory usage at the cost of higher CPU usage. See https://en.wikipedia.org/wiki/String_interning . See also -internStringCacheExpireDuration and -internStringMaxLen
-internStringMaxLen int
The maximum length for strings to intern. A lower limit may save memory at the cost of higher CPU usage. See https://en.wikipedia.org/wiki/String_interning . See also -internStringDisableCache and -internStringCacheExpireDuration (default 500)
-loggerDisableTimestamps
Whether to disable writing timestamps in logs
-loggerErrorsPerSecondLimit int
Per-second limit on the number of ERROR messages. If more than the given number of errors are emitted per second, the remaining errors are suppressed. Zero values disable the rate limit
-loggerFormat string
Format for logs. Possible values: default, json (default "default")
-loggerJSONFields string
Allows renaming fields in JSON formatted logs. Example: "ts:timestamp,msg:message" renames "ts" to "timestamp" and "msg" to "message". Supported fields: ts, level, caller, msg
-loggerLevel string
Minimum level of errors to log. Possible values: INFO, WARN, ERROR, FATAL, PANIC (default "INFO")
-loggerMaxArgLen int
The maximum length of a single logged argument. Longer arguments are replaced with 'arg_start..arg_end', where 'arg_start' and 'arg_end' is prefix and suffix of the arg with the length not exceeding -loggerMaxArgLen / 2 (default 5000)
-loggerOutput string
Output for the logs. Supported values: stderr, stdout (default "stderr")
-loggerTimezone string
Timezone to use for timestamps in logs. Timezone must be a valid IANA Time Zone. For example: America/New_York, Europe/Berlin, Etc/GMT+3 or Local (default "UTC")
-loggerWarnsPerSecondLimit int
Per-second limit on the number of WARN messages. If more than the given number of warns are emitted per second, then the remaining warns are suppressed. Zero values disable the rate limit
-maxConcurrentInserts int
The maximum number of concurrent insert requests. Set higher value when clients send data over slow networks. Default value depends on the number of available CPU cores. It should work fine in most cases since it minimizes resource usage. See also -insert.maxQueueDuration (default 20)
-maxInsertRequestSize size
The maximum size in bytes of a single Prometheus remote_write API request
Supports the following optional suffixes for size values: KB, MB, GB, TB, KiB, MiB, GiB, TiB (default 33554432)
-memory.allowedBytes size
Allowed size of system memory VictoriaMetrics caches may occupy. This option overrides -memory.allowedPercent if set to a non-zero value. Too low a value may increase the cache miss rate usually resulting in higher CPU and disk IO usage. Too high a value may evict too much data from the OS page cache resulting in higher disk IO usage
Supports the following optional suffixes for size values: KB, MB, GB, TB, KiB, MiB, GiB, TiB (default 0)
-memory.allowedPercent float
Allowed percent of system memory VictoriaMetrics caches may occupy. See also -memory.allowedBytes. Too low a value may increase cache miss rate usually resulting in higher CPU and disk IO usage. Too high a value may evict too much data from the OS page cache which will result in higher disk IO usage (default 60)
-metrics.exposeMetadata
Whether to expose TYPE and HELP metadata at the /metrics page, which is exposed at -httpListenAddr . The metadata may be needed when the /metrics page is consumed by systems, which require this information. For example, Managed Prometheus in Google Cloud - https://cloud.google.com/stackdriver/docs/managed-prometheus/troubleshooting#missing-metric-type
-metricsAuthKey value
Auth key for /metrics endpoint. It must be passed via authKey query arg. It overrides -httpAuth.*
Flag value can be read from the given file when using -metricsAuthKey=file:///abs/path/to/file or -metricsAuthKey=file://./relative/path/to/file.
Flag value can be read from the given http/https url when using -metricsAuthKey=http://host/path or -metricsAuthKey=https://host/path
-pprofAuthKey value
Auth key for /debug/pprof/* endpoints. It must be passed via authKey query arg. It overrides -httpAuth.*
Flag value can be read from the given file when using -pprofAuthKey=file:///abs/path/to/file or -pprofAuthKey=file://./relative/path/to/file.
Flag value can be read from the given http/https url when using -pprofAuthKey=http://host/path or -pprofAuthKey=https://host/path
-pushmetrics.disableCompression
Whether to disable request body compression when pushing metrics to every -pushmetrics.url
-pushmetrics.extraLabel array
Optional labels to add to metrics pushed to every -pushmetrics.url . For example, -pushmetrics.extraLabel='instance="foo"' adds instance="foo" label to all the metrics pushed to every -pushmetrics.url
Supports an array of values separated by comma or specified via multiple flags.
Each array item can contain comma inside single-quoted or double-quoted string, {}, [] and () braces.
-pushmetrics.header array
Optional HTTP request header to send to every -pushmetrics.url . For example, -pushmetrics.header='Authorization: Basic foobar' adds 'Authorization: Basic foobar' header to every request to every -pushmetrics.url
Supports an array of values separated by comma or specified via multiple flags.
Each array item can contain comma inside single-quoted or double-quoted string, {}, [] and () braces.
-pushmetrics.interval duration
Interval for pushing metrics to every -pushmetrics.url (default 10s)
-pushmetrics.url array
Optional URL to push metrics exposed at /metrics page. See https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#push-metrics . By default, metrics exposed at /metrics page aren't pushed to any remote storage
Supports an array of values separated by comma or specified via multiple flags.
Each array item can contain comma inside single-quoted or double-quoted string, {}, [] and () braces.
-secret.flags array
Comma-separated list of flag names with secret values. Values for these flags are hidden in logs and on /metrics page
Supports an array of values separated by comma or specified via multiple flags.
Each array item can contain comma inside single-quoted or double-quoted string, {}, [] and () braces.
-storageNode array
HTTP URLs of remote vmestimator nodes to query for cardinality snapshots, e.g. http://vmestimator-2:8490
Supports an array of values separated by comma or specified via multiple flags.
Each array item can contain comma inside single-quoted or double-quoted string, {}, [] and () braces.
-tls array
Whether to enable TLS for incoming HTTP requests at the given -httpListenAddr (aka https). -tlsCertFile and -tlsKeyFile must be set if -tls is set. See also -mtls
Supports array of values separated by comma or specified via multiple flags.
Empty values are set to false.
-tlsCertFile array
Path to file with TLS certificate for the corresponding -httpListenAddr if -tls is set. Prefer ECDSA certs instead of RSA certs as RSA certs are slower. The provided certificate file is automatically re-read every second, so it can be dynamically updated. See also -tlsAutocertHosts
Supports an array of values separated by comma or specified via multiple flags.
Each array item can contain comma inside single-quoted or double-quoted string, {}, [] and () braces.
-tlsCipherSuites array
Optional list of TLS cipher suites for incoming requests over HTTPS if -tls is set. See the list of supported cipher suites at https://pkg.go.dev/crypto/tls#pkg-constants
Supports an array of values separated by comma or specified via multiple flags.
Each array item can contain comma inside single-quoted or double-quoted string, {}, [] and () braces.
-tlsKeyFile array
Path to file with TLS key for the corresponding -httpListenAddr if -tls is set. The provided key file is automatically re-read every second, so it can be dynamically updated. See also -tlsAutocertHosts
Supports an array of values separated by comma or specified via multiple flags.
Each array item can contain comma inside single-quoted or double-quoted string, {}, [] and () braces.
-tlsMinVersion array
Optional minimum TLS version to use for the corresponding -httpListenAddr if -tls is set. Supported values: TLS10, TLS11, TLS12, TLS13
Supports an array of values separated by comma or specified via multiple flags.
Each array item can contain comma inside single-quoted or double-quoted string, {}, [] and () braces.
-version
Show VictoriaMetrics version
```

4
go.mod
View File

@@ -7,11 +7,11 @@ require (
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.22.0
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.14.0
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.8.0
github.com/VictoriaMetrics/VictoriaLogs v1.121.1-0.20260616132739-c901a1e31cb3
github.com/VictoriaMetrics/VictoriaLogs v1.51.1-0.20260624061259-dc94972a8708
github.com/VictoriaMetrics/easyproto v1.2.0
github.com/VictoriaMetrics/fastcache v1.13.3
github.com/VictoriaMetrics/metrics v1.44.0
github.com/VictoriaMetrics/metricsql v0.87.1
github.com/VictoriaMetrics/metricsql v0.87.2
github.com/aws/aws-sdk-go-v2 v1.42.0
github.com/aws/aws-sdk-go-v2/config v1.32.25
github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.22.27

12
go.sum
View File

@@ -52,20 +52,16 @@ github.com/GoogleCloudPlatform/opentelemetry-operations-go/internal/resourcemapp
github.com/GoogleCloudPlatform/opentelemetry-operations-go/internal/resourcemapping v0.57.0/go.mod h1:YqwkQPrWSC7+byyc1VlKbWLBF5JsW5IoL6xUkemYSXk=
github.com/Microsoft/go-winio v0.6.2 h1:F2VQgta7ecxGYO8k3ZZz3RS8fVIXVxONVUPlNERoyfY=
github.com/Microsoft/go-winio v0.6.2/go.mod h1:yd8OoFMLzJbo9gZq8j5qaps8bJ9aShtEA8Ipt1oGCvU=
github.com/VictoriaMetrics/VictoriaLogs v1.50.1-0.20260415124154-6b7a6357aec0 h1:2x1Tszv41PnCdSMumEtejz/On1RQ45kHQ+hhKT53sOk=
github.com/VictoriaMetrics/VictoriaLogs v1.50.1-0.20260415124154-6b7a6357aec0/go.mod h1:fQtmzaSUL+HJmHozeAKmnTJTOMBT+vBccv/VWQEwhUQ=
github.com/VictoriaMetrics/VictoriaLogs v1.121.1-0.20260616132739-c901a1e31cb3 h1:3eP8RRZitbga5EYiQ3IANrMPxpBwMAX4VA6akDaXwpU=
github.com/VictoriaMetrics/VictoriaLogs v1.121.1-0.20260616132739-c901a1e31cb3/go.mod h1:H4sDxcvk6OmC6zOt++IlDyrwfbn4F1eSLwMpR+kpRt8=
github.com/VictoriaMetrics/VictoriaLogs v1.51.1-0.20260624061259-dc94972a8708 h1:D9/Jzlm3B8PBnrWxg4ft8KYZdG607dV3lpBfPCoiJD8=
github.com/VictoriaMetrics/VictoriaLogs v1.51.1-0.20260624061259-dc94972a8708/go.mod h1:H4sDxcvk6OmC6zOt++IlDyrwfbn4F1eSLwMpR+kpRt8=
github.com/VictoriaMetrics/easyproto v1.2.0 h1:FJT9uNXA2isppFuJErbLqD306KoFlehl7Wn2dg/6oIE=
github.com/VictoriaMetrics/easyproto v1.2.0/go.mod h1:QlGlzaJnDfFd8Lk6Ci/fuLxfTo3/GThPs2KH23mv710=
github.com/VictoriaMetrics/fastcache v1.13.3 h1:rBabE0iIxcqKEMCwUmwHZ9dgEqXerg8FRbRDUvC7OVc=
github.com/VictoriaMetrics/fastcache v1.13.3/go.mod h1:hHXhl4DA2fTL2HTZDJFXWgW0LNjo6B+4aj2Wmng3TjU=
github.com/VictoriaMetrics/metrics v1.43.2 h1:+8pIQEGwchKS5CYFyvv3LKvNXGi7baZ9hmIV4RHqibY=
github.com/VictoriaMetrics/metrics v1.43.2/go.mod h1:xDM82ULLYCYdFRgQ2JBxi8Uf1+8En1So9YUwlGTOqTc=
github.com/VictoriaMetrics/metrics v1.44.0 h1:Fr8yqQSV+ZfYaDD/anqk1E8e9YPgfleSleJmAI0M0Tw=
github.com/VictoriaMetrics/metrics v1.44.0/go.mod h1:xDM82ULLYCYdFRgQ2JBxi8Uf1+8En1So9YUwlGTOqTc=
github.com/VictoriaMetrics/metricsql v0.87.1 h1:GdIblCDgXsrBJcBSDtFT8SLK7P+QHijdQmcr4L/f0Go=
github.com/VictoriaMetrics/metricsql v0.87.1/go.mod h1:d4EisFO6ONP/HIGDYTAtwrejJBBeKGQYiRl095bS4QQ=
github.com/VictoriaMetrics/metricsql v0.87.2 h1:7OsrcDBWREWKqqpnFyIUEOM4FNv2qHvCoww2GYz3Tc0=
github.com/VictoriaMetrics/metricsql v0.87.2/go.mod h1:d4EisFO6ONP/HIGDYTAtwrejJBBeKGQYiRl095bS4QQ=
github.com/VividCortex/ewma v1.2.0 h1:f58SaIzcDXrSy3kWaHNvuJgJ3Nmz59Zji6XoJR/q1ow=
github.com/VividCortex/ewma v1.2.0/go.mod h1:nz4BbCtbLyFDeC9SUHbtcT5644juEuWfUAUnGx7j5l4=
github.com/alecthomas/units v0.0.0-20240927000941-0f3dac36c52b h1:mimo19zliBX/vSQ6PWWSL9lK8qwHozUj03+zLoEB8O0=

View File

@@ -36,10 +36,12 @@ function submitRelabelDebugForm(e) {
<div class="container-fluid">
<a href="https://docs.victoriametrics.com/victoriametrics/relabeling/" target="_blank">Relabeling docs</a>{% space %}
{% if targetURL != "" %}
<a href="metric-relabel-debug{% if targetID != "" %}?id={%s targetID %}{% endif %}">Metric relabel debug</a>
{% else %}
<a href="target-relabel-debug{% if targetID != "" %}?id={%s targetID %}{% endif %}">Target relabel debug</a>
{% if targetID != "" %}
{% if targetURL != "" %}
<a href="metric-relabel-debug?id={%s targetID %}">Metric relabel debug</a>
{% else %}
<a href="target-relabel-debug?id={%s targetID %}">Target relabel debug</a>
{% endif %}
{% endif %}
<br>

View File

@@ -80,425 +80,417 @@ func StreamRelabelDebugStepsHTML(qw422016 *qt422016.Writer, targetURL, targetID
//line lib/promrelabel/debug.qtpl:37
qw422016.N().S(` `)
//line lib/promrelabel/debug.qtpl:39
if targetURL != "" {
//line lib/promrelabel/debug.qtpl:39
qw422016.N().S(`<a href="metric-relabel-debug`)
if targetID != "" {
//line lib/promrelabel/debug.qtpl:40
if targetID != "" {
if targetURL != "" {
//line lib/promrelabel/debug.qtpl:40
qw422016.N().S(`?id=`)
//line lib/promrelabel/debug.qtpl:40
qw422016.E().S(targetID)
//line lib/promrelabel/debug.qtpl:40
}
//line lib/promrelabel/debug.qtpl:40
qw422016.N().S(`">Metric relabel debug</a>`)
qw422016.N().S(`<a href="metric-relabel-debug?id=`)
//line lib/promrelabel/debug.qtpl:41
} else {
//line lib/promrelabel/debug.qtpl:41
qw422016.N().S(`<a href="target-relabel-debug`)
//line lib/promrelabel/debug.qtpl:42
if targetID != "" {
//line lib/promrelabel/debug.qtpl:42
qw422016.N().S(`?id=`)
//line lib/promrelabel/debug.qtpl:42
qw422016.E().S(targetID)
//line lib/promrelabel/debug.qtpl:41
qw422016.N().S(`">Metric relabel debug</a>`)
//line lib/promrelabel/debug.qtpl:42
}
} else {
//line lib/promrelabel/debug.qtpl:42
qw422016.N().S(`">Target relabel debug</a>`)
qw422016.N().S(`<a href="target-relabel-debug?id=`)
//line lib/promrelabel/debug.qtpl:43
qw422016.E().S(targetID)
//line lib/promrelabel/debug.qtpl:43
qw422016.N().S(`">Target relabel debug</a>`)
//line lib/promrelabel/debug.qtpl:44
}
//line lib/promrelabel/debug.qtpl:45
}
//line lib/promrelabel/debug.qtpl:43
//line lib/promrelabel/debug.qtpl:45
qw422016.N().S(`<br>`)
//line lib/promrelabel/debug.qtpl:46
//line lib/promrelabel/debug.qtpl:48
if err != nil {
//line lib/promrelabel/debug.qtpl:47
//line lib/promrelabel/debug.qtpl:49
htmlcomponents.StreamErrorNotification(qw422016, err)
//line lib/promrelabel/debug.qtpl:48
//line lib/promrelabel/debug.qtpl:50
}
//line lib/promrelabel/debug.qtpl:48
//line lib/promrelabel/debug.qtpl:50
qw422016.N().S(`<div class="m-3"><form method="POST" onsubmit="submitRelabelDebugForm(event)">`)
//line lib/promrelabel/debug.qtpl:52
//line lib/promrelabel/debug.qtpl:54
streamrelabelDebugFormInputs(qw422016, metric, relabelConfigs)
//line lib/promrelabel/debug.qtpl:53
//line lib/promrelabel/debug.qtpl:55
if targetID != "" {
//line lib/promrelabel/debug.qtpl:53
//line lib/promrelabel/debug.qtpl:55
qw422016.N().S(`<input type="hidden" name="id" value="`)
//line lib/promrelabel/debug.qtpl:54
//line lib/promrelabel/debug.qtpl:56
qw422016.E().S(targetID)
//line lib/promrelabel/debug.qtpl:54
//line lib/promrelabel/debug.qtpl:56
qw422016.N().S(`" />`)
//line lib/promrelabel/debug.qtpl:55
//line lib/promrelabel/debug.qtpl:57
}
//line lib/promrelabel/debug.qtpl:55
//line lib/promrelabel/debug.qtpl:57
qw422016.N().S(`<input type="submit" value="Submit" class="btn btn-primary m-1" />`)
//line lib/promrelabel/debug.qtpl:57
//line lib/promrelabel/debug.qtpl:59
if targetID != "" {
//line lib/promrelabel/debug.qtpl:57
//line lib/promrelabel/debug.qtpl:59
qw422016.N().S(`<button type="button" onclick="location.href='?id=`)
//line lib/promrelabel/debug.qtpl:58
//line lib/promrelabel/debug.qtpl:60
qw422016.E().S(targetID)
//line lib/promrelabel/debug.qtpl:58
//line lib/promrelabel/debug.qtpl:60
qw422016.N().S(`'" class="btn btn-secondary m-1">Reset</button>`)
//line lib/promrelabel/debug.qtpl:59
//line lib/promrelabel/debug.qtpl:61
}
//line lib/promrelabel/debug.qtpl:59
//line lib/promrelabel/debug.qtpl:61
qw422016.N().S(`</form></div><div class="row"><main class="col-12">`)
//line lib/promrelabel/debug.qtpl:65
//line lib/promrelabel/debug.qtpl:67
streamrelabelDebugSteps(qw422016, dss, targetURL, targetID)
//line lib/promrelabel/debug.qtpl:65
//line lib/promrelabel/debug.qtpl:67
qw422016.N().S(`</main></div></div></body></html>`)
//line lib/promrelabel/debug.qtpl:71
//line lib/promrelabel/debug.qtpl:73
}
//line lib/promrelabel/debug.qtpl:71
//line lib/promrelabel/debug.qtpl:73
func WriteRelabelDebugStepsHTML(qq422016 qtio422016.Writer, targetURL, targetID string, dss []DebugStep, metric, relabelConfigs string, err error) {
//line lib/promrelabel/debug.qtpl:71
//line lib/promrelabel/debug.qtpl:73
qw422016 := qt422016.AcquireWriter(qq422016)
//line lib/promrelabel/debug.qtpl:71
//line lib/promrelabel/debug.qtpl:73
StreamRelabelDebugStepsHTML(qw422016, targetURL, targetID, dss, metric, relabelConfigs, err)
//line lib/promrelabel/debug.qtpl:71
//line lib/promrelabel/debug.qtpl:73
qt422016.ReleaseWriter(qw422016)
//line lib/promrelabel/debug.qtpl:71
//line lib/promrelabel/debug.qtpl:73
}
//line lib/promrelabel/debug.qtpl:71
//line lib/promrelabel/debug.qtpl:73
func RelabelDebugStepsHTML(targetURL, targetID string, dss []DebugStep, metric, relabelConfigs string, err error) string {
//line lib/promrelabel/debug.qtpl:71
//line lib/promrelabel/debug.qtpl:73
qb422016 := qt422016.AcquireByteBuffer()
//line lib/promrelabel/debug.qtpl:71
//line lib/promrelabel/debug.qtpl:73
WriteRelabelDebugStepsHTML(qb422016, targetURL, targetID, dss, metric, relabelConfigs, err)
//line lib/promrelabel/debug.qtpl:71
//line lib/promrelabel/debug.qtpl:73
qs422016 := string(qb422016.B)
//line lib/promrelabel/debug.qtpl:71
//line lib/promrelabel/debug.qtpl:73
qt422016.ReleaseByteBuffer(qb422016)
//line lib/promrelabel/debug.qtpl:71
//line lib/promrelabel/debug.qtpl:73
return qs422016
//line lib/promrelabel/debug.qtpl:71
//line lib/promrelabel/debug.qtpl:73
}
//line lib/promrelabel/debug.qtpl:73
//line lib/promrelabel/debug.qtpl:75
func streamrelabelDebugFormInputs(qw422016 *qt422016.Writer, metric, relabelConfigs string) {
//line lib/promrelabel/debug.qtpl:73
//line lib/promrelabel/debug.qtpl:75
qw422016.N().S(`<div>Relabel configs:<br/><textarea name="relabel_configs" style="width: 100%; height: 15em; font-family: monospace" class="m-1">`)
//line lib/promrelabel/debug.qtpl:76
//line lib/promrelabel/debug.qtpl:78
qw422016.E().S(relabelConfigs)
//line lib/promrelabel/debug.qtpl:76
//line lib/promrelabel/debug.qtpl:78
qw422016.N().S(`</textarea></div><div>Labels:<br/><textarea name="metric" style="width: 100%; height: 5em; font-family: monospace" class="m-1">`)
//line lib/promrelabel/debug.qtpl:81
//line lib/promrelabel/debug.qtpl:83
qw422016.E().S(metric)
//line lib/promrelabel/debug.qtpl:81
//line lib/promrelabel/debug.qtpl:83
qw422016.N().S(`</textarea></div>`)
//line lib/promrelabel/debug.qtpl:83
}
//line lib/promrelabel/debug.qtpl:83
func writerelabelDebugFormInputs(qq422016 qtio422016.Writer, metric, relabelConfigs string) {
//line lib/promrelabel/debug.qtpl:83
qw422016 := qt422016.AcquireWriter(qq422016)
//line lib/promrelabel/debug.qtpl:83
streamrelabelDebugFormInputs(qw422016, metric, relabelConfigs)
//line lib/promrelabel/debug.qtpl:83
qt422016.ReleaseWriter(qw422016)
//line lib/promrelabel/debug.qtpl:83
}
//line lib/promrelabel/debug.qtpl:83
func relabelDebugFormInputs(metric, relabelConfigs string) string {
//line lib/promrelabel/debug.qtpl:83
qb422016 := qt422016.AcquireByteBuffer()
//line lib/promrelabel/debug.qtpl:83
writerelabelDebugFormInputs(qb422016, metric, relabelConfigs)
//line lib/promrelabel/debug.qtpl:83
qs422016 := string(qb422016.B)
//line lib/promrelabel/debug.qtpl:83
qt422016.ReleaseByteBuffer(qb422016)
//line lib/promrelabel/debug.qtpl:83
return qs422016
//line lib/promrelabel/debug.qtpl:83
//line lib/promrelabel/debug.qtpl:85
}
//line lib/promrelabel/debug.qtpl:85
func writerelabelDebugFormInputs(qq422016 qtio422016.Writer, metric, relabelConfigs string) {
//line lib/promrelabel/debug.qtpl:85
qw422016 := qt422016.AcquireWriter(qq422016)
//line lib/promrelabel/debug.qtpl:85
streamrelabelDebugFormInputs(qw422016, metric, relabelConfigs)
//line lib/promrelabel/debug.qtpl:85
qt422016.ReleaseWriter(qw422016)
//line lib/promrelabel/debug.qtpl:85
}
//line lib/promrelabel/debug.qtpl:85
func relabelDebugFormInputs(metric, relabelConfigs string) string {
//line lib/promrelabel/debug.qtpl:85
qb422016 := qt422016.AcquireByteBuffer()
//line lib/promrelabel/debug.qtpl:85
writerelabelDebugFormInputs(qb422016, metric, relabelConfigs)
//line lib/promrelabel/debug.qtpl:85
qs422016 := string(qb422016.B)
//line lib/promrelabel/debug.qtpl:85
qt422016.ReleaseByteBuffer(qb422016)
//line lib/promrelabel/debug.qtpl:85
return qs422016
//line lib/promrelabel/debug.qtpl:85
}
//line lib/promrelabel/debug.qtpl:87
func streamrelabelDebugSteps(qw422016 *qt422016.Writer, dss []DebugStep, targetURL, targetID string) {
//line lib/promrelabel/debug.qtpl:86
//line lib/promrelabel/debug.qtpl:88
if len(dss) > 0 {
//line lib/promrelabel/debug.qtpl:86
//line lib/promrelabel/debug.qtpl:88
qw422016.N().S(`<div class="m-3"><b>Original labels:</b> <samp>`)
//line lib/promrelabel/debug.qtpl:88
//line lib/promrelabel/debug.qtpl:90
streammustFormatLabels(qw422016, dss[0].In)
//line lib/promrelabel/debug.qtpl:88
//line lib/promrelabel/debug.qtpl:90
qw422016.N().S(`</samp></div>`)
//line lib/promrelabel/debug.qtpl:90
//line lib/promrelabel/debug.qtpl:92
}
//line lib/promrelabel/debug.qtpl:90
//line lib/promrelabel/debug.qtpl:92
qw422016.N().S(`<table class="table table-striped table-hover table-bordered table-sm"><thead><tr><th scope="col" style="width: 5%">Step</th><th scope="col" style="width: 25%">Relabeling Rule</th><th scope="col" style="width: 35%">Input Labels</th><th scope="col" stile="width: 35%">Output labels</a></tr></thead><tbody>`)
//line lib/promrelabel/debug.qtpl:101
for i, ds := range dss {
//line lib/promrelabel/debug.qtpl:103
for i, ds := range dss {
//line lib/promrelabel/debug.qtpl:105
inLabels, inErr := promutil.NewLabelsFromString(ds.In)
outLabels, outErr := promutil.NewLabelsFromString(ds.Out)
changedLabels := getChangedLabelNames(inLabels, outLabels)
//line lib/promrelabel/debug.qtpl:106
//line lib/promrelabel/debug.qtpl:108
qw422016.N().S(`<tr><td>`)
//line lib/promrelabel/debug.qtpl:108
//line lib/promrelabel/debug.qtpl:110
qw422016.N().D(i)
//line lib/promrelabel/debug.qtpl:108
//line lib/promrelabel/debug.qtpl:110
qw422016.N().S(`</td><td><b><pre class="m-2">`)
//line lib/promrelabel/debug.qtpl:109
//line lib/promrelabel/debug.qtpl:111
qw422016.E().S(ds.Rule)
//line lib/promrelabel/debug.qtpl:109
//line lib/promrelabel/debug.qtpl:111
qw422016.N().S(`</pre></b></td><td>`)
//line lib/promrelabel/debug.qtpl:111
//line lib/promrelabel/debug.qtpl:113
if inErr == nil {
//line lib/promrelabel/debug.qtpl:111
//line lib/promrelabel/debug.qtpl:113
qw422016.N().S(`<div class="m-2" style="font-size: 0.9em" title="deleted and updated labels highlighted in red">`)
//line lib/promrelabel/debug.qtpl:113
//line lib/promrelabel/debug.qtpl:115
streamlabelsWithHighlight(qw422016, inLabels, changedLabels, "#D15757")
//line lib/promrelabel/debug.qtpl:113
//line lib/promrelabel/debug.qtpl:115
qw422016.N().S(`</div>`)
//line lib/promrelabel/debug.qtpl:115
//line lib/promrelabel/debug.qtpl:117
} else {
//line lib/promrelabel/debug.qtpl:115
//line lib/promrelabel/debug.qtpl:117
qw422016.N().S(`<div class="m-2" style="font-size: 0.9em; color: red" title="error parsing input labels"><pre>`)
//line lib/promrelabel/debug.qtpl:117
qw422016.E().S(inErr.Error())
//line lib/promrelabel/debug.qtpl:117
qw422016.N().S(`</pre></div>`)
//line lib/promrelabel/debug.qtpl:119
break
//line lib/promrelabel/debug.qtpl:120
}
//line lib/promrelabel/debug.qtpl:120
qw422016.N().S(`</td><td>`)
//line lib/promrelabel/debug.qtpl:123
if outErr == nil {
//line lib/promrelabel/debug.qtpl:123
qw422016.N().S(`<div class="m-2" style="font-size: 0.9em" title="added and updated labels highlighted in blue">`)
//line lib/promrelabel/debug.qtpl:125
streamlabelsWithHighlight(qw422016, outLabels, changedLabels, "#4495e0")
//line lib/promrelabel/debug.qtpl:125
qw422016.N().S(`</div>`)
//line lib/promrelabel/debug.qtpl:127
} else {
//line lib/promrelabel/debug.qtpl:127
qw422016.N().S(`<div class="m-2" style="font-size: 0.9em; color: red" title="error parsing output labels"><pre>`)
//line lib/promrelabel/debug.qtpl:129
qw422016.E().S(outErr.Error())
//line lib/promrelabel/debug.qtpl:129
qw422016.E().S(inErr.Error())
//line lib/promrelabel/debug.qtpl:119
qw422016.N().S(`</pre></div>`)
//line lib/promrelabel/debug.qtpl:131
//line lib/promrelabel/debug.qtpl:121
break
//line lib/promrelabel/debug.qtpl:132
//line lib/promrelabel/debug.qtpl:122
}
//line lib/promrelabel/debug.qtpl:132
qw422016.N().S(`</td></tr>`)
//line lib/promrelabel/debug.qtpl:135
}
//line lib/promrelabel/debug.qtpl:135
qw422016.N().S(`</tbody></table>`)
//line lib/promrelabel/debug.qtpl:138
if len(dss) > 0 {
//line lib/promrelabel/debug.qtpl:138
qw422016.N().S(`<div class="m-3"><b>Resulting labels:</b> <samp>`)
//line lib/promrelabel/debug.qtpl:140
streammustFormatLabels(qw422016, dss[len(dss)-1].Out)
//line lib/promrelabel/debug.qtpl:140
qw422016.N().S(`</samp>`)
//line lib/promrelabel/debug.qtpl:141
if targetURL != "" {
//line lib/promrelabel/debug.qtpl:141
qw422016.N().S(`<div><b>Target URL:</b>`)
//line lib/promrelabel/debug.qtpl:143
qw422016.N().S(` `)
//line lib/promrelabel/debug.qtpl:143
qw422016.N().S(`<a href="`)
//line lib/promrelabel/debug.qtpl:143
qw422016.E().S(targetURL)
//line lib/promrelabel/debug.qtpl:143
qw422016.N().S(`" target="_blank">`)
//line lib/promrelabel/debug.qtpl:143
qw422016.E().S(targetURL)
//line lib/promrelabel/debug.qtpl:143
qw422016.N().S(`</a>`)
//line lib/promrelabel/debug.qtpl:144
if targetID != "" {
//line lib/promrelabel/debug.qtpl:145
qw422016.N().S(` `)
//line lib/promrelabel/debug.qtpl:145
qw422016.N().S(`(<a href="target_response?id=`)
//line lib/promrelabel/debug.qtpl:146
qw422016.E().S(targetID)
//line lib/promrelabel/debug.qtpl:146
qw422016.N().S(`" target="_blank" title="click to fetch target response on behalf of the scraper">response</a>)`)
//line lib/promrelabel/debug.qtpl:147
}
//line lib/promrelabel/debug.qtpl:147
//line lib/promrelabel/debug.qtpl:122
qw422016.N().S(`</td><td>`)
//line lib/promrelabel/debug.qtpl:125
if outErr == nil {
//line lib/promrelabel/debug.qtpl:125
qw422016.N().S(`<div class="m-2" style="font-size: 0.9em" title="added and updated labels highlighted in blue">`)
//line lib/promrelabel/debug.qtpl:127
streamlabelsWithHighlight(qw422016, outLabels, changedLabels, "#4495e0")
//line lib/promrelabel/debug.qtpl:127
qw422016.N().S(`</div>`)
//line lib/promrelabel/debug.qtpl:149
//line lib/promrelabel/debug.qtpl:129
} else {
//line lib/promrelabel/debug.qtpl:129
qw422016.N().S(`<div class="m-2" style="font-size: 0.9em; color: red" title="error parsing output labels"><pre>`)
//line lib/promrelabel/debug.qtpl:131
qw422016.E().S(outErr.Error())
//line lib/promrelabel/debug.qtpl:131
qw422016.N().S(`</pre></div>`)
//line lib/promrelabel/debug.qtpl:133
break
//line lib/promrelabel/debug.qtpl:134
}
//line lib/promrelabel/debug.qtpl:149
qw422016.N().S(`</div>`)
//line lib/promrelabel/debug.qtpl:151
//line lib/promrelabel/debug.qtpl:134
qw422016.N().S(`</td></tr>`)
//line lib/promrelabel/debug.qtpl:137
}
//line lib/promrelabel/debug.qtpl:152
//line lib/promrelabel/debug.qtpl:137
qw422016.N().S(`</tbody></table>`)
//line lib/promrelabel/debug.qtpl:140
if len(dss) > 0 {
//line lib/promrelabel/debug.qtpl:140
qw422016.N().S(`<div class="m-3"><b>Resulting labels:</b> <samp>`)
//line lib/promrelabel/debug.qtpl:142
streammustFormatLabels(qw422016, dss[len(dss)-1].Out)
//line lib/promrelabel/debug.qtpl:142
qw422016.N().S(`</samp>`)
//line lib/promrelabel/debug.qtpl:143
if targetURL != "" {
//line lib/promrelabel/debug.qtpl:143
qw422016.N().S(`<div><b>Target URL:</b>`)
//line lib/promrelabel/debug.qtpl:145
qw422016.N().S(` `)
//line lib/promrelabel/debug.qtpl:145
qw422016.N().S(`<a href="`)
//line lib/promrelabel/debug.qtpl:145
qw422016.E().S(targetURL)
//line lib/promrelabel/debug.qtpl:145
qw422016.N().S(`" target="_blank">`)
//line lib/promrelabel/debug.qtpl:145
qw422016.E().S(targetURL)
//line lib/promrelabel/debug.qtpl:145
qw422016.N().S(`</a>`)
//line lib/promrelabel/debug.qtpl:146
if targetID != "" {
//line lib/promrelabel/debug.qtpl:147
qw422016.N().S(` `)
//line lib/promrelabel/debug.qtpl:147
qw422016.N().S(`(<a href="target_response?id=`)
//line lib/promrelabel/debug.qtpl:148
qw422016.E().S(targetID)
//line lib/promrelabel/debug.qtpl:148
qw422016.N().S(`" target="_blank" title="click to fetch target response on behalf of the scraper">response</a>)`)
//line lib/promrelabel/debug.qtpl:149
}
//line lib/promrelabel/debug.qtpl:149
qw422016.N().S(`</div>`)
//line lib/promrelabel/debug.qtpl:151
}
//line lib/promrelabel/debug.qtpl:151
qw422016.N().S(`</div>`)
//line lib/promrelabel/debug.qtpl:153
}
//line lib/promrelabel/debug.qtpl:154
}
//line lib/promrelabel/debug.qtpl:152
//line lib/promrelabel/debug.qtpl:154
func writerelabelDebugSteps(qq422016 qtio422016.Writer, dss []DebugStep, targetURL, targetID string) {
//line lib/promrelabel/debug.qtpl:152
//line lib/promrelabel/debug.qtpl:154
qw422016 := qt422016.AcquireWriter(qq422016)
//line lib/promrelabel/debug.qtpl:152
//line lib/promrelabel/debug.qtpl:154
streamrelabelDebugSteps(qw422016, dss, targetURL, targetID)
//line lib/promrelabel/debug.qtpl:152
//line lib/promrelabel/debug.qtpl:154
qt422016.ReleaseWriter(qw422016)
//line lib/promrelabel/debug.qtpl:152
//line lib/promrelabel/debug.qtpl:154
}
//line lib/promrelabel/debug.qtpl:152
//line lib/promrelabel/debug.qtpl:154
func relabelDebugSteps(dss []DebugStep, targetURL, targetID string) string {
//line lib/promrelabel/debug.qtpl:152
//line lib/promrelabel/debug.qtpl:154
qb422016 := qt422016.AcquireByteBuffer()
//line lib/promrelabel/debug.qtpl:152
//line lib/promrelabel/debug.qtpl:154
writerelabelDebugSteps(qb422016, dss, targetURL, targetID)
//line lib/promrelabel/debug.qtpl:152
//line lib/promrelabel/debug.qtpl:154
qs422016 := string(qb422016.B)
//line lib/promrelabel/debug.qtpl:152
//line lib/promrelabel/debug.qtpl:154
qt422016.ReleaseByteBuffer(qb422016)
//line lib/promrelabel/debug.qtpl:152
//line lib/promrelabel/debug.qtpl:154
return qs422016
//line lib/promrelabel/debug.qtpl:152
//line lib/promrelabel/debug.qtpl:154
}
//line lib/promrelabel/debug.qtpl:154
//line lib/promrelabel/debug.qtpl:156
func StreamRelabelDebugStepsJSON(qw422016 *qt422016.Writer, targetURL, targetID string, dss []DebugStep, metric, relabelConfigs string, err error) {
//line lib/promrelabel/debug.qtpl:154
//line lib/promrelabel/debug.qtpl:156
qw422016.N().S(`{`)
//line lib/promrelabel/debug.qtpl:156
if err != nil {
//line lib/promrelabel/debug.qtpl:156
qw422016.N().S(`"status": "error","error":`)
//line lib/promrelabel/debug.qtpl:158
qw422016.N().Q(fmt.Sprintf("Error: %s", err))
//line lib/promrelabel/debug.qtpl:159
} else {
if err != nil {
//line lib/promrelabel/debug.qtpl:158
qw422016.N().S(`"status": "error","error":`)
//line lib/promrelabel/debug.qtpl:160
qw422016.N().Q(fmt.Sprintf("Error: %s", err))
//line lib/promrelabel/debug.qtpl:161
} else {
//line lib/promrelabel/debug.qtpl:162
var hasError bool
//line lib/promrelabel/debug.qtpl:160
//line lib/promrelabel/debug.qtpl:162
qw422016.N().S(`"status": "success","steps": [`)
//line lib/promrelabel/debug.qtpl:163
for i, ds := range dss {
//line lib/promrelabel/debug.qtpl:165
for i, ds := range dss {
//line lib/promrelabel/debug.qtpl:167
inLabels, inErr := promutil.NewLabelsFromString(ds.In)
outLabels, outErr := promutil.NewLabelsFromString(ds.Out)
changedLabels := getChangedLabelNames(inLabels, outLabels)
//line lib/promrelabel/debug.qtpl:168
//line lib/promrelabel/debug.qtpl:170
qw422016.N().S(`{"inLabels":`)
//line lib/promrelabel/debug.qtpl:170
//line lib/promrelabel/debug.qtpl:172
qw422016.N().Q(labelsWithHighlight(inLabels, changedLabels, "#D15757"))
//line lib/promrelabel/debug.qtpl:170
//line lib/promrelabel/debug.qtpl:172
qw422016.N().S(`,"outLabels":`)
//line lib/promrelabel/debug.qtpl:171
//line lib/promrelabel/debug.qtpl:173
qw422016.N().Q(labelsWithHighlight(outLabels, changedLabels, "#4495e0"))
//line lib/promrelabel/debug.qtpl:171
//line lib/promrelabel/debug.qtpl:173
qw422016.N().S(`,"rule":`)
//line lib/promrelabel/debug.qtpl:172
//line lib/promrelabel/debug.qtpl:174
qw422016.N().Q(ds.Rule)
//line lib/promrelabel/debug.qtpl:172
//line lib/promrelabel/debug.qtpl:174
qw422016.N().S(`,"errors": {`)
//line lib/promrelabel/debug.qtpl:174
if inErr != nil {
//line lib/promrelabel/debug.qtpl:174
qw422016.N().S(`"inLabels":`)
//line lib/promrelabel/debug.qtpl:175
qw422016.N().Q(`<span style="color: #D15757">` + inErr.Error() + `</span>`)
//line lib/promrelabel/debug.qtpl:175
if outErr != nil {
//line lib/promrelabel/debug.qtpl:175
qw422016.N().S(`,`)
//line lib/promrelabel/debug.qtpl:175
}
//line lib/promrelabel/debug.qtpl:176
hasError = true
if inErr != nil {
//line lib/promrelabel/debug.qtpl:176
qw422016.N().S(`"inLabels":`)
//line lib/promrelabel/debug.qtpl:177
} else {
qw422016.N().Q(`<span style="color: #D15757">` + inErr.Error() + `</span>`)
//line lib/promrelabel/debug.qtpl:177
if outErr != nil {
//line lib/promrelabel/debug.qtpl:177
qw422016.N().S(`,`)
//line lib/promrelabel/debug.qtpl:177
}
//line lib/promrelabel/debug.qtpl:178
}
//line lib/promrelabel/debug.qtpl:179
if outErr != nil {
//line lib/promrelabel/debug.qtpl:179
qw422016.N().S(`"outLabels":`)
//line lib/promrelabel/debug.qtpl:180
qw422016.N().Q(`<span style="color: #D15757">` + outErr.Error() + `</span>`)
//line lib/promrelabel/debug.qtpl:181
hasError = true
//line lib/promrelabel/debug.qtpl:182
//line lib/promrelabel/debug.qtpl:179
} else {
//line lib/promrelabel/debug.qtpl:180
}
//line lib/promrelabel/debug.qtpl:181
if outErr != nil {
//line lib/promrelabel/debug.qtpl:181
qw422016.N().S(`"outLabels":`)
//line lib/promrelabel/debug.qtpl:182
qw422016.N().Q(`<span style="color: #D15757">` + outErr.Error() + `</span>`)
//line lib/promrelabel/debug.qtpl:183
hasError = true
//line lib/promrelabel/debug.qtpl:184
}
//line lib/promrelabel/debug.qtpl:184
qw422016.N().S(`}}`)
//line lib/promrelabel/debug.qtpl:185
//line lib/promrelabel/debug.qtpl:187
if i != len(dss)-1 {
//line lib/promrelabel/debug.qtpl:185
//line lib/promrelabel/debug.qtpl:187
qw422016.N().S(`,`)
//line lib/promrelabel/debug.qtpl:185
//line lib/promrelabel/debug.qtpl:187
}
//line lib/promrelabel/debug.qtpl:186
//line lib/promrelabel/debug.qtpl:188
}
//line lib/promrelabel/debug.qtpl:186
//line lib/promrelabel/debug.qtpl:188
qw422016.N().S(`]`)
//line lib/promrelabel/debug.qtpl:188
//line lib/promrelabel/debug.qtpl:190
if len(dss) > 0 && !hasError {
//line lib/promrelabel/debug.qtpl:188
//line lib/promrelabel/debug.qtpl:190
qw422016.N().S(`,"originalLabels":`)
//line lib/promrelabel/debug.qtpl:190
qw422016.N().Q(mustFormatLabels(dss[0].In))
//line lib/promrelabel/debug.qtpl:190
qw422016.N().S(`,"resultingLabels":`)
//line lib/promrelabel/debug.qtpl:191
qw422016.N().Q(mustFormatLabels(dss[len(dss)-1].Out))
//line lib/promrelabel/debug.qtpl:192
qw422016.N().Q(mustFormatLabels(dss[0].In))
//line lib/promrelabel/debug.qtpl:192
qw422016.N().S(`,"resultingLabels":`)
//line lib/promrelabel/debug.qtpl:193
qw422016.N().Q(mustFormatLabels(dss[len(dss)-1].Out))
//line lib/promrelabel/debug.qtpl:194
}
//line lib/promrelabel/debug.qtpl:193
//line lib/promrelabel/debug.qtpl:195
}
//line lib/promrelabel/debug.qtpl:193
//line lib/promrelabel/debug.qtpl:195
qw422016.N().S(`}`)
//line lib/promrelabel/debug.qtpl:195
}
//line lib/promrelabel/debug.qtpl:195
func WriteRelabelDebugStepsJSON(qq422016 qtio422016.Writer, targetURL, targetID string, dss []DebugStep, metric, relabelConfigs string, err error) {
//line lib/promrelabel/debug.qtpl:195
qw422016 := qt422016.AcquireWriter(qq422016)
//line lib/promrelabel/debug.qtpl:195
StreamRelabelDebugStepsJSON(qw422016, targetURL, targetID, dss, metric, relabelConfigs, err)
//line lib/promrelabel/debug.qtpl:195
qt422016.ReleaseWriter(qw422016)
//line lib/promrelabel/debug.qtpl:195
}
//line lib/promrelabel/debug.qtpl:195
func RelabelDebugStepsJSON(targetURL, targetID string, dss []DebugStep, metric, relabelConfigs string, err error) string {
//line lib/promrelabel/debug.qtpl:195
qb422016 := qt422016.AcquireByteBuffer()
//line lib/promrelabel/debug.qtpl:195
WriteRelabelDebugStepsJSON(qb422016, targetURL, targetID, dss, metric, relabelConfigs, err)
//line lib/promrelabel/debug.qtpl:195
qs422016 := string(qb422016.B)
//line lib/promrelabel/debug.qtpl:195
qt422016.ReleaseByteBuffer(qb422016)
//line lib/promrelabel/debug.qtpl:195
return qs422016
//line lib/promrelabel/debug.qtpl:195
//line lib/promrelabel/debug.qtpl:197
}
//line lib/promrelabel/debug.qtpl:197
func streamlabelsWithHighlight(qw422016 *qt422016.Writer, labels *promutil.Labels, highlight map[string]struct{}, color string) {
func WriteRelabelDebugStepsJSON(qq422016 qtio422016.Writer, targetURL, targetID string, dss []DebugStep, metric, relabelConfigs string, err error) {
//line lib/promrelabel/debug.qtpl:197
qw422016 := qt422016.AcquireWriter(qq422016)
//line lib/promrelabel/debug.qtpl:197
StreamRelabelDebugStepsJSON(qw422016, targetURL, targetID, dss, metric, relabelConfigs, err)
//line lib/promrelabel/debug.qtpl:197
qt422016.ReleaseWriter(qw422016)
//line lib/promrelabel/debug.qtpl:197
}
//line lib/promrelabel/debug.qtpl:197
func RelabelDebugStepsJSON(targetURL, targetID string, dss []DebugStep, metric, relabelConfigs string, err error) string {
//line lib/promrelabel/debug.qtpl:197
qb422016 := qt422016.AcquireByteBuffer()
//line lib/promrelabel/debug.qtpl:197
WriteRelabelDebugStepsJSON(qb422016, targetURL, targetID, dss, metric, relabelConfigs, err)
//line lib/promrelabel/debug.qtpl:197
qs422016 := string(qb422016.B)
//line lib/promrelabel/debug.qtpl:197
qt422016.ReleaseByteBuffer(qb422016)
//line lib/promrelabel/debug.qtpl:197
return qs422016
//line lib/promrelabel/debug.qtpl:197
}
//line lib/promrelabel/debug.qtpl:199
func streamlabelsWithHighlight(qw422016 *qt422016.Writer, labels *promutil.Labels, highlight map[string]struct{}, color string) {
//line lib/promrelabel/debug.qtpl:201
labelsList := labels.GetLabels()
metricName := ""
for i, label := range labelsList {
@@ -509,153 +501,153 @@ func streamlabelsWithHighlight(qw422016 *qt422016.Writer, labels *promutil.Label
}
}
//line lib/promrelabel/debug.qtpl:209
//line lib/promrelabel/debug.qtpl:211
if metricName != "" {
//line lib/promrelabel/debug.qtpl:210
if _, ok := highlight["__name__"]; ok {
//line lib/promrelabel/debug.qtpl:210
qw422016.N().S(`<span style="font-weight:bold;color:`)
//line lib/promrelabel/debug.qtpl:211
qw422016.E().S(color)
//line lib/promrelabel/debug.qtpl:211
qw422016.N().S(`">`)
//line lib/promrelabel/debug.qtpl:211
qw422016.E().S(metricName)
//line lib/promrelabel/debug.qtpl:211
qw422016.N().S(`</span>`)
//line lib/promrelabel/debug.qtpl:212
} else {
if _, ok := highlight["__name__"]; ok {
//line lib/promrelabel/debug.qtpl:212
qw422016.N().S(`<span style="font-weight:bold;color:`)
//line lib/promrelabel/debug.qtpl:213
qw422016.E().S(color)
//line lib/promrelabel/debug.qtpl:213
qw422016.N().S(`">`)
//line lib/promrelabel/debug.qtpl:213
qw422016.E().S(metricName)
//line lib/promrelabel/debug.qtpl:214
}
//line lib/promrelabel/debug.qtpl:215
if len(labelsList) == 0 {
//line lib/promrelabel/debug.qtpl:215
return
//line lib/promrelabel/debug.qtpl:215
}
//line lib/promrelabel/debug.qtpl:216
}
//line lib/promrelabel/debug.qtpl:216
qw422016.N().S(`{`)
//line lib/promrelabel/debug.qtpl:218
for i, label := range labelsList {
//line lib/promrelabel/debug.qtpl:219
if _, ok := highlight[label.Name]; ok {
//line lib/promrelabel/debug.qtpl:219
qw422016.N().S(`<span style="font-weight:bold;color:`)
//line lib/promrelabel/debug.qtpl:220
qw422016.E().S(color)
//line lib/promrelabel/debug.qtpl:220
qw422016.N().S(`">`)
//line lib/promrelabel/debug.qtpl:220
qw422016.E().S(label.Name)
//line lib/promrelabel/debug.qtpl:220
qw422016.N().S(`=`)
//line lib/promrelabel/debug.qtpl:220
qw422016.E().Q(label.Value)
//line lib/promrelabel/debug.qtpl:220
//line lib/promrelabel/debug.qtpl:213
qw422016.N().S(`</span>`)
//line lib/promrelabel/debug.qtpl:221
//line lib/promrelabel/debug.qtpl:214
} else {
//line lib/promrelabel/debug.qtpl:215
qw422016.E().S(metricName)
//line lib/promrelabel/debug.qtpl:216
}
//line lib/promrelabel/debug.qtpl:217
if len(labelsList) == 0 {
//line lib/promrelabel/debug.qtpl:217
return
//line lib/promrelabel/debug.qtpl:217
}
//line lib/promrelabel/debug.qtpl:218
}
//line lib/promrelabel/debug.qtpl:218
qw422016.N().S(`{`)
//line lib/promrelabel/debug.qtpl:220
for i, label := range labelsList {
//line lib/promrelabel/debug.qtpl:221
if _, ok := highlight[label.Name]; ok {
//line lib/promrelabel/debug.qtpl:221
qw422016.N().S(`<span style="font-weight:bold;color:`)
//line lib/promrelabel/debug.qtpl:222
qw422016.E().S(color)
//line lib/promrelabel/debug.qtpl:222
qw422016.N().S(`">`)
//line lib/promrelabel/debug.qtpl:222
qw422016.E().S(label.Name)
//line lib/promrelabel/debug.qtpl:222
qw422016.N().S(`=`)
//line lib/promrelabel/debug.qtpl:222
qw422016.E().Q(label.Value)
//line lib/promrelabel/debug.qtpl:222
qw422016.N().S(`</span>`)
//line lib/promrelabel/debug.qtpl:223
}
} else {
//line lib/promrelabel/debug.qtpl:224
qw422016.E().S(label.Name)
//line lib/promrelabel/debug.qtpl:224
qw422016.N().S(`=`)
//line lib/promrelabel/debug.qtpl:224
qw422016.E().Q(label.Value)
//line lib/promrelabel/debug.qtpl:225
}
//line lib/promrelabel/debug.qtpl:226
if i < len(labelsList)-1 {
//line lib/promrelabel/debug.qtpl:224
//line lib/promrelabel/debug.qtpl:226
qw422016.N().S(`,`)
//line lib/promrelabel/debug.qtpl:224
//line lib/promrelabel/debug.qtpl:226
qw422016.N().S(` `)
//line lib/promrelabel/debug.qtpl:224
//line lib/promrelabel/debug.qtpl:226
}
//line lib/promrelabel/debug.qtpl:225
//line lib/promrelabel/debug.qtpl:227
}
//line lib/promrelabel/debug.qtpl:225
//line lib/promrelabel/debug.qtpl:227
qw422016.N().S(`}`)
//line lib/promrelabel/debug.qtpl:227
}
//line lib/promrelabel/debug.qtpl:227
func writelabelsWithHighlight(qq422016 qtio422016.Writer, labels *promutil.Labels, highlight map[string]struct{}, color string) {
//line lib/promrelabel/debug.qtpl:227
qw422016 := qt422016.AcquireWriter(qq422016)
//line lib/promrelabel/debug.qtpl:227
streamlabelsWithHighlight(qw422016, labels, highlight, color)
//line lib/promrelabel/debug.qtpl:227
qt422016.ReleaseWriter(qw422016)
//line lib/promrelabel/debug.qtpl:227
}
//line lib/promrelabel/debug.qtpl:227
func labelsWithHighlight(labels *promutil.Labels, highlight map[string]struct{}, color string) string {
//line lib/promrelabel/debug.qtpl:227
qb422016 := qt422016.AcquireByteBuffer()
//line lib/promrelabel/debug.qtpl:227
writelabelsWithHighlight(qb422016, labels, highlight, color)
//line lib/promrelabel/debug.qtpl:227
qs422016 := string(qb422016.B)
//line lib/promrelabel/debug.qtpl:227
qt422016.ReleaseByteBuffer(qb422016)
//line lib/promrelabel/debug.qtpl:227
return qs422016
//line lib/promrelabel/debug.qtpl:227
//line lib/promrelabel/debug.qtpl:229
}
//line lib/promrelabel/debug.qtpl:229
func writelabelsWithHighlight(qq422016 qtio422016.Writer, labels *promutil.Labels, highlight map[string]struct{}, color string) {
//line lib/promrelabel/debug.qtpl:229
qw422016 := qt422016.AcquireWriter(qq422016)
//line lib/promrelabel/debug.qtpl:229
streamlabelsWithHighlight(qw422016, labels, highlight, color)
//line lib/promrelabel/debug.qtpl:229
qt422016.ReleaseWriter(qw422016)
//line lib/promrelabel/debug.qtpl:229
}
//line lib/promrelabel/debug.qtpl:229
func labelsWithHighlight(labels *promutil.Labels, highlight map[string]struct{}, color string) string {
//line lib/promrelabel/debug.qtpl:229
qb422016 := qt422016.AcquireByteBuffer()
//line lib/promrelabel/debug.qtpl:229
writelabelsWithHighlight(qb422016, labels, highlight, color)
//line lib/promrelabel/debug.qtpl:229
qs422016 := string(qb422016.B)
//line lib/promrelabel/debug.qtpl:229
qt422016.ReleaseByteBuffer(qb422016)
//line lib/promrelabel/debug.qtpl:229
return qs422016
//line lib/promrelabel/debug.qtpl:229
}
//line lib/promrelabel/debug.qtpl:231
func streammustFormatLabels(qw422016 *qt422016.Writer, s string) {
//line lib/promrelabel/debug.qtpl:230
//line lib/promrelabel/debug.qtpl:232
labels, err := promutil.NewLabelsFromString(s)
//line lib/promrelabel/debug.qtpl:231
if err != nil {
//line lib/promrelabel/debug.qtpl:231
qw422016.N().S(`<span style="color: red" title="error parsing labels:`)
//line lib/promrelabel/debug.qtpl:232
qw422016.E().S(err.Error())
//line lib/promrelabel/debug.qtpl:232
qw422016.N().S(`">`)
//line lib/promrelabel/debug.qtpl:232
qw422016.E().S("error parsing labels: " + err.Error())
//line lib/promrelabel/debug.qtpl:232
qw422016.N().S(`</span>`)
//line lib/promrelabel/debug.qtpl:233
} else {
if err != nil {
//line lib/promrelabel/debug.qtpl:233
qw422016.N().S(`<span style="color: red" title="error parsing labels:`)
//line lib/promrelabel/debug.qtpl:234
streamlabelsWithHighlight(qw422016, labels, nil, "")
qw422016.E().S(err.Error())
//line lib/promrelabel/debug.qtpl:234
qw422016.N().S(`">`)
//line lib/promrelabel/debug.qtpl:234
qw422016.E().S("error parsing labels: " + err.Error())
//line lib/promrelabel/debug.qtpl:234
qw422016.N().S(`</span>`)
//line lib/promrelabel/debug.qtpl:235
} else {
//line lib/promrelabel/debug.qtpl:236
streamlabelsWithHighlight(qw422016, labels, nil, "")
//line lib/promrelabel/debug.qtpl:237
}
//line lib/promrelabel/debug.qtpl:236
//line lib/promrelabel/debug.qtpl:238
}
//line lib/promrelabel/debug.qtpl:236
//line lib/promrelabel/debug.qtpl:238
func writemustFormatLabels(qq422016 qtio422016.Writer, s string) {
//line lib/promrelabel/debug.qtpl:236
//line lib/promrelabel/debug.qtpl:238
qw422016 := qt422016.AcquireWriter(qq422016)
//line lib/promrelabel/debug.qtpl:236
//line lib/promrelabel/debug.qtpl:238
streammustFormatLabels(qw422016, s)
//line lib/promrelabel/debug.qtpl:236
//line lib/promrelabel/debug.qtpl:238
qt422016.ReleaseWriter(qw422016)
//line lib/promrelabel/debug.qtpl:236
//line lib/promrelabel/debug.qtpl:238
}
//line lib/promrelabel/debug.qtpl:236
//line lib/promrelabel/debug.qtpl:238
func mustFormatLabels(s string) string {
//line lib/promrelabel/debug.qtpl:236
//line lib/promrelabel/debug.qtpl:238
qb422016 := qt422016.AcquireByteBuffer()
//line lib/promrelabel/debug.qtpl:236
//line lib/promrelabel/debug.qtpl:238
writemustFormatLabels(qb422016, s)
//line lib/promrelabel/debug.qtpl:236
//line lib/promrelabel/debug.qtpl:238
qs422016 := string(qb422016.B)
//line lib/promrelabel/debug.qtpl:236
//line lib/promrelabel/debug.qtpl:238
qt422016.ReleaseByteBuffer(qb422016)
//line lib/promrelabel/debug.qtpl:236
//line lib/promrelabel/debug.qtpl:238
return qs422016
//line lib/promrelabel/debug.qtpl:236
//line lib/promrelabel/debug.qtpl:238
}

View File

@@ -108,7 +108,7 @@ func (av *rateAggrValue) pushSample(c aggrConfig, sample *pushSample, key string
}
if ok {
state = sv.getState(av.isGreen)
if sample.timestamp < state.timestamp {
if sample.timestamp < state.timestamp || sample.timestamp < sv.prevTimestamp {
// Skip out of order sample
return
}
@@ -143,9 +143,6 @@ func (av *rateAggrValue) flush(c aggrConfig, ctx *flushCtx, key string, isLast b
putRateAggrSharedValue(sv)
continue
}
if sv.prevTimestamp == 0 {
continue
}
state = sv.getState(av.isGreen)
if state.timestamp > 0 {
d := float64(state.timestamp-sv.prevTimestamp) / 1000

View File

@@ -789,6 +789,24 @@ foo:1m_by_cde_rate_sum{cde="1"} 0.125
outputs: [rate_sum, rate_avg]
`, "11111")
// test rate_sum with out of order samples
f([]string{`
foo 1
`, `
foo 61
`, `
foo 31 -70
foo 91
`, `
foo 121
`}, time.Minute, `foo:1m_rate_sum 1
foo:1m_rate_sum 0.5
foo:1m_rate_sum 0.5
`, `
- interval: 1m
outputs: [rate_sum]
`, "11111")
// test rate_sum and rate_avg with different staleness intervals
f([]string{`
foo{abc="123", cde="1"} 1

View File

@@ -30,7 +30,7 @@ func parseIfFilter(lex *lexer) (*ifFilter, error) {
return newIfFilter(newFilterNoop()), nil
}
f, err := parseFilter(lex, true)
f, err := parseFilter(lex)
if err != nil {
return nil, fmt.Errorf("cannot parse 'if' filter: %w", err)
}

View File

@@ -301,38 +301,43 @@ func (lr *LogRows) NeedFlush() bool {
// MustAddInsertRow adds r to lr.
func (lr *LogRows) MustAddInsertRow(r *InsertRow) {
// verify r.StreamTagsCanonical
if err := verifyStreamTagsCanonical(r.StreamTagsCanonical, r.Fields); err != nil {
st := GetStreamTags()
streamTagsCanonical := r.StreamTagsCanonical
if err := parseStreamTagsCanonical(st, streamTagsCanonical); err != nil {
line := MarshalFieldsToJSON(nil, r.Fields)
invalidStreamTagsLogger.Warnf("cannot unmarshal streamTagsCanonical: %s; skipping the log entry; log entry: %s", err, line)
PutStreamTags(st)
return
}
if st.normalize(r.Fields) {
bLen := len(lr.a.b)
lr.a.b = st.MarshalCanonical(lr.a.b)
streamTagsCanonical = bytesutil.ToUnsafeString(lr.a.b[bLen:])
}
PutStreamTags(st)
// Calculate the id for the StreamTags
var sid streamID
sid.tenantID = r.TenantID
streamTagsCanonical := bytesutil.ToUnsafeBytes(r.StreamTagsCanonical)
sid.id = hash128(streamTagsCanonical)
sid.id = hash128(bytesutil.ToUnsafeBytes(streamTagsCanonical))
// Store the row
lr.mustAddInternal(sid, r.Timestamp, r.Fields, r.StreamTagsCanonical)
lr.mustAddInternal(sid, r.Timestamp, r.Fields, streamTagsCanonical)
}
var invalidStreamTagsLogger = logger.WithThrottler("invalid_stream_tags", 5*time.Second)
func verifyStreamTagsCanonical(streamTagsCanonical string, fields []Field) error {
st := GetStreamTags()
defer PutStreamTags(st)
func parseStreamTagsCanonical(dst *StreamTags, streamTagsCanonical string) error {
src := bytesutil.ToUnsafeBytes(streamTagsCanonical)
tail, err := st.UnmarshalCanonicalInplace(src)
tail, err := dst.UnmarshalCanonicalInplace(src)
if err != nil {
return fmt.Errorf("cannot unmarshal streamTagsCanonical: %w", err)
}
if len(tail) > 0 {
return fmt.Errorf("unexpected tail left after unmarshaling streamTagsCanonical; len(tail)=%d; streamTags: %s", len(tail), st)
return fmt.Errorf("unexpected tail left after unmarshaling streamTagsCanonical; len(tail)=%d; streamTags: %s", len(tail), dst)
}
return st.verifyCanonicalFieldValues(fields)
return nil
}
func (lr *LogRows) mustAdd(tenantID TenantID, timestamp int64, fields []Field) {
@@ -394,11 +399,7 @@ func (lr *LogRows) MustAdd(tenantID TenantID, timestamp int64, fields []Field, s
invalidStreamTagsLogger.Warnf("cannot parse _stream=%s: %s; skipping the log entry; log entry: %s", f.Value, err, line)
return
}
if err := st.verifyCanonicalFieldValues(fields); err != nil {
line := MarshalFieldsToJSON(nil, fields)
invalidStreamTagsLogger.Warnf("invalid _stream=%s: %s; skipping the log entry; log entry: %s", f.Value, err, line)
return
}
st.normalize(fields)
// Remove _stream field, since it is re-generated from st below.
f.Value = ""
case "_stream_id":

View File

@@ -1928,7 +1928,7 @@ func parseQuery(lex *lexer) (*Query, error) {
lex.pushQueryOptions(&q.opts)
defer lex.popQueryOptions()
f, err := parseFilter(lex, true)
f, err := parseFilter(lex)
if err != nil {
return nil, fmt.Errorf("%w; context: [%s]", err, lex.context())
}
@@ -2113,20 +2113,11 @@ func parseQueryOptions(dstOpts *queryOptions, lex *lexer) error {
}
}
func parseFilter(lex *lexer, allowPipeKeywords bool) (filter, error) {
func parseFilter(lex *lexer) (filter, error) {
if lex.isQueryPartTrailer() {
return nil, fmt.Errorf("missing query")
}
if !allowPipeKeywords {
// Verify the first token in the filter doesn't match pipe names.
firstToken := strings.ToLower(lex.rawToken)
if firstToken == "by" || isPipeName(firstToken) || isStatsFuncName(firstToken) {
return nil, fmt.Errorf("query filter cannot start with pipe keyword %q; see https://docs.victoriametrics.com/victorialogs/logsql/#query-syntax; "+
"please put the first word of the filter into quotes", firstToken)
}
}
fo, err := parseFilterOr(lex, "")
if err != nil {
return nil, err
@@ -3875,7 +3866,7 @@ func quoteFieldFilterIfNeeded(s string) string {
if wildcard == "" || !needQuoteToken(wildcard) {
return s
}
return strconv.Quote(s)
return strconv.Quote(wildcard) + "*"
}
func quoteTokenIfNeeded(s string) string {
@@ -3912,7 +3903,7 @@ func isNumberPrefix(s string) bool {
}
func needQuoteToken(s string) bool {
if s == "." {
if !isWord(s) {
return true
}
@@ -3920,14 +3911,10 @@ func needQuoteToken(s string) bool {
if _, ok := reservedKeywords[sLower]; ok {
return true
}
if isPipeName(sLower) || isStatsFuncName(sLower) {
if isPipeName(sLower) || isStatsFuncName(sLower) || isMathFuncName(sLower) {
return true
}
for _, r := range s {
if !isTokenRune(r) && r != '.' {
return true
}
}
return false
}
@@ -4008,6 +3995,9 @@ var reservedKeywords = func() map[string]struct{} {
// 'as' is used in various pipes such as `format ... as ...`
"as",
// 'from' is used in various pipes such as `split ... from ...` and `unpack_json from ...`
"from",
}
m := make(map[string]struct{}, len(kws))
for _, kw := range kws {
@@ -4057,12 +4047,12 @@ func toFieldsFilters(pf *prefixfilter.Filter) string {
denyFilters := pf.GetDenyFilters()
if len(denyFilters) > 0 {
qStr += " | delete " + fieldNamesString(denyFilters)
qStr += " | delete " + fieldFiltersString(denyFilters)
}
allowFilters := pf.GetAllowFilters()
if len(allowFilters) > 0 && !prefixfilter.MatchAll(allowFilters) {
qStr += " | fields " + fieldNamesString(allowFilters)
qStr += " | fields " + fieldFiltersString(allowFilters)
}
return qStr

View File

@@ -0,0 +1,123 @@
package logstorage
import (
"fmt"
"strings"
"github.com/VictoriaMetrics/VictoriaLogs/lib/prefixfilter"
)
func parseFieldNamesInParens(lex *lexer) ([]string, error) {
fieldNames, err := parseFieldFiltersInParens(lex)
if err != nil {
return nil, err
}
for _, fieldName := range fieldNames {
if prefixfilter.IsWildcardFilter(fieldName) {
return nil, fmt.Errorf("the field name %q cannot end with '*'", fieldName)
}
}
return fieldNames, nil
}
func parseFieldFiltersInParens(lex *lexer) ([]string, error) {
if !lex.isKeyword("(") {
return nil, fmt.Errorf("missing `(`")
}
var fields []string
for {
lex.nextToken()
if lex.isKeyword(")") {
lex.nextToken()
return fields, nil
}
if lex.isKeyword(",") {
return nil, fmt.Errorf("unexpected `,`")
}
field, err := parseFieldFilter(lex)
if err != nil {
return nil, err
}
fields = append(fields, field)
switch {
case lex.isKeyword(")"):
lex.nextToken()
return fields, nil
case lex.isKeyword(","):
default:
return nil, fmt.Errorf("unexpected token: %q; expecting ',' or ')'", lex.token)
}
}
}
func parseCommaSeparatedFieldNames(lex *lexer) ([]string, error) {
fieldNames, err := parseCommaSeparatedFieldFilters(lex)
if err != nil {
return nil, err
}
for _, fieldName := range fieldNames {
if prefixfilter.IsWildcardFilter(fieldName) {
return nil, fmt.Errorf("the field name %q cannot end with '*'", fieldName)
}
}
return fieldNames, nil
}
func parseCommaSeparatedFieldFilters(lex *lexer) ([]string, error) {
var fields []string
for {
field, err := parseFieldFilter(lex)
if err != nil {
return nil, fmt.Errorf("cannot parse field name: %w", err)
}
fields = append(fields, field)
if !lex.isKeyword(",") {
return fields, nil
}
lex.nextToken()
}
}
func parseFieldName(lex *lexer) (string, error) {
fieldName, err := lex.nextCompoundToken()
if err != nil {
return "", err
}
fieldName = getCanonicalColumnName(fieldName)
return fieldName, nil
}
func parseFieldFilter(lex *lexer) (string, error) {
if lex.isKeyword("*") {
lex.nextToken()
return "*", nil
}
fieldName, err := lex.nextCompoundToken()
if err != nil {
return "", err
}
fieldName = getCanonicalColumnName(fieldName)
if !lex.isSkippedSpace && lex.isKeyword("*") {
lex.nextToken()
fieldName += "*"
}
return fieldName, nil
}
func fieldNamesString(fieldNames []string) string {
a := make([]string, len(fieldNames))
for i, f := range fieldNames {
a[i] = quoteTokenIfNeeded(f)
}
return strings.Join(a, ", ")
}
func fieldFiltersString(fieldFilters []string) string {
a := make([]string, len(fieldFilters))
for i, f := range fieldFilters {
a[i] = quoteFieldFilterIfNeeded(f)
}
return strings.Join(a, ", ")
}

View File

@@ -146,23 +146,54 @@ func parsePipe(lex *lexer) (pipe, error) {
return p, nil
}
lexState := lex.backupState()
// Try parsing stats pipe without 'stats' keyword
ps, err := parsePipeStatsNoStatsKeyword(lex)
if err == nil {
if isLikelyStatsPipe(lex) {
// Try parsing stats pipe without 'stats' keyword
ps, err := parsePipeStatsNoStatsKeyword(lex)
if err != nil {
return nil, fmt.Errorf("cannot parse 'stats' pipe: %w", err)
}
return ps, nil
}
lex.restoreState(lexState)
// Try parsing filter pipe without 'filter' keyword
pf, err := parsePipeFilterNoFilterKeyword(lex)
if err == nil {
if isLikelyFilterPipe(lex) {
// Try parsing filter pipe without 'filter' keyword
pf, err := parsePipeFilterNoFilterKeyword(lex)
if err != nil {
return nil, fmt.Errorf("cannot parse 'filter' pipe: %w", err)
}
return pf, nil
}
lex.restoreState(lexState)
return nil, fmt.Errorf("unexpected pipe %q", lex.token)
return nil, fmt.Errorf("unexpected pipe name %q; probably, 'filter' is missing in front of %q; "+
"see https://docs.victoriametrics.com/victorialogs/logsql/#filter-pipe", lex.token, lex.token)
}
func isLikelyStatsPipe(lex *lexer) bool {
return isStatsFuncName(lex.rawToken) || lex.isKeyword("by", "(")
}
func isLikelyFilterPipe(lex *lexer) bool {
if lex.isQuotedToken() {
return true
}
if !isWord(lex.token) {
// Any token that isn't a word cannot clash with a pipe name,
// since all pipe names are words. So treat it as a filter.
return true
}
if lex.isKeyword("not") {
// 'not' is a logical filter operator rather than a pipe name.
return true
}
lexState := lex.backupState()
defer lex.restoreState(lexState)
stopTokens := []string{":"}
if _, err := lex.nextCompoundTokenExt(stopTokens); err != nil {
return false
}
return lex.isKeyword(":")
}
var pipeParsers map[string]pipeParseFunc
@@ -201,6 +232,7 @@ func initPipeParsers() {
"generate_sequence": parsePipeGenerateSequence,
"hash": parsePipeHash,
"join": parsePipeJoin,
"json_array_concat": parsePipeJSONArrayConcat,
"json_array_len": parsePipeJSONArrayLen,
"head": parsePipeLimit,
"keep": parsePipeFields,

View File

@@ -23,7 +23,7 @@ func (pc *pipeCoalesce) String() string {
logger.Panicf("BUG: pipeCoalesce must contain at least one srcField")
}
s := "coalesce(" + fieldNamesString(pc.srcFieldFilters) + ")"
s := "coalesce(" + fieldFiltersString(pc.srcFieldFilters) + ")"
if pc.defaultValue != "" {
s += " default " + quoteTokenIfNeeded(pc.defaultValue)
}

View File

@@ -21,7 +21,7 @@ func (pd *pipeDelete) String() string {
logger.Panicf("BUG: pipeDelete must contain at least a single field")
}
return "delete " + fieldNamesString(pd.fieldFilters)
return "delete " + fieldFiltersString(pd.fieldFilters)
}
func (pd *pipeDelete) splitToRemoteAndLocal(_ int64) (pipe, []pipe) {
@@ -87,7 +87,7 @@ func parsePipeDelete(lex *lexer) (pipe, error) {
}
lex.nextToken()
fieldFilters, err := parseCommaSeparatedFields(lex)
fieldFilters, err := parseCommaSeparatedFieldFilters(lex)
if err != nil {
return nil, err
}

View File

@@ -20,7 +20,7 @@ func (pf *pipeFields) String() string {
if len(pf.fieldFilters) == 0 {
logger.Panicf("BUG: pipeFields must contain at least a single field filter")
}
return "fields " + fieldNamesString(pf.fieldFilters)
return "fields " + fieldFiltersString(pf.fieldFilters)
}
func (pf *pipeFields) splitToRemoteAndLocal(_ int64) (pipe, []pipe) {
@@ -100,7 +100,7 @@ func parsePipeFields(lex *lexer) (pipe, error) {
}
lex.nextToken()
fieldFilters, err := parseCommaSeparatedFields(lex)
fieldFilters, err := parseCommaSeparatedFieldFilters(lex)
if err != nil {
return nil, err
}
@@ -109,18 +109,3 @@ func parsePipeFields(lex *lexer) (pipe, error) {
}
return pf, nil
}
func parseCommaSeparatedFields(lex *lexer) ([]string, error) {
var fields []string
for {
field, err := parseFieldFilter(lex)
if err != nil {
return nil, fmt.Errorf("cannot parse field name: %w", err)
}
fields = append(fields, field)
if !lex.isKeyword(",") {
return fields, nil
}
lex.nextToken()
}
}

View File

@@ -124,7 +124,7 @@ func parsePipeFilterExt(lex *lexer, needFilterKeyword bool) (pipe, error) {
lex.nextToken()
}
f, err := parseFilter(lex, needFilterKeyword)
f, err := parseFilter(lex)
if err != nil {
return nil, fmt.Errorf("cannot parse 'filter': %w", err)
}

View File

@@ -192,7 +192,6 @@ func (pjp *pipeJoinProcessor) writeBlock(workerID uint, br *blockResult) {
for i := range cs {
name := cs[i].name
byValuesIdxs[i] = slices.Index(pj.byFields, name)
}
for rowIdx := range br.rowsLen {

View File

@@ -0,0 +1,197 @@
package logstorage
import (
"fmt"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/atomicutil"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/bytesutil"
"github.com/VictoriaMetrics/VictoriaLogs/lib/prefixfilter"
)
// pipeJSONArrayConcat processes '| json_array_concat ...' pipe.
//
// See https://docs.victoriametrics.com/victorialogs/logsql/#json_array_concat-pipe
type pipeJSONArrayConcat struct {
delimiter string
fromField string
resultField string
}
func (pc *pipeJSONArrayConcat) String() string {
s := "json_array_concat"
if pc.delimiter != "" {
s += " " + quoteTokenIfNeeded(pc.delimiter)
}
if !isMsgFieldName(pc.fromField) {
s += " from " + quoteTokenIfNeeded(pc.fromField)
}
if pc.resultField != pc.fromField {
s += " as " + quoteTokenIfNeeded(pc.resultField)
}
return s
}
func (pc *pipeJSONArrayConcat) splitToRemoteAndLocal(_ int64) (pipe, []pipe) {
return pc, nil
}
func (pc *pipeJSONArrayConcat) canLiveTail() bool {
return true
}
func (pc *pipeJSONArrayConcat) canReturnLastNResults() bool {
return pc.resultField != "_time"
}
func (pc *pipeJSONArrayConcat) isFixedOutputFieldsOrder() bool {
return false
}
func (pc *pipeJSONArrayConcat) updateNeededFields(pf *prefixfilter.Filter) {
if pf.MatchString(pc.resultField) {
pf.AddDenyFilter(pc.resultField)
pf.AddAllowFilter(pc.fromField)
}
}
func (pc *pipeJSONArrayConcat) hasFilterInWithQuery() bool {
return false
}
func (pc *pipeJSONArrayConcat) initFilterInValues(_ *inValuesCache, _ getFieldValuesFunc) (pipe, error) {
return pc, nil
}
func (pc *pipeJSONArrayConcat) visitSubqueries(_ func(q *Query)) {
// nothing to do
}
func (pc *pipeJSONArrayConcat) newPipeProcessor(_ int, _ <-chan struct{}, _ func(), ppNext pipeProcessor) pipeProcessor {
pcp := &pipeJSONArrayConcatProcessor{
pc: pc,
ppNext: ppNext,
}
pcp.shards.Init = func(shard *pipeJSONArrayConcatProcessorShard) {
shard.reset()
}
return pcp
}
type pipeJSONArrayConcatProcessor struct {
pc *pipeJSONArrayConcat
ppNext pipeProcessor
shards atomicutil.Slice[pipeJSONArrayConcatProcessorShard]
}
func (pcp *pipeJSONArrayConcatProcessor) writeBlock(workerID uint, br *blockResult) {
if br.rowsLen == 0 {
return
}
shard := pcp.shards.Get(workerID)
shard.rc.name = pcp.pc.resultField
c := br.getColumnByName(pcp.pc.fromField)
delimiter := pcp.pc.delimiter
if c.isConst {
// Fast path for const column
v := c.valuesEncoded[0]
out := shard.concat(v, delimiter)
shard.rc.addValue(out)
br.addResultColumnConst(shard.rc)
} else {
// Slow path for other columns
values := c.getValues(br)
prevOut := ""
for rowIdx := range values {
if rowIdx == 0 || values[rowIdx] != values[rowIdx-1] {
prevOut = shard.concat(values[rowIdx], delimiter)
}
shard.rc.addValue(prevOut)
}
br.addResultColumn(shard.rc)
}
pcp.ppNext.writeBlock(workerID, br)
shard.reset()
}
type pipeJSONArrayConcatProcessorShard struct {
a arena
rc resultColumn
tmpValues []string
}
func (shard *pipeJSONArrayConcatProcessorShard) reset() {
shard.a.reset()
shard.rc.reset()
shard.tmpValues = shard.tmpValues[:0]
}
func (shard *pipeJSONArrayConcatProcessorShard) concat(arrayStr, delimiter string) string {
shard.tmpValues = unpackJSONArray(shard.tmpValues[:0], &shard.a, arrayStr)
bLen := len(shard.a.b)
for i, v := range shard.tmpValues {
if i > 0 {
shard.a.b = append(shard.a.b, delimiter...)
}
shard.a.b = append(shard.a.b, v...)
}
return bytesutil.ToUnsafeString(shard.a.b[bLen:])
}
func (pcp *pipeJSONArrayConcatProcessor) flush() error {
return nil
}
func parsePipeJSONArrayConcat(lex *lexer) (pipe, error) {
if !lex.isKeyword("json_array_concat") {
return nil, fmt.Errorf("unexpected token: %q; want %q", lex.token, "json_array_concat")
}
lex.nextToken()
delimiter := ""
if !lex.isQueryPartTrailer() && !lex.isKeyword("from", "as") {
s, err := lex.nextCompoundToken()
if err != nil {
return nil, fmt.Errorf("cannot parse delimiter for 'json_array_concat': %w", err)
}
delimiter = s
}
fromField := "_msg"
if !lex.isQueryPartTrailer() && !lex.isKeyword("as") {
if lex.isKeyword("from") {
lex.nextToken()
}
f, err := parseFieldName(lex)
if err != nil {
return nil, fmt.Errorf("cannot parse 'from' field for 'json_array_concat': %w", err)
}
fromField = f
}
resultField := fromField
if !lex.isQueryPartTrailer() {
if lex.isKeyword("as") {
lex.nextToken()
}
f, err := parseFieldName(lex)
if err != nil {
return nil, fmt.Errorf("cannot parse result field for 'json_array_concat': %w", err)
}
resultField = f
}
return &pipeJSONArrayConcat{
delimiter: delimiter,
fromField: fromField,
resultField: resultField,
}, nil
}

View File

@@ -4,6 +4,7 @@ import (
"fmt"
"math"
"strings"
"sync"
"time"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/atomicutil"
@@ -567,32 +568,51 @@ func parseMathExprInParens(lex *lexer) (*mathExpr, error) {
return me, nil
}
type mathFuncParser func(lex *lexer) (*mathExpr, error)
var mathFuncParsers map[string]mathFuncParser
var mathFuncParsersOnce sync.Once
func getMathFuncParsers() map[string]mathFuncParser {
mathFuncParsersOnce.Do(initMathFuncParsers)
return mathFuncParsers
}
func initMathFuncParsers() {
mathFuncParsers = map[string]mathFuncParser{
"abs": parseMathExprAbs,
"ceil": parseMathExprCeil,
"exp": parseMathExprExp,
"floor": parseMathExprFloor,
"ln": parseMathExprLn,
"max": parseMathExprMax,
"min": parseMathExprMin,
"now": parseMathExprNow,
"rand": parseMathExprRand,
"round": parseMathExprRound,
}
}
func isMathFuncName(s string) bool {
mps := getMathFuncParsers()
sLower := strings.ToLower(s)
return mps[sLower] != nil
}
func parseMathExprOperand(lex *lexer) (*mathExpr, error) {
if lex.isKeyword("(") {
return parseMathExprInParens(lex)
}
// A quoted token (e.g. "abs") isn't a keyword, so isKeyword() returns false for it
// and it falls through to parseMathExprFieldName below as a field name.
for funcName, parseFunc := range getMathFuncParsers() {
if lex.isKeyword(funcName) {
return parseFunc(lex)
}
}
switch {
case lex.isKeyword("abs"):
return parseMathExprAbs(lex)
case lex.isKeyword("exp"):
return parseMathExprExp(lex)
case lex.isKeyword("ln"):
return parseMathExprLn(lex)
case lex.isKeyword("max"):
return parseMathExprMax(lex)
case lex.isKeyword("min"):
return parseMathExprMin(lex)
case lex.isKeyword("now"):
return parseMathExprNow(lex)
case lex.isKeyword("rand"):
return parseMathExprRand(lex)
case lex.isKeyword("round"):
return parseMathExprRound(lex)
case lex.isKeyword("ceil"):
return parseMathExprCeil(lex)
case lex.isKeyword("floor"):
return parseMathExprFloor(lex)
case lex.isKeyword("-"):
return parseMathExprUnaryMinus(lex)
case lex.isKeyword("+"):

View File

@@ -20,7 +20,7 @@ type pipePackJSON struct {
func (pp *pipePackJSON) String() string {
s := "pack_json"
if len(pp.fieldFilters) > 0 {
s += " fields (" + fieldNamesString(pp.fieldFilters) + ")"
s += " fields (" + fieldFiltersString(pp.fieldFilters) + ")"
}
if !isMsgFieldName(pp.resultField) {
s += " as " + quoteTokenIfNeeded(pp.resultField)

View File

@@ -20,7 +20,7 @@ type pipePackLogfmt struct {
func (pp *pipePackLogfmt) String() string {
s := "pack_logfmt"
if len(pp.fieldFilters) > 0 {
s += " fields (" + fieldNamesString(pp.fieldFilters) + ")"
s += " fields (" + fieldFiltersString(pp.fieldFilters) + ")"
}
if !isMsgFieldName(pp.resultField) {
s += " as " + quoteTokenIfNeeded(pp.resultField)

View File

@@ -92,7 +92,6 @@ func (pr *pipeReplaceRegexp) newPipeProcessor(_ int, _ <-chan struct{}, _ func()
}
return newPipeUpdateProcessor(updateFunc, ppNext, pr.field, pr.iff)
}
func parsePipeReplaceRegexp(lex *lexer) (pipe, error) {

View File

@@ -25,7 +25,7 @@ func (ps *pipeSetStreamFields) String() string {
if ps.iff != nil {
s += " " + ps.iff.String()
}
s += " " + fieldNamesString(ps.streamFieldFilters)
s += " " + fieldFiltersString(ps.streamFieldFilters)
return s
}
@@ -186,7 +186,7 @@ func parsePipeSetStreamFields(lex *lexer) (pipe, error) {
}
// Parse stream fields
streamFieldFilters, err := parseCommaSeparatedFields(lex)
streamFieldFilters, err := parseCommaSeparatedFieldFilters(lex)
if err != nil {
return nil, err
}

View File

@@ -1395,7 +1395,7 @@ func parsePipeStatsExt(lex *lexer, needStatsKeyword bool) (pipe, error) {
for {
e, err := parseStatsEntry(lex)
if err != nil {
return nil, fmt.Errorf("cannot parse 'stats' entry: %w", err)
return nil, err
}
ps.entries = append(ps.entries, e)
@@ -1779,85 +1779,6 @@ func tryParseBucketSize(s string) (float64, bool) {
return 0, false
}
func parseFieldNamesInParens(lex *lexer) ([]string, error) {
fieldNames, err := parseFieldFiltersInParens(lex)
if err != nil {
return nil, err
}
for _, fieldName := range fieldNames {
if prefixfilter.IsWildcardFilter(fieldName) {
return nil, fmt.Errorf("the field name %q cannot end with '*'", fieldName)
}
}
return fieldNames, nil
}
func parseFieldFiltersInParens(lex *lexer) ([]string, error) {
if !lex.isKeyword("(") {
return nil, fmt.Errorf("missing `(`")
}
var fields []string
for {
lex.nextToken()
if lex.isKeyword(")") {
lex.nextToken()
return fields, nil
}
if lex.isKeyword(",") {
return nil, fmt.Errorf("unexpected `,`")
}
field, err := parseFieldFilter(lex)
if err != nil {
return nil, err
}
fields = append(fields, field)
switch {
case lex.isKeyword(")"):
lex.nextToken()
return fields, nil
case lex.isKeyword(","):
default:
return nil, fmt.Errorf("unexpected token: %q; expecting ',' or ')'", lex.token)
}
}
}
func parseFieldName(lex *lexer) (string, error) {
fieldName, err := lex.nextCompoundToken()
if err != nil {
return "", err
}
fieldName = getCanonicalColumnName(fieldName)
return fieldName, nil
}
func parseFieldFilter(lex *lexer) (string, error) {
if lex.isKeyword("*") {
lex.nextToken()
return "*", nil
}
fieldName, err := lex.nextCompoundToken()
if err != nil {
return "", err
}
fieldName = getCanonicalColumnName(fieldName)
if !lex.isSkippedSpace && lex.isKeyword("*") {
lex.nextToken()
fieldName += "*"
}
return fieldName, nil
}
func fieldNamesString(fields []string) string {
a := make([]string, len(fields))
for i, f := range fields {
a[i] = quoteFieldFilterIfNeeded(f)
}
return strings.Join(a, ", ")
}
func areConstValues(values []string) bool {
if len(values) == 0 {
return false

View File

@@ -621,7 +621,7 @@ func parsePipeTop(lex *lexer) (pipe, error) {
}
byFields = bfs
} else if !lex.isKeyword("hits", "rank") && !lex.isQueryPartTrailer() {
bfs, err := parseCommaSeparatedFields(lex)
bfs, err := parseCommaSeparatedFieldNames(lex)
if err != nil {
return nil, fmt.Errorf("cannot parse 'by ...': %w", err)
}

View File

@@ -551,7 +551,7 @@ func parsePipeUniq(lex *lexer) (pipe, error) {
}
byFields = bfs
} else if !lex.isKeyword("filter", "with", "hits", "limit") && !lex.isQueryPartTrailer() {
bfs, err := parseCommaSeparatedFields(lex)
bfs, err := parseCommaSeparatedFieldNames(lex)
if err != nil {
return nil, fmt.Errorf("cannot parse 'by ...': %w", err)
}

View File

@@ -83,7 +83,6 @@ func (uctx *fieldsUnpackerContext) addField(name, value string) {
func newPipeUnpackProcessor(unpackFunc func(uctx *fieldsUnpackerContext, s string), ppNext pipeProcessor,
fromField string, fieldPrefix string, keepOriginalFields, skipEmptyResults bool, iff *ifFilter) *pipeUnpackProcessor {
return &pipeUnpackProcessor{
unpackFunc: unpackFunc,
ppNext: ppNext,

View File

@@ -41,7 +41,7 @@ func (pu *pipeUnpackJSON) String() string {
s += " from " + quoteTokenIfNeeded(pu.fromField)
}
if !prefixfilter.MatchAll(pu.fieldFilters) {
s += " fields (" + fieldNamesString(pu.fieldFilters) + ")"
s += " fields (" + fieldFiltersString(pu.fieldFilters) + ")"
}
if len(pu.preserveKeys) > 0 {
s += " preserve_keys (" + fieldNamesString(pu.preserveKeys) + ")"

View File

@@ -35,7 +35,7 @@ func (pu *pipeUnpackLogfmt) String() string {
s += " from " + quoteTokenIfNeeded(pu.fromField)
}
if !prefixfilter.MatchAll(pu.fieldFilters) {
s += " fields (" + fieldNamesString(pu.fieldFilters) + ")"
s += " fields (" + fieldFiltersString(pu.fieldFilters) + ")"
}
if pu.resultPrefix != "" {
s += " result_prefix " + quoteTokenIfNeeded(pu.resultPrefix)

View File

@@ -234,7 +234,7 @@ func parsePipeUnroll(lex *lexer) (pipe, error) {
}
fields = fs
} else {
fs, err := parseCommaSeparatedFields(lex)
fs, err := parseCommaSeparatedFieldNames(lex)
if err != nil {
return nil, fmt.Errorf("cannot parse 'by ...': %w", err)
}

View File

@@ -11,7 +11,7 @@ type runningStatsCount struct {
}
func (sc *runningStatsCount) String() string {
return "count(" + fieldNamesString(sc.fieldFilters) + ")"
return "count(" + fieldFiltersString(sc.fieldFilters) + ")"
}
func (sc *runningStatsCount) updateNeededFields(pf *prefixfilter.Filter) {

View File

@@ -9,7 +9,7 @@ type runningStatsMax struct {
}
func (sm *runningStatsMax) String() string {
return "max(" + fieldNamesString(sm.fieldFilters) + ")"
return "max(" + fieldFiltersString(sm.fieldFilters) + ")"
}
func (sm *runningStatsMax) updateNeededFields(pf *prefixfilter.Filter) {

View File

@@ -9,7 +9,7 @@ type runningStatsMin struct {
}
func (sm *runningStatsMin) String() string {
return "min(" + fieldNamesString(sm.fieldFilters) + ")"
return "min(" + fieldFiltersString(sm.fieldFilters) + ")"
}
func (sm *runningStatsMin) updateNeededFields(pf *prefixfilter.Filter) {

View File

@@ -12,7 +12,7 @@ type runningStatsSum struct {
}
func (ss *runningStatsSum) String() string {
return "sum(" + fieldNamesString(ss.fieldFilters) + ")"
return "sum(" + fieldFiltersString(ss.fieldFilters) + ")"
}
func (ss *runningStatsSum) updateNeededFields(pf *prefixfilter.Filter) {

View File

@@ -15,7 +15,7 @@ type statsAvg struct {
}
func (sa *statsAvg) String() string {
return "avg(" + fieldNamesString(sa.fieldFilters) + ")"
return "avg(" + fieldFiltersString(sa.fieldFilters) + ")"
}
func (sa *statsAvg) updateNeededFields(pf *prefixfilter.Filter) {

View File

@@ -16,7 +16,7 @@ type statsCount struct {
}
func (sc *statsCount) String() string {
return "count(" + fieldNamesString(sc.fieldFilters) + ")"
return "count(" + fieldFiltersString(sc.fieldFilters) + ")"
}
func (sc *statsCount) updateNeededFields(pf *prefixfilter.Filter) {

View File

@@ -16,7 +16,7 @@ type statsCountEmpty struct {
}
func (sc *statsCountEmpty) String() string {
return "count_empty(" + fieldNamesString(sc.fieldFilters) + ")"
return "count_empty(" + fieldFiltersString(sc.fieldFilters) + ")"
}
func (sc *statsCountEmpty) updateNeededFields(pf *prefixfilter.Filter) {

View File

@@ -28,7 +28,7 @@ type statsJSONValues struct {
}
func (sv *statsJSONValues) String() string {
s := "json_values(" + fieldNamesString(sv.fieldFilters) + ")"
s := "json_values(" + fieldFiltersString(sv.fieldFilters) + ")"
if len(sv.sortFields) > 0 {
a := make([]string, len(sv.sortFields))

View File

@@ -17,7 +17,7 @@ type statsMax struct {
}
func (sm *statsMax) String() string {
return "max(" + fieldNamesString(sm.fieldFilters) + ")"
return "max(" + fieldFiltersString(sm.fieldFilters) + ")"
}
func (sm *statsMax) updateNeededFields(pf *prefixfilter.Filter) {

View File

@@ -9,7 +9,7 @@ type statsMedian struct {
}
func (sm *statsMedian) String() string {
return "median(" + fieldNamesString(sm.sq.fieldFilters) + ")"
return "median(" + fieldFiltersString(sm.sq.fieldFilters) + ")"
}
func (sm *statsMedian) updateNeededFields(pf *prefixfilter.Filter) {

View File

@@ -17,7 +17,7 @@ type statsMin struct {
}
func (sm *statsMin) String() string {
return "min(" + fieldNamesString(sm.fieldFilters) + ")"
return "min(" + fieldFiltersString(sm.fieldFilters) + ")"
}
func (sm *statsMin) updateNeededFields(pf *prefixfilter.Filter) {

View File

@@ -24,7 +24,7 @@ type statsQuantile struct {
func (sq *statsQuantile) String() string {
s := "quantile(" + sq.phiStr
if !prefixfilter.MatchAll(sq.fieldFilters) {
s += ", " + fieldNamesString(sq.fieldFilters)
s += ", " + fieldFiltersString(sq.fieldFilters)
}
s += ")"
return s

View File

@@ -14,7 +14,7 @@ type statsRateSum struct {
}
func (sr *statsRateSum) String() string {
return "rate_sum(" + fieldNamesString(sr.ss.fieldFilters) + ")"
return "rate_sum(" + fieldFiltersString(sr.ss.fieldFilters) + ")"
}
func (sr *statsRateSum) updateNeededFields(pf *prefixfilter.Filter) {

View File

@@ -16,7 +16,7 @@ type statsRowAny struct {
}
func (sa *statsRowAny) String() string {
return "row_any(" + fieldNamesString(sa.fieldFilters) + ")"
return "row_any(" + fieldFiltersString(sa.fieldFilters) + ")"
}
func (sa *statsRowAny) updateNeededFields(pf *prefixfilter.Filter) {

View File

@@ -21,7 +21,7 @@ type statsRowMax struct {
func (sm *statsRowMax) String() string {
s := "row_max(" + quoteTokenIfNeeded(sm.srcField)
if !prefixfilter.MatchAll(sm.fieldFilters) {
s += ", " + fieldNamesString(sm.fieldFilters)
s += ", " + fieldFiltersString(sm.fieldFilters)
}
s += ")"
return s

View File

@@ -21,7 +21,7 @@ type statsRowMin struct {
func (sm *statsRowMin) String() string {
s := "row_min(" + quoteTokenIfNeeded(sm.srcField)
if !prefixfilter.MatchAll(sm.fieldFilters) {
s += ", " + fieldNamesString(sm.fieldFilters)
s += ", " + fieldFiltersString(sm.fieldFilters)
}
s += ")"
return s

View File

@@ -15,7 +15,7 @@ type statsStddev struct {
}
func (ss *statsStddev) String() string {
return "stddev(" + fieldNamesString(ss.fieldFilters) + ")"
return "stddev(" + fieldFiltersString(ss.fieldFilters) + ")"
}
func (ss *statsStddev) updateNeededFields(pf *prefixfilter.Filter) {

View File

@@ -15,7 +15,7 @@ type statsSum struct {
}
func (ss *statsSum) String() string {
return "sum(" + fieldNamesString(ss.fieldFilters) + ")"
return "sum(" + fieldFiltersString(ss.fieldFilters) + ")"
}
func (ss *statsSum) updateNeededFields(pf *prefixfilter.Filter) {

Some files were not shown because too many files have changed in this diff Show More