Compare commits

..

4 Commits

Author SHA1 Message Date
Fred Navruzov
3809c54af4 docs/vmanomaly: improve navigation and diagrams 2026-07-27 21:22:12 +02:00
Artem Fetishev
fae6656aa3 lib/storage: rewrite indexdb tests (#11302)
This PR rewrites the `TestSearchTSIDWithTimeRange`
`TestSearchLabelValues` tests:

- `TestSearchTSIDWithTimeRange` is split into several tests:
`TestIndexDBSearchTSIDs`, `TestIndexDBSearchLabelNames`,
`TestIndexDBSearchLabelValues`, `TestIndexDBGetTSDBStatus`,
`TestIndexDBDeleteSeries`.
- Previous `TestSearchLabelValues` was merged into
`TestIndexDBSearchLabelValues`
- All tests support disabling per-day index and can be easily extended
to support global index disabling ( related to #11196).

---------

Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
2026-07-27 15:11:11 +02:00
Max Kotliar
2ca332b332 app/vmui: sync ui with vmanget recent changes (#11301)
In https://github.com/VictoriaMetrics/VictoriaMetrics/pull/10607 vmagent
metrics relabling debug UI has been refined. There are a [few
changes](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/10607#issuecomment-4956698085)
that make sense in VMUI too:
- Rename "Labels" to "A time series" to highlight that a single
Prometheus time series can be provided in text format.
- Rename "Relabel Configs" to "Config". It should be clear from the page
contex what config is meant. Also there are links to documention.
- Add placeholder. It hints what could be placed into "A time series"
textarea.
- Add link to Relabel Stages. Same as in vmagent.
2026-07-27 15:55:35 +03:00
Zasda Yusuf Mikail
07b6070193 app/vmselect/promql: drop data points with missing samples on the right side of comparison operations
NaN values in non-scalar operands of comparison operations mean missing
samples, so they cannot match any sample on the other side. Previously
such data points were passed to the comparison function, so queries like
`foo != (bar > 100)` could return unexpected results, since `value !=
NaN` evaluates to `true`. This PR drops such data points in the same way
as Prometheus does.

Comparisons to scalars such as `foo != nan` keep the previous behavior,
since a scalar `NaN` is an explicit value rather than a missing sample
(see the existing `compare_to_nan_right` test case).

The check is applied only to the right side of the operation, since a
missing sample on the left side already results in `NaN` for every
comparison operation:
- `==`, `>`, `<`, `>=`, `<=`: `cf(NaN, b)` is `false`, so the data point
is dropped;
- `!=`: `cf(NaN, b)` is `true`, but the returned value is the left one,
which is `NaN`;
- `bool` modifier: the left `NaN` is handled explicitly in
`newBinaryOpCmpFunc`.

Verified with the reproduction steps from the linked issue: the query
`tsafdb0_value{table="t1", timeSeries="c0"} != ((272) <=
(tsafdb0_value{table="t1", timeSeries="c0"}))` now returns an empty
result, matching Prometheus behavior.

Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/10018
2026-07-24 15:59:45 +02:00
33 changed files with 1959 additions and 699 deletions

View File

@@ -18,7 +18,7 @@ groups:
concurrency: 2
rules:
- alert: RequestErrorsToAPI
expr: increase(vm_http_request_errors_total{path=~".+"}[5m]) > 0
expr: increase(vm_http_request_errors_total[5m]) > 0
for: 15m
labels:
severity: warning

View File

@@ -164,11 +164,11 @@ func newBinaryOpFunc(bf func(left, right float64, isBool bool) float64) binaryOp
left := bfa.left
right := bfa.right
op := bfa.be.Op
switch true {
case metricsql.IsBinaryOpCmp(op):
// Do not remove empty series for comparison operations,
// since this may lead to missing result.
default:
isCmpOp := metricsql.IsBinaryOpCmp(op)
if !isCmpOp {
// Do not remove empty series for comparison operations, since NaN can be an
// explicitly present sample value. In particular, `value != NaN` is true.
// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/150.
left = removeEmptySeries(left)
right = removeEmptySeries(right)
}
@@ -191,6 +191,14 @@ func newBinaryOpFunc(bf func(left, right float64, isBool bool) float64) binaryOp
isBool := bfa.be.Bool
fillLeft := bfa.be.FillLeft
fillRight := bfa.be.FillRight
// A NaN produced by a vector comparison denotes a filtered-out sample rather
// than an explicitly present NaN value. Drop it when this result is used as
// the right-hand side of another comparison.
// Note that filtered-out samples surviving as NaN inside other wrappers such as
// transform functions or arithmetic operations aren't detected, since such NaN
// is indistinguishable from an explicitly present NaN value there.
// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/10018.
dropNaNRight := isCmpOp && isVectorComparisonExpr(bfa.be.Right)
for i, tsLeft := range left {
leftValues := tsLeft.Values
rightValues := right[i].Values
@@ -203,11 +211,16 @@ func newBinaryOpFunc(bf func(left, right float64, isBool bool) float64) binaryOp
b := rightValues[j]
leftIsNaN := math.IsNaN(a)
rightIsNaN := math.IsNaN(b)
// apply the fill value when either the left or right side is NaN, but not both.
// Both sides are NaN.
if leftIsNaN && rightIsNaN {
dstValues[j] = bf(a, b, isBool)
continue
}
if dropNaNRight && rightIsNaN && fillRight == nil {
dstValues[j] = nan
continue
}
// Apply the fill value when either the left or right side is NaN, but not both.
if leftIsNaN && fillLeft != nil {
a = fillLeft.N
}
@@ -223,6 +236,38 @@ func newBinaryOpFunc(bf func(left, right float64, isBool bool) float64) binaryOp
}
}
// isVectorComparisonExpr returns whether e is a comparison operation
// with at least one non-scalar operand.
func isVectorComparisonExpr(e metricsql.Expr) bool {
if re, ok := e.(*metricsql.RollupExpr); ok && re.Window == nil {
// Unwrap `(...) offset <d>` and `(...) @ <t>`, which do not change
// the shape of the result. Subqueries are left as is, since rollup
// functions skip NaN values on their own.
e = re.Expr
}
be, ok := e.(*metricsql.BinaryOpExpr)
if !ok || !metricsql.IsBinaryOpCmp(be.Op) {
return false
}
return !isScalarLikeExpr(be.Left) || !isScalarLikeExpr(be.Right)
}
// isScalarLikeExpr returns whether e always evaluates to a scalar value.
func isScalarLikeExpr(e metricsql.Expr) bool {
switch e := e.(type) {
case *metricsql.NumberExpr, *metricsql.DurationExpr:
return true
case *metricsql.BinaryOpExpr:
return isScalarLikeExpr(e.Left) && isScalarLikeExpr(e.Right)
case *metricsql.FuncExpr:
switch strings.ToLower(e.Name) {
case "now", "pi", "scalar", "start", "end", "step", "time", "timezone_offset":
return true
}
}
return false
}
func adjustBinaryOpTags(be *metricsql.BinaryOpExpr, left, right []*timeseries) ([]*timeseries, []*timeseries, []*timeseries, error) {
if len(be.GroupModifier.Op) == 0 && len(be.JoinModifier.Op) == 0 {
if isScalar(left) {

View File

@@ -2994,6 +2994,136 @@ func TestExecSuccess(t *testing.T) {
resultExpected := []netstorage.Result{}
f(q, resultExpected)
})
t.Run(`compare_to_nan_left_vector_right_scalar`, func(t *testing.T) {
t.Parallel()
q := `label_set(time(), "foo", "bar") != NaN`
r := netstorage.Result{
MetricName: metricNameExpected,
Values: []float64{1000, 1200, 1400, 1600, 1800, 2000},
Timestamps: timestampsExpected,
}
r.MetricName.Tags = []storage.Tag{{
Key: []byte("foo"),
Value: []byte("bar"),
}}
resultExpected := []netstorage.Result{r}
f(q, resultExpected)
})
t.Run(`compare_to_non_nan_scalar_right`, func(t *testing.T) {
t.Parallel()
q := `label_set(time(), "foo", "bar") != 1200`
r := netstorage.Result{
MetricName: metricNameExpected,
Values: []float64{1000, nan, 1400, 1600, 1800, 2000},
Timestamps: timestampsExpected,
}
r.MetricName.Tags = []storage.Tag{{
Key: []byte("foo"),
Value: []byte("bar"),
}}
resultExpected := []netstorage.Result{r}
f(q, resultExpected)
})
t.Run(`compare_to_nan_vector_right`, func(t *testing.T) {
t.Parallel()
q := `label_set(time(), "foo", "bar") != label_set(NaN, "foo", "bar")`
r := netstorage.Result{
MetricName: metricNameExpected,
Values: []float64{1000, 1200, 1400, 1600, 1800, 2000},
Timestamps: timestampsExpected,
}
r.MetricName.Tags = []storage.Tag{{
Key: []byte("foo"),
Value: []byte("bar"),
}}
resultExpected := []netstorage.Result{r}
f(q, resultExpected)
})
t.Run(`compare_to_nan_scalar_comparison_right`, func(t *testing.T) {
t.Parallel()
q := `label_set(time(), "foo", "bar") != (1 > 2)`
r := netstorage.Result{
MetricName: metricNameExpected,
Values: []float64{1000, 1200, 1400, 1600, 1800, 2000},
Timestamps: timestampsExpected,
}
r.MetricName.Tags = []storage.Tag{{
Key: []byte("foo"),
Value: []byte("bar"),
}}
resultExpected := []netstorage.Result{r}
f(q, resultExpected)
})
t.Run(`compare_to_empty_vector_right`, func(t *testing.T) {
t.Parallel()
q := `label_set(time(), "foo", "bar") != (label_set(time(), "foo", "bar") > 100000)`
resultExpected := []netstorage.Result{}
f(q, resultExpected)
})
t.Run(`compare_to_empty_vector_right_offset`, func(t *testing.T) {
t.Parallel()
q := `label_set(time(), "foo", "bar") != ((label_set(time(), "foo", "bar") > 100000) offset 0s)`
resultExpected := []netstorage.Result{}
f(q, resultExpected)
})
t.Run(`compare_to_empty_vector_left`, func(t *testing.T) {
// A missing sample on the left side results in NaN for every comparison
// operation, so no special handling is needed there.
t.Parallel()
q := `(label_set(time(), "foo", "bar") > 100000) != label_set(time(), "foo", "bar")`
resultExpected := []netstorage.Result{}
f(q, resultExpected)
})
t.Run(`compare_to_empty_series_right_bool`, func(t *testing.T) {
t.Parallel()
q := `label_set(time(), "foo", "bar") == bool (label_set(time(), "foo", "bar") > 100000)`
resultExpected := []netstorage.Result{}
f(q, resultExpected)
})
t.Run(`compare_to_partially_empty_series_right`, func(t *testing.T) {
// Prometheus drops the timestamps filtered out by the comparison on the right.
// See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/11100#issuecomment-4933952390.
t.Parallel()
q := `label_set(time(), "foo", "bar") != (label_set(time(), "foo", "bar") * 2 > 2800)`
r := netstorage.Result{
MetricName: metricNameExpected,
Values: []float64{nan, nan, nan, 1600, 1800, 2000},
Timestamps: timestampsExpected,
}
r.MetricName.Tags = []storage.Tag{{
Key: []byte("foo"),
Value: []byte("bar"),
}}
resultExpected := []netstorage.Result{r}
f(q, resultExpected)
})
t.Run(`compare_to_empty_unlabeled_vector_right`, func(t *testing.T) {
t.Parallel()
q := `sum(label_set(time(), "foo", "bar")) != (sum(label_set(time(), "foo", "bar")) > 100000)`
resultExpected := []netstorage.Result{}
f(q, resultExpected)
})
t.Run(`compare_to_empty_series_right_with_fill_left`, func(t *testing.T) {
t.Parallel()
q := `label_set(time(), "foo", "bar") != fill_left(0) (label_set(time(), "foo", "bar") > 100000)`
resultExpected := []netstorage.Result{}
f(q, resultExpected)
})
t.Run(`compare_to_empty_series_right_with_fill_right`, func(t *testing.T) {
t.Parallel()
q := `label_set(time(), "foo", "bar") != fill_right(0) (label_set(time(), "foo", "bar") > 100000)`
r := netstorage.Result{
MetricName: metricNameExpected,
Values: []float64{1000, 1200, 1400, 1600, 1800, 2000},
Timestamps: timestampsExpected,
}
r.MetricName.Tags = []storage.Tag{{
Key: []byte("foo"),
Value: []byte("bar"),
}}
resultExpected := []netstorage.Result{r}
f(q, resultExpected)
})
t.Run(`-1 < 2`, func(t *testing.T) {
t.Parallel()
q := `-1 < 2`

View File

@@ -72,7 +72,7 @@ const Relabel: FC = () => {
<div className="vm-relabeling-header-configs">
<TextField
type="textarea"
label="Relabel configs"
label="Config"
value={config}
autofocus
onChange={handleChangeConfig}
@@ -82,8 +82,9 @@ const Relabel: FC = () => {
<div className="vm-relabeling-header__labels">
<TextField
type="textarea"
label="Labels"
label="A time series"
value={labels}
placeholder="up{job=&quot;job_name&quot;,instance=&quot;host:port&quot;}"
onChange={handleChangeLabels}
onEnter={handleRunQuery}
/>
@@ -97,22 +98,22 @@ const Relabel: FC = () => {
<Button
variant="text"
color="gray"
startIcon={<InfoIcon/>}
startIcon={<WikiIcon/>}
>
Relabeling cookbook
</Button>
</a>
<a
target="_blank"
href="https://docs.victoriametrics.com/victoriametrics/relabeling/"
href="https://docs.victoriametrics.com/victoriametrics/relabeling/#relabeling-stages"
rel="help noreferrer"
>
<Button
variant="text"
color="gray"
startIcon={<WikiIcon/>}
startIcon={<InfoIcon/>}
>
Documentation
Relabeling Stages
</Button>
</a>
<Button

View File

@@ -75,7 +75,7 @@ groups:
Consider to limit the ingestion rate, decrease retention or scale the disk space if possible."
- alert: RequestErrorsToAPI
expr: increase(vm_http_request_errors_total{path=~".+", path!="*"}[5m]) > 0
expr: increase(vm_http_request_errors_total[5m]) > 0
for: 15m
labels:
severity: warning
@@ -86,18 +86,6 @@ groups:
description: "Requests to path {{ $labels.path }} are receiving errors.
Please verify if clients are sending correct requests."
- alert: RequestErrorsToUnknownPaths
expr: sum(increase(vm_http_request_errors_total{path="*"}[5m])) by(job, instance, reason) > 0
for: 15m
labels:
severity: warning
show_at: dashboard
annotations:
dashboard: "{{ $externalURL }}/d/oS7Bi_0Wz?viewPanel=52&var-instance={{ $labels.instance }}"
summary: "Too many errors served for {{ $labels.job }} with reason {{ $labels.reason }} (instance {{ $labels.instance }})"
description: "Requests are failing with reason {{ $labels.reason }}.
Please verify if clients are sending correct requests."
- alert: RPCErrors
expr: |
(

View File

@@ -75,7 +75,7 @@ groups:
Consider to limit the ingestion rate, decrease retention or scale the disk space if possible."
- alert: RequestErrorsToAPI
expr: increase(vm_http_request_errors_total{path=~".+"}[5m]) > 0
expr: increase(vm_http_request_errors_total[5m]) > 0
for: 15m
labels:
severity: warning

View File

@@ -118,27 +118,6 @@ To visualize and interact with both [self-monitoring metrics](https://docs.victo
- {{% available_from "v1.26.0" anomaly %}} For rapid exploration of how different models, their configurations and included domain knowledge impacts the results of anomaly detection, use the built-in [vmanomaly UI](https://docs.victoriametrics.com/anomaly-detection/ui/).
![vmanomaly-ui-overview](vmanomaly-ui-overview.webp)
## Is vmanomaly stateful?
By default, `vmanomaly` is **stateless**, meaning it does not retain any state between service restarts. However, it can be configured {{% available_from "v1.24.0" anomaly %}} to be **stateful** by enabling the `restore_state` setting in the [settings section](https://docs.victoriametrics.com/anomaly-detection/components/settings/). This allows the service to restore its state from a previous run (training data, trained models), ensuring that models continue to produce [anomaly scores](#what-is-anomaly-score) right after restart and without requiring a full retraining process or re-querying training data from VictoriaMetrics. This is particularly useful for long-running services that need to maintain continuity in anomaly detection without losing previously learned patterns, especially when using [online models](https://docs.victoriametrics.com/anomaly-detection/components/models/#online-models) that continuously adapt to new data and update their internal state. Also, [hot-reloading](https://docs.victoriametrics.com/anomaly-detection/components/#hot-reload) works well with state restoration, allowing for on-the-fly configuration changes without losing the current state of the models and reusing unchanged models/data/scheduler combinations.
Please refer to the [state restoration section](https://docs.victoriametrics.com/anomaly-detection/components/settings/#state-restoration) for more details on how it works and how to configure it.
## Config hot-reloading
`vmanomaly` supports [hot reload](https://docs.victoriametrics.com/anomaly-detection/components/#hot-reload) {{% available_from "v1.25.0" anomaly %}} to apply configuration-file changes automatically. Enable it with the `--watch` [CLI argument](https://docs.victoriametrics.com/anomaly-detection/quickstart/#command-line-arguments) to update the service without an explicit restart.
## Environment variables
`vmanomaly` supports {{% available_from "v1.25.0" anomaly %}} an option to reference environment variables in [configuration files](https://docs.victoriametrics.com/anomaly-detection/components/) using scalar string placeholders `%{ENV_NAME}`. This feature is particularly useful for managing sensitive information like API keys or database credentials while still making it accessible to the service. Please refer to the [environment variables section](https://docs.victoriametrics.com/anomaly-detection/components/#environment-variables) for more details and examples.
## Deploying vmanomaly
`vmanomaly` can be deployed in various environments, including Docker, Kubernetes, and VM Operator. For detailed deployment instructions, refer to the [QuickStart section](https://docs.victoriametrics.com/anomaly-detection/quickstart/#how-to-install-and-run-vmanomaly).
## Migration
For information on migrating between different versions of `vmanomaly`, please refer to the [Migration section](https://docs.victoriametrics.com/anomaly-detection/migration/) for compatibility considerations and steps for a smooth transition.
## Choosing the right model for vmanomaly
Selecting the best model for `vmanomaly` depends on the data's nature and the [types of anomalies](https://victoriametrics.com/blog/victoriametrics-anomaly-detection-handbook-chapter-2/#categories-of-anomalies) to detect:
@@ -155,16 +134,6 @@ Please refer to [respective blogpost on anomaly types and alerting heuristics](h
Still not 100% sure what to use? We are [here to help](https://docs.victoriametrics.com/anomaly-detection/#get-in-touch).
## Can AI help configure vmanomaly?
Yes. The available tools serve different workflows:
- [UI Copilot](https://docs.victoriametrics.com/anomaly-detection/ui/#ai-assistance) provides interactive guidance and can apply query, model, and alerting changes in the UI.
- The [vmanomaly MCP server](https://docs.victoriametrics.com/ai-tools/#vmanomaly-mcp-server) gives compatible AI clients access to live schemas, documentation, time-series characteristics, configuration validation, and autotune tasks.
- [Agent skills](https://docs.victoriametrics.com/ai-tools/#agent-skills) provide repeatable workflows for investigating data and generating or reviewing `vmanomaly` and `vmalert` configurations.
Treat AI-generated configuration as a proposal. Review it and validate it through the UI or with [`--dryRun`](https://docs.victoriametrics.com/anomaly-detection/quickstart/#command-line-arguments) before deployment.
## Incorporating domain knowledge
Anomaly detection models can significantly improve when incorporating business-specific assumptions about the data and what constitutes an anomaly. `vmanomaly` supports various [business-side configuration parameters](https://docs.victoriametrics.com/anomaly-detection/components/models/#common-args) across all built-in models to **reduce [false positives](https://victoriametrics.com/blog/victoriametrics-anomaly-detection-handbook-chapter-1/#false-positive)** and **align model behavior with business needs**, for example:
@@ -222,39 +191,15 @@ models:
provide_series: ['anomaly_score', 'y', 'yhat', 'yhat_lower', 'yhat_upper']
```
## Alert generation in vmanomaly
While `vmanomaly` detects anomalies and produces scores, it *does not directly generate alerts*. The anomaly scores are written back to VictoriaMetrics, where respective alerting tool, like [`vmalert`](https://docs.victoriametrics.com/victoriametrics/vmalert/), can be used to create alerts based on these scores for integrating it with your alerting management system. See an example diagram of how `vmanomaly` integrates into observability pipeline for anomaly detection on `node_exporter` metrics:
## Can AI help configure vmanomaly?
<img src="https://docs.victoriametrics.com/anomaly-detection/guides/guide-vmanomaly-vmalert/guide-vmanomaly-vmalert_overview.webp" alt="node_exporter_example_diagram" style="width:60%"/>
Yes. The available tools serve different workflows:
Once anomaly scores are written back to VictoriaMetrics, you can use [MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/) expressions in `vmalert` to define alerting rules based on these scores. Reasonable defaults are based around default threshold of `anomaly_score > 1`:
- [UI Copilot](https://docs.victoriametrics.com/anomaly-detection/ui/#ai-assistance) provides interactive guidance and can apply query, model, and alerting changes in the UI.
- The [vmanomaly MCP server](https://docs.victoriametrics.com/ai-tools/#vmanomaly-mcp-server) gives compatible AI clients access to live schemas, documentation, time-series characteristics, configuration validation, and autotune tasks.
- [Agent skills](https://docs.victoriametrics.com/ai-tools/#agent-skills) provide repeatable workflows for investigating data and generating or reviewing `vmanomaly` and `vmalert` configurations.
```yaml
groups:
- name: VMAnomalyAlerts
interval: 60s
rules:
- alert: HighAnomalyScore
expr: min(anomaly_score) without (model_alias, scheduler_alias) >= 1
for: 5m # adjust to your needs based on data frequency and alerting policies
labels:
severity: warning
query_alias: explore
model_alias: default
scheduler_alias: periodic
preset: ui
annotations:
summary: High anomaly score detected.
description: Anomaly score exceeded threshold ({{ $value }}) for more than
{{ $for }} for query {{ $labels.for }}.
```
> {{% available_from "v1.27.0" anomaly %}} You can also use the [vmanomaly UI](https://docs.victoriametrics.com/anomaly-detection/ui/) to generate alerting rules automatically based on your model configurations and selected thresholds.
> {{% available_from "v1.28.3" anomaly %}} Check out our [MCP Server](https://github.com/VictoriaMetrics/mcp-vmanomaly) to get AI-assisted recommendations on setting up alerting rules based on produced anomaly scores. See [installation guide](https://github.com/VictoriaMetrics/mcp-vmanomaly#installation) for more details.
## Preventing alert fatigue
Produced anomaly scores are designed in such a way that values from 0.0 to 1.0 indicate non-anomalous data, while a value greater than 1.0 is generally classified as an anomaly. However, there are no perfect models for anomaly detection, that's why reasonable defaults expressions like `anomaly_score > 1` may not work 100% of the time. However, anomaly scores, produced by `vmanomaly` are written back as metrics to VictoriaMetrics, where tools like [`vmalert`](https://docs.victoriametrics.com/victoriametrics/vmalert/) can use [MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/) expressions to fine-tune alerting thresholds and conditions, balancing between avoiding [false negatives](https://victoriametrics.com/blog/victoriametrics-anomaly-detection-handbook-chapter-1/#false-negative) and reducing [false positives](https://victoriametrics.com/blog/victoriametrics-anomaly-detection-handbook-chapter-1/#false-positive).
Treat AI-generated configuration as a proposal. Review it and validate it through the UI or with [`--dryRun`](https://docs.victoriametrics.com/anomaly-detection/quickstart/#command-line-arguments) before deployment.
## How to backtest particular configuration on historical data?
@@ -417,6 +362,61 @@ groups:
description: "Disk usage is forecasted to exceed 95% in the next 3 days for instance {{ $labels.instance }}. Forecasted value: {{ $value }}."
```
## Alert generation in vmanomaly
While `vmanomaly` detects anomalies and produces scores, it *does not directly generate alerts*. The anomaly scores are written back to VictoriaMetrics, where respective alerting tool, like [`vmalert`](https://docs.victoriametrics.com/victoriametrics/vmalert/), can be used to create alerts based on these scores for integrating it with your alerting management system. See an example diagram of how `vmanomaly` integrates into observability pipeline for anomaly detection on `node_exporter` metrics:
<img src="/anomaly-detection/guides/guide-vmanomaly-vmalert/guide-vmanomaly-vmalert_overview.svg" alt="Typical vmanomaly observability pipeline using node-exporter, vmagent, VictoriaMetrics, Grafana, vmalert, and Alertmanager" style="width:60%"/>
Once anomaly scores are written back to VictoriaMetrics, you can use [MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/) expressions in `vmalert` to define alerting rules based on these scores. Reasonable defaults are based around default threshold of `anomaly_score > 1`:
```yaml
groups:
- name: VMAnomalyAlerts
interval: 60s
rules:
- alert: HighAnomalyScore
expr: min(anomaly_score) without (model_alias, scheduler_alias) >= 1
for: 5m # adjust to your needs based on data frequency and alerting policies
labels:
severity: warning
query_alias: explore
model_alias: default
scheduler_alias: periodic
preset: ui
annotations:
summary: High anomaly score detected.
description: Anomaly score exceeded threshold ({{ $value }}) for more than
{{ $for }} for query {{ $labels.for }}.
```
> {{% available_from "v1.27.0" anomaly %}} You can also use the [vmanomaly UI](https://docs.victoriametrics.com/anomaly-detection/ui/) to generate alerting rules automatically based on your model configurations and selected thresholds.
> {{% available_from "v1.28.3" anomaly %}} Check out our [MCP Server](https://github.com/VictoriaMetrics/mcp-vmanomaly) to get AI-assisted recommendations on setting up alerting rules based on produced anomaly scores. See [installation guide](https://github.com/VictoriaMetrics/mcp-vmanomaly#installation) for more details.
## Preventing alert fatigue
Produced anomaly scores are designed in such a way that values from 0.0 to 1.0 indicate non-anomalous data, while a value greater than 1.0 is generally classified as an anomaly. However, there are no perfect models for anomaly detection, that's why reasonable defaults expressions like `anomaly_score > 1` may not work 100% of the time. However, anomaly scores, produced by `vmanomaly` are written back as metrics to VictoriaMetrics, where tools like [`vmalert`](https://docs.victoriametrics.com/victoriametrics/vmalert/) can use [MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/) expressions to fine-tune alerting thresholds and conditions, balancing between avoiding [false negatives](https://victoriametrics.com/blog/victoriametrics-anomaly-detection-handbook-chapter-1/#false-negative) and reducing [false positives](https://victoriametrics.com/blog/victoriametrics-anomaly-detection-handbook-chapter-1/#false-positive).
## Deploying vmanomaly
`vmanomaly` can be deployed in various environments, including Docker, Kubernetes, and VM Operator. For detailed deployment instructions, refer to the [QuickStart section](https://docs.victoriametrics.com/anomaly-detection/quickstart/#how-to-install-and-run-vmanomaly).
## Environment variables
`vmanomaly` supports {{% available_from "v1.25.0" anomaly %}} an option to reference environment variables in [configuration files](https://docs.victoriametrics.com/anomaly-detection/components/) using scalar string placeholders `%{ENV_NAME}`. This feature is particularly useful for managing sensitive information like API keys or database credentials while still making it accessible to the service. Please refer to the [environment variables section](https://docs.victoriametrics.com/anomaly-detection/components/#environment-variables) for more details and examples.
## Is vmanomaly stateful?
By default, `vmanomaly` is **stateless**, meaning it does not retain any state between service restarts. However, it can be configured {{% available_from "v1.24.0" anomaly %}} to be **stateful** by enabling the `restore_state` setting in the [settings section](https://docs.victoriametrics.com/anomaly-detection/components/settings/). This allows the service to restore its state from a previous run (training data, trained models), ensuring that models continue to produce [anomaly scores](#what-is-anomaly-score) right after restart and without requiring a full retraining process or re-querying training data from VictoriaMetrics. This is particularly useful for long-running services that need to maintain continuity in anomaly detection without losing previously learned patterns, especially when using [online models](https://docs.victoriametrics.com/anomaly-detection/components/models/#online-models) that continuously adapt to new data and update their internal state. Also, [hot-reloading](https://docs.victoriametrics.com/anomaly-detection/components/#hot-reload) works well with state restoration, allowing for on-the-fly configuration changes without losing the current state of the models and reusing unchanged models/data/scheduler combinations.
Please refer to the [state restoration section](https://docs.victoriametrics.com/anomaly-detection/components/settings/#state-restoration) for more details on how it works and how to configure it.
## Config hot-reloading
`vmanomaly` supports [hot reload](https://docs.victoriametrics.com/anomaly-detection/components/#hot-reload) {{% available_from "v1.25.0" anomaly %}} to apply configuration-file changes automatically. Enable it with the `--watch` [CLI argument](https://docs.victoriametrics.com/anomaly-detection/quickstart/#command-line-arguments) to update the service without an explicit restart.
## Migration
For information on migrating between different versions of `vmanomaly`, please refer to the [Migration section](https://docs.victoriametrics.com/anomaly-detection/migration/) for compatibility considerations and steps for a smooth transition.
## Resource consumption of vmanomaly
`vmanomaly` itself is a lightweight service, resource usage is primarily dependent on [scheduling](https://docs.victoriametrics.com/anomaly-detection/components/scheduler/) (how often and on what data to fit/infer your models), [# and size of timeseries returned by your queries](https://docs.victoriametrics.com/anomaly-detection/components/reader/#vm-reader), and the complexity of the employed [models](https://docs.victoriametrics.com/anomaly-detection/components/models/). Its resource usage is directly related to these factors, making it adaptable to various operational scales. Various optimizations are available to balance between RAM usage, processing speed, and model capacity. These options are described in the sections below.

View File

@@ -27,7 +27,7 @@ Key functions:
The diagram below illustrates how `vmanomaly` fits into an observability setup, such as detecting anomalies in metrics collected by `node_exporter`:
<img src="https://docs.victoriametrics.com/anomaly-detection/guides/guide-vmanomaly-vmalert/guide-vmanomaly-vmalert_overview.webp" alt="node_exporter_example_diagram" style="width:60%"/>
<img src="/anomaly-detection/guides/guide-vmanomaly-vmalert/guide-vmanomaly-vmalert_overview.svg" alt="Typical vmanomaly observability pipeline using node-exporter, vmagent, VictoriaMetrics, Grafana, vmalert, and Alertmanager" style="width:60%"/>
## How does it work?
@@ -40,7 +40,7 @@ VictoriaMetrics Anomaly Detection **continuously re-fit and apply machine learni
- **Confidence intervals** (`[yhat_lower, yhat_upper]`)
These outputs integrate seamlessly into downstream applications, making it easier to **visually inspect anomalies**, e.g. in respective [Grafana dashboards](https://docs.victoriametrics.com/anomaly-detection/presets/#grafana-dashboard).
<img src="https://docs.victoriametrics.com/anomaly-detection/components/vmanomaly-components.webp" alt="node_exporter_example_diagram" style="width:80%"/>
{{% content "components/vmanomaly-components-diagram.md" %}}
## Key benefits

View File

@@ -78,9 +78,7 @@ These [sub-configurations](#sub-configuration) can be assigned to a specific sha
Additionally, a replication factor `R ≥ 1` ensures [high availability](#high-availability) by enforcing redundancy across shards.
<p></p>
![vmanomaly-sharding-ha-diagram](vmanomaly-sharding-ha-diagram.webp)
{{% content "vmanomaly-sharding-ha-diagram.md" %}}
> Please [refer to deployment options section](#deployment-options) for the examples (Docker, Docker Compose, Helm). To avoid duplicate metrics being reported from each vmanomaly service used in sharded mode, make sure that [deduplication](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#deduplication) is configured on vmsingle or vmselect and vmstorage for the VictoriaMetrics instance used in the [writer section of the configuration](https://docs.victoriametrics.com/anomaly-detection/components/writer/).
@@ -130,9 +128,7 @@ Similar to other VictoriaMetrics ecosystem components, like [VMAgent](https://do
When `VMANOMALY_REPLICATION_FACTOR` > 1, each [sub-config](#sub-configuration) `n` from `{0, N-1}` is assigned to exactly `R` nodes. This ensures redundancy, preventing single-node failures from causing data loss.
<p></p>
![vmanomaly-sharding-ha-diagram](vmanomaly-sharding-ha-diagram.webp)
{{% content "vmanomaly-sharding-ha-diagram.md" %}}
> Please [refer to deployment options section](#deployment-options) for the examples (Docker, Docker Compose, Helm). To avoid duplicate metrics being reported from each vmanomaly service used in sharded mode, make sure that [deduplication](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#deduplication) is configured on vmsingle or vmselect and vmstorage for the VictoriaMetrics instance used in the [writer section of the configuration](https://docs.victoriametrics.com/anomaly-detection/components/writer/).

View File

@@ -28,7 +28,7 @@ Below, you will find an example illustrating how the components of `vmanomaly` i
> [Reader](https://docs.victoriametrics.com/anomaly-detection/components/reader/#vm-reader) and [Writer](https://docs.victoriametrics.com/anomaly-detection/components/writer/#vm-writer) also support [multitenancy](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#multitenancy), so you can read/write from/to different locations - see `tenant_id` param description.
![vmanomaly-components](vmanomaly-components.webp)
{{% content "vmanomaly-components-diagram.md" %}}
## Example config

View File

@@ -0,0 +1,68 @@
<?xml version="1.0" encoding="UTF-8"?>
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 968 420" role="img" aria-labelledby="title description">
<title id="title">AutoTunedModel tuning and inference lifecycle</title>
<desc id="description">The tuning process tests and scores model candidates across n time-series splits until the trial count or timeout is reached. Each fold fits a candidate on training data and predicts anomalies on its validation segment. The best model is then used on inference data until the next fit call.</desc>
<defs>
<marker id="arrow" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
<path d="M0 0 10 5 0 10Z" fill="#202124"/>
</marker>
<style>
text { font-family: Arial, Helvetica, sans-serif; fill: #202124; font-size: 18px; }
.line { fill: none; stroke: #202124; stroke-width: 2.5; }
.arrow { fill: none; stroke: #202124; stroke-width: 2.5; marker-end: url(#arrow); }
.dash { fill: none; stroke: #202124; stroke-width: 2.5; stroke-dasharray: 8 9; }
.box { fill: #fff; stroke: #202124; stroke-width: 1.5; }
</style>
</defs>
<rect width="968" height="420" fill="#fff"/>
<rect x="112" y="10" width="242" height="83" class="box"/>
<text x="233" y="38" text-anchor="middle">
<tspan x="233" dy="0">test and score candidates</tspan>
<tspan x="233" dy="21">until `n_trials` or `timeout`</tspan>
<tspan x="233" dy="21">is reached</tspan>
</text>
<path d="M354 49 V67" class="arrow"/>
<text x="355" y="90" text-anchor="middle">Tuning</text>
<text x="355" y="112" text-anchor="middle">process</text>
<path d="M162 157 V136 H549 V157" class="line"/>
<path d="M355 112 V136" class="line"/>
<path d="M399 93 H653 V152" class="line"/>
<path d="M113 161 H75 V350 H113" class="line"/>
<path d="M75 236 H39" class="line"/>
<text x="28" y="250" transform="rotate(-90 28 250)" text-anchor="middle">n splits</text>
<text x="102" y="227">Fold 1</text>
<text x="102" y="255">Fold 2</text>
<text x="102" y="283">Fold 3</text>
<text x="120" y="327">...</text>
<text x="260" y="179" text-anchor="middle">Fit</text>
<text x="260" y="202" text-anchor="middle">candidate</text>
<text x="413" y="179" text-anchor="middle">Predict</text>
<text x="413" y="202" text-anchor="middle">anomalies</text>
<path d="M173 221 H354 V207" class="line"/>
<path d="M232 249 H412 V235" class="line"/>
<path d="M293 280 H472 V265" class="line"/>
<path d="M354 221 H412" class="dash"/>
<path d="M412 249 H474" class="dash"/>
<path d="M472 280 H513" class="dash"/>
<path d="M548 144 V415" class="dash" style="stroke-width:1.5;stroke-dasharray:4 6"/>
<rect x="577" y="152" width="136" height="61" rx="12" class="box"/>
<text x="645" y="187" text-anchor="middle">best model</text>
<path d="M653 213 V348" class="arrow"/>
<text x="815" y="168" text-anchor="middle">
<tspan x="815" dy="0">used to predict</tspan>
<tspan x="815" dy="21">on inference data</tspan>
<tspan x="815" dy="21">until the next `fit` call</tspan>
</text>
<path d="M40 350 H895" class="arrow"/>
<text x="455" y="402" text-anchor="middle">Training data</text>
<text x="623" y="402" text-anchor="middle">Inference data</text>
<text x="856" y="402">Time axis, t</text>
</svg>

After

Width:  |  Height:  |  Size: 3.3 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 15 KiB

View File

@@ -0,0 +1,139 @@
<?xml version="1.0" encoding="UTF-8"?>
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1920 1295" role="img" aria-labelledby="title description">
<title id="title">Multivariate models lifecycle</title>
<desc id="description">MetricsQL queries retrieve an aligned set of series from VictoriaMetrics. Reader data fits one shared multivariate model. The exact same series set produces one anomaly score series with the intersected label set; inference is skipped when the fit and inference series sets differ. Writer stores produced scores in VictoriaMetrics.</desc>
<defs>
<marker id="arrow" viewBox="0 0 12 12" refX="10.5" refY="6" markerWidth="16" markerHeight="16" markerUnits="userSpaceOnUse" orient="auto-start-reverse">
<path d="M1.5 1.5 10.5 6 1.5 10.5" fill="none" stroke="#303038" stroke-width="2.25" stroke-linecap="round" stroke-linejoin="round"/>
</marker>
<marker id="arrow-blue" viewBox="0 0 12 12" refX="10.5" refY="6" markerWidth="16" markerHeight="16" markerUnits="userSpaceOnUse" orient="auto-start-reverse">
<path d="M1.5 1.5 10.5 6 1.5 10.5" fill="none" stroke="#1478c9" stroke-width="2.25" stroke-linecap="round" stroke-linejoin="round"/>
</marker>
<style>
text { font-family: Arial, Helvetica, sans-serif; fill: #303038; }
.title { font-size: 50px; font-weight: 400; }
.node { font-size: 34px; font-weight: 400; }
.label { font-size: 29px; }
.blue-label { font-size: 26px; }
.small { font-size: 25px; }
.box { fill: #fff; stroke: #303038; stroke-width: 3; }
.group { fill: #fff; stroke: #303038; stroke-width: 3; }
.model-group { fill: #fff; stroke: #303038; stroke-width: 3; stroke-dasharray: 14 12; }
.line { fill: none; stroke: #303038; stroke-width: 3; marker-end: url(#arrow); }
.blue-line { fill: none; stroke: #1478c9; stroke-width: 4; marker-end: url(#arrow-blue); }
.blue { fill: #1478c9; }
</style>
</defs>
<rect width="1920" height="1295" fill="#fff"/>
<text x="20" y="70" class="title">Multivariate Models Lifecycle</text>
<!-- VictoriaMetrics and query configuration -->
<rect x="535" y="175" width="420" height="215" class="box"/>
<text x="745" y="265" class="node" text-anchor="middle">VictoriaMetrics TSDB</text>
<text x="745" y="310" class="node" text-anchor="middle">(Single-Node or Cluster)</text>
<rect x="270" y="270" width="150" height="155" class="box"/>
<text x="345" y="305" class="small" text-anchor="middle">Config.yml</text>
<text x="285" y="360" class="label">MetricsQL</text>
<text x="285" y="395" class="label">queries</text>
<path d="M420 346 H535" class="line"/>
<!-- Reader and datasource exchange -->
<rect x="580" y="500" width="310" height="100" class="box"/>
<text x="735" y="562" class="node" text-anchor="middle">Reader</text>
<path d="M580 545 H455 V365 H535" class="line"/>
<text x="465" y="478" class="label">1. Request data</text>
<path d="M955 270 H1040 V550 H890" class="line"/>
<text x="810" y="478" class="label">2. Get metrics</text>
<!-- Historical fit data returned by the configured queries -->
<g aria-label="Historical fit data">
<rect x="1090" y="85" width="430" height="505" class="group"/>
<rect x="1120" y="135" width="340" height="170" class="group"/>
<text x="1140" y="180" class="node">Query 1</text>
<rect x="1135" y="195" width="300" height="42" class="box"/>
<text x="1150" y="228" class="label">Metric 1.1</text>
<text x="1150" y="262" class="label">...</text>
<rect x="1135" y="265" width="300" height="42" class="box"/>
<text x="1150" y="298" class="label">Metric 1.M₁</text>
<text x="1305" y="355" class="node" text-anchor="middle">...</text>
<rect x="1120" y="390" width="340" height="170" class="group"/>
<text x="1140" y="435" class="node">Query N</text>
<rect x="1135" y="450" width="300" height="42" class="box"/>
<text x="1150" y="483" class="label">Metric N.1</text>
<text x="1150" y="517" class="label">...</text>
<rect x="1135" y="520" width="300" height="42" class="box"/>
<text x="1150" y="553" class="label">Metric N.Mₙ</text>
</g>
<!-- One shared model is fitted on the complete aligned set -->
<rect x="1215" y="795" width="310" height="115" class="box"/>
<text x="1370" y="865" class="node" text-anchor="middle">Model</text>
<rect x="1365" y="635" width="155" height="130" class="box"/>
<text x="1443" y="675" class="small" text-anchor="middle">Config.yml</text>
<text x="1382" y="723" class="label">Model</text>
<text x="1382" y="757" class="label">config</text>
<path d="M1305 590 V795" class="line"/>
<text x="1055" y="672" class="label">
<tspan x="1055" dy="0">3. Fit one model</tspan>
<tspan x="1055" dy="38">on all historical series</tspan>
</text>
<path d="M1443 765 V795" class="line"/>
<!-- Inference data must contain the same set of series -->
<g aria-label="Inference data">
<rect x="25" y="615" width="430" height="500" class="group"/>
<rect x="55" y="645" width="340" height="170" class="group"/>
<text x="75" y="690" class="node">Query 1</text>
<rect x="70" y="705" width="300" height="42" class="box"/>
<text x="85" y="738" class="label">Metric 1.1</text>
<text x="85" y="772" class="label">...</text>
<rect x="70" y="775" width="300" height="42" class="box"/>
<text x="85" y="808" class="label">Metric 1.M₁</text>
<text x="225" y="870" class="node" text-anchor="middle">...</text>
<rect x="55" y="900" width="340" height="175" class="group"/>
<text x="75" y="945" class="node">Query N</text>
<rect x="70" y="960" width="300" height="42" class="box"/>
<text x="85" y="993" class="label">Metric N.1</text>
<text x="85" y="1027" class="label">...</text>
<rect x="70" y="1030" width="300" height="42" fill="#fff" stroke="#1478c9" stroke-width="4"/>
<text x="85" y="1063" class="label blue">Metric N.Mₖ</text>
</g>
<path d="M580 575 H500 V650 H455" class="line"/>
<text x="465" y="680" class="label">4. Provide inference data</text>
<!-- Single multivariate model registry -->
<g aria-label="Multivariate model registry">
<rect x="580" y="750" width="420" height="480" class="group"/>
<text x="790" y="800" class="node" text-anchor="middle">Model registry</text>
<rect x="610" y="835" width="360" height="220" class="model-group"/>
<rect x="640" y="885" width="300" height="115" class="box"/>
<text x="790" y="955" class="node" text-anchor="middle">Model (single)</text>
</g>
<path d="M455 1050 H580" class="line"/>
<path d="M1215 852 H1000" class="line"/>
<!-- Joint output and mismatched-series skip path -->
<rect x="1600" y="1015" width="285" height="105" class="box"/>
<text x="1743" y="1080" class="node" text-anchor="middle">Writer</text>
<rect x="1740" y="805" width="145" height="145" class="box"/>
<text x="1813" y="845" class="small" text-anchor="middle">Config.yml</text>
<text x="1758" y="900" class="label">Writer</text>
<text x="1758" y="935" class="label">config</text>
<path d="M1813 950 V1015" class="line"/>
<path d="M1000 1040 H1600" class="line"/>
<text x="1295" y="965" class="label" text-anchor="middle">
<tspan x="1295" dy="0">5.a Produce one anomaly-score series</tspan>
<tspan x="1295" dy="38">with the intersected label set</tspan>
</text>
<path d="M1000 1110 H1600" class="blue-line"/>
<text x="1320" y="1145" class="blue-label blue" text-anchor="middle">
<tspan x="1320" dy="0">5.b Skip inference when the fit and inference</tspan>
<tspan x="1320" dy="32">series sets differ;</tspan>
<tspan x="1320" dy="32">update the model_runs_skipped counter</tspan>
</text>
<!-- Persist the joint anomaly score -->
<path d="M1743 1015 V80 H745 V175" class="line"/>
<text x="1170" y="55" class="label" text-anchor="middle">6. Write one anomaly-score series (label set = intersection)</text>
</svg>

After

Width:  |  Height:  |  Size: 7.8 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 66 KiB

View File

@@ -0,0 +1,148 @@
<?xml version="1.0" encoding="UTF-8"?>
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1920 1295" role="img" aria-labelledby="title description">
<title id="title">Univariate models lifecycle</title>
<desc id="description">MetricsQL queries retrieve multiple series from VictoriaMetrics. Reader data fits one model per series in the model registry. Known series produce individual anomaly scores for Writer; an unseen series is skipped until a fitted model exists. Writer stores produced scores in VictoriaMetrics.</desc>
<defs>
<marker id="arrow" viewBox="0 0 12 12" refX="10.5" refY="6" markerWidth="16" markerHeight="16" markerUnits="userSpaceOnUse" orient="auto-start-reverse">
<path d="M1.5 1.5 10.5 6 1.5 10.5" fill="none" stroke="#303038" stroke-width="2.25" stroke-linecap="round" stroke-linejoin="round"/>
</marker>
<marker id="arrow-blue" viewBox="0 0 12 12" refX="10.5" refY="6" markerWidth="16" markerHeight="16" markerUnits="userSpaceOnUse" orient="auto-start-reverse">
<path d="M1.5 1.5 10.5 6 1.5 10.5" fill="none" stroke="#1478c9" stroke-width="2.25" stroke-linecap="round" stroke-linejoin="round"/>
</marker>
<style>
text { font-family: Arial, Helvetica, sans-serif; fill: #303038; }
.title { font-size: 50px; font-weight: 400; }
.node { font-size: 34px; font-weight: 400; }
.label { font-size: 29px; }
.blue-label { font-size: 26px; }
.small { font-size: 25px; }
.box { fill: #fff; stroke: #303038; stroke-width: 3; }
.group { fill: #fff; stroke: #303038; stroke-width: 3; }
.model-group { fill: #fff; stroke: #303038; stroke-width: 3; stroke-dasharray: 14 12; }
.line { fill: none; stroke: #303038; stroke-width: 3; marker-end: url(#arrow); }
.blue-line { fill: none; stroke: #1478c9; stroke-width: 4; marker-end: url(#arrow-blue); }
.blue { fill: #1478c9; }
</style>
</defs>
<rect width="1920" height="1295" fill="#fff"/>
<text x="20" y="70" class="title">Univariate Models Lifecycle</text>
<!-- VictoriaMetrics and query configuration -->
<rect x="535" y="175" width="420" height="215" class="box"/>
<text x="745" y="265" class="node" text-anchor="middle">VictoriaMetrics TSDB</text>
<text x="745" y="310" class="node" text-anchor="middle">(Single-Node or Cluster)</text>
<rect x="270" y="270" width="150" height="155" class="box"/>
<text x="345" y="305" class="small" text-anchor="middle">Config.yml</text>
<text x="285" y="360" class="label">MetricsQL</text>
<text x="285" y="395" class="label">queries</text>
<path d="M420 346 H535" class="line"/>
<!-- Reader and datasource exchange -->
<rect x="580" y="500" width="310" height="100" class="box"/>
<text x="735" y="562" class="node" text-anchor="middle">Reader</text>
<path d="M580 545 H455 V365 H535" class="line"/>
<text x="465" y="478" class="label">1. Request data</text>
<path d="M955 270 H1040 V550 H890" class="line"/>
<text x="810" y="478" class="label">2. Get metrics</text>
<!-- Historical fit data returned by the configured queries -->
<g aria-label="Historical fit data">
<rect x="1090" y="85" width="430" height="505" class="group"/>
<rect x="1120" y="135" width="340" height="170" class="group"/>
<text x="1140" y="180" class="node">Query 1</text>
<rect x="1135" y="195" width="300" height="42" class="box"/>
<text x="1150" y="228" class="label">Metric 1.1</text>
<text x="1150" y="262" class="label">...</text>
<rect x="1135" y="265" width="300" height="42" class="box"/>
<text x="1150" y="298" class="label">Metric 1.M₁</text>
<text x="1305" y="355" class="node" text-anchor="middle">...</text>
<rect x="1120" y="390" width="340" height="170" class="group"/>
<text x="1140" y="435" class="node">Query N</text>
<rect x="1135" y="450" width="300" height="42" class="box"/>
<text x="1150" y="483" class="label">Metric N.1</text>
<text x="1150" y="517" class="label">...</text>
<rect x="1135" y="520" width="300" height="42" class="box"/>
<text x="1150" y="553" class="label">Metric N.Mₙ</text>
</g>
<!-- Model fitting -->
<rect x="1215" y="795" width="310" height="115" class="box"/>
<text x="1370" y="865" class="node" text-anchor="middle">Model</text>
<rect x="1365" y="635" width="155" height="130" class="box"/>
<text x="1443" y="675" class="small" text-anchor="middle">Config.yml</text>
<text x="1382" y="723" class="label">Model</text>
<text x="1382" y="757" class="label">config</text>
<path d="M1305 590 V795" class="line"/>
<text x="1055" y="672" class="label">
<tspan x="1055" dy="0">3. Fit models</tspan>
<tspan x="1055" dy="38">on historical data</tspan>
</text>
<path d="M1443 765 V795" class="line"/>
<!-- Inference data, including a new unseen series -->
<g aria-label="Inference data">
<rect x="25" y="615" width="430" height="500" class="group"/>
<rect x="55" y="645" width="340" height="170" class="group"/>
<text x="75" y="690" class="node">Query 1 (has fit model)</text>
<rect x="70" y="705" width="300" height="42" class="box"/>
<text x="85" y="738" class="label">Metric 1.1</text>
<text x="85" y="772" class="label">...</text>
<rect x="70" y="775" width="300" height="42" class="box"/>
<text x="85" y="808" class="label">Metric 1.M₁</text>
<text x="225" y="870" class="node" text-anchor="middle">...</text>
<rect x="55" y="900" width="340" height="175" class="group"/>
<text x="75" y="945" class="node">Query N</text>
<rect x="70" y="960" width="300" height="42" class="box"/>
<text x="85" y="993" class="label">Metric N.1</text>
<text x="85" y="1027" class="label">...</text>
<rect x="70" y="1030" width="300" height="42" fill="#fff" stroke="#1478c9" stroke-width="4"/>
<text x="85" y="1063" class="label blue">Metric N.Mₖ</text>
</g>
<path d="M580 575 H500 V650 H455" class="line"/>
<text x="465" y="680" class="label">4. Provide inference data</text>
<!-- One fitted model per known series -->
<g aria-label="Univariate model registry">
<rect x="580" y="750" width="420" height="480" class="group"/>
<text x="790" y="800" class="node" text-anchor="middle">Model registry</text>
<rect x="610" y="830" width="360" height="170" class="model-group"/>
<rect x="630" y="865" width="300" height="42" class="box"/>
<text x="645" y="898" class="label">Model 1.1</text>
<text x="645" y="932" class="label">...</text>
<rect x="630" y="935" width="300" height="42" class="box"/>
<text x="645" y="968" class="label">Model 1.M₁</text>
<text x="790" y="1045" class="node" text-anchor="middle">...</text>
<rect x="610" y="1070" width="360" height="135" class="model-group"/>
<rect x="630" y="1090" width="300" height="42" class="box"/>
<text x="645" y="1123" class="label">Model N.1</text>
<rect x="630" y="1145" width="300" height="42" class="box"/>
<text x="645" y="1178" class="label">Model N.Mₙ</text>
</g>
<path d="M455 1050 H580" class="line"/>
<path d="M1215 852 H1000" class="line"/>
<!-- Known-series output and unseen-series skip path -->
<rect x="1600" y="1015" width="285" height="105" class="box"/>
<text x="1743" y="1080" class="node" text-anchor="middle">Writer</text>
<rect x="1740" y="805" width="145" height="145" class="box"/>
<text x="1813" y="845" class="small" text-anchor="middle">Config.yml</text>
<text x="1758" y="900" class="label">Writer</text>
<text x="1758" y="935" class="label">config</text>
<path d="M1813 950 V1015" class="line"/>
<path d="M1000 1040 H1600" class="line"/>
<text x="1275" y="975" class="label" text-anchor="middle">
<tspan x="1275" dy="0">5.a Produce anomaly scores</tspan>
<tspan x="1275" dy="38">for known series</tspan>
</text>
<path d="M1000 1110 H1600" class="blue-line"/>
<text x="1320" y="1145" class="blue-label blue" text-anchor="middle">
<tspan x="1320" dy="0">5.b Skip inference until a fitted model exists</tspan>
<tspan x="1320" dy="32">for Metric N.Mₖ;</tspan>
<tspan x="1320" dy="32">update the model_runs_skipped counter</tspan>
</text>
<!-- Persist anomaly scores -->
<path d="M1743 1015 V80 H745 V175" class="line"/>
<text x="1130" y="55" class="label" text-anchor="middle">6. Write back produced anomaly scores</text>
</svg>

After

Width:  |  Height:  |  Size: 8.2 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 66 KiB

View File

@@ -68,6 +68,10 @@ models:
Common arguments supported by every model were introduced in [v1.10.0](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1100).
<div class="collapse-group">
{{% collapse name="Queries" %}}
### Queries
The `queries` argument selects the [reader queries](https://docs.victoriametrics.com/anomaly-detection/components/reader/#config-parameters) used to fit and run a particular model{{% available_from "v1.10.0" anomaly %}}. Every series returned by a selected query is passed to that model.
@@ -93,6 +97,10 @@ models:
queries: ['q1', 'q2', 'q3'] # i.e., if your `queries` in `reader` section has exactly q1, q2, q3 aliases
```
{{% /collapse %}}
{{% collapse name="Schedulers" %}}
### Schedulers
The `schedulers` argument selects the [schedulers](https://docs.victoriametrics.com/anomaly-detection/components/scheduler/) that run a particular model{{% available_from "v1.11.0" anomaly %}}.
@@ -118,6 +126,10 @@ models:
schedulers: ['s1', 's2', 's3'] # i.e., if your `schedulers` section has exactly s1, s2, s3 aliases
```
{{% /collapse %}}
{{% collapse name="Provide series" %}}
### Provide series
The `provide_series` argument{{% available_from "v1.12.0" anomaly %}} limits the [model output](#vmanomaly-output) sent to the writer. For example, a model may produce `['anomaly_score', 'yhat', 'yhat_lower', 'yhat_upper']` by default, while the following configuration writes only `anomaly_score` for each input series:
@@ -131,6 +143,10 @@ models:
> If `provide_series` is not specified in model config, the model will produce its default [model-dependent output](#vmanomaly-output). The output can't be less than `['anomaly_score']`. Even if `timestamp` column is omitted, it will be implicitly added to `provide_series` list, as it's required for metrics to be properly written.
{{% /collapse %}}
{{% collapse name="Detection direction" %}}
### Detection direction
The `detection_direction` argument{{% available_from "v1.13.0" anomaly %}} can reduce [false positives](https://victoriametrics.com/blog/victoriametrics-anomaly-detection-handbook-chapter-1/#false-positive) when domain knowledge indicates that only values above or below the expected value are anomalous. Available values are `both`, `above_expected`, and `below_expected`.
@@ -190,6 +206,10 @@ reader:
# other components like writer, schedule, monitoring
```
{{% /collapse %}}
{{% collapse name="Minimal deviation from expected" %}}
### Minimal deviation from expected
`min_dev_from_expected`{{% available_from "v1.13.0" anomaly %}} argument is designed to **reduce [false positives](https://victoriametrics.com/blog/victoriametrics-anomaly-detection-handbook-chapter-1/#false-positive)** in scenarios where deviations between the actual value (`y`) and the expected value (`yhat`) are **relatively** high. Such deviations can cause models to generate high [anomaly scores](https://docs.victoriametrics.com/anomaly-detection/faq/#what-is-anomaly-score). However, these deviations may not be significant enough in **absolute values** from a business perspective to be considered anomalies. This parameter ensures that anomaly scores for data points where `|y - yhat| < min_dev_from_expected` are explicitly set to 0. By default, if this parameter is not set, it is set to `0` to maintain backward compatibility.
@@ -238,6 +258,10 @@ models:
queries: ['normal_behavior'] # use the default where it's not needed
```
{{% /collapse %}}
{{% collapse name="Minimal relative deviation from expected" %}}
### Minimal relative deviation from expected
{{% available_from "v1.29.1" anomaly %}} `min_rel_dev_from_expected` argument serves a similar purpose to `min_dev_from_expected` (see [section above](#minimal-deviation-from-expected)), but focuses on **relative deviations** rather than absolute ones. It is designed to reduce [false positives](https://victoriametrics.com/blog/victoriametrics-anomaly-detection-handbook-chapter-1/#false-positive) in scenarios where the relative deviation between the actual value (`y`) and the expected value (`yhat`) is high, but the absolute deviation is not significant enough to be considered an anomaly from a business perspective. This parameter ensures that anomaly scores for data points where `|y - yhat| / |yhat| < min_rel_dev_from_expected` are explicitly set to 0. By default, if this parameter is not set, it is set to `0` to maintain backward compatibility.
@@ -278,6 +302,10 @@ models:
```
{{% /collapse %}}
{{% collapse name="Group by" %}}
### Group by
> The `groupby` argument works only in combination with [multivariate models](#multivariate-models).
@@ -316,6 +344,10 @@ models:
groupby: [host]
```
{{% /collapse %}}
{{% collapse name="Scale" %}}
### Scale
Previously available only to [ProphetModel](#prophet) and [OnlineQuantileModel](#online-seasonal-quantile), the `scale` {{% available_from "v1.20.0" anomaly %}} parameter is now applicable to all models that support generating predictions (`yhat`, `yhat_lower`, `yhat_upper`). Also, it is **two-sided** now, represented as a list of two positive float values, allowing separate scaling for the intervals `[yhat, yhat_upper]` and `[yhat_lower, yhat]`. The new margins are calculated as:
@@ -346,6 +378,10 @@ models:
scale: [1.2, 0.75]
```
{{% /collapse %}}
{{% collapse name="Clip predictions" %}}
### Clip predictions
A post-processing step to **clip model predictions** (`yhat`, `yhat_lower`, and `yhat_upper` series) to the configured [`data_range` values](https://docs.victoriametrics.com/anomaly-detection/components/reader/#config-parameters) in `VmReader` is available.
@@ -400,6 +436,10 @@ models:
]
```
{{% /collapse %}}
{{% collapse name="Score outside data range" %}}
### Score outside data range
The `anomaly_score_outside_data_range` {{% available_from "v1.20.0" anomaly %}} parameter allows overriding the default **anomaly score (`1.01`)** assigned when actual values (`y`) fall **outside the defined `data_range` if defined in [reader](https://docs.victoriametrics.com/anomaly-detection/components/reader/)**. This provides greater flexibility for **alerting rule configurations** and enables **clearer visual differentiation** between different types of anomalies:
@@ -445,6 +485,10 @@ models:
anomaly_score_outside_data_range: 3.0
```
{{% /collapse %}}
{{% collapse name="Decay" %}}
### Decay
> The `decay` argument works only in combination with [online models](#online-models) like [`ZScoreOnlineModel`](#online-z-score) or [`OnlineQuantileModel`](#online-seasonal-quantile).
@@ -477,6 +521,10 @@ models:
queries: ['q1']
```
{{% /collapse %}}
</div>
## Model types
@@ -502,7 +550,7 @@ If during an inference, you got a series having **new labelset** (not present in
**Examples:** [Prophet](#prophet), [Holt-Winters](#holt-winters)
![vmanomaly-model-type-univariate](model-lifecycle-univariate.webp)
![Univariate model lifecycle](model-lifecycle-univariate.svg)
### Multivariate Models
@@ -517,9 +565,34 @@ If during an inference, you got a **different amount of series** or some series
**Implications:** Multivariate models are a go-to default, when your queries returns **fixed** amount of **individual** time series (say, some aggregations), to be used for adding cross-series (and cross-query) context, useful for catching [collective anomalies](https://victoriametrics.com/blog/victoriametrics-anomaly-detection-handbook-chapter-2/#collective-anomalies) or [novelties](https://victoriametrics.com/blog/victoriametrics-anomaly-detection-handbook-chapter-2/#novelties) (expanded to multi-input scenario). For example, you may set it up for anomaly detection of CPU usage in different modes (`idle`, `user`, `system`, etc.) and use its cross-dependencies to detect **unseen (in fit data)** behavior.
**Examples:** [IsolationForest](#isolation-forest-multivariate)
**Examples:** [Temporal Envelope](#temporal-envelope), [Isolation Forest](#isolation-forest-multivariate)
![vmanomaly-model-type-multivariate](model-lifecycle-multivariate.webp)
![Multivariate model lifecycle](model-lifecycle-multivariate.svg)
The following configuration applies both models to the same aligned input series. Start with Temporal Envelope when temporal profiles and online adaptation matter; use Isolation Forest as an offline alternative when feature-space outliers are the primary concern.
```yaml
models:
service_dependency_envelope:
class: temporal_envelope_multivariate
queries: [request_rate, error_rate, latency]
groupby: [cluster]
dependency_rank: 8
score_aggregation: l2
seasonalities: [hod_smooth, dow_smooth]
provide_series: [anomaly_score]
service_dependency_isolation_forest:
class: isolation_forest_multivariate
queries: [request_rate, error_rate, latency]
groupby: [cluster]
contamination: 0.01
seasonal_features: [hod, dow]
args:
n_estimators: 100
random_state: 42
provide_series: [anomaly_score]
```
### Online Models
@@ -664,7 +737,7 @@ models:
</div>
![vmanomaly-autotune-schema](autotune.webp)
![AutoTunedModel tuning and inference lifecycle](autotune.svg)
#### Shared asynchronous autotune workflow
@@ -1344,6 +1417,10 @@ This guide shows how to:
> The file containing the model must be written in [Python](https://www.python.org/) 3.14 or later. A custom model runs inside the `vmanomaly` Python environment, so keep its dependencies compatible with the target image and keep the module available when restoring serialized model state after a restart.
<div class="collapse-group">
{{% collapse name="Custom model implementation guide" %}}
### 1. Custom model
Create `custom_model.py` with a `CustomModel` class derived from `Model`. A concrete model must implement:
@@ -1510,6 +1587,10 @@ The writer emits one `custom_anomaly_score` series for each input series. It ret
{__name__="custom_anomaly_score", for="churn_rate", model_alias="custom_model", scheduler_alias="s1", run="test-format"}
```
{{% /collapse %}}
</div>
## Deprecations
{{% collapse name="Deprecated model types and models" %}}

View File

@@ -24,6 +24,10 @@ There are 2 models to monitor VictoriaMetrics Anomaly Detection behavior - [push
- Adding `preset` and `scheduler_alias` keys to [VmReader](https://docs.victoriametrics.com/anomaly-detection/components/monitoring/#reader-behaviour-metrics) and [VmWriter](https://docs.victoriametrics.com/anomaly-detection/components/monitoring/#writer-behaviour-metrics) metrics for consistency in multi-[scheduler](https://docs.victoriametrics.com/anomaly-detection/components/scheduler/) setups.
- Renaming [Counters](https://prometheus.io/docs/concepts/metric_types/#counter) `vmanomaly_reader_response_count` to `vmanomaly_reader_responses` and `vmanomaly_writer_response_count` to `vmanomaly_writer_responses`.
<div class="collapse-group">
{{% collapse name="Pull model config parameters" %}}
## Pull Model Config parameters
<table class="params">
@@ -60,6 +64,10 @@ There are 2 models to monitor VictoriaMetrics Anomaly Detection behavior - [push
</tbody>
</table>
{{% /collapse %}}
{{% collapse name="Push config parameters" %}}
## Push Config parameters
By default, metrics are pushed only after the completion of specific stages, e.g., `fit`, `infer`, or `fit_infer` (for each [scheduler](https://docs.victoriametrics.com/anomaly-detection/components/scheduler/) if using a multi-scheduler configuration).
@@ -227,6 +235,10 @@ Path to a file with the client certificate key, i.e. `client.key`{{% available_f
</tbody>
</table>
{{% /collapse %}}
</div>
## Monitoring section config example
``` yaml
@@ -260,6 +272,10 @@ For detailed guidance on configuring mTLS parameters such as `verify_tls`, `tls_
- [Model metrics](#models-behaviour-metrics)
- [Writer metrics](#writer-behaviour-metrics)
<div class="collapse-group">
{{% collapse name="Startup metrics" %}}
### Startup metrics
<table class="params">
@@ -385,6 +401,10 @@ For detailed guidance on configuring mTLS parameters such as `verify_tls`, `tls_
[Back to metric sections](#metrics-generated-by-vmanomaly)
{{% /collapse %}}
{{% collapse name="Reader behaviour metrics" %}}
### Reader behaviour metrics
Label names [description](#labelnames)
@@ -509,6 +529,10 @@ Label names [description](#labelnames)
[Back to metric sections](#metrics-generated-by-vmanomaly)
{{% /collapse %}}
{{% collapse name="Models behaviour metrics" %}}
### Models behaviour metrics
Label names [description](#labelnames)
@@ -635,6 +659,10 @@ Label names [description](#labelnames)
[Back to metric sections](#metrics-generated-by-vmanomaly)
{{% /collapse %}}
{{% collapse name="Writer behaviour metrics" %}}
### Writer behaviour metrics
Label names [description](#labelnames)
@@ -746,6 +774,10 @@ Label names [description](#labelnames)
[Back to metric sections](#metrics-generated-by-vmanomaly)
{{% /collapse %}}
</div>
### Labelnames
* `stage` - model execution stage: `fit`, `infer`, or `fit_infer` for a combined fit/inference scheduler run. See [model types](https://docs.victoriametrics.com/anomaly-detection/components/models/#model-types).
@@ -793,6 +825,10 @@ and the [command-line arguments](https://docs.victoriametrics.com/anomaly-detect
- [Query server and task logs](#query-server-and-task-logs)
- [AI Copilot logs](#ai-copilot-logs)
<div class="collapse-group">
{{% collapse name="Startup logs" %}}
### Startup logs
@@ -812,6 +848,10 @@ server addresses, hot-reload state, and active schedulers. The most useful prefi
[Back to logging sections](#logs-generated-by-vmanomaly)
{{% /collapse %}}
{{% collapse name="Reader logs" %}}
### Reader logs
Reader logs cover endpoint checks, request splitting, network failures, response parsing, and coordination between
@@ -863,6 +903,10 @@ or parsed. See [reader behaviour metrics](#reader-behaviour-metrics).
[Back to logging sections](#logs-generated-by-vmanomaly)
{{% /collapse %}}
{{% collapse name="Service logs" %}}
### Service logs
The service logs `fit`, `infer`, and combined `fit_infer`/backtesting work for each model alias and scheduler.
@@ -891,6 +935,10 @@ an unsuccessful stage. See [models behaviour metrics](#models-behaviour-metrics)
[Back to logging sections](#logs-generated-by-vmanomaly)
{{% /collapse %}}
{{% collapse name="Writer logs" %}}
### Writer logs
Writer logs cover serialization and delivery of produced series such as
@@ -918,6 +966,10 @@ and datapoints are recorded only after a successful response. See [writer behavi
[Back to logging sections](#logs-generated-by-vmanomaly)
{{% /collapse %}}
{{% collapse name="Scheduler supervision logs" %}}
### Scheduler supervision logs
Scheduler supervision{{% available_from "v1.30.0" anomaly %}} logs a dead worker, automatic restart, successful
@@ -927,6 +979,10 @@ Correlate them with `vmanomaly_scheduler_alive` and `vmanomaly_scheduler_restart
[Back to logging sections](#logs-generated-by-vmanomaly)
{{% /collapse %}}
{{% collapse name="Hot-reload logs" %}}
### Hot-reload logs
Hot reload logs config-change detection, validation, staged service restart, success, and rollback. `Reload aborted
@@ -936,6 +992,10 @@ without restarting services`.
[Back to logging sections](#logs-generated-by-vmanomaly)
{{% /collapse %}}
{{% collapse name="Persisted-state logs" %}}
### Persisted-state logs
With `settings.restore_state`, startup logs the stored/runtime version assessment, reusable components, required
@@ -944,6 +1004,10 @@ stored artifacts completely` indicates a full reset; missing or unreadable model
[Back to logging sections](#logs-generated-by-vmanomaly)
{{% /collapse %}}
{{% collapse name="Query server and task logs" %}}
### Query server and task logs
The query server logs its listening address and datasource-proxy timeouts/failures. Background anomaly-detection
@@ -952,6 +1016,10 @@ background raw query finishing cleanly.
[Back to logging sections](#logs-generated-by-vmanomaly)
{{% /collapse %}}
{{% collapse name="AI Copilot logs" %}}
### AI Copilot logs
AI Copilot{{% available_from "v1.30.0" anomaly %}} reports whether it is initialized, disabled, misconfigured, or
@@ -960,3 +1028,7 @@ request failed` identifies provider execution failure, and `MCP server unreachab
guidance tools.
[Back to logging sections](#logs-generated-by-vmanomaly)
{{% /collapse %}}
</div>

View File

@@ -25,6 +25,8 @@ Use the following playgrounds to develop and test input queries:
## VM reader
<div class="collapse-group">
{{% collapse name="Queries format migration (to v1.13.0+)" %}}
> The backward-compatible `queries` format introduced in v1.13.0 allows [VmReader](#vm-reader) parameters such as `step` to be configured per query. This can reduce the amount of data read from VictoriaMetrics. See [per-query parameters](#per-query-parameters) for details.
@@ -61,6 +63,8 @@ reader:
```
{{% /collapse %}}
{{% collapse name="VM reader per-query parameters and example" %}}
### Per-query parameters
There is change {{% available_from "v1.13.0" anomaly %}} of [`queries`](https://docs.victoriametrics.com/anomaly-detection/components/reader/#vm-reader) arg format. Now each query alias supports the next (sub)fields, which *override reader-level parameters*, if set:
@@ -125,6 +129,10 @@ reader:
offset: '-15s' # to override reader-wise `offset` and query data 15 seconds earlier to account for data collection delays
```
{{% /collapse %}}
{{% collapse name="VM reader config parameters and example" %}}
### Config parameters
<table class="params">
@@ -507,10 +515,16 @@ reader:
series_processing_batch_size: 8
```
{{% /collapse %}}
</div>
### MetricsQL Playground
To experiment with MetricsQL queries for `VmReader`, you can use the [VictoriaMetrics MetricsQL Playground](https://play.victoriametrics.com/), which provides an interactive environment to test and visualize your queries against sample data. You can also access embedded version of the playground below:
<div class="collapse-group">
{{% collapse name="VictoriaMetrics Playground" %}}
<div class="position-relative mb-3">
@@ -536,6 +550,8 @@ To experiment with MetricsQL queries for `VmReader`, you can use the [VictoriaMe
{{% /collapse %}}
</div>
### mTLS protection
`vmanomaly` supports [mutual TLS (mTLS)](https://en.wikipedia.org/wiki/Mutual_authentication){{% available_from "v1.16.3" anomaly %}} for secure communication across its components, including [VmReader](https://docs.victoriametrics.com/anomaly-detection/components/reader/#vm-reader), [VmWriter](https://docs.victoriametrics.com/anomaly-detection/components/writer/#vm-writer), and [Monitoring/Push](https://docs.victoriametrics.com/anomaly-detection/components/monitoring/#push-config-parameters). This allows for mutual authentication between the client and server when querying or writing data to [VictoriaMetrics Enterprise, configured for mTLS](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#mtls-protection).
@@ -680,6 +696,8 @@ Similarly, [VictoriaTraces LogsQL Playground](https://play-vtraces.victoriametri
You can also access **embedded version of the playground below** (VictoriaLogs datasource):
<div class="collapse-group">
{{% collapse name="VictoriaLogs LogsQL Playground" %}}
<div class="position-relative mb-3">
@@ -705,6 +723,11 @@ You can also access **embedded version of the playground below** (VictoriaLogs d
{{% /collapse %}}
</div>
<div class="collapse-group">
{{% collapse name="VictoriaLogs reader config parameters" %}}
### Config parameters
@@ -989,6 +1012,10 @@ Optional hard cap {{% available_from "v1.30.0" anomaly %}} for how far last-seen
</tbody>
</table>
{{% /collapse %}}
{{% collapse name="VictoriaLogs reader per-query parameters and example" %}}
### Per-query parameters
The names, types and the logic of the per-query parameters subset used in `VLogsReader` are exactly the same as those of [`VmReader`](#vm-reader), please see [per-query parameters](#per-query-parameters) section above for the details. The only difference is that `expr` parameter should contain a valid [LogsQL](https://docs.victoriametrics.com/victorialogs/logsql/) expression with `stats` [pipe](https://docs.victoriametrics.com/victorialogs/logsql/#stats-pipe), as described in [query examples](#query-examples) section above.
@@ -1036,6 +1063,10 @@ reader:
# other config sections, like models, schedulers, writer, ...
```
{{% /collapse %}}
</div>
### mTLS protection
Please refer to the [mTLS protection](#mtls-protection) section above for details on how to configure mTLS for `VLogsReader`. It uses the same config parameters as `VmReader` for mTLS setup.

View File

@@ -226,6 +226,10 @@ monitoring:
# other monitoring settings
```
<div class="collapse-group">
{{% collapse name="State restoration example" %}}
### Example
For a configuration with the following models, queries and schedulers:
@@ -305,6 +309,10 @@ This means that the service upon restart:
1. Won't restore the state of `zscore_online` model, because its `z_threshold` argument **has changed**, retraining from scratch is needed on the last `fit_window` = 24 hours of data for `q1`, `q2` and `q3` (as model's `queries` arg is not set so it defaults to all queries found in the reader).
2. Will **partially** restore the state of `prophet` model, because its class and schedulers are unchanged, but **only instances trained on timeseries returned by `q1` query**. New fit/infer jobs will be set for new query `q3`. The old query `q2` artifacts will be dropped upon restart - all respective models and data for (`prophet`, `q2`) combination will be removed from the database file and from the disk.
{{% /collapse %}}
</div>
## Retention
{{% available_from "v1.28.1" anomaly %}} The `retention` argument sets a [time to live](https://en.wikipedia.org/wiki/Time_to_live) (TTL) for service artifacts such as stored model instances and training data. At each `check_interval`, the service removes artifacts that have not been used for inference or refitting within `ttl`. This bounds stale resource usage in long-running deployments.

View File

@@ -0,0 +1,15 @@
---
build:
list: never
publishResources: false
render: never
sitemap:
disable: true
---
The required path is `config.yml` → Scheduler → Reader → Model → Writer. The Reader queries the configured VictoriaMetrics, VictoriaLogs, or VictoriaTraces datasource; the Writer stores inferred anomaly scores in VictoriaMetrics. Monitoring is optional and can push metrics or expose them for collection.
Solid nodes and arrows show the required anomaly-detection path. Dashed nodes and arrows show optional self-monitoring integrations.
![vmanomaly component interaction: the Scheduler starts Reader, Model, and Writer work against a configured datasource; Monitoring is optional](/anomaly-detection/components/vmanomaly-components.svg)
{style="display:block; width:80%; min-width:320px; margin:1.5rem auto"}

View File

@@ -0,0 +1,152 @@
<?xml version="1.0" encoding="UTF-8"?>
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1920 1600" role="img" aria-labelledby="title description">
<title id="title">How vmanomaly operates during a scheduled iteration</title>
<desc id="description">The required flow starts from config.yml and the Scheduler. The Reader queries either VictoriaMetrics through query_range or VictoriaLogs and VictoriaTraces through stats_query_range, then sends data to a Model. The Model produces anomaly scores, and the Writer stores them in VictoriaMetrics through import. Reader, Model, and Writer can optionally report self-monitoring metrics using push or pull monitoring.</desc>
<defs>
<marker id="arrow" viewBox="0 0 12 12" refX="10.5" refY="6" markerWidth="16" markerHeight="16" markerUnits="userSpaceOnUse" orient="auto-start-reverse">
<path d="M1.5 1.5 10.5 6 1.5 10.5" fill="none" stroke="#303038" stroke-width="2.25" stroke-linecap="round" stroke-linejoin="round"/>
</marker>
<marker id="arrow-optional" viewBox="0 0 12 12" refX="10.5" refY="6" markerWidth="16" markerHeight="16" markerUnits="userSpaceOnUse" orient="auto-start-reverse">
<path d="M1.5 1.5 10.5 6 1.5 10.5" fill="none" stroke="#666a73" stroke-width="2.25" stroke-linecap="round" stroke-linejoin="round"/>
</marker>
<style>
.label { font-family: Arial, Helvetica, sans-serif; fill: #303038; }
.title { font-size: 54px; font-weight: 400; }
.service-title { font-size: 46px; font-weight: 400; }
.node-text { font-size: 34px; font-weight: 600; text-anchor: middle; }
.body { font-size: 28px; }
.edge-label { font-size: 25px; }
.endpoint-text { font-size: 23px; }
.source-text { font-size: 21px; }
.required-node { fill: #fff; stroke: #303038; stroke-width: 4; }
.optional-node { fill: #fff; stroke: #666a73; stroke-width: 4; stroke-dasharray: 14 12; }
.required-line { fill: none; stroke: #303038; stroke-width: 4; marker-end: url(#arrow); }
.optional-line { fill: none; stroke: #666a73; stroke-width: 4; stroke-dasharray: 14 12; marker-end: url(#arrow-optional); }
.boundary { fill: none; stroke: #303038; stroke-width: 4; }
.optional-boundary { fill: none; stroke: #666a73; stroke-width: 4; stroke-dasharray: 14 12; }
</style>
</defs>
<rect width="1920" height="1600" fill="#fff"/>
<text x="105" y="72" class="label title">How does vmanomaly operate (scheduled iteration example)</text>
<!-- Required / optional legend -->
<g aria-label="Legend" transform="translate(1425 145)">
<rect x="0" y="0" width="135" height="62" class="optional-node"/>
<rect x="205" y="0" width="135" height="62" class="required-node"/>
<path d="M0 100 H135" class="optional-line"/>
<path d="M205 100 H340" class="required-line"/>
<text x="67" y="162" class="label body" text-anchor="middle">Optional</text>
<text x="272" y="162" class="label body" text-anchor="middle">Required</text>
</g>
<!-- Exactly one configured read datasource is used. -->
<g aria-label="Configured datasource choice">
<rect x="25" y="315" width="300" height="550" class="optional-boundary"/>
<g transform="translate(34 326)">
<path d="M8 32 V184 C8 224 280 224 280 184 V32 C280 70 8 70 8 32Z" class="required-node"/>
<ellipse cx="144" cy="32" rx="136" ry="33" class="required-node"/>
<rect x="30" y="72" width="228" height="66" rx="16" class="required-node"/>
<text x="144" y="99" class="label endpoint-text" text-anchor="middle">/query_range</text>
<text x="144" y="126" class="label endpoint-text" text-anchor="middle">endpoint</text>
<text x="144" y="158" class="label source-text" text-anchor="middle">VictoriaMetrics</text>
<text x="144" y="182" class="label source-text" text-anchor="middle">TSDB</text>
</g>
<text x="178" y="600" class="label body" text-anchor="middle">OR</text>
<g transform="translate(34 632)">
<path d="M8 32 V184 C8 224 280 224 280 184 V32 C280 70 8 70 8 32Z" class="required-node"/>
<ellipse cx="144" cy="32" rx="136" ry="33" class="required-node"/>
<rect x="30" y="72" width="228" height="66" rx="16" class="required-node"/>
<text x="144" y="99" class="label endpoint-text" text-anchor="middle">/stats_query_range</text>
<text x="144" y="126" class="label endpoint-text" text-anchor="middle">endpoint</text>
<text x="144" y="158" class="label source-text" text-anchor="middle">VictoriaLogs /</text>
<text x="144" y="182" class="label source-text" text-anchor="middle">VictoriaTraces</text>
</g>
</g>
<!-- vmanomaly service boundary and components -->
<g aria-label="vmanomaly service">
<rect x="720" y="145" width="610" height="1145" class="boundary"/>
<rect x="1005" y="175" width="270" height="100" class="required-node"/>
<text x="1140" y="235" class="label node-text">config.yml</text>
<text x="1025" y="352" class="label service-title" text-anchor="middle">vmanomaly</text>
<text x="1025" y="405" class="label service-title" text-anchor="middle">service</text>
<rect x="765" y="448" width="445" height="76" rx="16" class="required-node"/>
<text x="987" y="500" class="label node-text">Scheduler</text>
<rect x="765" y="590" width="445" height="76" rx="16" class="required-node"/>
<text x="987" y="642" class="label node-text">Reader</text>
<rect x="765" y="732" width="445" height="76" rx="16" class="required-node"/>
<text x="987" y="784" class="label node-text">Model</text>
<rect x="765" y="874" width="445" height="76" rx="16" class="required-node"/>
<text x="987" y="926" class="label node-text">Writer</text>
<rect x="778" y="1180" width="420" height="82" rx="16" class="optional-node"/>
<text x="988" y="1235" class="label node-text">Monitoring</text>
<path d="M1005 225 H900 V448" class="required-line"/>
<path d="M1210 486 H1290 V628 H1210" class="required-line"/>
<path d="M1210 628 H1290 V770 H1210" class="required-line"/>
<path d="M1210 770 H1290 V912 H1210" class="required-line"/>
<path d="M875 666 V1178" class="optional-line"/>
<path d="M987 808 V1178" class="optional-line"/>
<path d="M1100 950 V1178" class="optional-line"/>
</g>
<!-- Required flow labels -->
<text x="1365" y="455" class="label body">
<tspan x="1365" dy="0">1. Get the metrics to</tspan>
<tspan x="1365" dy="37">a. fit the model, or</tspan>
<tspan x="1365" dy="37">b. produce anomaly scores</tspan>
</text>
<text x="1365" y="620" class="label body">
<tspan x="1365" dy="0">3. Model receives data</tspan>
<tspan x="1365" dy="37">to train or infer on</tspan>
</text>
<text x="1365" y="775" class="label body">
<tspan x="1365" dy="0">4. Produce anomaly</tspan>
<tspan x="1365" dy="37">scores (inference)</tspan>
</text>
<!-- Datasource request and response -->
<path d="M765 610 H338" class="required-line"/>
<path d="M338 652 H765" class="required-line"/>
<text x="370" y="580" class="label edge-label">2.1 Query request</text>
<text x="370" y="700" class="label edge-label">2.2 Metrics response</text>
<!-- Anomaly-score output -->
<g aria-label="VictoriaMetrics write endpoint" transform="translate(26 1145)">
<path d="M8 32 V184 C8 224 290 224 290 184 V32 C290 70 8 70 8 32Z" class="required-node"/>
<ellipse cx="149" cy="32" rx="141" ry="33" class="required-node"/>
<rect x="34" y="72" width="230" height="66" rx="16" class="required-node"/>
<text x="149" y="99" class="label endpoint-text" text-anchor="middle">/import</text>
<text x="149" y="126" class="label endpoint-text" text-anchor="middle">endpoint</text>
<text x="149" y="158" class="label source-text" text-anchor="middle">VictoriaMetrics</text>
<text x="149" y="182" class="label source-text" text-anchor="middle">TSDB</text>
</g>
<path d="M765 912 H650 V1095 L316 1197" class="required-line"/>
<rect x="338" y="1018" width="292" height="74" fill="#fff"/>
<text x="350" y="1048" class="label edge-label">
<tspan x="340" dy="0">5. Write anomaly scores</tspan>
<tspan x="340" dy="33">to VictoriaMetrics</tspan>
</text>
<!-- Optional monitoring integrations -->
<rect x="748" y="1083" width="490" height="48" fill="#fff"/>
<text x="993" y="1116" class="label edge-label" text-anchor="middle">6. Produce self-monitoring metrics</text>
<rect x="390" y="1460" width="300" height="118" rx="45" class="optional-node"/>
<text x="540" y="1515" class="label body" text-anchor="middle">Monitoring system</text>
<text x="540" y="1552" class="label body" text-anchor="middle">push approach</text>
<rect x="1490" y="1460" width="300" height="118" rx="45" class="optional-node"/>
<text x="1640" y="1515" class="label body" text-anchor="middle">Monitoring system</text>
<text x="1640" y="1552" class="label body" text-anchor="middle">pull approach</text>
<path d="M850 1262 L540 1460" class="optional-line"/>
<path d="M1198 1222 L1640 1460" class="optional-line"/>
<rect x="590" y="1350" width="180" height="42" fill="#fff"/>
<text x="680" y="1380" class="label edge-label" text-anchor="middle">Push metrics</text>
<rect x="1230" y="1340" width="415" height="42" fill="#fff"/>
<text x="1438" y="1370" class="label edge-label" text-anchor="middle">HTTP GET /metrics or /health</text>
</svg>

After

Width:  |  Height:  |  Size: 9.2 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 107 KiB

View File

@@ -17,6 +17,10 @@ Future updates will introduce additional export methods, offering users more fle
## VM writer
<div class="collapse-group">
{{% collapse name="VM writer config parameters and example" %}}
### Config parameters
<table class="params">
@@ -35,8 +39,7 @@ Future updates will introduce additional export methods, offering users more fle
</td>
<td>
<span style="white-space: nowrap;">`writer.vm.VmWriter` or `vm`{{% available_from "v1.13.0" anomaly %}}
</span>
`writer.vm.VmWriter` or `vm`{{% available_from "v1.13.0" anomaly %}}
</td>
<td>
@@ -63,10 +66,7 @@ Datasource URL address
<span style="white-space: nowrap;">`tenant_id`</span>
</td>
<td>
<span>
`0:0`, `multitenant`{{% available_from "v1.16.2" anomaly %}}
</span>
</td>
<td>
@@ -253,9 +253,8 @@ Token is passed in the standard format with header: `Authorization: bearer {toke
`path_to_file`
</td>
<td>
<span>
Path to a file, which contains token, that is passed in the standard format with header: `Authorization: bearer {token}`{{% available_from "v1.15.9" anomaly %}}
</span> </td>
</td>
</tr>
<tr>
<td>
@@ -267,9 +266,8 @@ Path to a file, which contains token, that is passed in the standard format with
`1`
</td>
<td>
<span>
Number of attempts to retry the connection in case of failure {{% available_from "v1.29.2" anomaly %}}.
</span> </td>
</td>
</tr>
</tbody>
</table>
@@ -293,6 +291,10 @@ writer:
connection_retry_attempts: 2 # if not specified, it will be 1 by default
```
{{% /collapse %}}
</div>
### Multitenancy support
> This feature applies to the VictoriaMetrics Cluster version only. Tenants are identified by either `accountID` or `accountID:projectID`. `multitenant` [endpoint](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#multitenancy-via-labels){{% available_from "v1.15.9" anomaly %}} is supported for writing data across multiple [tenants](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#multitenancy). For more details, refer to the VictoriaMetrics Cluster [multitenancy documentation](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#multitenancy).

View File

@@ -17,7 +17,7 @@ sitemap:
- [Docker](https://docs.docker.com/get-docker/) and [Docker Compose](https://docs.docker.com/compose/)
- [Node exporter](https://github.com/prometheus/node_exporter#node-exporter) (v1.9.1) and [Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/) (v0.28.1)
![typical setup diagram](guide-vmanomaly-vmalert_overview.webp)
![Typical vmanomaly observability pipeline](guide-vmanomaly-vmalert_overview.svg)
> **Configurations used throughout this guide can be found [here](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker/vmanomaly/vmanomaly-integration/)**

View File

@@ -0,0 +1,65 @@
<?xml version="1.0" encoding="UTF-8"?>
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1706 1069" role="img" aria-labelledby="title description">
<title id="title">Typical vmanomaly observability pipeline</title>
<desc id="description">vmagent scrapes node-exporter metrics and writes them to VictoriaMetrics. vmanomaly reads those metrics and writes anomaly scores back. Grafana visualizes the results. vmalert evaluates rules based on anomaly scores and sends alerts to Alertmanager.</desc>
<defs>
<marker id="arrow" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="10" markerHeight="10" orient="auto-start-reverse">
<path d="M0 0 10 5 0 10Z" fill="#252525"/>
</marker>
<style>
text { font-family: Arial, Helvetica, sans-serif; fill: #252525; font-size: 40px; }
.box { fill: #fff; stroke: #252525; stroke-width: 3; }
.arrow { fill: none; stroke: #252525; stroke-width: 4; marker-end: url(#arrow); }
</style>
</defs>
<rect width="1706" height="1069" fill="#fff"/>
<rect x="20" y="55" width="390" height="100" class="box"/>
<text x="215" y="119" text-anchor="middle">node-exporter</text>
<rect x="730" y="50" width="395" height="100" class="box"/>
<text x="928" y="115" text-anchor="middle">vmagent</text>
<path d="M710 100 H435" class="arrow"/>
<text x="570" y="56" text-anchor="middle">Scrape metrics</text>
<rect x="745" y="325" width="400" height="180" class="box"/>
<text x="945" y="405" text-anchor="middle">VictoriaMetrics</text>
<text x="945" y="457" text-anchor="middle">TSDB</text>
<path d="M945 150 V322" class="arrow"/>
<text x="1018" y="204">
<tspan x="1018" dy="0">Push node</tspan>
<tspan x="1018" dy="50">exporter</tspan>
<tspan x="1018" dy="50">metrics</tspan>
</text>
<rect x="25" y="340" width="395" height="100" class="box"/>
<text x="222" y="405" text-anchor="middle">vmanomaly</text>
<path d="M725 365 H440" class="arrow"/>
<text x="575" y="319" text-anchor="middle">Read metrics</text>
<path d="M440 421 H725" class="arrow"/>
<text x="570" y="508" text-anchor="middle">
<tspan x="570" dy="0">Write produced</tspan>
<tspan x="570" dy="50">anomaly scores</tspan>
</text>
<rect x="1440" y="330" width="245" height="170" class="box"/>
<text x="1562" y="430" text-anchor="middle">Grafana</text>
<path d="M1160 417 H1418" class="arrow"/>
<text x="1285" y="486" text-anchor="middle">
<tspan x="1285" dy="0">Visualize the</tspan>
<tspan x="1285" dy="50">results</tspan>
</text>
<rect x="750" y="738" width="395" height="100" class="box"/>
<text x="948" y="805" text-anchor="middle">vmalert</text>
<path d="M946 525 V716" class="arrow"/>
<text x="1000" y="644">
<tspan x="1000" dy="0">Evaluate rules based</tspan>
<tspan x="1000" dy="50">on anomaly scores</tspan>
</text>
<rect x="755" y="948" width="395" height="100" class="box"/>
<text x="952" y="1015" text-anchor="middle">alertmanager</text>
<path d="M948 858 V928" class="arrow"/>
<text x="1004" y="906">Send alerts</text>
</svg>

After

Width:  |  Height:  |  Size: 3.0 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 29 KiB

View File

@@ -0,0 +1,107 @@
# Render with D2 v0.7.1:
# d2 --layout elk --theme 0 --pad 20 vmanomaly-sharding-ha-diagram.d2 vmanomaly-sharding-ha-diagram.svg
grid-columns: 1
grid-gap: 36
classes: {
boundary: {
style.fill: "#F7F7F8"
style.stroke: "#A7AAB2"
style.stroke-width: 2
}
process: {
style.fill: "#FFFFFF"
style.stroke: "#303038"
style.stroke-width: 2
style.border-radius: 8
}
member: {
style.fill: "#F1F3F5"
style.stroke: "#59616A"
style.stroke-width: 2
style.border-radius: 8
}
}
flow: "" {
grid-columns: 3
grid-gap: 36
style.fill: transparent
style.stroke: transparent
global: "Global YAML\nconfiguration" {
shape: page
class: process
}
splitting: Configuration splitting {
class: boundary
grid-columns: 1
grid-gap: 24
split: "Split by VMANOMALY_SPLIT_BY\ndefault: complete" {
class: process
}
subconfigs: "N valid sub-configurations\nn = 0 ... N-1" {
class: process
style.multiple: true
}
split -> subconfigs: {
style.stroke: "#303038"
}
}
placement: Deterministic placement {
class: boundary
grid-columns: 1
grid-gap: 24
settings: "K members: VMANOMALY_MEMBERS_COUNT\nMember index: VMANOMALY_MEMBER_NUM\nR replicas: VMANOMALY_REPLICATION_FACTOR" {
class: process
}
assign: "Assign every sub-configuration\nto exactly R members" {
class: process
}
settings -> assign: {
style.stroke: "#303038"
}
}
}
members: K vmanomaly members {
class: boundary
grid-columns: 3
grid-gap: 30
member0: "Member 0\nvmanomaly service 1" {
class: member
}
member1: "Member 1\nvmanomaly service 2" {
class: member
}
memberK: "Member K-1\nvmanomaly service K" {
class: member
}
}
flow.global -> flow.splitting.split: {
style.stroke: "#303038"
}
flow.splitting.subconfigs -> flow.placement.assign: {
style.stroke: "#303038"
}
flow.placement.assign -> members.member0: "assigned subset" {
style.stroke: "#59616A"
style.stroke-width: 2
}
flow.placement.assign -> members.member1: "assigned subset" {
style.stroke: "#59616A"
style.stroke-width: 2
}
flow.placement.assign -> members.memberK: "assigned subset" {
style.stroke: "#59616A"
style.stroke-width: 2
}

View File

@@ -0,0 +1,13 @@
---
build:
list: never
publishResources: false
render: never
sitemap:
disable: true
---
The global configuration is split into `N` independently valid sub-configurations. Deterministic placement distributes them across `K` members, and each sub-configuration is assigned to exactly `R` members when replication is enabled. Member `k` processes only its assigned subset.
![vmanomaly sharding and high availability: a global configuration is split into N sub-configurations and replicated across K members](/anomaly-detection/vmanomaly-sharding-ha-diagram.svg)
{style="display:block; width:50%; min-width:320px; margin:1.5rem auto"}

File diff suppressed because one or more lines are too long

After

Width:  |  Height:  |  Size: 33 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 87 KiB

View File

@@ -34,6 +34,7 @@ See also [LTS releases](https://docs.victoriametrics.com/victoriametrics/lts-rel
* FEATURE: [dashboards](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/dashboards): add `Fsync avg duration` panel to the Troubleshooting section of the single-node, cluster, and vmagent dashboards. This panel surfaces degradation of IO operation for faster incident triage. See [#10432](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/10432).
* FEATURE: [vmbackupmanager](https://docs.victoriametrics.com/victoriametrics/vmbackupmanager/) and [alerts](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/rules): introduce `vm_backup_last_success_at` metric to track the last successful backup by type. Add [alerting rules](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/rules/alerts-vmbackupmanager.yml) `NoLatestBackupWithinLastDay`, `NoHourlyBackupWithinLastDay`, `NoDailyBackupWithinLast3Days`, `NoWeeklyBackupWithinLast14Days` and `NoMonthlyBackupWithinLast62Days` to remind users about the missing backups. See [#11217](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/11217).
* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/): properly drop data points filtered out by an inner [comparison operation](https://prometheus.io/docs/prometheus/latest/querying/operators/#comparison-binary-operators) when its result is used on the right side of another comparison. Previously, queries like `foo != (bar > 100)` could return unexpected results because filtered-out data points are represented internally as `NaN`, and `value != NaN` evaluates to `true`. Comparisons against explicitly present `NaN` values keep the previous behavior. See [#10018](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/10018). Thanks to @zasdaym for contribution.
* BUGFIX: [vmalert](https://docs.victoriametrics.com/victoriametrics/vmalert/): fixed the display of rule state badges on the `Groups` page in the web UI. See [#11160](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/11160).
* BUGFIX: [vmbackupmanager](https://docs.victoriametrics.com/victoriametrics/vmbackupmanager/): previously, `vmbackupmanager` was crashing on startup when it failed to restore backup state from remote storage, causing a crash loop. Now it logs the error and continues running, retrying the state restore before each scheduled backup. Added `vm_backup_errors_total{type="restoreState"}` metric to track backup state restore failures. See [#11217](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/11217).
@@ -122,6 +123,7 @@ Released at 2026-06-22
* BUGFIX: [vmsingle](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/) and `vmselect` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/): properly escape `metricFamilyName` at metrics metadata response. See [#11129](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/11129). Thanks for @fxrlv for the contribution.
* BUGFIX: [vmsingle](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/) and `vmstorage` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/): prevent more cases of panic during directory deletion on `NFS`-based mounts. See [#11060](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/11060).
## [v1.145.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.145.0)
Released at 2026-06-08

File diff suppressed because it is too large Load Diff