Improve the `/anomaly-detection` documentation by:
- adding collapsible reference sections for better navigation and UX;
- reordering FAQ questions into a clearer usage and operations flow;
- replacing raster diagrams with maintainable SVGs and adding model
lifecycle diagrams;
- other small improvements
NaN values in non-scalar operands of comparison operations mean missing
samples, so they cannot match any sample on the other side. Previously
such data points were passed to the comparison function, so queries like
`foo != (bar > 100)` could return unexpected results, since `value !=
NaN` evaluates to `true`. This PR drops such data points in the same way
as Prometheus does.
Comparisons to scalars such as `foo != nan` keep the previous behavior,
since a scalar `NaN` is an explicit value rather than a missing sample
(see the existing `compare_to_nan_right` test case).
The check is applied only to the right side of the operation, since a
missing sample on the left side already results in `NaN` for every
comparison operation:
- `==`, `>`, `<`, `>=`, `<=`: `cf(NaN, b)` is `false`, so the data point
is dropped;
- `!=`: `cf(NaN, b)` is `true`, but the returned value is the left one,
which is `NaN`;
- `bool` modifier: the left `NaN` is handled explicitly in
`newBinaryOpCmpFunc`.
Verified with the reproduction steps from the linked issue: the query
`tsafdb0_value{table="t1", timeSeries="c0"} != ((272) <=
(tsafdb0_value{table="t1", timeSeries="c0"}))` now returns an empty
result, matching Prometheus behavior.
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/10018
Docs upgrade for v1.30.0 release of vmanomaly, including:
- UX fixes and refreshed UI screenshots on `/anomaly-detection/` pages;
- revised AI tooling guidance and new cross-references to MCP, skills,
and UI Copilot;
- Temporal Envelope, autotune, reader, settings, and custom-model
documentation aligned with v1.30.0;
- updated self-monitoring metric names, labels, stage semantics,
resource-capacity descriptions, and license health metrics;
- corrections to outdated instructions across the anomaly-detection
documentation.
TODOs:
- [x] update UI screenshots
- [x] add cross-references
- [x] update stale monitoring metrics/logs page
The commit
dd2d6807e4
added the following metrics to VictoriaMetrics:
- vm_filestream_fsync_duration_seconds_total - it measures the summary
duration for
[fsync()](https://man7.org/linux/man-pages/man2/fsync.2.html) calls
during background merges at VictoriaMetrics.
- vm_filestream_fsync_calls_total - it counts the number of fsync()
calls.
This commit adds a `Fsync avg duration ($instance)` panel to the
`Troubleshooting` section of the `VictoriaMetrics - single-node`,
`VictoriaMetrics - cluster`, and `vmagent` dashboards.
PR https://github.com/VictoriaMetrics/VictoriaMetrics/pull/11079
---------
Co-authored-by: Dominic Polizzi <dpolizzi@victoriametrics.com>
Co-authored-by: Max Kotliar <mkotlyar@victoriametrics.com>
Slowness rerouting was disabled by default in
a1b298a842
due to rerouting storm issues.
PRs https://github.com/VictoriaMetrics/VictoriaMetrics/pull/9945 and
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/10901 fixed the
storm: now only the single slowest storage node
is rerouted away from. With these fixes in place, most users benefit
from
slowness rerouting being enabled by default.
Slowness rerouting is forcefully disabled when replicationFactor > 1
because:
- rerouting does not guarantee that replicated copies land on distinct
storage nodes,
which violates the replication contract
- rerouting under replication creates significant additional load on the
next node
This commit adds http client load-balancing with DNS discovery. For both A and SRV records.
It helps to route HTTP requests evenly for across discovered backends.
Discovered IP addresses are cached locally.
The main benefit of this feature is to remove intermediate vmauth as a load-balancer between
vmagent (vmalert) and remote targets. Which simplifies components management and
reduces operational overhead.
By default, it's disable and could be used with `dns+` or `srv+` hostname suffix at url.
For example, `-remoteWrite.url=http://dns+victoria-metrics:8428/api/v1/write`.
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2388
This commit adds `name` label to the `vm_persistentqueue_*` metrics. It provides more context for queues with human read-able format. Because `path` label contains only on-disk path, which is not really useful.
For vmagent case it will contain remote.writeURL position at command-line args.
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7944
A slow vmagent can cause reading the request body to exceed the deadline configured by `-maxQueueDuration`. In this case, vmauth returned 400 Bad Request, causing vmagent to downgrade the remote write protocol from zstd to snappy. If subsequent requests also failed, vmagent could drop data blocks without retrying.
This change returns 408 Request Timeout instead of 400 Bad Request when reading the request body exceeds `-maxQueueDuration`, preventing the unnecessary protocol downgrade and subsequent data loss.
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/11272
PR https://github.com/VictoriaMetrics/VictoriaMetrics/pull/11273
---------
Signed-off-by: “Jayice” <jzhou@victoriametrics.com>
Co-authored-by: Max Kotliar <mkotlyar@victoriametrics.com>
Previously queue metainfo was flushed with fsync. However it may
corrupt file if OS filesystem not properly flush data.
This commit adds atomic file write with temporary file.
It must prevent possible file corruption issue.
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/11192
Target metrics relabel debug UI now preloads relabeling rules configured in `-promscrape.config`, `-remoteWrite.relabelConfig` and `-remoteWrite.urlRelabelConfig` flags.
Metrics relabel debug UI now preloads relabeling rules configured in `-remoteWrite.relabelConfig` and `-remoteWrite.urlRelabelConfig` flags.
The configs are ordered in the same order they are executed by vmagent. They also contain hint comments that highlight from which flag they come.
The select at the top lets you change which remote write configuration to use. The first one is preloaded by default.
Other UI improvements:
- renamed "Labels" -> "A Time Series", "Reset" -> "Reset All",
"Relabel Configs" -> "Configs"
- pre-populate "A Time Series" field with an `up` metric placeholder
- display YAML comments (lines starting with #) in gray
- added links to "Relabeling Cookbook" and "Relabeling Stages" docs
- added JSONPath hints for -promscrape.config relabel config fields
- use POST when reloading config content larger than 1KB
- improved error messages when multiple time series are provided
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/9918
PR https://github.com/VictoriaMetrics/VictoriaMetrics/pull/10607
---------
Signed-off-by: Zhu Jiekun <jiekun@victoriametrics.com>
Co-authored-by: Max Kotliar <mkotlyar@victoriametrics.com>
Fixes#11154
## What
Hide the `Total metric names` widget in the cardinality page when
`match` or `focusLabel` is not empty.
## Why
`Total metric names` is relevant for the unfiltered overview, but
becomes misleading in filtered cardinality views.
## Changes
- hide `Total metric names` when `match` is set
- hide `Total metric names` when `focusLabel` is set
- keep existing totals behavior otherwise
## Validation
- `npm run test`
- `npm run lint:local`
- `npm run typecheck`
---------
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
Recommendation to gracefuly shutdown process is sometimes overlooked by
users. Mentioning it explicitly so it is present in docs as
recommendation and can be linkable when sharing with users.
---------
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Signed-off-by: Roman Khavronenko <hagen1778@gmail.com>
Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>
Co-authored-by: Kirill Yurkov <kirillyu@users.noreply.github.com>
The idea here is to give a very direct path to start trying
VictoriaMetrics with a very clear, simple and straight forward guidance.
We keep the full instalation process in the same document on purpose to
keep any external link working.
---------
Signed-off-by: Jesús Espino <jespinog@gmail.com>
Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>
Previously, reader could advance writer offset at metadata without actual data write.
It may produce corrupted persistent queue, if vmagent crashed.
This commit changes flush behavior to perform f-sync flush, only if there is some recently written data.
Related to https://github.com/VictoriaMetrics/VictoriaMetrics/issues/11192
Previously, slow http_sd target may slow discovery pipeline. And for
example, if configuration contains 100 targets which respond in 1
second. It will take 100 seconds to refresh targets.
This commit adds background discovery for `http_sd` scrape config.
A new goroutine starts for each http_sd url. Which reduces discovery
time.
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8838
This commit adds new metrics:
* vm_data_size_bytes{type="storage/metaindex"}
* vm_data_size_bytes{type="indexdb/metaindex"}
Which are useful to track memory used to store meta index.
Related to https://github.com/VictoriaMetrics/VictoriaMetrics/issues/10203
Allows rejecting ingested samples with historical timestamps older than
now-maxBackfillAge, independently of -retentionPeriod. This is useful
for limiting ingestion of historical samples, for example when older
data has been moved between storage tiers (nvme/hdd, hot/cold).
Mirrors the -maxBackfillAge flag already available in VictoriaLogs.
Defaults to 0, in which case it is clamped to -retentionPeriod,
preserving previous ingestion behavior.
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/11199
---------
Signed-off-by: Ashwin Ramani <ashwinramanipsg@gmail.com>
Co-authored-by: Artem Fetishev <rtm@victoriametrics.com>
Previously, in case when connection to the vmstorage was established in
legacy mode. VMinsert could still forward non metric rows to this
connection. It caused parsing errors at legacy vmstorage.
This commit drops any non-metric rows designated to the legacy
vmstorages. Because it's not possible correctly re-route it, since most
likely other vmstorages also don't support RPC protocol. Instead
vminsert logs warning message with suggestion to perform an upgrade for
vmstorage.
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/11146
