mirror of
https://github.com/VictoriaMetrics/VictoriaMetrics.git
synced 2026-07-15 13:20:36 +03:00
Compare commits
2 Commits
docs-artic
...
jaycie-tes
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
43a198782e | ||
|
|
8f06486f75 |
@@ -154,13 +154,6 @@ See [our blog](https://victoriametrics.com/blog) for the latest articles written
|
||||
* [Why irate from Prometheus doesn't capture spikes](https://valyala.medium.com/why-irate-from-prometheus-doesnt-capture-spikes-45f9896d7832)
|
||||
* [VictoriaMetrics: PromQL compliance](https://medium.com/@romanhavronenko/victoriametrics-promql-compliance-d4318203f51e)
|
||||
* [How do open source solutions for logs work: Elasticsearch, Loki and VictoriaLogs](https://itnext.io/how-do-open-source-solutions-for-logs-work-elasticsearch-loki-and-victorialogs-9f7097ecbc2f)
|
||||
* [How vmagent Collects and Ships Metrics Fast with Aggregation, Deduplication, and More](https://victoriametrics.com/blog/vmagent-how-it-works/)
|
||||
* [When Metrics Meet vminsert: A Data-Delivery Story](https://victoriametrics.com/blog/vminsert-how-it-works/)
|
||||
* [How vmstorage Handles Data Ingestion From vminsert](https://victoriametrics.com/blog/vmstorage-how-it-handles-data-ingestion/)
|
||||
* [How vmstorage Processes Data: Retention, Merging, Deduplication,...](https://victoriametrics.com/blog/vmstorage-retention-merging-deduplication/)
|
||||
* [How vmstorage's IndexDB Works](https://victoriametrics.com/blog/vmstorage-how-indexdb-works/)
|
||||
* [How vmstorage Handles Query Requests From vmselect](https://victoriametrics.com/blog/vmstorage-how-it-handles-query-requests/)
|
||||
* [Inside vmselect: The Query Processing Engine of VictoriaMetrics](https://victoriametrics.com/blog/vmselect-how-it-works/)
|
||||
|
||||
### Tutorials, guides and how-to articles
|
||||
|
||||
@@ -178,12 +171,6 @@ See [our guides](https://docs.victoriametrics.com/guides/) for the up-to-date gu
|
||||
* [Prometheus storage: tech terms for humans](https://valyala.medium.com/prometheus-storage-technical-terms-for-humans-4ab4de6c3d48)
|
||||
* [Cardinality explorer](https://victoriametrics.com/blog/cardinality-explorer/)
|
||||
* [Rules backfilling via vmalert](https://victoriametrics.com/blog/rules-replay/)
|
||||
* [vmagent: Key Features Explained in Under 15 Minutes](https://victoriametrics.com/blog/vmagent-key-features-explained/)
|
||||
* [Prometheus Metrics Explained: Counters, Gauges, Histograms & Summaries](https://victoriametrics.com/blog/prometheus-monitoring-metrics-counters-gauges-histogram-summaries/)
|
||||
* [Prometheus Monitoring: Instant Queries and Range Queries Explained](https://victoriametrics.com/blog/prometheus-monitoring-instant-range-query/)
|
||||
* [Prometheus Monitoring: Functions, Subqueries, Operators, and Modifiers](https://victoriametrics.com/blog/prometheus-monitoring-function-operator-modifier/)
|
||||
* [Prometheus Alerting 101: Rules, Recording Rules, and Alertmanager](https://victoriametrics.com/blog/alerting-recording-rules-alertmanager/)
|
||||
* [Alerting Best Practices](https://victoriametrics.com/blog/alerting-best-practices/)
|
||||
|
||||
### Other articles
|
||||
|
||||
|
||||
@@ -57,10 +57,6 @@ Each service may scale independently and may run on the most suitable hardware.
|
||||
This is a [shared nothing architecture](https://en.wikipedia.org/wiki/Shared-nothing_architecture).
|
||||
It increases cluster availability, and simplifies cluster maintenance as well as cluster scaling.
|
||||
|
||||
> Further reading, deep dives into how each service works internally:
|
||||
> - `vmstorage`: [How vmstorage Handles Data Ingestion From vminsert](https://victoriametrics.com/blog/vmstorage-how-it-handles-data-ingestion/), [How vmstorage's IndexDB Works](https://victoriametrics.com/blog/vmstorage-how-indexdb-works/), [How vmstorage Handles Query Requests From vmselect](https://victoriametrics.com/blog/vmstorage-how-it-handles-query-requests/).
|
||||
> - `vmselect`: [Inside vmselect: The Query Processing Engine of VictoriaMetrics](https://victoriametrics.com/blog/vmselect-how-it-works/).
|
||||
|
||||

|
||||
|
||||
## vmui
|
||||
@@ -838,7 +834,7 @@ This ensures that incoming metrics are evenly distributed across all `vmstorage`
|
||||
The downside is that a single slow vmstorage node can throttle the entire cluster.
|
||||
|
||||
When `-disableRerouting=false` is enabled on `vminsert`,
|
||||
the cluster will automatically [re-route writes](https://victoriametrics.com/blog/vminsert-how-it-works/#31-rerouting) away from the slowest vmstorage node to preserve maximum ingestion throughput.
|
||||
the cluster will automatically re-route writes away from the slowest vmstorage node to preserve maximum ingestion throughput.
|
||||
|
||||
Re-routing occurs only when all of the following conditions hold:
|
||||
- the storage send buffer is full.
|
||||
@@ -881,7 +877,7 @@ See also [resource usage limits docs](#resource-usage-limits).
|
||||
|
||||
## Rebalancing
|
||||
|
||||
Every `vminsert` node [evenly spreads (shards) incoming data](https://victoriametrics.com/blog/vminsert-how-it-works/#3-sharding-and-buffering) among `vmstorage` nodes specified in the `-storageNode` command-line flag.
|
||||
Every `vminsert` node evenly spreads (shards) incoming data among `vmstorage` nodes specified in the `-storageNode` command-line flag.
|
||||
This guarantees even distribution of the ingested data among `vmstorage` nodes. When new `vmstorage` nodes are added to the `-storageNode`
|
||||
command-line flag at `vminsert`, then only newly ingested data is distributed evenly among old and new `vmstorage` nodes, while
|
||||
historical data remains on the old `vmstorage` nodes. This speeds up data ingestion and querying for the majority of production workloads,
|
||||
@@ -1029,7 +1025,7 @@ By default, VictoriaMetrics offloads replication to the underlying storage point
|
||||
which guarantees data durability. VictoriaMetrics supports application-level replication if replicated durable persistent disks cannot be used for some reason.
|
||||
|
||||
The replication can be enabled by passing `-replicationFactor=N` command-line flag to `vminsert`. This instructs `vminsert` to store `N` copies for every ingested sample
|
||||
on `N` distinct `vmstorage` nodes. This guarantees that all the stored data remains available for querying if up to `N-1` `vmstorage` nodes are unavailable. See [how `vminsert` replicates each sample to `N` `vmstorage` nodes](https://victoriametrics.com/blog/vminsert-how-it-works/#4-replication-and-sending-data-to-vmstorage) for details.
|
||||
on `N` distinct `vmstorage` nodes. This guarantees that all the stored data remains available for querying if up to `N-1` `vmstorage` nodes are unavailable.
|
||||
|
||||
Passing `-replicationFactor=N` command-line flag to `vmselect` instructs it to not mark responses as `partial` if less than `-replicationFactor` vmstorage nodes are unavailable during the query.
|
||||
See [cluster availability docs](#cluster-availability) for details.
|
||||
@@ -1064,7 +1060,7 @@ deduplication can't be guaranteed when samples and sample duplicates for the sam
|
||||
- when `vmstorage` node has no enough capacity for processing incoming data stream. Then `vminsert` re-routes new samples to other `vmstorage` nodes.
|
||||
|
||||
It is recommended to set **the same** `-dedup.minScrapeInterval` command-line flag value to both `vmselect` and `vmstorage` nodes
|
||||
to ensure query results consistency, even if [storage layer didn't complete deduplication](https://victoriametrics.com/blog/vmstorage-retention-merging-deduplication/#deduplication) yet.
|
||||
to ensure query results consistency, even if storage layer didn't complete deduplication yet.
|
||||
|
||||
## Metrics Metadata
|
||||
|
||||
|
||||
@@ -59,7 +59,6 @@ Other PromQL functionality should work the same in MetricsQL.
|
||||
|
||||
MetricsQL implements [PromQL](https://medium.com/@valyala/promql-tutorial-for-beginners-9ab455142085)
|
||||
and provides additional functionality mentioned below, which is aimed towards solving practical cases.
|
||||
See [operators and modifiers in MetricsQL](https://victoriametrics.com/blog/prometheus-monitoring-function-operator-modifier/#operators--modifiers) for details.
|
||||
Feel free [filing a feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues) if you think MetricsQL misses certain useful functionality.
|
||||
|
||||
This functionality can be evaluated at [VictoriaMetrics demo playground](https://play.victoriametrics.com/select/accounting/1/6a716b0f-38bc-4856-90ce-448fd713e3fe/prometheus/graph/)
|
||||
@@ -157,7 +156,7 @@ MetricsQL provides the following functions:
|
||||
|
||||
### Rollup functions
|
||||
|
||||
**[Rollup functions](https://victoriametrics.com/blog/prometheus-monitoring-function-operator-modifier/#rollup-functions)** (aka range functions or window functions) calculate rollups over [raw samples](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#raw-samples)
|
||||
**Rollup functions** (aka range functions or window functions) calculate rollups over [raw samples](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#raw-samples)
|
||||
on the given lookbehind window for the [selected time series](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#filtering).
|
||||
For example, `avg_over_time(temperature[24h])` calculates the average temperature over [raw samples](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#raw-samples) for the last 24 hours.
|
||||
|
||||
@@ -349,7 +348,6 @@ If the lookbehind window is skipped in square brackets, then it is automatically
|
||||
passed to [/api/v1/query_range](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#range-query) or [/api/v1/query](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#instant-query),
|
||||
while `scrape_interval` is the interval between [raw samples](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#raw-samples) for the selected time series.
|
||||
This allows avoiding unexpected gaps on the graph when `step` is smaller than the `scrape_interval`.
|
||||
See [range vector selectors](https://victoriametrics.com/blog/prometheus-monitoring-instant-range-query/#range-vector-selector) for details.
|
||||
|
||||
#### delta
|
||||
|
||||
@@ -1112,7 +1110,7 @@ See also [zscore](#zscore), [range_trim_zscore](#range_trim_zscore) and [outlier
|
||||
|
||||
### Transform functions
|
||||
|
||||
**[Transform functions](https://victoriametrics.com/blog/prometheus-monitoring-function-operator-modifier/#transformation-functions)** calculate transformations over [rollup results](#rollup-functions).
|
||||
**Transform functions** calculate transformations over [rollup results](#rollup-functions).
|
||||
For example, `abs(delta(temperature[24h]))` calculates the absolute value for every point of every time series
|
||||
returned from the rollup `delta(temperature[24h])`.
|
||||
|
||||
@@ -1848,7 +1846,7 @@ This function is supported by PromQL.
|
||||
|
||||
### Label manipulation functions
|
||||
|
||||
**[Label manipulation functions](https://victoriametrics.com/blog/prometheus-monitoring-function-operator-modifier/#label-manipulation-functions)** perform manipulations with labels on the selected [rollup results](#rollup-functions).
|
||||
**Label manipulation functions** perform manipulations with labels on the selected [rollup results](#rollup-functions).
|
||||
|
||||
Additional details:
|
||||
|
||||
@@ -2018,7 +2016,7 @@ See also [sort_by_label_numeric](#sort_by_label_numeric) and [sort_by_label_desc
|
||||
|
||||
### Aggregate functions
|
||||
|
||||
**[Aggregate functions](https://victoriametrics.com/blog/prometheus-monitoring-function-operator-modifier/#aggregation-functions)** calculate aggregates over groups of [rollup results](#rollup-functions).
|
||||
**Aggregate functions** calculate aggregates over groups of [rollup results](#rollup-functions).
|
||||
|
||||
Additional details:
|
||||
|
||||
@@ -2341,7 +2339,7 @@ See also [zscore_over_time](#zscore_over_time), [range_trim_zscore](#range_trim_
|
||||
|
||||
## Subqueries
|
||||
|
||||
MetricsQL supports and extends PromQL [subqueries](https://victoriametrics.com/blog/prometheus-monitoring-function-operator-modifier/#subqueries). See [this article](https://valyala.medium.com/prometheus-subqueries-in-victoriametrics-9b1492b720b3) for details.
|
||||
MetricsQL supports and extends PromQL subqueries. See [this article](https://valyala.medium.com/prometheus-subqueries-in-victoriametrics-9b1492b720b3) for details.
|
||||
Any [rollup function](#rollup-functions) for something other than [series selector](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#filtering) form a subquery.
|
||||
Nested rollup functions can be implicit thanks to the [implicit query conversions](#implicit-query-conversions).
|
||||
For example, `delta(sum(m))` is implicitly converted to `delta(sum(default_rollup(m))[1i:1i])`, so it becomes a subquery,
|
||||
|
||||
@@ -1314,7 +1314,6 @@ per each `-dedup.minScrapeInterval` discrete interval if `-dedup.minScrapeInterv
|
||||
For example, `-dedup.minScrapeInterval=60s` would leave a single raw sample with the biggest timestamp per each discrete
|
||||
`60s` interval.
|
||||
This aligns with the [staleness rules in Prometheus](https://prometheus.io/docs/prometheus/latest/querying/basics/#staleness).
|
||||
See [how deduplication works](https://victoriametrics.com/blog/vmstorage-retention-merging-deduplication/#deduplication) for details.
|
||||
|
||||
If multiple raw samples have **the same timestamp** on the given `-dedup.minScrapeInterval` discrete interval,
|
||||
then the sample with **the biggest value** is kept.
|
||||
@@ -1398,7 +1397,7 @@ in separate files under `part` directory - `timestamps.bin` and `values.bin`.
|
||||
The `part` directory also contains `index.bin` and `metaindex.bin` files - these files contain index
|
||||
for fast block lookups, which belong to the given `TSID` and cover the given time range.
|
||||
|
||||
`Parts` are periodically merged into bigger parts in background. The [background merge](https://victoriametrics.com/blog/vmstorage-retention-merging-deduplication/#merge-process) provides the following benefits:
|
||||
`Parts` are periodically merged into bigger parts in background. The background merge provides the following benefits:
|
||||
|
||||
* keeping the number of data files under control, so they don't exceed limits on open files
|
||||
* improved data compression, since bigger parts are usually compressed better than smaller parts
|
||||
@@ -1533,7 +1532,6 @@ are **eventually deleted** during [background merge](https://medium.com/@valyala
|
||||
The time range covered by data part is **not limited by retention period unit**. One data part can cover hours or days of
|
||||
data. Hence, a data part can be deleted only **when fully outside the configured retention**.
|
||||
See more about partitions and parts in the [Storage section](#storage).
|
||||
See [how retention frees disk space](https://victoriametrics.com/blog/vmstorage-retention-merging-deduplication/#retention-free-disk-space-guard-and-downsampling) for details.
|
||||
|
||||
The maximum disk space usage for a given `-retentionPeriod` is going to be (`-retentionPeriod` + 1) months.
|
||||
For example, if `-retentionPeriod` is set to 1, data for January is deleted on March 1st.
|
||||
@@ -1636,7 +1634,6 @@ See how to request a [free trial license](https://victoriametrics.com/products/e
|
||||
This command-line flag instructs leaving the last sample per each `interval` for [time series](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#time-series)
|
||||
[samples](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#raw-samples) older than the `offset`. The `offset` must be a multiple of `interval`. For example, `-downsampling.period=30d:5m` instructs leaving the last sample
|
||||
per each 5-minute interval for samples older than 30 days, while the rest of samples are dropped.
|
||||
See [Enterprise downsampling internals](https://victoriametrics.com/blog/vmstorage-retention-merging-deduplication/#retention-filters-and-downsampling-enterprise-plan) for details.
|
||||
|
||||
The `-downsampling.period` command-line flag can be specified multiple times in order to apply different downsampling levels for different time ranges (aka multi-level downsampling).
|
||||
For example, `-downsampling.period=30d:5m,180d:1h` instructs leaving the last sample per each 5-minute interval for samples older than 30 days,
|
||||
|
||||
@@ -40,7 +40,7 @@ requests_total{path="/", code="403"}
|
||||
The meta-information - a set of `labels` in curly braces - gives us a context for which `path` and with what `code`
|
||||
the `request` was served. Label-value pairs are always of a `string` type. VictoriaMetrics data model is schemaless,
|
||||
which means there is no need to define metric names or their labels in advance. The user is free to add or change ingested
|
||||
metrics anytime. See [metric names and labels explained](https://victoriametrics.com/blog/prometheus-monitoring-metrics-counters-gauges-histogram-summaries/#metric-name-and-labels) for details.
|
||||
metrics anytime.
|
||||
|
||||
Actually, the metric name is also a label with a special name `__name__`.
|
||||
The `__name__` key could be omitted {{% available_from "v1.111.0" %}} for simplicity. So the following series are identical:
|
||||
@@ -75,7 +75,7 @@ See [these docs](https://docs.victoriametrics.com/victoriametrics/faq/#what-is-h
|
||||
|
||||
#### Raw samples
|
||||
|
||||
Every unique time series may consist of an arbitrary number of `(value, timestamp)` data points (aka [`raw samples`](https://victoriametrics.com/blog/prometheus-monitoring-metrics-counters-gauges-histogram-summaries/#sample)) sorted by `timestamp`.
|
||||
Every unique time series may consist of an arbitrary number of `(value, timestamp)` data points (aka `raw samples`) sorted by `timestamp`.
|
||||
VictoriaMetrics stores all the `values` as [float64](https://en.wikipedia.org/wiki/Double-precision_floating-point_format)
|
||||
with [extra compression](https://faun.pub/victoriametrics-achieving-better-compression-for-time-series-data-than-gorilla-317bc1f95932) applied.
|
||||
This allows storing precise integer values with up to 12 decimal digits and any floating-point values with up to 12 significant decimal digits.
|
||||
@@ -126,7 +126,7 @@ type exists specifically to help users to understand how the metric was measured
|
||||
|
||||
#### Counter
|
||||
|
||||
[Counter](https://victoriametrics.com/blog/prometheus-monitoring-metrics-counters-gauges-histogram-summaries/#counter) is a metric, which counts some events. Its value increases or stays the same over time.
|
||||
Counter is a metric, which counts some events. Its value increases or stays the same over time.
|
||||
It cannot decrease in general case. The only exception is e.g. `counter reset`,
|
||||
when the metric resets to zero. The `counter reset` can occur when the service, which exposes the counter, restarts.
|
||||
So, the `counter` metric shows the number of observed events since the service start.
|
||||
@@ -157,7 +157,7 @@ by humans from other metric types.
|
||||
|
||||
#### Gauge
|
||||
|
||||
[Gauge](https://victoriametrics.com/blog/prometheus-monitoring-metrics-counters-gauges-histogram-summaries/#gauge) is used for measuring a value that can go up and down:
|
||||
Gauge is used for measuring a value that can go up and down:
|
||||
|
||||

|
||||
|
||||
@@ -178,7 +178,7 @@ and [rollup functions](https://docs.victoriametrics.com/victoriametrics/metricsq
|
||||
|
||||
#### Histogram
|
||||
|
||||
[Histogram](https://victoriametrics.com/blog/prometheus-monitoring-metrics-counters-gauges-histogram-summaries/#histogram) is a set of [counter](#counter) metrics with different `vmrange` or `le` labels.
|
||||
Histogram is a set of [counter](#counter) metrics with different `vmrange` or `le` labels.
|
||||
The `vmrange` or `le` labels define measurement boundaries of a particular bucket.
|
||||
When the observed measurement hits a particular bucket, then the corresponding counter is incremented.
|
||||
|
||||
@@ -282,7 +282,7 @@ We recommend reading the following articles before you start using histograms:
|
||||
|
||||
#### Summary
|
||||
|
||||
[Summary](https://victoriametrics.com/blog/prometheus-monitoring-metrics-counters-gauges-histogram-summaries/#summary) metric type is quite similar to [histogram](#histogram) and is used for
|
||||
Summary metric type is quite similar to [histogram](#histogram) and is used for
|
||||
[quantiles](https://prometheus.io/docs/practices/histograms/#quantiles) calculations. The main difference
|
||||
is that calculations are made on the client-side, so metrics exposition format already contains pre-defined
|
||||
quantiles:
|
||||
@@ -504,7 +504,6 @@ Params:
|
||||
The result of Instant query is a list of [time series](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#time-series)
|
||||
matching the filter in `query` expression. Each returned series contains exactly one `(timestamp, value)` entry,
|
||||
where `timestamp` equals to the `time` query arg, while the `value` contains `query` result at the requested `time`.
|
||||
See [instant vectors explained](https://victoriametrics.com/blog/prometheus-monitoring-instant-range-query/#instant-vector) for details.
|
||||
|
||||
To understand how instant queries work, let's begin with a data sample:
|
||||
|
||||
@@ -607,7 +606,6 @@ at `start`, `start+step`, `start+2*step`, ..., `start+N*step` timestamps. In oth
|
||||
executed independently at `start`, `start+step`, ..., `start+N*step` timestamps with the only difference that an instant query
|
||||
does not return `ephemeral` samples (see below). Instead, if the database does not contain any samples for the requested time and step,
|
||||
it simply returns an empty result.
|
||||
See [range vectors explained](https://victoriametrics.com/blog/prometheus-monitoring-instant-range-query/#range-vector) for details.
|
||||
|
||||
|
||||
For example, to get the values of `foo_bar` during the time range from `2022-05-10T07:59:00Z` to `2022-05-10T08:17:00Z`,
|
||||
|
||||
@@ -127,8 +127,7 @@ See [these docs](#how-to-collect-metrics-in-prometheus-format) for details.
|
||||
|
||||
`vmagent` can be used as an alternative to [StatsD](https://github.com/statsd/statsd)
|
||||
when [stream aggregation](https://docs.victoriametrics.com/victoriametrics/stream-aggregation/) is enabled.
|
||||
See [these docs](https://docs.victoriametrics.com/victoriametrics/stream-aggregation/#statsd-alternative) for details,
|
||||
or the blog post on [how vmagent's stream aggregation works](https://victoriametrics.com/blog/vmagent-key-features-explained/#stream-aggregation).
|
||||
See [these docs](https://docs.victoriametrics.com/victoriametrics/stream-aggregation/#statsd-alternative) for details.
|
||||
|
||||
### Flexible metrics relay
|
||||
|
||||
@@ -138,7 +137,7 @@ to other remote storage systems that support Prometheus `remote_write` protocol
|
||||
|
||||
### Replication and high availability
|
||||
|
||||
`vmagent` [replicates the collected metrics](https://victoriametrics.com/blog/vmagent-how-it-works/#step-4-sharding--replication) among multiple remote storage instances configured via `-remoteWrite.url` args.
|
||||
`vmagent` replicates the collected metrics among multiple remote storage instances configured via `-remoteWrite.url` args.
|
||||
If a single remote storage instance is temporarily unavailable, the collected data remains available on the other remote storage instances.
|
||||
`vmagent` buffers the collected data in files at `-remoteWrite.tmpDataPath` until the remote storage becomes available again.
|
||||
Then it sends the buffered data to the remote storage in order to prevent data gaps.
|
||||
@@ -151,8 +150,7 @@ See [these docs](https://docs.victoriametrics.com/victoriametrics/cluster-victor
|
||||
|
||||
`vmagent` can add, remove, or update labels on the collected data before sending it to the remote storage.
|
||||
It can filter scrape targets or remove unwanted samples via Prometheus-like relabeling.
|
||||
Please see the [Relabeling cookbook](https://docs.victoriametrics.com/victoriametrics/relabeling/) for details,
|
||||
or the blog post on [how vmagent applies global relabeling to reduce cardinality](https://victoriametrics.com/blog/vmagent-how-it-works/#step-2-global-relabeling-cardinality-reduction).
|
||||
Please see the [Relabeling cookbook](https://docs.victoriametrics.com/victoriametrics/relabeling/) for details.
|
||||
|
||||
### Sharding among remote storages
|
||||
|
||||
@@ -160,7 +158,6 @@ By default, `vmagent` replicates data to remote storage systems via the `-remote
|
||||
If the `-remoteWrite.shardByURL` command-line flag is set, then `vmagent` spreads
|
||||
the outgoing [time series](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#time-series) evenly among all the remote storage
|
||||
systems listed in `-remoteWrite.url`.
|
||||
See the blog post on [how vmagent shards data across remote storage systems](https://victoriametrics.com/blog/vmagent-key-features-explained/#sharding) for the details.
|
||||
|
||||
It is possible to replicate samples among remote storage systems by passing `-remoteWrite.shardByURLReplicas=N`
|
||||
to `vmagent` in addition to the `-remoteWrite.shardByURL` command-line flag.
|
||||
@@ -271,9 +268,7 @@ for the collected samples. Examples:
|
||||
```sh
|
||||
./vmagent -remoteWrite.url=http://remote-storage/api/v1/write -streamAggr.dropInputLabels=replica -streamAggr.dedupInterval=60s
|
||||
```
|
||||
|
||||
See the blog post on [how vmagent performs global deduplication and stream aggregation](https://victoriametrics.com/blog/vmagent-how-it-works/#step-3-global-deduplication--stream-aggregation) for the details.
|
||||
|
||||
|
||||
### Monitoring Data eXchange
|
||||
|
||||
The Monitoring Data eXchange (MDX){{% available_from "v1.147.0" %}} feature allows `vmagent` to forward only VictoriaMetrics metrics to selected `-remoteWrite.url` destinations while dropping metrics from non-VictoriaMetrics services.
|
||||
@@ -362,8 +357,6 @@ in addition to the pull-based Prometheus-compatible targets' scraping:
|
||||
* Prometheus exposition format via `http://<vmagent>:8429/api/v1/import/prometheus`. See [these docs](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#how-to-import-data-in-prometheus-exposition-format) for details.
|
||||
* Arbitrary CSV data via `http://<vmagent>:8429/api/v1/import/csv`. See [these docs](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#how-to-import-csv-data).
|
||||
|
||||
See the blog post on [how vmagent receives data via push APIs or scraping](https://victoriametrics.com/blog/vmagent-how-it-works/#step-1-receiving-data-via-api-or-scrape) for the details.
|
||||
|
||||
## How to collect metrics in Prometheus format
|
||||
|
||||
Specify the path to the `prometheus.yml` file via the `-promscrape.config` command-line flag. `vmagent` takes into account the following
|
||||
@@ -497,13 +490,15 @@ SRV URLs are useful when HTTP services run on different TCP ports or when their
|
||||
|
||||
When comparing the remote protocols between VictoriaMetrics and Prometheus, VictoriaMetrics provides the following benefits:
|
||||
|
||||
* Reduced network bandwidth usage by 2x-5x. This allows [saving network bandwidth usage costs](https://victoriametrics.com/blog/victoriametrics-remote-write/) when `vmagent` and
|
||||
* Reduced network bandwidth usage by 2x-5x. This allows saving network bandwidth usage costs when `vmagent` and
|
||||
the configured remote storage systems are located in different datacenters, availability zones, or regions.
|
||||
|
||||
* Reduced disk read/write IO and disk space usage at `vmagent` when the remote storage is temporarily unavailable.
|
||||
In this case, `vmagent` buffers incoming data to disk using the VictoriaMetrics remote write format.
|
||||
This reduces disk read/write IO and disk space usage by 2x-5x compared to the Prometheus remote write format.
|
||||
|
||||
> See blogpost [Save network costs with VictoriaMetrics remote write protocol](https://victoriametrics.com/blog/victoriametrics-remote-write/).
|
||||
|
||||
`vmagent` uses VictoriaMetrics remote write protocol by default {{% available_from "v1.116.0" %}} when it sends data to VictoriaMetrics components such as other `vmagent` instances,
|
||||
[single-node VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/)
|
||||
or `vminsert` at [cluster version](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/). If needed, it can automatically downgrade to a Prometheus protocol at runtime.
|
||||
@@ -840,7 +835,7 @@ as soon as it is parsed in stream parsing mode.
|
||||
## Scraping big number of targets
|
||||
|
||||
A single `vmagent` instance can scrape tens of thousands of scrape targets. Sometimes this isn't enough due to limitations on CPU, network, RAM, etc.
|
||||
In this case, scrape targets can be split among multiple `vmagent` instances (aka [`vmagent` horizontal scaling](https://victoriametrics.com/blog/vmagent-key-features-explained/#scaling-vmagent), sharding, and clustering).
|
||||
In this case, scrape targets can be split among multiple `vmagent` instances (aka `vmagent` horizontal scaling, sharding, and clustering).
|
||||
The number of `vmagent` instances in the cluster must be passed to the `-promscrape.cluster.membersCount` command-line flag.
|
||||
Each `vmagent` instance in the cluster must use identical `-promscrape.config` files with distinct `-promscrape.cluster.memberNum` values
|
||||
in the range `0 ... N-1`, where `N` is the number of `vmagent` instances in the cluster specified via `-promscrape.cluster.membersCount`.
|
||||
@@ -953,7 +948,7 @@ scrape_configs:
|
||||
|
||||
## On-disk persistence
|
||||
|
||||
`vmagent` [stores pending data that cannot be sent to the configured remote storage systems](https://victoriametrics.com/blog/vmagent-key-features-explained/#persistent-disk-for-remote-write) in a timely manner.
|
||||
`vmagent` stores pending data that cannot be sent to the configured remote storage systems in a timely manner.
|
||||
By default, `vmagent` writes all the pending data to the folder configured via `-remoteWrite.tmpDataPath` cmd-line flag
|
||||
until this data is sent to the configured `-remoteWrite.url` systems or until the folder becomes full.
|
||||
The maximum data size that can be saved to `-remoteWrite.tmpDataPath` per every configured `-remoteWrite.url` can be
|
||||
@@ -1001,7 +996,7 @@ moment it becomes visible at the remote storage.
|
||||
|
||||
This behavior can be changed with the `-remoteWrite.inmemoryQueues` {{% available_from "v1.146.0" %}} command-line flag.
|
||||
When set to a non-zero value, vmagent starts the given number of additional workers,
|
||||
which send only recently ingested data from the [in-memory queue](https://victoriametrics.com/blog/vmagent-how-it-works/#in-memory-queue), while the workers configured via `-remoteWrite.queues` drain the file-based backlog concurrently.
|
||||
which send only recently ingested data from the in-memory queue, while the workers configured via `-remoteWrite.queues` drain the file-based backlog concurrently.
|
||||
This reduces the delivery lag for fresh samples after remote storage outages or slowdowns. The flag can be set individually per each `-remoteWrite.url`.
|
||||
|
||||
Note that these workers are started in addition to the workers configured via `-remoteWrite.queues`, so the total number of concurrent connections to
|
||||
|
||||
@@ -209,8 +209,6 @@ rules:
|
||||
[ debug: <bool> | default = false ]
|
||||
```
|
||||
|
||||
See a blogpost about [how rule groups work](https://victoriametrics.com/blog/alerting-recording-rules-alertmanager/#groups).
|
||||
|
||||
### Rules
|
||||
|
||||
Every rule contains an `expr` field for the expression to evaluate against the configured datasource.
|
||||
@@ -287,8 +285,6 @@ annotations:
|
||||
[ <labelname>: <tmpl_string> ]
|
||||
```
|
||||
|
||||
See a blogpost about [how alerting rules work](https://victoriametrics.com/blog/alerting-recording-rules-alertmanager/#alerting-rules).
|
||||
|
||||
#### Recording rules
|
||||
|
||||
The syntax for recording rules is the following:
|
||||
@@ -326,13 +322,10 @@ labels:
|
||||
|
||||
For recording rules to work `-remoteWrite.url` must be specified.
|
||||
|
||||
See a blogpost about [how recording rules work](https://victoriametrics.com/blog/alerting-recording-rules-alertmanager/#recording-rules).
|
||||
|
||||
## Templating
|
||||
|
||||
It is allowed to use [Go templating](https://golang.org/pkg/text/template/) in annotations and labels(with limited support) to format data, iterate over
|
||||
or execute expressions.
|
||||
See a blogpost about [templating alerts](https://victoriametrics.com/blog/alerting-recording-rules-alertmanager/#templates).
|
||||
The following variables are available in templating:
|
||||
|
||||
| Variable | Description | Example |
|
||||
@@ -935,7 +928,7 @@ See full description for these flags in `./vmalert -help`.
|
||||
|
||||
## Unit Testing for Rules
|
||||
|
||||
You can use `vmalert-tool` to [test your alerting and recording rules](https://victoriametrics.com/blog/alerting-best-practices/#testing-alerts) like [promtool does](https://prometheus.io/docs/prometheus/latest/configuration/unit_testing_rules/).
|
||||
You can use `vmalert-tool` to test your alerting and recording rules like [promtool does](https://prometheus.io/docs/prometheus/latest/configuration/unit_testing_rules/).
|
||||
See more details in [vmalert-tool](https://docs.victoriametrics.com/victoriametrics/vmalert-tool/#unit-testing-for-rules).
|
||||
|
||||
## Monitoring
|
||||
@@ -989,8 +982,6 @@ Try the following tips to avoid common issues:
|
||||
In that case, the default step will be used (`-datasource.queryStep`) and may cause unexpected results compared to
|
||||
executing this query in vmui/Grafana, where step is adjusted differently.
|
||||
|
||||
See a blogpost about [reducing alert noise](https://victoriametrics.com/blog/alerting-best-practices/#reducing-noise).
|
||||
|
||||
### Rule state
|
||||
|
||||
vmalert keeps the last `-rule.updateEntriesLimit` updates (or `update_entries_limit` [per-rule config](https://docs.victoriametrics.com/victoriametrics/vmalert/#alerting-rules))
|
||||
@@ -1046,8 +1037,6 @@ Sometimes, it's hard to understand why a specific alert fired or not. Keep in mi
|
||||
If evaluation returns error (i.e. datasource is unavailable), alert state doesn't change.
|
||||
If at least one evaluation returns no data, then alert's `for` state resets.
|
||||
|
||||
See a blogpost about [tuning the `for` param](https://victoriametrics.com/blog/alerting-best-practices/#the-for-param).
|
||||
|
||||
> Note: The alert state is tracked separately for each time series returned during evaluation.
|
||||
> For example, if the 1st evaluation returns series A and B, and the 2nd evaluation returns only B – the alert will remain active **only for B**.
|
||||
|
||||
@@ -1121,8 +1110,6 @@ How to reduce the chance for a rule to flap:
|
||||
|
||||
See [common mistakes](#common-mistakes) for rules config.
|
||||
|
||||
See a blogpost about [tuning `keep_firing_for`](https://victoriametrics.com/blog/alerting-best-practices/#the-keep_firing_for-param).
|
||||
|
||||
### Never-firing alerts
|
||||
|
||||
vmalert can detect {{% available_from "v1.91.0" %}} if alert's expression doesn't match any time series in runtime.
|
||||
@@ -1373,8 +1360,6 @@ The list of configured or discovered Notifiers can be explored via [UI](#web).
|
||||
If Alertmanager runs in cluster mode then all its URLs needs to be available during discovery
|
||||
to ensure [high availability](https://github.com/prometheus/alertmanager#high-availability).
|
||||
|
||||
See a blogpost about [how Alertmanager processes alerts](https://victoriametrics.com/blog/alerting-recording-rules-alertmanager/#alertmanager).
|
||||
|
||||
The configuration file [specification](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmalert/notifier/config.go)
|
||||
is the following:
|
||||
|
||||
|
||||
@@ -3479,7 +3479,6 @@ func TestStorageAddRows_SamplesWithZeroDate(t *testing.T) {
|
||||
|
||||
func TestStorageAddRows_currHourMetricIDs(t *testing.T) {
|
||||
defer testRemoveAll(t)
|
||||
|
||||
f := func(t *testing.T, disablePerDayIndex bool) {
|
||||
t.Helper()
|
||||
|
||||
|
||||
Reference in New Issue
Block a user