Compare commits

..

5 Commits

Author SHA1 Message Date
Pablo Fernandez
e19d09ce9d Apply suggestions from cubic 2026-08-14 17:01:36 +01:00
Victoria Nduka
f4c13d3e11 Apply suggestions from code review
Co-authored-by: Pablo (Tomas) Fernandez <46322567+TomFern@users.noreply.github.com>
Signed-off-by: Victoria Nduka <122698422+nwanduka@users.noreply.github.com>
2026-08-14 14:46:19 +01:00
Victoria Nduka
1f9d32a28c Merge branch 'master' into nwanduka-patch-2
Signed-off-by: Victoria Nduka <122698422+nwanduka@users.noreply.github.com>
2026-08-14 09:11:20 +01:00
Victoria Nduka
6a975da6ed Merge branch 'master' into nwanduka-patch-2 2026-08-11 22:03:13 +01:00
Victoria Nduka
b346d0437e Update links and wording in keyConcepts.md
Signed-off-by: Victoria Nduka <122698422+nwanduka@users.noreply.github.com>
2026-08-11 22:01:00 +01:00
6 changed files with 43 additions and 96 deletions

View File

@@ -1,4 +1,4 @@
FROM golang:1.26.6 AS build-web-stage
FROM golang:1.26.5 AS build-web-stage
COPY build /build
WORKDIR /build

View File

@@ -7,7 +7,7 @@ ROOT_IMAGE ?= alpine:3.24.1
ROOT_IMAGE_SCRATCH ?= scratch
CERTS_IMAGE := alpine:3.24.1
GO_BUILDER_IMAGE := golang:1.26.6
GO_BUILDER_IMAGE := golang:1.26.5
BUILDER_IMAGE := local/builder:2.0.0-$(shell echo $(GO_BUILDER_IMAGE) | tr :/ __)-1
BASE_IMAGE := local/base:1.1.4-$(shell echo $(ROOT_IMAGE) | tr :/ __)-$(shell echo $(CERTS_IMAGE) | tr :/ __)

View File

@@ -1266,96 +1266,47 @@ See also [resource usage limits at VictoriaMetrics cluster](https://docs.victori
## High availability
VictoriaMetrics supports high availability for both writes and reads by combining replication with multiple instances.
The general approach for achieving high availability is the following:
You can achieve **high availability for writes** using replication:
* To run two identically configured VictoriaMetrics instances in distinct datacenters (availability zones);
* To store the collected data simultaneously into these instances via [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/) or Prometheus.
* To query the first VictoriaMetrics instance and to fail over to the second instance when the first instance becomes temporarily unavailable.
This can be done via [vmauth](https://docs.victoriametrics.com/victoriametrics/vmauth/) according to [these docs](https://docs.victoriametrics.com/victoriametrics/vmauth/#high-availability).
* Run two or more identically configured VictoriaMetrics instances in distinct datacenters (availability zones);
* Replicate collected metrics simultaneously into all these instances via one or more [vmagents](https://docs.victoriametrics.com/victoriametrics/vmagent/).
Such a setup guarantees that the collected data isn't lost when one of VictoriaMetrics instance becomes unavailable.
The collected data continues to be written to the available VictoriaMetrics instance, so it should be available for querying.
Both [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/) and Prometheus buffer the collected data locally if they cannot send it
to the configured remote storage. So the collected data will be written to the temporarily unavailable VictoriaMetrics instance
after it becomes available.
If you use [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/) for storing the data into VictoriaMetrics,
then it can be configured with multiple `-remoteWrite.url` command-line flags, where every flag points to the VictoriaMetrics
instance in a particular availability zone, in order to replicate the collected data to all the VictoriaMetrics instances.
For example, the following command instructs `vmagent` to replicate data to `vm-az1` and `vm-az2` instances of VictoriaMetrics:
In this setup, configure vmagent [to replicate data](https://docs.victoriametrics.com/victoriametrics/vmagent/#replication-and-high-availability)
to each remote destination:
```sh
/path/to/vmagent \
-remoteWrite.url=https://victoriametrics-1:8428/api/v1/write \
-remoteWrite.url=https://victoriametrics-2:8428/api/v1/write
-remoteWrite.url=http://<vm-az1>:8428/api/v1/write \
-remoteWrite.url=http://<vm-az2>:8428/api/v1/write
```
Each `--remoteWrite.url` creates its own replication queue. The queue temporarily stores data on disk while a remote destination is unavailable.
See more about [on-disk persistence in vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/#on-disk-persistence).
If you use Prometheus for collecting and writing the data to VictoriaMetrics,
then the following [`remote_write`](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write) section
in Prometheus config can be used for replicating the collected data to `vm-az1` and `vm-az2` VictoriaMetrics instances:
When the remote destination becomes available, vmagent drains the queue and restores data consistency across destinations.
> The max size of the on-disk queue can be increased by [horizontally sharding vmagents](https://docs.victoriametrics.com/victoriametrics/vmagent/#scraping-big-number-of-targets).
> To achieve high availability for vmagent itself, run multiple identically configured vmagent replicas.
> In this case, the load on the remote destinations will increase proportionally to the number of vmagent replicas. The duplicated data in remote destinations
> has to be [deduplicated](https://docs.victoriametrics.com/victoriametrics/#deduplication) on the VictoriaMetrics side.
You can achieve **high availability for reads** by choosing one of the following options:
- Load balancer: Use a load balancer to ensure read operations are always routed to an available VictoriaMetrics instance.
- Top-level vmselect: Use vmselect to query all available VictoriaMetrics instances and merge the results
### Load balancer
In this mode, we use a load balancer to query the main VictoriaMetrics instance and fail over to a secondary instance if the first one becomes temporarily unavailable.
This can be done using [vmauth](https://docs.victoriametrics.com/victoriametrics/vmauth/) configured in [high-availability mode](https://docs.victoriametrics.com/victoriametrics/vmauth/#high-availability).
```mermaid
flowchart LR
Client["Query Client<br/>Grafana/vmalert"]
VMAUTH["vmauth<br/>Load Balancer / Failover"]
VM1["VictoriaMetrics-1<br/>Primary read target"]
VM2["VictoriaMetrics-2<br/>Failover read target"]
Client -->|"Read query"| VMAUTH
VMAUTH -->|"1. Send queries"| VM1
VMAUTH -.->|"2. Fail over if VM1<br/>is unavailable"| VM2
```yaml
remote_write:
- url: http://<vm-az1>:8428/api/v1/write
- url: http://<vm-az2>:8428/api/v1/write
```
This is the most cost-efficient option because only one VictoriaMetrics instance is queried at a time.
It is recommended to use [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/) instead of Prometheus for highly loaded setups,
since it uses lower amounts of RAM, CPU and network bandwidth than Prometheus.
The downside is that when one instance goes down and then comes back up, the load balancer may immediately start sending read queries to the recovering instance, even though it hasn't caught up yet. During this short recovery window:
If you use identically configured [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/) instances for collecting the same data
and sending it to VictoriaMetrics, then do not forget enabling [deduplication](#deduplication) at VictoriaMetrics side.
- The recovering VictoriaMetrics instance has not yet ingested all the data that was queued while it was down.
- vmagent nodes may still be draining their backlogged queues into that instance
- Read queries routed to the recovering instance can return incomplete results, because some recent data may still be in transit or not yet visible.
### Top-level vmselect
In this option, we use a top-level [vmselect](https://docs.victoriametrics.com/victoriametrics/vmselect/) to query all remote destinations simultaneously and merge the results.
This option is only possible if VictoriaMetrics single-node instances are configured with the `-vmselectAddr` flag. See more details in the [VictoriaMetrics multi-tenancy section](https://docs.victoriametrics.com/victoriametrics/#multi-tenancy).
```mermaid
flowchart LR
Client["Query Client<br/>Grafana / vmalert"]
VMSELECT["vmselect<br/>Query all destinations<br/>
<code><pre>-dedup.minScrapeInterval=1ms<br/>-replicationFactor=2</pre></code>"]
VM1["VictoriaMetrics-1<br/>Single-node<br/><code>-vmselectAddr=:8401</code>"]
VM2["VictoriaMetrics-2<br/>Single-node<br/><code>-vmselectAddr=:8401</code>"]
Client -->|"Read query"| VMSELECT
VMSELECT --> VM1
VMSELECT --> VM2
VMSELECT -->|"Merged and deduplicated results"| Client
```
This option requires more resources because it queries all remote destinations simultaneously and merges their responses before returning the final result.
The benefit is that it can handle data gaps across destinations by merging responses from all VictoriaMetrics instances. Thus, a single recovering instance is less likely to cause incomplete results, unlike in the load balancer case.
Since vmselect queries identical data across multiple VictoriaMetrics instances, configure it with `-dedup.minScrapeInterval=1ms` to remove duplicate samples during merging. Also set `-replicationFactor=N` on vmselect, where `N` equals the number of remote storage destinations, so that queries can tolerate the unavailability of up to `N-1` destinations.
See [VMDistributed](https://docs.victoriametrics.com/operator/resources/vmdistributed/) Kubernetes operator resource for an example of configuring vmselect with deduplication and replication.
See [VMDistributed](https://docs.victoriametrics.com/operator/resources/vmdistributed/) Kubernetes operator resource for an example.
## Deduplication

View File

@@ -28,8 +28,6 @@ See also [LTS releases](https://docs.victoriametrics.com/victoriametrics/lts-rel
**Update Note 1:** `vmselect` and `vminsert` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/), and `vmagent`: default value of `-enableMultitenancyViaHeaders` command-line flag has changed from `false` to `true`. This change enables support of [multitenancy via headers for cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#multitenancy-via-headers) and [for vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/#multitenancy-via-headers) by default. With this change, mentioned components will start supporting URLs with omitted tenant ID in the path: `https://<vmselect>:8481/select/prometheus/api/v1/query` will become a valid URL. To disable multitenancy via headers and simplified URLs set `--enableMultitenancyViaHeaders=false` on vmagent, vminsert and vmselect.
* SECURITY: upgrade Go builder from Go1.26.5 to Go1.26.6. See [the list of issues addressed in Go1.26.6](https://github.com/golang/go/issues?q=milestone%3AGo1.26.6%20label%3ACherryPickApproved).
* FEATURE: [relabeling](https://docs.victoriametrics.com/victoriametrics/relabeling/): reduce CPU usage up to 30% when matching relabeling rules with multiple `if` expressions containing exact metric names. Expressions for other metric names are now skipped before evaluating their remaining label filters. See [#11341](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/11341). Thanks to @nevgeny for contribution.
* FEATURE: [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/) and [vmsingle](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/): add support for [linode_sd_configs](https://docs.victoriametrics.com/victoriametrics/sd_configs/#linode_sd_configs) for discovering scrape targets from Linode instances. See [#9118](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/9118). Thanks to @cxdy for contribution.
* FEATURE: [vmalert](https://docs.victoriametrics.com/victoriametrics/vmalert/): extend `-replay.continueWithExecutionErr` to also handle the `400 Bad Request` response code, since it is used for Prometheus querying API requests when request parameters are missing or incorrect. See [#11352](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/11352).

View File

@@ -71,8 +71,7 @@ See [what is an active time series](https://docs.victoriametrics.com/victoriamet
#### Cardinality
The number of unique [time series](#time-series) is named `cardinality`. Having too many unique time series is named `high cardinality`.
High cardinality may result in increased resource usage in VictoriaMetrics.
See [these docs](https://docs.victoriametrics.com/victoriametrics/faq/#what-is-high-cardinality) for more details.
[High cardinality](https://docs.victoriametrics.com/victoriametrics/faq/#what-is-high-cardinality) may result in increased resource usage in VictoriaMetrics.
#### Raw samples
@@ -307,7 +306,7 @@ Such an approach makes summaries easier to use but also puts significant limitat
- It is impossible to calculate a quantile over multiple summary metrics, e.g. `sum(go_gc_duration_seconds{quantile="0.75"})`,
`avg(go_gc_duration_seconds{quantile="0.75"})` or `max(go_gc_duration_seconds{quantile="0.75"})`
won't return the expected 75th percentile over `go_gc_duration_seconds` metrics collected from multiple instances
of the application. See [this article](https://latencytipoftheday.blogspot.de/2014/06/latencytipoftheday-you-cant-average.html) for details.
of the application. See [Latency Tip of the Day: You Can't Average Percentiles](https://latencytipoftheday.blogspot.de/2014/06/latencytipoftheday-you-cant-average.html) for details.
- It is impossible to calculate quantiles other than the already pre-calculated quantiles.
@@ -322,9 +321,9 @@ As was said at the beginning of the [types of metrics](#types-of-metrics) sectio
measured. VictoriaMetrics TSDB doesn't know about metric types. All it sees are metric names, labels, values, and timestamps.
What these metrics are, what they measure, and how - all these depend on the application which emits them.
To instrument your application with metrics compatible with VictoriaMetrics, we recommend
To instrument your application with metrics compatible with VictoriaMetrics we recommend
using the [github.com/VictoriaMetrics/metrics](https://github.com/VictoriaMetrics/metrics) package.
See more details on how to use it in [this article](https://victoriametrics.medium.com/how-to-monitor-go-applications-with-victoriametrics-c04703110870).
See [How to monitor Go applications with VictoriaMetrics](https://victoriametrics.medium.com/how-to-monitor-go-applications-with-victoriametrics-c04703110870).
VictoriaMetrics is also compatible with [Prometheus client libraries for metrics instrumentation](https://prometheus.io/docs/instrumenting/clientlibs/).
@@ -422,7 +421,7 @@ In the pull model, the monitoring system needs to be aware of all the applicatio
scraped (pulled) from the known applications (aka `scrape targets`) via HTTP protocol on a regular basis (aka `scrape_interval`).
VictoriaMetrics supports discovering Prometheus-compatible targets and scraping metrics from them in the same way as Prometheus does -
see [these docs](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#how-to-scrape-prometheus-exporters-such-as-node-exporter).
see [how to scrape Prometheus exporters in VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#how-to-scrape-prometheus-exporters-such-as-node-exporter).
Metrics scraping is supported by [single-node VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#how-to-scrape-prometheus-exporters-such-as-node-exporter)
and by [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/).
@@ -776,7 +775,7 @@ VictoriaMetrics provides a special query language for executing read queries - [
It is a [PromQL](https://prometheus.io/docs/prometheus/latest/querying/basics)-like query language with a powerful set of
functions and features for working specifically with time series data. MetricsQL is backward-compatible with PromQL,
so it shares most of the query concepts. The basic concepts for PromQL and MetricsQL are
described [here](https://valyala.medium.com/promql-tutorial-for-beginners-9ab455142085).
described in this [PromQL tutorial for beginners](https://valyala.medium.com/promql-tutorial-for-beginners-9ab455142085).
#### Filtering
@@ -883,7 +882,7 @@ query may break or may lead to incorrect results. The basics of the matching rul
with the same set of labels, applies the operation for each data point, and returns the resulting time series with the
same set of labels. If there are no matches, then the time series is dropped from the result.
* The matching rules may be augmented with `ignoring`, `on`, `group_left` and `group_right` modifiers.
See [these docs](https://prometheus.io/docs/prometheus/latest/querying/operators/#vector-matching) for details.
See [Prometheus's vector matching documentation](https://prometheus.io/docs/prometheus/latest/querying/operators/#vector-matching) for details.
#### Comparison operations
@@ -975,15 +974,14 @@ See [How to delete time series](https://docs.victoriametrics.com/victoriametrics
### Relabeling
Relabeling is a powerful mechanism for modifying time series before they have been written to the database. Relabeling
may be applied for both [push](#push-model) and [pull](#pull-model) models. See more
details [here](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#relabeling).
[Relabeling](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#relabeling) is a powerful mechanism for modifying time series before they have been written to the database. Relabeling
may be applied for both [push](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#push-model) and [pull](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#pull-model) models.
### Deduplication
VictoriaMetrics supports data deduplication. See [these docs](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#deduplication).
VictoriaMetrics supports data [deduplication](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#deduplication).
### Downsampling
VictoriaMetrics supports data downsampling. See [these docs](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#downsampling).
VictoriaMetrics Enterprise supports data [downsampling](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#downsampling). Downsampling can reduce disk space usage and improve query performance by reducing the number samples in a time series.

2
go.mod
View File

@@ -1,6 +1,6 @@
module github.com/VictoriaMetrics/VictoriaMetrics
go 1.26.6
go 1.26.5
require (
cloud.google.com/go/storage v1.64.0