mirror of
https://github.com/VictoriaMetrics/VictoriaMetrics.git
synced 2026-08-15 04:02:25 +03:00
Compare commits
5 Commits
docs-singl
...
nwanduka-p
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
e19d09ce9d | ||
|
|
f4c13d3e11 | ||
|
|
1f9d32a28c | ||
|
|
6a975da6ed | ||
|
|
b346d0437e |
@@ -1,4 +1,4 @@
|
||||
FROM golang:1.26.6 AS build-web-stage
|
||||
FROM golang:1.26.5 AS build-web-stage
|
||||
COPY build /build
|
||||
|
||||
WORKDIR /build
|
||||
|
||||
@@ -7,7 +7,7 @@ ROOT_IMAGE ?= alpine:3.24.1
|
||||
ROOT_IMAGE_SCRATCH ?= scratch
|
||||
CERTS_IMAGE := alpine:3.24.1
|
||||
|
||||
GO_BUILDER_IMAGE := golang:1.26.6
|
||||
GO_BUILDER_IMAGE := golang:1.26.5
|
||||
|
||||
BUILDER_IMAGE := local/builder:2.0.0-$(shell echo $(GO_BUILDER_IMAGE) | tr :/ __)-1
|
||||
BASE_IMAGE := local/base:1.1.4-$(shell echo $(ROOT_IMAGE) | tr :/ __)-$(shell echo $(CERTS_IMAGE) | tr :/ __)
|
||||
|
||||
@@ -1266,96 +1266,47 @@ See also [resource usage limits at VictoriaMetrics cluster](https://docs.victori
|
||||
|
||||
## High availability
|
||||
|
||||
VictoriaMetrics supports high availability for both writes and reads by combining replication with multiple instances.
|
||||
The general approach for achieving high availability is the following:
|
||||
|
||||
You can achieve **high availability for writes** using replication:
|
||||
* To run two identically configured VictoriaMetrics instances in distinct datacenters (availability zones);
|
||||
* To store the collected data simultaneously into these instances via [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/) or Prometheus.
|
||||
* To query the first VictoriaMetrics instance and to fail over to the second instance when the first instance becomes temporarily unavailable.
|
||||
This can be done via [vmauth](https://docs.victoriametrics.com/victoriametrics/vmauth/) according to [these docs](https://docs.victoriametrics.com/victoriametrics/vmauth/#high-availability).
|
||||
|
||||
* Run two or more identically configured VictoriaMetrics instances in distinct datacenters (availability zones);
|
||||
* Replicate collected metrics simultaneously into all these instances via one or more [vmagents](https://docs.victoriametrics.com/victoriametrics/vmagent/).
|
||||
Such a setup guarantees that the collected data isn't lost when one of VictoriaMetrics instance becomes unavailable.
|
||||
The collected data continues to be written to the available VictoriaMetrics instance, so it should be available for querying.
|
||||
Both [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/) and Prometheus buffer the collected data locally if they cannot send it
|
||||
to the configured remote storage. So the collected data will be written to the temporarily unavailable VictoriaMetrics instance
|
||||
after it becomes available.
|
||||
|
||||
If you use [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/) for storing the data into VictoriaMetrics,
|
||||
then it can be configured with multiple `-remoteWrite.url` command-line flags, where every flag points to the VictoriaMetrics
|
||||
instance in a particular availability zone, in order to replicate the collected data to all the VictoriaMetrics instances.
|
||||
For example, the following command instructs `vmagent` to replicate data to `vm-az1` and `vm-az2` instances of VictoriaMetrics:
|
||||
|
||||
In this setup, configure vmagent [to replicate data](https://docs.victoriametrics.com/victoriametrics/vmagent/#replication-and-high-availability)
|
||||
to each remote destination:
|
||||
```sh
|
||||
/path/to/vmagent \
|
||||
-remoteWrite.url=https://victoriametrics-1:8428/api/v1/write \
|
||||
-remoteWrite.url=https://victoriametrics-2:8428/api/v1/write
|
||||
-remoteWrite.url=http://<vm-az1>:8428/api/v1/write \
|
||||
-remoteWrite.url=http://<vm-az2>:8428/api/v1/write
|
||||
```
|
||||
|
||||
Each `--remoteWrite.url` creates its own replication queue. The queue temporarily stores data on disk while a remote destination is unavailable.
|
||||
See more about [on-disk persistence in vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/#on-disk-persistence).
|
||||
If you use Prometheus for collecting and writing the data to VictoriaMetrics,
|
||||
then the following [`remote_write`](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write) section
|
||||
in Prometheus config can be used for replicating the collected data to `vm-az1` and `vm-az2` VictoriaMetrics instances:
|
||||
|
||||
When the remote destination becomes available, vmagent drains the queue and restores data consistency across destinations.
|
||||
|
||||
> The max size of the on-disk queue can be increased by [horizontally sharding vmagents](https://docs.victoriametrics.com/victoriametrics/vmagent/#scraping-big-number-of-targets).
|
||||
> To achieve high availability for vmagent itself, run multiple identically configured vmagent replicas.
|
||||
> In this case, the load on the remote destinations will increase proportionally to the number of vmagent replicas. The duplicated data in remote destinations
|
||||
> has to be [deduplicated](https://docs.victoriametrics.com/victoriametrics/#deduplication) on the VictoriaMetrics side.
|
||||
|
||||
You can achieve **high availability for reads** by choosing one of the following options:
|
||||
|
||||
- Load balancer: Use a load balancer to ensure read operations are always routed to an available VictoriaMetrics instance.
|
||||
- Top-level vmselect: Use vmselect to query all available VictoriaMetrics instances and merge the results
|
||||
|
||||
### Load balancer
|
||||
|
||||
In this mode, we use a load balancer to query the main VictoriaMetrics instance and fail over to a secondary instance if the first one becomes temporarily unavailable.
|
||||
|
||||
This can be done using [vmauth](https://docs.victoriametrics.com/victoriametrics/vmauth/) configured in [high-availability mode](https://docs.victoriametrics.com/victoriametrics/vmauth/#high-availability).
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
Client["Query Client<br/>Grafana/vmalert"]
|
||||
|
||||
VMAUTH["vmauth<br/>Load Balancer / Failover"]
|
||||
|
||||
VM1["VictoriaMetrics-1<br/>Primary read target"]
|
||||
VM2["VictoriaMetrics-2<br/>Failover read target"]
|
||||
|
||||
Client -->|"Read query"| VMAUTH
|
||||
|
||||
VMAUTH -->|"1. Send queries"| VM1
|
||||
VMAUTH -.->|"2. Fail over if VM1<br/>is unavailable"| VM2
|
||||
```yaml
|
||||
remote_write:
|
||||
- url: http://<vm-az1>:8428/api/v1/write
|
||||
- url: http://<vm-az2>:8428/api/v1/write
|
||||
```
|
||||
|
||||
This is the most cost-efficient option because only one VictoriaMetrics instance is queried at a time.
|
||||
It is recommended to use [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/) instead of Prometheus for highly loaded setups,
|
||||
since it uses lower amounts of RAM, CPU and network bandwidth than Prometheus.
|
||||
|
||||
The downside is that when one instance goes down and then comes back up, the load balancer may immediately start sending read queries to the recovering instance, even though it hasn't caught up yet. During this short recovery window:
|
||||
If you use identically configured [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/) instances for collecting the same data
|
||||
and sending it to VictoriaMetrics, then do not forget enabling [deduplication](#deduplication) at VictoriaMetrics side.
|
||||
|
||||
- The recovering VictoriaMetrics instance has not yet ingested all the data that was queued while it was down.
|
||||
- vmagent nodes may still be draining their backlogged queues into that instance
|
||||
- Read queries routed to the recovering instance can return incomplete results, because some recent data may still be in transit or not yet visible.
|
||||
|
||||
### Top-level vmselect
|
||||
|
||||
In this option, we use a top-level [vmselect](https://docs.victoriametrics.com/victoriametrics/vmselect/) to query all remote destinations simultaneously and merge the results.
|
||||
|
||||
This option is only possible if VictoriaMetrics single-node instances are configured with the `-vmselectAddr` flag. See more details in the [VictoriaMetrics multi-tenancy section](https://docs.victoriametrics.com/victoriametrics/#multi-tenancy).
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
Client["Query Client<br/>Grafana / vmalert"]
|
||||
|
||||
VMSELECT["vmselect<br/>Query all destinations<br/>
|
||||
<code><pre>-dedup.minScrapeInterval=1ms<br/>-replicationFactor=2</pre></code>"]
|
||||
|
||||
VM1["VictoriaMetrics-1<br/>Single-node<br/><code>-vmselectAddr=:8401</code>"]
|
||||
VM2["VictoriaMetrics-2<br/>Single-node<br/><code>-vmselectAddr=:8401</code>"]
|
||||
|
||||
Client -->|"Read query"| VMSELECT
|
||||
|
||||
VMSELECT --> VM1
|
||||
VMSELECT --> VM2
|
||||
|
||||
VMSELECT -->|"Merged and deduplicated results"| Client
|
||||
```
|
||||
|
||||
This option requires more resources because it queries all remote destinations simultaneously and merges their responses before returning the final result.
|
||||
|
||||
The benefit is that it can handle data gaps across destinations by merging responses from all VictoriaMetrics instances. Thus, a single recovering instance is less likely to cause incomplete results, unlike in the load balancer case.
|
||||
|
||||
Since vmselect queries identical data across multiple VictoriaMetrics instances, configure it with `-dedup.minScrapeInterval=1ms` to remove duplicate samples during merging. Also set `-replicationFactor=N` on vmselect, where `N` equals the number of remote storage destinations, so that queries can tolerate the unavailability of up to `N-1` destinations.
|
||||
|
||||
See [VMDistributed](https://docs.victoriametrics.com/operator/resources/vmdistributed/) Kubernetes operator resource for an example of configuring vmselect with deduplication and replication.
|
||||
See [VMDistributed](https://docs.victoriametrics.com/operator/resources/vmdistributed/) Kubernetes operator resource for an example.
|
||||
|
||||
## Deduplication
|
||||
|
||||
|
||||
@@ -28,8 +28,6 @@ See also [LTS releases](https://docs.victoriametrics.com/victoriametrics/lts-rel
|
||||
|
||||
**Update Note 1:** `vmselect` and `vminsert` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/), and `vmagent`: default value of `-enableMultitenancyViaHeaders` command-line flag has changed from `false` to `true`. This change enables support of [multitenancy via headers for cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#multitenancy-via-headers) and [for vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/#multitenancy-via-headers) by default. With this change, mentioned components will start supporting URLs with omitted tenant ID in the path: `https://<vmselect>:8481/select/prometheus/api/v1/query` will become a valid URL. To disable multitenancy via headers and simplified URLs set `--enableMultitenancyViaHeaders=false` on vmagent, vminsert and vmselect.
|
||||
|
||||
* SECURITY: upgrade Go builder from Go1.26.5 to Go1.26.6. See [the list of issues addressed in Go1.26.6](https://github.com/golang/go/issues?q=milestone%3AGo1.26.6%20label%3ACherryPickApproved).
|
||||
|
||||
* FEATURE: [relabeling](https://docs.victoriametrics.com/victoriametrics/relabeling/): reduce CPU usage up to 30% when matching relabeling rules with multiple `if` expressions containing exact metric names. Expressions for other metric names are now skipped before evaluating their remaining label filters. See [#11341](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/11341). Thanks to @nevgeny for contribution.
|
||||
* FEATURE: [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/) and [vmsingle](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/): add support for [linode_sd_configs](https://docs.victoriametrics.com/victoriametrics/sd_configs/#linode_sd_configs) for discovering scrape targets from Linode instances. See [#9118](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/9118). Thanks to @cxdy for contribution.
|
||||
* FEATURE: [vmalert](https://docs.victoriametrics.com/victoriametrics/vmalert/): extend `-replay.continueWithExecutionErr` to also handle the `400 Bad Request` response code, since it is used for Prometheus querying API requests when request parameters are missing or incorrect. See [#11352](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/11352).
|
||||
|
||||
@@ -71,8 +71,7 @@ See [what is an active time series](https://docs.victoriametrics.com/victoriamet
|
||||
#### Cardinality
|
||||
|
||||
The number of unique [time series](#time-series) is named `cardinality`. Having too many unique time series is named `high cardinality`.
|
||||
High cardinality may result in increased resource usage in VictoriaMetrics.
|
||||
See [these docs](https://docs.victoriametrics.com/victoriametrics/faq/#what-is-high-cardinality) for more details.
|
||||
[High cardinality](https://docs.victoriametrics.com/victoriametrics/faq/#what-is-high-cardinality) may result in increased resource usage in VictoriaMetrics.
|
||||
|
||||
#### Raw samples
|
||||
|
||||
@@ -307,7 +306,7 @@ Such an approach makes summaries easier to use but also puts significant limitat
|
||||
- It is impossible to calculate a quantile over multiple summary metrics, e.g. `sum(go_gc_duration_seconds{quantile="0.75"})`,
|
||||
`avg(go_gc_duration_seconds{quantile="0.75"})` or `max(go_gc_duration_seconds{quantile="0.75"})`
|
||||
won't return the expected 75th percentile over `go_gc_duration_seconds` metrics collected from multiple instances
|
||||
of the application. See [this article](https://latencytipoftheday.blogspot.de/2014/06/latencytipoftheday-you-cant-average.html) for details.
|
||||
of the application. See [Latency Tip of the Day: You Can't Average Percentiles](https://latencytipoftheday.blogspot.de/2014/06/latencytipoftheday-you-cant-average.html) for details.
|
||||
|
||||
- It is impossible to calculate quantiles other than the already pre-calculated quantiles.
|
||||
|
||||
@@ -322,9 +321,9 @@ As was said at the beginning of the [types of metrics](#types-of-metrics) sectio
|
||||
measured. VictoriaMetrics TSDB doesn't know about metric types. All it sees are metric names, labels, values, and timestamps.
|
||||
What these metrics are, what they measure, and how - all these depend on the application which emits them.
|
||||
|
||||
To instrument your application with metrics compatible with VictoriaMetrics, we recommend
|
||||
To instrument your application with metrics compatible with VictoriaMetrics we recommend
|
||||
using the [github.com/VictoriaMetrics/metrics](https://github.com/VictoriaMetrics/metrics) package.
|
||||
See more details on how to use it in [this article](https://victoriametrics.medium.com/how-to-monitor-go-applications-with-victoriametrics-c04703110870).
|
||||
See [How to monitor Go applications with VictoriaMetrics](https://victoriametrics.medium.com/how-to-monitor-go-applications-with-victoriametrics-c04703110870).
|
||||
|
||||
VictoriaMetrics is also compatible with [Prometheus client libraries for metrics instrumentation](https://prometheus.io/docs/instrumenting/clientlibs/).
|
||||
|
||||
@@ -422,7 +421,7 @@ In the pull model, the monitoring system needs to be aware of all the applicatio
|
||||
scraped (pulled) from the known applications (aka `scrape targets`) via HTTP protocol on a regular basis (aka `scrape_interval`).
|
||||
|
||||
VictoriaMetrics supports discovering Prometheus-compatible targets and scraping metrics from them in the same way as Prometheus does -
|
||||
see [these docs](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#how-to-scrape-prometheus-exporters-such-as-node-exporter).
|
||||
see [how to scrape Prometheus exporters in VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#how-to-scrape-prometheus-exporters-such-as-node-exporter).
|
||||
|
||||
Metrics scraping is supported by [single-node VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#how-to-scrape-prometheus-exporters-such-as-node-exporter)
|
||||
and by [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/).
|
||||
@@ -776,7 +775,7 @@ VictoriaMetrics provides a special query language for executing read queries - [
|
||||
It is a [PromQL](https://prometheus.io/docs/prometheus/latest/querying/basics)-like query language with a powerful set of
|
||||
functions and features for working specifically with time series data. MetricsQL is backward-compatible with PromQL,
|
||||
so it shares most of the query concepts. The basic concepts for PromQL and MetricsQL are
|
||||
described [here](https://valyala.medium.com/promql-tutorial-for-beginners-9ab455142085).
|
||||
described in this [PromQL tutorial for beginners](https://valyala.medium.com/promql-tutorial-for-beginners-9ab455142085).
|
||||
|
||||
#### Filtering
|
||||
|
||||
@@ -883,7 +882,7 @@ query may break or may lead to incorrect results. The basics of the matching rul
|
||||
with the same set of labels, applies the operation for each data point, and returns the resulting time series with the
|
||||
same set of labels. If there are no matches, then the time series is dropped from the result.
|
||||
* The matching rules may be augmented with `ignoring`, `on`, `group_left` and `group_right` modifiers.
|
||||
See [these docs](https://prometheus.io/docs/prometheus/latest/querying/operators/#vector-matching) for details.
|
||||
See [Prometheus's vector matching documentation](https://prometheus.io/docs/prometheus/latest/querying/operators/#vector-matching) for details.
|
||||
|
||||
#### Comparison operations
|
||||
|
||||
@@ -975,15 +974,14 @@ See [How to delete time series](https://docs.victoriametrics.com/victoriametrics
|
||||
|
||||
### Relabeling
|
||||
|
||||
Relabeling is a powerful mechanism for modifying time series before they have been written to the database. Relabeling
|
||||
may be applied for both [push](#push-model) and [pull](#pull-model) models. See more
|
||||
details [here](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#relabeling).
|
||||
[Relabeling](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#relabeling) is a powerful mechanism for modifying time series before they have been written to the database. Relabeling
|
||||
may be applied for both [push](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#push-model) and [pull](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#pull-model) models.
|
||||
|
||||
### Deduplication
|
||||
|
||||
VictoriaMetrics supports data deduplication. See [these docs](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#deduplication).
|
||||
VictoriaMetrics supports data [deduplication](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#deduplication).
|
||||
|
||||
|
||||
### Downsampling
|
||||
|
||||
VictoriaMetrics supports data downsampling. See [these docs](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#downsampling).
|
||||
VictoriaMetrics Enterprise supports data [downsampling](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#downsampling). Downsampling can reduce disk space usage and improve query performance by reducing the number samples in a time series.
|
||||
|
||||
Reference in New Issue
Block a user