mirror of
https://github.com/VictoriaMetrics/VictoriaMetrics.git
synced 2026-05-19 09:46:57 +03:00
Compare commits
5 Commits
debug-grou
...
v1.136.3
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
94af588e92 | ||
|
|
e5c194cc10 | ||
|
|
7c65e3daca | ||
|
|
a6532c28b2 | ||
|
|
ec26ebb803 |
@@ -7,6 +7,9 @@ sitemap:
|
||||
disable: true
|
||||
---
|
||||
|
||||
> [!NOTE] Tip
|
||||
> Store every configuration file you use during this guide in version control. You may need them for reference or to change the configuration of your installation.
|
||||
|
||||
This guide walks you through deploying a VictoriaMetrics cluster version on Kubernetes.
|
||||
|
||||
By the end of this guide, you will know:
|
||||
@@ -382,7 +385,7 @@ Consider reading these resources to complete your setup:
|
||||
- VictoriaMetrics
|
||||
- [Learn more about the cluster version](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/)
|
||||
- [Migrate existing metric data into VictoriaMetrics with vmctl](https://docs.victoriametrics.com/victoriametrics/vmctl/)
|
||||
- [Setup alerts](https://docs.victoriametrics.com/victoriametrics/vmalert/)
|
||||
- [Setup alerts](https://docs.victoriametrics.com/guides/vmalert-datasource-managed-alerts-grafana/)
|
||||
- Grafana
|
||||
- [Enable persistent storage](https://grafana.com/docs/grafana/latest/setup-grafana/installation/helm/#enable-persistent-storage-recommended)
|
||||
- [Configure private TLS authority](https://grafana.com/docs/grafana/latest/setup-grafana/installation/helm/#configure-a-private-ca-certificate-authority)
|
||||
|
||||
@@ -7,6 +7,9 @@ sitemap:
|
||||
disable: true
|
||||
---
|
||||
|
||||
> [!NOTE] Tip
|
||||
> Store every configuration file you use during this guide in version control. You may need them for reference or to change the configuration of your installation.
|
||||
|
||||
This guide walks you through deploying a [single-node version of VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/) on Kubernetes using Helm.
|
||||
|
||||
At the end of this guide, you will know:
|
||||
@@ -56,10 +59,19 @@ vm/victoria-metrics-anomaly 1.12.9 v1.28.2 Victoria
|
||||
|
||||
## 2. Install [VictoriaMetrics single](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/) from Helm Chart
|
||||
|
||||
Download the Helm value files with the example configuration for VictoriaMetrics single-node:
|
||||
|
||||
```sh
|
||||
wget https://docs.victoriametrics.com/guides/examples/guide-vmsingle-values.yaml
|
||||
```
|
||||
|
||||
> [!NOTE] Tip
|
||||
> Keep this file in version control. You may need it to update the configuration of your VictoriaMetrics service
|
||||
|
||||
Run this command in your terminal to install [VictoriaMetrics single node](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/) to the default [namespace](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) in your cluster:
|
||||
|
||||
```shell
|
||||
helm install vmsingle vm/victoria-metrics-single -f https://docs.victoriametrics.com/guides/examples/guide-vmsingle-values.yaml
|
||||
helm install vmsingle vm/victoria-metrics-single -f guide-vmsingle-values.yaml
|
||||
```
|
||||
|
||||
Below are the key sections in the chart values file [`guide-vmsingle-values.yaml`](https://docs.victoriametrics.com/guides/examples/guide-vmsingle-values.yaml):
|
||||
@@ -315,7 +327,7 @@ Consider reading these resources to complete your setup:
|
||||
- VictoriaMetrics
|
||||
- [Learn more about the single-node version](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/)
|
||||
- [Migrate existing metric data into VictoriaMetrics with vmctl](https://docs.victoriametrics.com/victoriametrics/vmctl/)
|
||||
- [Setup alerts](https://docs.victoriametrics.com/victoriametrics/vmalert/)
|
||||
- [Setup alerts](https://docs.victoriametrics.com/guides/vmalert-datasource-managed-alerts-grafana/)
|
||||
- Grafana
|
||||
- [Enable persistent storage](https://grafana.com/docs/grafana/latest/setup-grafana/installation/helm/#enable-persistent-storage-recommended)
|
||||
- [Configure private TLS authority](https://grafana.com/docs/grafana/latest/setup-grafana/installation/helm/#configure-a-private-ca-certificate-authority)
|
||||
|
||||
615
docs/guides/vmalert-datasource-managed-alerts-grafana/README.md
Normal file
615
docs/guides/vmalert-datasource-managed-alerts-grafana/README.md
Normal file
@@ -0,0 +1,615 @@
|
||||
---
|
||||
build:
|
||||
list: never
|
||||
publishResources: false
|
||||
render: never
|
||||
sitemap:
|
||||
disable: true
|
||||
---
|
||||
|
||||
Grafana offers a rich alerting UI, including rule grouping, silences, and notification history. While Grafana-managed alerts are easy to use, [they can hit performance issues without additional configuration since they depend on a relational database by default](https://grafana.com/blog/how-we-improved-grafanas-alert-state-history-to-provide-better-insights-into-your-alerting-data/).
|
||||
|
||||
By moving rule evaluation to [vmalert](https://docs.victoriametrics.com/victoriametrics/vmalert/), you can move past these limitations while retaining Grafana's unified alerting UI. This guide shows the ideal topology for scalable alerting using vmalert, Alertmanager, and Grafana datasource-managed alerts.
|
||||
|
||||
## Grafana Alert Modes
|
||||
|
||||
Grafana supports two alert modes, which can run side by side:
|
||||
- Grafana-managed: alerts are created and evaluated entirely within Grafana itself. The alert state is stored in a SQL database by default, but [Grafana can be configured to store this data in a Prometheus-compatible database like VictoriaMetrics as well](https://grafana.com/docs/grafana/latest/alerting/set-up/configure-alert-state-history/#configure-prometheus-for-alert-state-grafana_alerts-metric).
|
||||
- Datasource-managed: alerts have their rules defined, stored, and evaluated in an external system like vmalert and Alertmanager, with Grafana just providing the UI. State is stored in VictoriaMetrics.
|
||||
|
||||
The following table compares the two modes:
|
||||
|
||||
| Aspect | Grafana-Managed | Data Source-Managed |
|
||||
| ------------------- | -------------------------------------------------- | ----------------------------- |
|
||||
| Where rules live | Grafana's SQL database | vmalert's YAML config |
|
||||
| Evaluation | Grafana's scheduler | vmalert |
|
||||
| Horizontal Scaling | Complex (requires HA SQL database) | Simple (add more pods) |
|
||||
| State storage | SQL backend or a Prometheus datasource | VictoriaMetrics |
|
||||
| UI Management | Full create/edit in Grafana | View-only |
|
||||
| Dependencies | SQL + Grafana | VictoriaMetrics |
|
||||
| Version control | Complex (rules must be exported/imported) | Easy (rules are in YAML file) |
|
||||
|
||||
## Datasource-managed Alert Topology
|
||||
|
||||
The proposed alert setup relies on the following services:
|
||||
|
||||
- VictoriaMetrics: provides the time-series database and persists alert status.
|
||||
- vmalert: evaluates alerting rules from its config file against VictoriaMetrics data and forwards firing alerts to Alertmanager.
|
||||
- Alertmanager: groups and routes alerts to the configured recipients.
|
||||
- Grafana: serves as the unified UI, connecting to VictoriaMetrics for rules and metrics, and to Alertmanager for notifications/silences.
|
||||
|
||||

|
||||
|
||||
## vmalert Demo with Docker {#docker}
|
||||
|
||||
In this section, we'll describe how you can try datasource-managed alerts on Grafana with Docker Compose. Follow the steps in this section to see how the Grafana UI looks in datasource-managed alerts.
|
||||
|
||||
First, create `alerts.yml`. The following configuration file creates an always-firing alert, which you can use to test datasource-managed alerts in Grafana.
|
||||
|
||||
```yaml
|
||||
# alerts.yml
|
||||
groups:
|
||||
- name: demo
|
||||
rules:
|
||||
# Always-firing demo alert so you see something immediately
|
||||
- alert: AlwaysFiring
|
||||
expr: vector(1)
|
||||
for: 10s
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Demo alert that always fires"
|
||||
description: "This is a demo alert from vmalert using vector(1)."
|
||||
|
||||
# Simple recording rule you can graph in Grafana
|
||||
- record: demo:vector_one
|
||||
expr: vector(1)
|
||||
```
|
||||
|
||||
Next, create a basic Alertmanager config called `alertmanager.yml`. This example does not forward alerts anywhere, but serves as a source for Grafana:
|
||||
|
||||
```yaml
|
||||
# alertmanager.yml
|
||||
global:
|
||||
resolve_timeout: 5m
|
||||
|
||||
route:
|
||||
receiver: "log"
|
||||
|
||||
receivers:
|
||||
- name: "log"
|
||||
```
|
||||
|
||||
Finally, create `grafana-datasources.yml` to configure Grafana to use VictoriaMetrics and Alertmanager as datasources for alerts and notifications:
|
||||
|
||||
```yaml
|
||||
# grafana-datasources.yml
|
||||
apiVersion: 1
|
||||
|
||||
datasources:
|
||||
- name: VictoriaMetrics
|
||||
type: prometheus
|
||||
access: proxy
|
||||
url: http://victoriametrics:8428
|
||||
isDefault: true
|
||||
|
||||
- name: Alertmanager
|
||||
type: alertmanager
|
||||
access: proxy
|
||||
url: http://alertmanager:9093
|
||||
jsonData:
|
||||
implementation: prometheus
|
||||
```
|
||||
|
||||
The final piece is the Docker Compose file. This ties all the services together and sets up the required command-line arguments:
|
||||
|
||||
```yaml
|
||||
# compose.yml
|
||||
services:
|
||||
victoriametrics:
|
||||
image: victoriametrics/victoria-metrics:v1.138.0
|
||||
command:
|
||||
- "--storageDataPath=/victoria-metrics-data"
|
||||
- "--selfScrapeInterval=10s"
|
||||
# Proxy vmalert APIs so Grafana can see rules via VictoriaMetrics
|
||||
- "--vmalert.proxyURL=http://vmalert:8880"
|
||||
ports:
|
||||
- "8428:8428"
|
||||
volumes:
|
||||
- vm-data:/victoria-metrics-data
|
||||
|
||||
alertmanager:
|
||||
image: prom/alertmanager:v0.31.1
|
||||
command:
|
||||
- "--config.file=/etc/alertmanager/alertmanager.yml"
|
||||
ports:
|
||||
- "9093:9093"
|
||||
volumes:
|
||||
- ./alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro
|
||||
|
||||
vmalert:
|
||||
image: victoriametrics/vmalert:v1.138.0
|
||||
depends_on:
|
||||
- victoriametrics
|
||||
- alertmanager
|
||||
command:
|
||||
# read metrics from VictoriaMetrics
|
||||
- "--datasource.url=http://victoriametrics:8428"
|
||||
# store and retrieve alert state from VictoriaMetrics
|
||||
- "--remoteRead.url=http://victoriametrics:8428"
|
||||
- "--remoteWrite.url=http://victoriametrics:8428"
|
||||
# configure Alertmanager as the notifier
|
||||
- "--notifier.url=http://alertmanager:9093"
|
||||
- "--rule=/etc/vmalert/alerts.yml"
|
||||
- "--evaluationInterval=15s"
|
||||
# external settings link vmalerts and Alertmanager to Grafana
|
||||
- "--external.url=http://localhost:3000"
|
||||
- "--external.alert.source=explore?orgId=1&left={\"datasource\":\"VictoriaMetrics\",\"queries\":[{\"refId\":\"A\",\"expr\":\"{{.Expr|queryEscape}}\"}]}"
|
||||
ports:
|
||||
- "8880:8880"
|
||||
volumes:
|
||||
- ./alerts.yml:/etc/vmalert/alerts.yml:ro
|
||||
|
||||
grafana:
|
||||
image: grafana/grafana:12.4
|
||||
depends_on:
|
||||
- victoriametrics
|
||||
- alertmanager
|
||||
environment:
|
||||
GF_SECURITY_ADMIN_USER: admin
|
||||
GF_SECURITY_ADMIN_PASSWORD: admin
|
||||
GF_PATHS_PROVISIONING: /etc/grafana/provisioning
|
||||
ports:
|
||||
- "3000:3000"
|
||||
volumes:
|
||||
- ./grafana-datasources.yml:/etc/grafana/provisioning/datasources/datasources.yml:ro
|
||||
- grafana-data:/var/lib/grafana
|
||||
|
||||
volumes:
|
||||
vm-data:
|
||||
grafana-data:
|
||||
```
|
||||
|
||||
Let's break down the main command line arguments that connect every component:
|
||||
|
||||
- VictoriaMetrics
|
||||
- `-vmalert.proxyURL`: forwards Grafana requests for `/api/v1/rules` and `/api/v1/alerts` to vmalert, enabling rule visibility in Grafana UI
|
||||
|
||||
- vmalert
|
||||
- `-datasource.url`: configures VictoriaMetrics as the query source for rule evaluation
|
||||
- `-remoteWrite.url`: defines VictoriaMetrics as the backend used to persist rule state across restarts
|
||||
- `-remoteRead.url`: defines VictoriaMetrics as the backend used to read historical state for pending alerts
|
||||
- `-notifier.url`: directs firing alerts to Alertmanager
|
||||
- `-external.url`: defines the base URL for alert links so users see the public URL of Alertmanager, or an external alerting UI like Karma or Grafana in notifications.
|
||||
- `-external.alert.source`: creates a template for clickable alert links for Grafana. Allows Alertmanager UI to link directly to Grafana
|
||||
|
||||
Now, start the demo with:
|
||||
|
||||
```sh
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
Open your browser at `localhost:3000` and log in to Grafana with username `admin` and password `admin`.
|
||||
|
||||
If you open the sidebar and select **Alerting** > **Alert rules**, you should be able to see one alert pending or firing.
|
||||
|
||||

|
||||
<figcaption style="text-align: center; font-style: italic;">Datasource-managed alert firing in Grafana</figcaption>
|
||||
|
||||
Open the sidebar again and go to **Alerting** > **Active notifications** to see the active alert reported by Alertmanager.
|
||||
|
||||

|
||||
|
||||
You can also see the alerts in VMUI by opening the browser in `http://localhost:8428/vmui/?#/rules`. This is possible only when we have configured `-vmalert.proxyURL` in VictoriaMetrics.
|
||||
|
||||

|
||||
<figcaption style="text-align: center; font-style: italic;">Alerts can be visualized in VMUI directly</figcaption>
|
||||
|
||||
If you open the browser in `http://localhost:9093/#/alerts`, you will see the Alertmanager UI with the firing alert.
|
||||
|
||||

|
||||
<figcaption style="text-align: center; font-style: italic;">Alertmanager UI showing the firing alert</figcaption>
|
||||
|
||||
Clicking **Source** should take you back to Grafana and display the query that generated the alert.
|
||||
|
||||
## vmalert and VictoriaMetrics Single on Kubernetes {#vmsingle}
|
||||
|
||||
This section explains how to configure datasource-managed alerts on the VictoriaMetrics single-node version on Kubernetes.
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- A Kubernetes cluster
|
||||
- VictoriaMetrics single-node
|
||||
- Grafana
|
||||
- Helm values or config files used for installation
|
||||
|
||||
You can follow this guide to install all required components: [Kubernetes monitoring via VictoriaMetrics single](https://docs.victoriametrics.com/guides/k8s-monitoring-via-vm-single/).
|
||||
|
||||
### 1. Ensure VictoriaMetrics and Grafana are installed
|
||||
|
||||
Ensure you have added and updated the VictoriaMetrics Helm repository:
|
||||
|
||||
```sh
|
||||
helm repo add vm https://victoriametrics.github.io/helm-charts/
|
||||
helm repo update
|
||||
|
||||
```
|
||||
|
||||
Confirm that the VictoriaMetrics single-node version is installed (assuming the release name `vmsingle` from the [installation guide](https://docs.victoriametrics.com/guides/k8s-monitoring-via-vm-single/))
|
||||
|
||||
```sh
|
||||
kubectl get pods -l app.kubernetes.io/instance=vmsingle
|
||||
```
|
||||
|
||||
You should get a single running pod:
|
||||
|
||||
```sh
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
vmsingle-victoria-metrics-single-server-0 1/1 Running 0 48m
|
||||
```
|
||||
|
||||
Do the same for Grafana:
|
||||
|
||||
```sh
|
||||
kubectl get pod -l app.kubernetes.io/name=grafana
|
||||
```
|
||||
|
||||
You should get the name of the Grafana pod running in your cluster:
|
||||
|
||||
```sh
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
my-grafana-65d6d4ccbc-nxkxq 1/1 Running 0 58m
|
||||
```
|
||||
|
||||
### 2. Install vmalert and Alertmanager
|
||||
|
||||
Create a Helm values file for vmalert and Alertmanager called `vm-alerting-values.yml`.
|
||||
|
||||
The example below comes with two demo alerts. Add your own vmalert [alerting rules](https://docs.victoriametrics.com/victoriametrics/vmalert/#rules) in the `config: alerts:` section below.
|
||||
|
||||
```sh
|
||||
cat <<EOF > vm-alerting-values.yml
|
||||
# Enable and configure Alertmanager
|
||||
alertmanager:
|
||||
enabled: true
|
||||
config:
|
||||
global:
|
||||
resolve_timeout: 5m
|
||||
route:
|
||||
group_by: ["alertname"]
|
||||
group_wait: 30s
|
||||
group_interval: 5m
|
||||
repeat_interval: 12h
|
||||
receiver: "log"
|
||||
|
||||
receivers:
|
||||
- name: "log"
|
||||
# place your default route here for notifications
|
||||
|
||||
# Configure vmalert ("server" section in this chart)
|
||||
server:
|
||||
# vmalert evaluation datasource: point at vmsingle’s Prometheus-compatible API
|
||||
datasource:
|
||||
url: http://vmsingle-victoria-metrics-single-server.default.svc.cluster.local.:8428
|
||||
|
||||
# Where vmalert stores and reads alert state (remote write/read)
|
||||
remote:
|
||||
write:
|
||||
url: http://vmsingle-victoria-metrics-single-server.default.svc.cluster.local.:8428
|
||||
read:
|
||||
url: http://vmsingle-victoria-metrics-single-server.default.svc.cluster.local.:8428
|
||||
|
||||
# Configure Alertmanager as notifier
|
||||
notifier:
|
||||
alertmanager:
|
||||
# Adjust namespace/name if you install into a non-default namespace or change the release name
|
||||
url: http://vmalert-victoria-metrics-alert-alertmanager:9093
|
||||
|
||||
# Inline demo rules. Add your alerting groups and rules here
|
||||
config:
|
||||
alerts:
|
||||
groups:
|
||||
- name: vm-health
|
||||
rules:
|
||||
- alert: TooManyRestarts
|
||||
expr: changes(process_start_time_seconds{job=~"victoriametrics|vmagent|vmalert"}[15m]) > 2
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "{{ $labels.job }} too many restarts (instance {{ $labels.instance }})"
|
||||
description: "Job {{ $labels.job }} has restarted more than twice in the last 15 minutes. It might be crashlooping."
|
||||
- alert: ServiceDown
|
||||
expr: up{job=~"victoriametrics|vmagent|vmalert"} == 0
|
||||
for: 2m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "Service {{ $labels.job }} is down on {{ $labels.instance }}"
|
||||
description: "{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 2 minutes."
|
||||
EOF
|
||||
```
|
||||
|
||||
Install `vmalert` and Alertmanager with:
|
||||
|
||||
```sh
|
||||
helm install vmalert vm/victoria-metrics-alert -f vm-alerting-values.yml
|
||||
```
|
||||
|
||||
### 3. Configure VictoriaMetrics single to proxy to vmalert
|
||||
|
||||
For datasource-managed alerts, Grafana talks to VictoriaMetrics, and VictoriaMetrics proxies alerting-related API calls to vmalert via the `-vmalert.proxyURL` flag.
|
||||
|
||||
First, check the service name for `vmalert`:
|
||||
|
||||
```sh
|
||||
kubectl get svc -l app.kubernetes.io/instance=vmalert,app=server
|
||||
```
|
||||
|
||||
Get the name of the `vmalert` service:
|
||||
|
||||
```sh
|
||||
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
|
||||
vmalert-victoria-metrics-alert-server ClusterIP None <none> 8880/TCP 58m
|
||||
```
|
||||
|
||||
The internal DNS name for this service, in the default namespace, will be:
|
||||
|
||||
```text
|
||||
vmalert-victoria-metrics-alert-server.default.svc.cluster.local:8880
|
||||
```
|
||||
|
||||
Next, create a Helm values file with the internal Kubernetes URL for vmalert.
|
||||
|
||||
```sh
|
||||
cat <<EOF > vm-vmalert-proxy-values.yml
|
||||
# vm-vmalert-proxy-values.yaml
|
||||
server:
|
||||
extraArgs:
|
||||
vmalert.proxyURL: http://vmalert-victoria-metrics-alert-server.default.svc.cluster.local:8880
|
||||
EOF
|
||||
|
||||
```
|
||||
|
||||
Update the configuration of VictoriaMetrics single by applying both *your original Helm values* and this new overlay. In the example below, the original values file is called `vmsingle-values-file.yml` (this is the file you used when you first installed the cluster):
|
||||
|
||||
```sh
|
||||
helm upgrade vmsingle vm/victoria-metrics-single \
|
||||
-f vmsingle-values-file.yml \
|
||||
-f vm-vmalert-proxy-values.yml
|
||||
```
|
||||
|
||||
After this upgrade, vmsingle will start proxying `/api/v1/rules`, `/api/v1/alerts`, and other `vmalert` [endpoints](https://docs.victoriametrics.com/victoriametrics/vmalert/#web) to the vmalert service, enabling Grafana’s alerting UI and API to work through the VictoriaMetrics datasource.
|
||||
|
||||
To finish the setup, jump to the [Configure Grafana](https://docs.victoriametrics.com/guides/vmalert-datasource-managed-alerts-grafana/#grafana) section
|
||||
|
||||
## vmalert and VictoriaMetrics Cluster on Kubernetes {#vmcluster}
|
||||
|
||||
This section explains how to configure datasource-managed alerts on the VictoriaMetrics cluster version on Kubernetes.
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- A Kubernetes cluster
|
||||
- VictoriaMetrics cluster
|
||||
- Grafana
|
||||
- Helm values or config files used for the installation of the cluster
|
||||
|
||||
You can follow this guide to install the cluster and Grafana first: [Kubernetes monitoring with VictoriaMetrics cluster](https://docs.victoriametrics.com/guides/k8s-monitoring-via-vm-cluster/).
|
||||
|
||||
### 1. Ensure VictoriaMetrics and Grafana are installed
|
||||
|
||||
Ensure you have added and updated the VictoriaMetrics Helm repository:
|
||||
|
||||
```sh
|
||||
helm repo add vm https://victoriametrics.github.io/helm-charts/
|
||||
helm repo update
|
||||
```
|
||||
|
||||
Confirm that the VictoriaMetrics cluster is installed (assuming the release name `vmcluster` from the [installation guide](https://docs.victoriametrics.com/guides/k8s-monitoring-via-vm-cluster/)):
|
||||
|
||||
```sh
|
||||
kubectl get pods -l app.kubernetes.io/instance=vmcluster
|
||||
```
|
||||
|
||||
You should see pods for `vmselect`, `vminsert`, and `vmstorage`, for example:
|
||||
|
||||
```text
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
vmcluster-victoria-metrics-cluster-vminsert-b4d494b4c-cx2m5 1/1 Running 0 3m8s
|
||||
vmcluster-victoria-metrics-cluster-vminsert-b4d494b4c-xdv76 1/1 Running 0 3m8s
|
||||
vmcluster-victoria-metrics-cluster-vmselect-67979c98fc-7hfdl 1/1 Running 0 3m8s
|
||||
vmcluster-victoria-metrics-cluster-vmselect-67979c98fc-ftpzn 1/1 Running 0 3m8s
|
||||
vmcluster-victoria-metrics-cluster-vmstorage-0 1/1 Running 0 3m8s
|
||||
vmcluster-victoria-metrics-cluster-vmstorage-1 1/1 Running 0 2m48s
|
||||
```
|
||||
|
||||
VictoriaMetrics exposes its write API via the `vminsert` service on port 8480 and its read (Prometheus-compatible) API via the `vmselect` service on port 8481 by default. For a default installation, these DNS names are:
|
||||
|
||||
- Write: `vmcluster-victoria-metrics-cluster-vminsert.default.svc.cluster.local.:8480`
|
||||
- Read: `vmcluster-victoria-metrics-cluster-vmselect.default.svc.cluster.local.:8481`
|
||||
|
||||
Now, ensure Grafana is installed:
|
||||
|
||||
```sh
|
||||
kubectl get pod -l app.kubernetes.io/name=grafana
|
||||
```
|
||||
|
||||
You should get the name of the Grafana pod running in your cluster:
|
||||
|
||||
```sh
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
my-grafana-65d6d4ccbc-nxkxq 1/1 Running 0 58m
|
||||
```
|
||||
|
||||
### 2. Install vmalert and Alertmanager
|
||||
|
||||
Create a Helm values file for vmalert and Alertmanager called `vm-alerting-values.yml`.
|
||||
|
||||
The example below comes with two demo alerts. Add your own vmalert [alerting rules](https://docs.victoriametrics.com/victoriametrics/vmalert/#rules) in the `config: alerts:` section below.
|
||||
|
||||
|
||||
```sh
|
||||
cat <<EOF > vm-alerting-values.yml
|
||||
# Enable and configure Alertmanager
|
||||
alertmanager:
|
||||
enabled: true
|
||||
config:
|
||||
global:
|
||||
resolve_timeout: 5m
|
||||
route:
|
||||
group_by: ["alertname"]
|
||||
group_wait: 30s
|
||||
group_interval: 5m
|
||||
repeat_interval: 12h
|
||||
receiver: "log"
|
||||
|
||||
receivers:
|
||||
- name: "log"
|
||||
# place your notification route here
|
||||
|
||||
# Configure vmalert ("server" section in this chart)
|
||||
server:
|
||||
# vmalert evaluation datasource: point at vmselect’s Prometheus-compatible API
|
||||
datasource:
|
||||
url: http://vmcluster-victoria-metrics-cluster-vmselect.default.svc.cluster.local.:8481/select/multitenant/prometheus/
|
||||
|
||||
# Where vmalert stores and reads alert state (remote write/read)
|
||||
remote:
|
||||
write:
|
||||
# send ALERTS / recording rule series into vminsert
|
||||
url: http://vmcluster-victoria-metrics-cluster-vminsert.default.svc.cluster.local.:8480/insert/0/prometheus/
|
||||
read:
|
||||
# read back alert state from vmselect
|
||||
url: http://vmcluster-victoria-metrics-cluster-vmselect.default.svc.cluster.local.:8481/select/0/prometheus/
|
||||
|
||||
# Configure Alertmanager as notifier
|
||||
notifier:
|
||||
alertmanager:
|
||||
# Adjust namespace/name if you install into a non-default namespace or change the release name
|
||||
url: http://vmalert-victoria-metrics-alert-alertmanager:9093
|
||||
|
||||
# Inline demo rules. Add your alerting groups and rules here
|
||||
config:
|
||||
alerts:
|
||||
groups:
|
||||
- name: vm-health
|
||||
rules:
|
||||
- alert: TooManyRestarts
|
||||
expr: changes(process_start_time_seconds{job=~"victoriametrics|vmagent|vmalert"}[15m]) > 2
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "{{ $labels.job }} too many restarts (instance {{ $labels.instance }})"
|
||||
description: "Job {{ $labels.job }} has restarted more than twice in the last 15 minutes.
|
||||
It might be crashlooping."
|
||||
- alert: ServiceDown
|
||||
expr: up{job=~"victoriametrics|vmagent|vmalert"} == 0
|
||||
for: 2m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "Service {{ $labels.job }} is down on {{ $labels.instance }}"
|
||||
description: "{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 2 minutes."
|
||||
EOF
|
||||
```
|
||||
|
||||
|
||||
The key differences from the [single-node setup](https://docs.victoriametrics.com/guides/vmalert-datasource-managed-alerts-grafana/#vmsingle) section
|
||||
- `server.datasource.url` and `server.remote.read.url` point to the `vmselect` read endpoint (`/select/multitenant/prometheus/`).
|
||||
- `server.remote.write.url` points to the `vminsert` write endpoint (`/insert/multitenant/prometheus/`).
|
||||
|
||||
Install `vmalert` and Alertmanager with:
|
||||
|
||||
```sh
|
||||
helm install vmalert vm/victoria-metrics-alert -f vm-alerting-values.yml
|
||||
```
|
||||
|
||||
### 3. Configure VictoriaMetrics Cluster to proxy to vmalert
|
||||
|
||||
For datasource-managed alerts, Grafana talks to VictoriaMetrics, and VictoriaMetrics proxies alerting-related API calls to vmalert via the `-vmalert.proxyURL` flag. In the cluster version, set this flag on `vmselect` and point it to the vmalert service so Grafana can reach the alert state.
|
||||
|
||||
First, check the service name for vmalert:
|
||||
|
||||
```sh
|
||||
kubectl get svc -l app.kubernetes.io/instance=vmalert,app=server
|
||||
```
|
||||
|
||||
You should see something like:
|
||||
|
||||
```sh
|
||||
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
|
||||
vmalert-victoria-metrics-alert-server ClusterIP None <none> 8880/TCP 58m
|
||||
```
|
||||
|
||||
The internal DNS name for this service, in the default namespace, will be:
|
||||
|
||||
```text
|
||||
vmalert-victoria-metrics-alert-server.default.svc.cluster.local:8880
|
||||
```
|
||||
|
||||
Create a Helm values overlay file for the cluster called `vmcluster-vmalert-proxy-values.yml`.
|
||||
|
||||
The `vmselect.extraArgs` map in the `victoria-metrics-cluster` chart allows you to pass arbitrary command-line flags to vmselect, including `-vmalert.proxyURL`.
|
||||
|
||||
```sh
|
||||
cat <<EOF > vmcluster-vmalert-proxy-values.yml
|
||||
vmselect:
|
||||
extraArgs:
|
||||
vmalert.proxyURL: http://vmalert-victoria-metrics-alert-server.default.svc.cluster.local:8880
|
||||
EOF
|
||||
```
|
||||
|
||||
Update the VictoriaMetrics cluster configuration by applying both your *original Helm values* and this new overlay. In the example below, the original values file is called `vmcluster-values-file.yml` (this is the file you used when you first installed the cluster):
|
||||
|
||||
```sh
|
||||
helm upgrade vmcluster vm/victoria-metrics-cluster \
|
||||
-f victoria-metrics-cluster-values.yml \
|
||||
-f vmcluster-vmalert-proxy-values.yml
|
||||
```
|
||||
|
||||
After this upgrade, vmselect will start proxying `/api/v1/rules`, `/api/v1/alerts`, and other `vmalert` endpoints to the vmalert service, enabling Grafana’s alerting UI and API to work through the VictoriaMetrics datasource.
|
||||
|
||||
To finalize the setup, continue on to the next section, [Configure Grafana](#grafana).
|
||||
|
||||
## Configure Grafana {#grafana}
|
||||
|
||||
The last step on any Kubernetes-based installation is to add Alertmanager to Grafana so that Grafana can show notifications alongside the rules it discovers via the VictoriaMetrics datasource.
|
||||
|
||||
Get the service name for Alertmanager in your cluster:
|
||||
|
||||
```sh
|
||||
kubectl get svc -l app.kubernetes.io/instance=vmalert,app=alertmanager
|
||||
```
|
||||
|
||||
You should see something like:
|
||||
|
||||
```text
|
||||
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
|
||||
vmalert-victoria-metrics-alert-alertmanager ClusterIP 10.43.114.243 <none> 9093/TCP 68m
|
||||
```
|
||||
|
||||
Next, add Alertmanager to Grafana:
|
||||
|
||||
1. Log in to your Grafana dashboard.
|
||||
2. Go to **Connections** > **Datasources**.
|
||||
3. Press **+ Add new data source**.
|
||||
4. Search and select “Alertmanager”.
|
||||
5. Fill in the following parameters (adjusting namespace/service name if needed):
|
||||
- Implementation: Prometheus
|
||||
- URL: `http://vmalert-victoria-metrics-alert-alertmanager.default.svc.cluster.local:9093`
|
||||
|
||||
Ensure the URL matches the Alertmanager service name you obtained earlier.
|
||||
|
||||
6. Press **Save & Test**.
|
||||
|
||||

|
||||
<figcaption style="text-align: center; font-style: italic;">Adding Alertmanager to Grafana</figcaption>
|
||||
|
||||
With vmselect’s `vmalert.proxyURL` set and Alertmanager configured as a datasource, Grafana should now be able to display vmalert rules and alert instances from the VictoriaMetrics Cluster datasource and show notifications managed by Alertmanager in its UI.
|
||||
|
||||
## See also
|
||||
|
||||
- [YouTube: Mathias Palmersheim - Who will be your Ruler](https://youtu.be/NfhVOEkznFY)
|
||||
- [Kubernetes monitoring via VictoriaMetrics single](https://docs.victoriametrics.com/guides/k8s-monitoring-via-vm-single/)
|
||||
- [Kubernetes monitoring with VictoriaMetrics cluster](https://docs.victoriametrics.com/guides/k8s-monitoring-via-vm-cluster/)
|
||||
- Learn more about [vmalert](https://docs.victoriametrics.com/victoriametrics/vmalert/)
|
||||
|
||||
@@ -0,0 +1,17 @@
|
||||
---
|
||||
weight: 16
|
||||
title: Datasource-Managed Alerts with vmalert and Grafana
|
||||
menu:
|
||||
docs:
|
||||
parent: "guides"
|
||||
weight: 16
|
||||
tags:
|
||||
- metrics
|
||||
- guide
|
||||
- kubernetes
|
||||
- alerts
|
||||
aliases:
|
||||
- /guides/vmalert-datasource-managed-alerts-grafana.html
|
||||
---
|
||||
{{% content "README.md" %}}
|
||||
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 403 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 634 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 373 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 557 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 185 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 114 KiB |
@@ -100,6 +100,10 @@ The following steps must be performed during the upgrade / downgrade procedure:
|
||||
* Wait until the process stops. This can take a few seconds.
|
||||
* Start the upgraded VictoriaMetrics.
|
||||
|
||||
> If you'd prefer not to manage upgrades yourself, [VictoriaMetrics Cloud](https://console.victoriametrics.cloud/signUp?utm_source=website&utm_campaign=docs_vm_bestpractices_upgrade)
|
||||
> performs version upgrades automatically during scheduled maintenance windows with no action required on your part.
|
||||
> See the [VictoriaMetrics Cloud documentation](https://docs.victoriametrics.com/victoriametrics-cloud/) to get started.
|
||||
|
||||
## Backup Recommendations
|
||||
|
||||
VictoriaMetrics supports backups via [vmbackup](https://docs.victoriametrics.com/victoriametrics/vmbackup/) and [vmrestore](https://docs.victoriametrics.com/victoriametrics/vmrestore/) tools. There is also [vmbackupmanager](https://docs.victoriametrics.com/victoriametrics/vmbackupmanager/), which simplifies backup automation.
|
||||
|
||||
@@ -755,6 +755,10 @@ The `minimum downtime` strategy has the following benefits comparing to `no down
|
||||
- It allows minimizing the duration of config update / version upgrade for clusters with big number of nodes
|
||||
of for clusters with big `vmstorage` nodes, which may take long time for graceful restart.
|
||||
|
||||
> If you'd prefer to avoid managing cluster upgrades yourself, [VictoriaMetrics Cloud](https://console.victoriametrics.cloud/signUp?utm_source=website&utm_campaign=docs_vm_cluster_upgrade)
|
||||
> handles version upgrades automatically during maintenance windows, and lets you create, resize, or delete deployments with a few clicks from the UI.
|
||||
> See the [VictoriaMetrics Cloud documentation](https://docs.victoriametrics.com/victoriametrics-cloud/) to get started.
|
||||
|
||||
### Improving re-routing performance during restart
|
||||
|
||||
`vmstorage` nodes may experience increased usage for CPU, RAM and disk IO during
|
||||
|
||||
@@ -247,6 +247,10 @@ The following steps must be performed during the upgrade / downgrade procedure:
|
||||
|
||||
Prometheus doesn't drop data during VictoriaMetrics restart. See [this article](https://grafana.com/blog/2019/03/25/whats-new-in-prometheus-2.8-wal-based-remote-write/) for details. The same applies also to [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/).
|
||||
|
||||
> If you'd prefer not to manage upgrades yourself, [VictoriaMetrics Cloud](https://console.victoriametrics.cloud/signUp?utm_source=website&utm_campaign=docs_vm_single_upgrade)
|
||||
> performs version upgrades automatically during maintenance windows with no action required on your part.
|
||||
> See the [VictoriaMetrics Cloud documentation](https://docs.victoriametrics.com/victoriametrics-cloud/) to get started.
|
||||
|
||||
## vmui
|
||||
|
||||
VictoriaMetrics provides UI for query troubleshooting and exploration. The UI is available at `http://victoriametrics:8428/vmui`
|
||||
@@ -1786,6 +1790,10 @@ via [vmalert](https://docs.victoriametrics.com/victoriametrics/vmalert/) or via
|
||||
See also [VictoriaMetrics Monitoring](https://victoriametrics.com/blog/victoriametrics-monitoring/)
|
||||
and [troubleshooting docs](https://docs.victoriametrics.com/victoriametrics/troubleshooting/).
|
||||
|
||||
> [VictoriaMetrics Cloud](https://console.victoriametrics.cloud/signUp?utm_source=website&utm_campaign=docs_vm_single_monitoring)
|
||||
> provides built-in monitoring dashboards and automatic alerts when resource consumption is high or configured limits are approached,
|
||||
> so you get notified before issues impact your workload. See the [VictoriaMetrics Cloud documentation](https://docs.victoriametrics.com/victoriametrics-cloud/) to get started.
|
||||
|
||||
> VictoriaMetrics components do not expose metadata `TYPE` and `HELP` fields on `/metrics` page.
|
||||
> Services like Google Cloud Managed Prometheus could require metadata to be present for scraping. In this case, pass `-metrics.exposeMetadata`
|
||||
command-line to them. See [these docs](https://cloud.google.com/stackdriver/docs/managed-prometheus/troubleshooting#missing-metric-type) for details.
|
||||
|
||||
@@ -39,7 +39,7 @@ Released at 2026-03-27
|
||||
* FEATURE: [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/): allow specifying `basic_auth` in scrape configs without `username`. Previously this resulted in a config error. Now a warning is logged instead. See [#6956](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6956). Thanks to @andriibeee for the contribution.
|
||||
* FEATURE: [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/) and [vmsingle](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/): add support for negative buckets in [OpenTelemetry](https://docs.victoriametrics.com/victoriametrics/integrations/opentelemetry/) `ExponentialHistogram` during ingestion. See [#9896-comment](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/9896#issuecomment-4037522985).
|
||||
* FEATURE: [vmrestore](https://docs.victoriametrics.com/victoriametrics/vmrestore/): Improve restore speed on linux systems by pre-allocating files and optimizing write path. This behavior can be disabled with `-skipFilePreallocation`. See [#10661](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/10661/). Thanks to @BenNF for the contribution.
|
||||
|
||||
* FEATURE: [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/): [yandexcloud_sd_configs](https://docs.victoriametrics.com/victoriametrics/sd_configs/#yandexcloud_sd_configs) now supports `folder_ids` for limiting discovery to specific folders. See [#10623](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/10623). Thanks to @Br1an67 for the contribution.
|
||||
|
||||
* BUGFIX: `vmselect` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/): retry RPC by dialing a new connection instead of reusing a pooled one when the previous attempt fails with `io.EOF`, `broken pipe` or `reset by peer`. This reduces query failures caused by stale connections to restarted vmstorage nodes. See [#10314](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/10314)
|
||||
* BUGFIX: [vmui](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#vmui): fix autocomplete dropdown not closing on the Raw Query page. See [#10665](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/10665)
|
||||
@@ -62,7 +62,6 @@ Released at 2026-03-13
|
||||
* FEATURE: [vmalert](https://docs.victoriametrics.com/victoriametrics/vmalert/): support negative values for the group `eval_offset` option, which allows starting group evaluation at `groupInterval-abs(eval_offset)` within `[0...groupInterval]`. See [#10424](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/10424).
|
||||
* FEATURE: [vmsingle](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/) and `vmselect` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/): Disable `/graphite/tags/tagSeries` and `/graphite/tags/tagMultiSeries` for Graphite tag registration since it is unlikely it is used in context of VictoriaMetrics. See [10544](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/10544).
|
||||
* FEATURE: [vmui](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#vmui): rename debug tools buttons for clarity. See [#10453](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/10453).
|
||||
* FEATURE: [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/): [yandexcloud_sd_configs](https://docs.victoriametrics.com/victoriametrics/sd_configs/#yandexcloud_sd_configs) now supports `folder_ids` for limiting discovery to specific folders. See [#10623](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/10623). Thanks to @Br1an67 for the contribution.
|
||||
|
||||
* BUGFIX: all VictoriaMetrics components: replace `histogram` with `untyped` metric metadata type for [VictoriaMetrics histograms](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#histogram) when `-metrics.exposeMetadata` is set. See [#82](https://github.com/VictoriaMetrics/metrics/issues/82).
|
||||
* BUGFIX: [vmauth](https://docs.victoriametrics.com/victoriametrics/vmauth/): properly route requests to `default_url`. Previously, `request_path` query arg could be set incorrectly during concurrent requests. See [#10626](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/10626).
|
||||
@@ -104,6 +103,20 @@ It enables back `Discovered targets` debug UI by default.
|
||||
* BUGFIX: `vmstorage` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/): properly apply `extra_filters[]` filter when querying `vm_account_id` or `vm_project_id` labels via [multitenant](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#multitenancy) request for `/api/v1/label/…/values` API. Before, `extra_filters` was ignored. See [#10503](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/10503).
|
||||
* BUGFIX: [vmsingle](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/) and `vmselect` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/): revert the use of rollup result cache for [instant queries](https://docs.victoriametrics.com/keyConcepts.html#instant-query) that contain [`rate`](https://docs.victoriametrics.com/MetricsQL.html#rate) function with a lookbehind window larger than `-search.minWindowForInstantRollupOptimization`. The cache usage was removed since [v1.132.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.132.0). See [#10098](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/10098#issuecomment-3895011084) for more details.
|
||||
|
||||
## [v1.136.3](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.136.3)
|
||||
|
||||
Released at 2026-03-27
|
||||
|
||||
**v1.136.x is a line of [LTS releases](https://docs.victoriametrics.com/victoriametrics/lts-releases/). It contains important up-to-date bugfixes for [VictoriaMetrics enterprise](https://docs.victoriametrics.com/victoriametrics/enterprise/).
|
||||
All these fixes are also included in [the latest community release](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest).
|
||||
The v1.136.x line will be supported for at least 12 months since [v1.136.0](https://docs.victoriametrics.com/victoriametrics/changelog/#v11360) release**
|
||||
|
||||
* BUGFIX: `vmselect` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/): retry RPC by dialing a new connection instead of reusing a pooled one when the previous attempt fails with `io.EOF`, `broken pipe` or `reset by peer`. This reduces query failures caused by stale connections to restarted vmstorage nodes. See [#10314](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/10314)
|
||||
* BUGFIX: [vmui](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#vmui): fix autocomplete dropdown not closing on the Raw Query page. See [#10665](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/10665)
|
||||
* BUGFIX: [vmsingle](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/) and `vmselect` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/): enforce `datasource_type=prometheus` when [proxying](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#vmalert) Grafana requests to [vmalert](https://docs.victoriametrics.com/victoriametrics/vmalert/). Grafana supports only `prometheus` and `loki` alerts. Without this fix, Grafana shows `Error loading alerts` when non-Prometheus alert types are returned. See [victoriametrics-datasource#329](https://github.com/VictoriaMetrics/victoriametrics-datasource/issues/329).
|
||||
* BUGFIX: [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/): stop logging `error`-level messages when scraping targets that expose OpenMetrics `info`, `gaugehistogram`, `stateset`, or `unknown` metric types. These are valid [OpenMetrics](https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md) types and should be parsed without error. See [#10685](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/10685). Thanks to @tsarna for the contribution.
|
||||
* BUGFIX: [vmsingle](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/) and `vmstorage` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/): prevent panic during directory deletion on `NFS`-based mounts. The bug was introduced in [83da33d8](https://github.com/VictoriaMetrics/VictoriaMetrics/commit/83da33d8cfe8352fd0022d05a8b6346ebb48420d) and included in [v1.123.0](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/docs/victoriametrics/changelog/CHANGELOG_2025.md#v11230). See [#9842](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/9842).
|
||||
|
||||
## [v1.136.2](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.136.2)
|
||||
|
||||
Released at 2026-03-13
|
||||
@@ -293,6 +306,17 @@ See changes [here](https://docs.victoriametrics.com/victoriametrics/changelog/ch
|
||||
|
||||
See changes [here](https://docs.victoriametrics.com/victoriametrics/changelog/changelog_2025/#v11230)
|
||||
|
||||
## [v1.122.18](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.122.18)
|
||||
|
||||
Released at 2026-03-27
|
||||
|
||||
**v1.122.x is a line of [LTS releases](https://docs.victoriametrics.com/victoriametrics/lts-releases/). It contains important up-to-date bugfixes for [VictoriaMetrics enterprise](https://docs.victoriametrics.com/victoriametrics/enterprise/).
|
||||
All these fixes are also included in [the latest community release](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest).
|
||||
The v1.122.x line will be supported for at least 12 months since [v1.122.0](https://docs.victoriametrics.com/victoriametrics/changelog/#v11220) release**
|
||||
|
||||
* BUGFIX: `vmselect` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/): retry RPC by dialing a new connection instead of reusing a pooled one when the previous attempt fails with `io.EOF`, `broken pipe` or `reset by peer`. This reduces query failures caused by stale connections to restarted vmstorage nodes. See [#10314](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/10314)
|
||||
* BUGFIX: [vmsingle](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/) and `vmselect` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/): enforce `datasource_type=prometheus` when [proxying](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#vmalert) Grafana requests to [vmalert](https://docs.victoriametrics.com/victoriametrics/vmalert/). Grafana supports only `prometheus` and `loki` alerts. Without this fix, Grafana shows `Error loading alerts` when non-Prometheus alert types are returned. See [victoriametrics-datasource#329](https://github.com/VictoriaMetrics/victoriametrics-datasource/issues/329).
|
||||
|
||||
## [v1.122.17](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.122.17)
|
||||
|
||||
Released at 2026-03-13
|
||||
@@ -1418,4 +1442,4 @@ See changes [here](https://docs.victoriametrics.com/victoriametrics/changelog/ch
|
||||
|
||||
## Previous releases
|
||||
|
||||
See [releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases).
|
||||
See [releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases).
|
||||
@@ -36,6 +36,10 @@ Pass `-help` to `vmauth` in order to see all the supported command-line flags wi
|
||||
|
||||
Feel free to [contact us](mailto:info@victoriametrics.com) if you need a customized auth proxy for VictoriaMetrics with the support of LDAP, SSO, RBAC, SAML, accounting, and rate limiting, such as [vmgateway](https://docs.victoriametrics.com/victoriametrics/vmgateway/).
|
||||
|
||||
> [VictoriaMetrics Cloud](https://console.victoriametrics.cloud/signUp?utm_source=website&utm_campaign=docs_vm_vmauth_access)
|
||||
> provides built-in access control with organization-level roles, and lets you create and revoke read, write, or read/write access tokens with a couple of clicks from the UI, without the need of additional proxies.
|
||||
> See the [VictoriaMetrics Cloud documentation](https://docs.victoriametrics.com/victoriametrics-cloud/) to get started.
|
||||
|
||||
## Use cases
|
||||
|
||||
* [Simple HTTP proxy](#simple-http-proxy)
|
||||
|
||||
@@ -26,6 +26,10 @@ This tool simplifies the creation of hourly, daily, weekly and monthly backups.
|
||||
Enterprise binaries can be downloaded and evaluated for free from [the releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest).
|
||||
See how to request a [free trial license](https://victoriametrics.com/products/enterprise/trial/).
|
||||
|
||||
> If you'd prefer not to manage backups at all, [VictoriaMetrics Cloud](https://console.victoriametrics.cloud/signUp?utm_source=website&utm_campaign=docs_vm_vmbackup_intro)
|
||||
> handles backups and restores automatically without the need of extra configurations.
|
||||
> See the [VictoriaMetrics Cloud documentation](https://docs.victoriametrics.com/victoriametrics-cloud/) to get started.
|
||||
|
||||
### Single node backup
|
||||
To make a complete backup for VictoriaMetrics single node run the following command:
|
||||
```sh
|
||||
|
||||
@@ -16,6 +16,10 @@ aliases:
|
||||
|
||||
Restore process can be interrupted at any time. It is automatically resumed when restarting `vmrestore` with the same args. If file preallocation is enabled{{% available_from "v1.139.0" %}}, it resumes from the last complete file, if file preallocation is disabled via `-skipFilePreallocation` then it resumes from the interruption point mid file.
|
||||
|
||||
> If you'd prefer not to manage backups/restores at all, [VictoriaMetrics Cloud](https://console.victoriametrics.cloud/signUp?utm_source=website&utm_campaign=docs_vm_vmbackup_intro)
|
||||
> handles backups and restores automatically without the need of extra configurations.
|
||||
> See the [VictoriaMetrics Cloud documentation](https://docs.victoriametrics.com/victoriametrics-cloud/) to get started.
|
||||
|
||||
## Usage
|
||||
|
||||
VictoriaMetrics must be stopped during the restore process.
|
||||
|
||||
Reference in New Issue
Block a user