Compare commits

...

5 Commits

Author SHA1 Message Date
Artem Fetishev
94af588e92 docs: forward port LTS v1.122.18 changelog to upstream
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
2026-04-01 14:24:20 +02:00
Artem Fetishev
e5c194cc10 docs: forward port LTS v1.136.3 changelog to upstream
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
2026-04-01 14:22:51 +02:00
Artem Fetishev
7c65e3daca docs: fix changelog
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
2026-04-01 13:17:16 +02:00
Pablo (Tomas) Fernandez
a6532c28b2 docs/guides: Add new guide "Set up Datasource-Managed Alerts with vmalert and Grafana" (#10691)
Create a guide to use datasource-managed alerts in Grafana

See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/10528

Signed-off-by: Pablo (Tomas) Fernandez <46322567+TomFern@users.noreply.github.com>
Co-authored-by: Mathias Palmersheim <mathias@victoriametrics.com>
2026-03-31 18:57:04 +03:00
Jose Gómez-Sellés
ec26ebb803 docs: raise cloud awareness in docs (#10716)
### Describe Your Changes

Some users may not know that VictoriaMetrics Cloud provides relevant
features to manage workloads. This change add notes in relevant places
in which users may find that a managed solution is what they need.

The intention is not to push users to Cloud, but giving the information.
That's why it's always phrased like: "If you don't want to do X, Cloud
can do it for you", instead of "Start for free, etc". This is an Open
Source first project, and shall remain as such.

After this gets proper review, VictoriaLogs and other repos may follow.

### Checklist

The following checks are **mandatory**:

- [X] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/#pull-request-checklist).
- [X] My change adheres to [VictoriaMetrics development
goals](https://docs.victoriametrics.com/victoriametrics/goals/).

---------

Signed-off-by: Jose Gómez-Sellés <14234281+jgomezselles@users.noreply.github.com>
Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>
2026-03-31 18:51:06 +03:00
17 changed files with 705 additions and 6 deletions

View File

@@ -7,6 +7,9 @@ sitemap:
disable: true
---
> [!NOTE] Tip
> Store every configuration file you use during this guide in version control. You may need them for reference or to change the configuration of your installation.
This guide walks you through deploying a VictoriaMetrics cluster version on Kubernetes.
By the end of this guide, you will know:
@@ -382,7 +385,7 @@ Consider reading these resources to complete your setup:
- VictoriaMetrics
- [Learn more about the cluster version](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/)
- [Migrate existing metric data into VictoriaMetrics with vmctl](https://docs.victoriametrics.com/victoriametrics/vmctl/)
- [Setup alerts](https://docs.victoriametrics.com/victoriametrics/vmalert/)
- [Setup alerts](https://docs.victoriametrics.com/guides/vmalert-datasource-managed-alerts-grafana/)
- Grafana
- [Enable persistent storage](https://grafana.com/docs/grafana/latest/setup-grafana/installation/helm/#enable-persistent-storage-recommended)
- [Configure private TLS authority](https://grafana.com/docs/grafana/latest/setup-grafana/installation/helm/#configure-a-private-ca-certificate-authority)

View File

@@ -7,6 +7,9 @@ sitemap:
disable: true
---
> [!NOTE] Tip
> Store every configuration file you use during this guide in version control. You may need them for reference or to change the configuration of your installation.
This guide walks you through deploying a [single-node version of VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/) on Kubernetes using Helm.
At the end of this guide, you will know:
@@ -56,10 +59,19 @@ vm/victoria-metrics-anomaly 1.12.9 v1.28.2 Victoria
## 2. Install [VictoriaMetrics single](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/) from Helm Chart
Download the Helm value files with the example configuration for VictoriaMetrics single-node:
```sh
wget https://docs.victoriametrics.com/guides/examples/guide-vmsingle-values.yaml
```
> [!NOTE] Tip
> Keep this file in version control. You may need it to update the configuration of your VictoriaMetrics service
Run this command in your terminal to install [VictoriaMetrics single node](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/) to the default [namespace](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) in your cluster:
```shell
helm install vmsingle vm/victoria-metrics-single -f https://docs.victoriametrics.com/guides/examples/guide-vmsingle-values.yaml
helm install vmsingle vm/victoria-metrics-single -f guide-vmsingle-values.yaml
```
Below are the key sections in the chart values file [`guide-vmsingle-values.yaml`](https://docs.victoriametrics.com/guides/examples/guide-vmsingle-values.yaml):
@@ -315,7 +327,7 @@ Consider reading these resources to complete your setup:
- VictoriaMetrics
- [Learn more about the single-node version](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/)
- [Migrate existing metric data into VictoriaMetrics with vmctl](https://docs.victoriametrics.com/victoriametrics/vmctl/)
- [Setup alerts](https://docs.victoriametrics.com/victoriametrics/vmalert/)
- [Setup alerts](https://docs.victoriametrics.com/guides/vmalert-datasource-managed-alerts-grafana/)
- Grafana
- [Enable persistent storage](https://grafana.com/docs/grafana/latest/setup-grafana/installation/helm/#enable-persistent-storage-recommended)
- [Configure private TLS authority](https://grafana.com/docs/grafana/latest/setup-grafana/installation/helm/#configure-a-private-ca-certificate-authority)

View File

@@ -0,0 +1,615 @@
---
build:
list: never
publishResources: false
render: never
sitemap:
disable: true
---
Grafana offers a rich alerting UI, including rule grouping, silences, and notification history. While Grafana-managed alerts are easy to use, [they can hit performance issues without additional configuration since they depend on a relational database by default](https://grafana.com/blog/how-we-improved-grafanas-alert-state-history-to-provide-better-insights-into-your-alerting-data/).
By moving rule evaluation to [vmalert](https://docs.victoriametrics.com/victoriametrics/vmalert/), you can move past these limitations while retaining Grafana's unified alerting UI. This guide shows the ideal topology for scalable alerting using vmalert, Alertmanager, and Grafana datasource-managed alerts.
## Grafana Alert Modes
Grafana supports two alert modes, which can run side by side:
- Grafana-managed: alerts are created and evaluated entirely within Grafana itself. The alert state is stored in a SQL database by default, but [Grafana can be configured to store this data in a Prometheus-compatible database like VictoriaMetrics as well](https://grafana.com/docs/grafana/latest/alerting/set-up/configure-alert-state-history/#configure-prometheus-for-alert-state-grafana_alerts-metric).
- Datasource-managed: alerts have their rules defined, stored, and evaluated in an external system like vmalert and Alertmanager, with Grafana just providing the UI. State is stored in VictoriaMetrics.
The following table compares the two modes:
| Aspect | Grafana-Managed | Data Source-Managed |
| ------------------- | -------------------------------------------------- | ----------------------------- |
| Where rules live | Grafana's SQL database | vmalert's YAML config |
| Evaluation | Grafana's scheduler | vmalert |
| Horizontal Scaling | Complex (requires HA SQL database) | Simple (add more pods) |
| State storage | SQL backend or a Prometheus datasource | VictoriaMetrics |
| UI Management | Full create/edit in Grafana | View-only |
| Dependencies | SQL + Grafana | VictoriaMetrics |
| Version control | Complex (rules must be exported/imported) | Easy (rules are in YAML file) |
## Datasource-managed Alert Topology
The proposed alert setup relies on the following services:
- VictoriaMetrics: provides the time-series database and persists alert status.
- vmalert: evaluates alerting rules from its config file against VictoriaMetrics data and forwards firing alerts to Alertmanager.
- Alertmanager: groups and routes alerts to the configured recipients.
- Grafana: serves as the unified UI, connecting to VictoriaMetrics for rules and metrics, and to Alertmanager for notifications/silences.
![Topology Diagram](topology.webp)
## vmalert Demo with Docker {#docker}
In this section, we'll describe how you can try datasource-managed alerts on Grafana with Docker Compose. Follow the steps in this section to see how the Grafana UI looks in datasource-managed alerts.
First, create `alerts.yml`. The following configuration file creates an always-firing alert, which you can use to test datasource-managed alerts in Grafana.
```yaml
# alerts.yml
groups:
- name: demo
rules:
# Always-firing demo alert so you see something immediately
- alert: AlwaysFiring
expr: vector(1)
for: 10s
labels:
severity: warning
annotations:
summary: "Demo alert that always fires"
description: "This is a demo alert from vmalert using vector(1)."
# Simple recording rule you can graph in Grafana
- record: demo:vector_one
expr: vector(1)
```
Next, create a basic Alertmanager config called `alertmanager.yml`. This example does not forward alerts anywhere, but serves as a source for Grafana:
```yaml
# alertmanager.yml
global:
resolve_timeout: 5m
route:
receiver: "log"
receivers:
- name: "log"
```
Finally, create `grafana-datasources.yml` to configure Grafana to use VictoriaMetrics and Alertmanager as datasources for alerts and notifications:
```yaml
# grafana-datasources.yml
apiVersion: 1
datasources:
- name: VictoriaMetrics
type: prometheus
access: proxy
url: http://victoriametrics:8428
isDefault: true
- name: Alertmanager
type: alertmanager
access: proxy
url: http://alertmanager:9093
jsonData:
implementation: prometheus
```
The final piece is the Docker Compose file. This ties all the services together and sets up the required command-line arguments:
```yaml
# compose.yml
services:
victoriametrics:
image: victoriametrics/victoria-metrics:v1.138.0
command:
- "--storageDataPath=/victoria-metrics-data"
- "--selfScrapeInterval=10s"
# Proxy vmalert APIs so Grafana can see rules via VictoriaMetrics
- "--vmalert.proxyURL=http://vmalert:8880"
ports:
- "8428:8428"
volumes:
- vm-data:/victoria-metrics-data
alertmanager:
image: prom/alertmanager:v0.31.1
command:
- "--config.file=/etc/alertmanager/alertmanager.yml"
ports:
- "9093:9093"
volumes:
- ./alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro
vmalert:
image: victoriametrics/vmalert:v1.138.0
depends_on:
- victoriametrics
- alertmanager
command:
# read metrics from VictoriaMetrics
- "--datasource.url=http://victoriametrics:8428"
# store and retrieve alert state from VictoriaMetrics
- "--remoteRead.url=http://victoriametrics:8428"
- "--remoteWrite.url=http://victoriametrics:8428"
# configure Alertmanager as the notifier
- "--notifier.url=http://alertmanager:9093"
- "--rule=/etc/vmalert/alerts.yml"
- "--evaluationInterval=15s"
# external settings link vmalerts and Alertmanager to Grafana
- "--external.url=http://localhost:3000"
- "--external.alert.source=explore?orgId=1&left={\"datasource\":\"VictoriaMetrics\",\"queries\":[{\"refId\":\"A\",\"expr\":\"{{.Expr|queryEscape}}\"}]}"
ports:
- "8880:8880"
volumes:
- ./alerts.yml:/etc/vmalert/alerts.yml:ro
grafana:
image: grafana/grafana:12.4
depends_on:
- victoriametrics
- alertmanager
environment:
GF_SECURITY_ADMIN_USER: admin
GF_SECURITY_ADMIN_PASSWORD: admin
GF_PATHS_PROVISIONING: /etc/grafana/provisioning
ports:
- "3000:3000"
volumes:
- ./grafana-datasources.yml:/etc/grafana/provisioning/datasources/datasources.yml:ro
- grafana-data:/var/lib/grafana
volumes:
vm-data:
grafana-data:
```
Let's break down the main command line arguments that connect every component:
- VictoriaMetrics
- `-vmalert.proxyURL`: forwards Grafana requests for `/api/v1/rules` and `/api/v1/alerts` to vmalert, enabling rule visibility in Grafana UI
- vmalert
- `-datasource.url`: configures VictoriaMetrics as the query source for rule evaluation
- `-remoteWrite.url`: defines VictoriaMetrics as the backend used to persist rule state across restarts
- `-remoteRead.url`: defines VictoriaMetrics as the backend used to read historical state for pending alerts
- `-notifier.url`: directs firing alerts to Alertmanager
- `-external.url`: defines the base URL for alert links so users see the public URL of Alertmanager, or an external alerting UI like Karma or Grafana in notifications.
- `-external.alert.source`: creates a template for clickable alert links for Grafana. Allows Alertmanager UI to link directly to Grafana
Now, start the demo with:
```sh
docker compose up -d
```
Open your browser at `localhost:3000` and log in to Grafana with username `admin` and password `admin`.
If you open the sidebar and select **Alerting** > **Alert rules**, you should be able to see one alert pending or firing.
![Screenshot of Grafana alert pane](grafana-alert-firing.webp)
<figcaption style="text-align: center; font-style: italic;">Datasource-managed alert firing in Grafana</figcaption>
Open the sidebar again and go to **Alerting** > **Active notifications** to see the active alert reported by Alertmanager.
![Screenshot of Grafana Active notifications Page](grafana-active-notifications.webp)
You can also see the alerts in VMUI by opening the browser in `http://localhost:8428/vmui/?#/rules`. This is possible only when we have configured `-vmalert.proxyURL` in VictoriaMetrics.
![Screenshot of VMUI](vmui-alerts.webp)
<figcaption style="text-align: center; font-style: italic;">Alerts can be visualized in VMUI directly</figcaption>
If you open the browser in `http://localhost:9093/#/alerts`, you will see the Alertmanager UI with the firing alert.
![Screenshot of Alertmanager](alertmanager-alerts.webp)
<figcaption style="text-align: center; font-style: italic;">Alertmanager UI showing the firing alert</figcaption>
Clicking **Source** should take you back to Grafana and display the query that generated the alert.
## vmalert and VictoriaMetrics Single on Kubernetes {#vmsingle}
This section explains how to configure datasource-managed alerts on the VictoriaMetrics single-node version on Kubernetes.
### Prerequisites
- A Kubernetes cluster
- VictoriaMetrics single-node
- Grafana
- Helm values or config files used for installation
You can follow this guide to install all required components: [Kubernetes monitoring via VictoriaMetrics single](https://docs.victoriametrics.com/guides/k8s-monitoring-via-vm-single/).
### 1. Ensure VictoriaMetrics and Grafana are installed
Ensure you have added and updated the VictoriaMetrics Helm repository:
```sh
helm repo add vm https://victoriametrics.github.io/helm-charts/
helm repo update
```
Confirm that the VictoriaMetrics single-node version is installed (assuming the release name `vmsingle` from the [installation guide](https://docs.victoriametrics.com/guides/k8s-monitoring-via-vm-single/))
```sh
kubectl get pods -l app.kubernetes.io/instance=vmsingle
```
You should get a single running pod:
```sh
NAME READY STATUS RESTARTS AGE
vmsingle-victoria-metrics-single-server-0 1/1 Running 0 48m
```
Do the same for Grafana:
```sh
kubectl get pod -l app.kubernetes.io/name=grafana
```
You should get the name of the Grafana pod running in your cluster:
```sh
NAME READY STATUS RESTARTS AGE
my-grafana-65d6d4ccbc-nxkxq 1/1 Running 0 58m
```
### 2. Install vmalert and Alertmanager
Create a Helm values file for vmalert and Alertmanager called `vm-alerting-values.yml`.
The example below comes with two demo alerts. Add your own vmalert [alerting rules](https://docs.victoriametrics.com/victoriametrics/vmalert/#rules) in the `config: alerts:` section below.
```sh
cat <<EOF > vm-alerting-values.yml
# Enable and configure Alertmanager
alertmanager:
enabled: true
config:
global:
resolve_timeout: 5m
route:
group_by: ["alertname"]
group_wait: 30s
group_interval: 5m
repeat_interval: 12h
receiver: "log"
receivers:
- name: "log"
# place your default route here for notifications
# Configure vmalert ("server" section in this chart)
server:
# vmalert evaluation datasource: point at vmsingles Prometheus-compatible API
datasource:
url: http://vmsingle-victoria-metrics-single-server.default.svc.cluster.local.:8428
# Where vmalert stores and reads alert state (remote write/read)
remote:
write:
url: http://vmsingle-victoria-metrics-single-server.default.svc.cluster.local.:8428
read:
url: http://vmsingle-victoria-metrics-single-server.default.svc.cluster.local.:8428
# Configure Alertmanager as notifier
notifier:
alertmanager:
# Adjust namespace/name if you install into a non-default namespace or change the release name
url: http://vmalert-victoria-metrics-alert-alertmanager:9093
# Inline demo rules. Add your alerting groups and rules here
config:
alerts:
groups:
- name: vm-health
rules:
- alert: TooManyRestarts
expr: changes(process_start_time_seconds{job=~"victoriametrics|vmagent|vmalert"}[15m]) > 2
labels:
severity: critical
annotations:
summary: "{{ $labels.job }} too many restarts (instance {{ $labels.instance }})"
description: "Job {{ $labels.job }} has restarted more than twice in the last 15 minutes. It might be crashlooping."
- alert: ServiceDown
expr: up{job=~"victoriametrics|vmagent|vmalert"} == 0
for: 2m
labels:
severity: critical
annotations:
summary: "Service {{ $labels.job }} is down on {{ $labels.instance }}"
description: "{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 2 minutes."
EOF
```
Install `vmalert` and Alertmanager with:
```sh
helm install vmalert vm/victoria-metrics-alert -f vm-alerting-values.yml
```
### 3. Configure VictoriaMetrics single to proxy to vmalert
For datasource-managed alerts, Grafana talks to VictoriaMetrics, and VictoriaMetrics proxies alerting-related API calls to vmalert via the `-vmalert.proxyURL` flag.
First, check the service name for `vmalert`:
```sh
kubectl get svc -l app.kubernetes.io/instance=vmalert,app=server
```
Get the name of the `vmalert` service:
```sh
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
vmalert-victoria-metrics-alert-server ClusterIP None <none> 8880/TCP 58m
```
The internal DNS name for this service, in the default namespace, will be:
```text
vmalert-victoria-metrics-alert-server.default.svc.cluster.local:8880
```
Next, create a Helm values file with the internal Kubernetes URL for vmalert.
```sh
cat <<EOF > vm-vmalert-proxy-values.yml
# vm-vmalert-proxy-values.yaml
server:
extraArgs:
vmalert.proxyURL: http://vmalert-victoria-metrics-alert-server.default.svc.cluster.local:8880
EOF
```
Update the configuration of VictoriaMetrics single by applying both *your original Helm values* and this new overlay. In the example below, the original values file is called `vmsingle-values-file.yml` (this is the file you used when you first installed the cluster):
```sh
helm upgrade vmsingle vm/victoria-metrics-single \
-f vmsingle-values-file.yml \
-f vm-vmalert-proxy-values.yml
```
After this upgrade, vmsingle will start proxying `/api/v1/rules`, `/api/v1/alerts`, and other `vmalert` [endpoints](https://docs.victoriametrics.com/victoriametrics/vmalert/#web) to the vmalert service, enabling Grafanas alerting UI and API to work through the VictoriaMetrics datasource.
To finish the setup, jump to the [Configure Grafana](https://docs.victoriametrics.com/guides/vmalert-datasource-managed-alerts-grafana/#grafana) section
## vmalert and VictoriaMetrics Cluster on Kubernetes {#vmcluster}
This section explains how to configure datasource-managed alerts on the VictoriaMetrics cluster version on Kubernetes.
### Prerequisites
- A Kubernetes cluster
- VictoriaMetrics cluster
- Grafana
- Helm values or config files used for the installation of the cluster
You can follow this guide to install the cluster and Grafana first: [Kubernetes monitoring with VictoriaMetrics cluster](https://docs.victoriametrics.com/guides/k8s-monitoring-via-vm-cluster/).
### 1. Ensure VictoriaMetrics and Grafana are installed
Ensure you have added and updated the VictoriaMetrics Helm repository:
```sh
helm repo add vm https://victoriametrics.github.io/helm-charts/
helm repo update
```
Confirm that the VictoriaMetrics cluster is installed (assuming the release name `vmcluster` from the [installation guide](https://docs.victoriametrics.com/guides/k8s-monitoring-via-vm-cluster/)):
```sh
kubectl get pods -l app.kubernetes.io/instance=vmcluster
```
You should see pods for `vmselect`, `vminsert`, and `vmstorage`, for example:
```text
NAME READY STATUS RESTARTS AGE
vmcluster-victoria-metrics-cluster-vminsert-b4d494b4c-cx2m5 1/1 Running 0 3m8s
vmcluster-victoria-metrics-cluster-vminsert-b4d494b4c-xdv76 1/1 Running 0 3m8s
vmcluster-victoria-metrics-cluster-vmselect-67979c98fc-7hfdl 1/1 Running 0 3m8s
vmcluster-victoria-metrics-cluster-vmselect-67979c98fc-ftpzn 1/1 Running 0 3m8s
vmcluster-victoria-metrics-cluster-vmstorage-0 1/1 Running 0 3m8s
vmcluster-victoria-metrics-cluster-vmstorage-1 1/1 Running 0 2m48s
```
VictoriaMetrics exposes its write API via the `vminsert` service on port 8480 and its read (Prometheus-compatible) API via the `vmselect` service on port 8481 by default. For a default installation, these DNS names are:
- Write: `vmcluster-victoria-metrics-cluster-vminsert.default.svc.cluster.local.:8480`
- Read: `vmcluster-victoria-metrics-cluster-vmselect.default.svc.cluster.local.:8481`
Now, ensure Grafana is installed:
```sh
kubectl get pod -l app.kubernetes.io/name=grafana
```
You should get the name of the Grafana pod running in your cluster:
```sh
NAME READY STATUS RESTARTS AGE
my-grafana-65d6d4ccbc-nxkxq 1/1 Running 0 58m
```
### 2. Install vmalert and Alertmanager
Create a Helm values file for vmalert and Alertmanager called `vm-alerting-values.yml`.
The example below comes with two demo alerts. Add your own vmalert [alerting rules](https://docs.victoriametrics.com/victoriametrics/vmalert/#rules) in the `config: alerts:` section below.
```sh
cat <<EOF > vm-alerting-values.yml
# Enable and configure Alertmanager
alertmanager:
enabled: true
config:
global:
resolve_timeout: 5m
route:
group_by: ["alertname"]
group_wait: 30s
group_interval: 5m
repeat_interval: 12h
receiver: "log"
receivers:
- name: "log"
# place your notification route here
# Configure vmalert ("server" section in this chart)
server:
# vmalert evaluation datasource: point at vmselects Prometheus-compatible API
datasource:
url: http://vmcluster-victoria-metrics-cluster-vmselect.default.svc.cluster.local.:8481/select/multitenant/prometheus/
# Where vmalert stores and reads alert state (remote write/read)
remote:
write:
# send ALERTS / recording rule series into vminsert
url: http://vmcluster-victoria-metrics-cluster-vminsert.default.svc.cluster.local.:8480/insert/0/prometheus/
read:
# read back alert state from vmselect
url: http://vmcluster-victoria-metrics-cluster-vmselect.default.svc.cluster.local.:8481/select/0/prometheus/
# Configure Alertmanager as notifier
notifier:
alertmanager:
# Adjust namespace/name if you install into a non-default namespace or change the release name
url: http://vmalert-victoria-metrics-alert-alertmanager:9093
# Inline demo rules. Add your alerting groups and rules here
config:
alerts:
groups:
- name: vm-health
rules:
- alert: TooManyRestarts
expr: changes(process_start_time_seconds{job=~"victoriametrics|vmagent|vmalert"}[15m]) > 2
labels:
severity: critical
annotations:
summary: "{{ $labels.job }} too many restarts (instance {{ $labels.instance }})"
description: "Job {{ $labels.job }} has restarted more than twice in the last 15 minutes.
It might be crashlooping."
- alert: ServiceDown
expr: up{job=~"victoriametrics|vmagent|vmalert"} == 0
for: 2m
labels:
severity: critical
annotations:
summary: "Service {{ $labels.job }} is down on {{ $labels.instance }}"
description: "{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 2 minutes."
EOF
```
The key differences from the [single-node setup](https://docs.victoriametrics.com/guides/vmalert-datasource-managed-alerts-grafana/#vmsingle) section
- `server.datasource.url` and `server.remote.read.url` point to the `vmselect` read endpoint (`/select/multitenant/prometheus/`).
- `server.remote.write.url` points to the `vminsert` write endpoint (`/insert/multitenant/prometheus/`).
Install `vmalert` and Alertmanager with:
```sh
helm install vmalert vm/victoria-metrics-alert -f vm-alerting-values.yml
```
### 3. Configure VictoriaMetrics Cluster to proxy to vmalert
For datasource-managed alerts, Grafana talks to VictoriaMetrics, and VictoriaMetrics proxies alerting-related API calls to vmalert via the `-vmalert.proxyURL` flag. In the cluster version, set this flag on `vmselect` and point it to the vmalert service so Grafana can reach the alert state.
First, check the service name for vmalert:
```sh
kubectl get svc -l app.kubernetes.io/instance=vmalert,app=server
```
You should see something like:
```sh
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
vmalert-victoria-metrics-alert-server ClusterIP None <none> 8880/TCP 58m
```
The internal DNS name for this service, in the default namespace, will be:
```text
vmalert-victoria-metrics-alert-server.default.svc.cluster.local:8880
```
Create a Helm values overlay file for the cluster called `vmcluster-vmalert-proxy-values.yml`.
The `vmselect.extraArgs` map in the `victoria-metrics-cluster` chart allows you to pass arbitrary command-line flags to vmselect, including `-vmalert.proxyURL`.
```sh
cat <<EOF > vmcluster-vmalert-proxy-values.yml
vmselect:
extraArgs:
vmalert.proxyURL: http://vmalert-victoria-metrics-alert-server.default.svc.cluster.local:8880
EOF
```
Update the VictoriaMetrics cluster configuration by applying both your *original Helm values* and this new overlay. In the example below, the original values file is called `vmcluster-values-file.yml` (this is the file you used when you first installed the cluster):
```sh
helm upgrade vmcluster vm/victoria-metrics-cluster \
-f victoria-metrics-cluster-values.yml \
-f vmcluster-vmalert-proxy-values.yml
```
After this upgrade, vmselect will start proxying `/api/v1/rules`, `/api/v1/alerts`, and other `vmalert` endpoints to the vmalert service, enabling Grafanas alerting UI and API to work through the VictoriaMetrics datasource.
To finalize the setup, continue on to the next section, [Configure Grafana](#grafana).
## Configure Grafana {#grafana}
The last step on any Kubernetes-based installation is to add Alertmanager to Grafana so that Grafana can show notifications alongside the rules it discovers via the VictoriaMetrics datasource.
Get the service name for Alertmanager in your cluster:
```sh
kubectl get svc -l app.kubernetes.io/instance=vmalert,app=alertmanager
```
You should see something like:
```text
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
vmalert-victoria-metrics-alert-alertmanager ClusterIP 10.43.114.243 <none> 9093/TCP 68m
```
Next, add Alertmanager to Grafana:
1. Log in to your Grafana dashboard.
2. Go to **Connections** > **Datasources**.
3. Press **+ Add new data source**.
4. Search and select “Alertmanager”.
5. Fill in the following parameters (adjusting namespace/service name if needed):
- Implementation: Prometheus
- URL: `http://vmalert-victoria-metrics-alert-alertmanager.default.svc.cluster.local:9093`
Ensure the URL matches the Alertmanager service name you obtained earlier.
6. Press **Save & Test**.
![Screenshot of Grafana](grafana-alertmanager.webp)
<figcaption style="text-align: center; font-style: italic;">Adding Alertmanager to Grafana</figcaption>
With vmselects `vmalert.proxyURL` set and Alertmanager configured as a datasource, Grafana should now be able to display vmalert rules and alert instances from the VictoriaMetrics Cluster datasource and show notifications managed by Alertmanager in its UI.
## See also
- [YouTube: Mathias Palmersheim - Who will be your Ruler](https://youtu.be/NfhVOEkznFY)
- [Kubernetes monitoring via VictoriaMetrics single](https://docs.victoriametrics.com/guides/k8s-monitoring-via-vm-single/)
- [Kubernetes monitoring with VictoriaMetrics cluster](https://docs.victoriametrics.com/guides/k8s-monitoring-via-vm-cluster/)
- Learn more about [vmalert](https://docs.victoriametrics.com/victoriametrics/vmalert/)

View File

@@ -0,0 +1,17 @@
---
weight: 16
title: Datasource-Managed Alerts with vmalert and Grafana
menu:
docs:
parent: "guides"
weight: 16
tags:
- metrics
- guide
- kubernetes
- alerts
aliases:
- /guides/vmalert-datasource-managed-alerts-grafana.html
---
{{% content "README.md" %}}

Binary file not shown.

After

Width:  |  Height:  |  Size: 403 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 634 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 373 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 557 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 185 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 114 KiB

View File

@@ -100,6 +100,10 @@ The following steps must be performed during the upgrade / downgrade procedure:
* Wait until the process stops. This can take a few seconds.
* Start the upgraded VictoriaMetrics.
> If you'd prefer not to manage upgrades yourself, [VictoriaMetrics Cloud](https://console.victoriametrics.cloud/signUp?utm_source=website&utm_campaign=docs_vm_bestpractices_upgrade)
> performs version upgrades automatically during scheduled maintenance windows with no action required on your part.
> See the [VictoriaMetrics Cloud documentation](https://docs.victoriametrics.com/victoriametrics-cloud/) to get started.
## Backup Recommendations
VictoriaMetrics supports backups via [vmbackup](https://docs.victoriametrics.com/victoriametrics/vmbackup/) and [vmrestore](https://docs.victoriametrics.com/victoriametrics/vmrestore/) tools. There is also [vmbackupmanager](https://docs.victoriametrics.com/victoriametrics/vmbackupmanager/), which simplifies backup automation.

View File

@@ -755,6 +755,10 @@ The `minimum downtime` strategy has the following benefits comparing to `no down
- It allows minimizing the duration of config update / version upgrade for clusters with big number of nodes
of for clusters with big `vmstorage` nodes, which may take long time for graceful restart.
> If you'd prefer to avoid managing cluster upgrades yourself, [VictoriaMetrics Cloud](https://console.victoriametrics.cloud/signUp?utm_source=website&utm_campaign=docs_vm_cluster_upgrade)
> handles version upgrades automatically during maintenance windows, and lets you create, resize, or delete deployments with a few clicks from the UI.
> See the [VictoriaMetrics Cloud documentation](https://docs.victoriametrics.com/victoriametrics-cloud/) to get started.
### Improving re-routing performance during restart
`vmstorage` nodes may experience increased usage for CPU, RAM and disk IO during

View File

@@ -247,6 +247,10 @@ The following steps must be performed during the upgrade / downgrade procedure:
Prometheus doesn't drop data during VictoriaMetrics restart. See [this article](https://grafana.com/blog/2019/03/25/whats-new-in-prometheus-2.8-wal-based-remote-write/) for details. The same applies also to [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/).
> If you'd prefer not to manage upgrades yourself, [VictoriaMetrics Cloud](https://console.victoriametrics.cloud/signUp?utm_source=website&utm_campaign=docs_vm_single_upgrade)
> performs version upgrades automatically during maintenance windows with no action required on your part.
> See the [VictoriaMetrics Cloud documentation](https://docs.victoriametrics.com/victoriametrics-cloud/) to get started.
## vmui
VictoriaMetrics provides UI for query troubleshooting and exploration. The UI is available at `http://victoriametrics:8428/vmui`
@@ -1786,6 +1790,10 @@ via [vmalert](https://docs.victoriametrics.com/victoriametrics/vmalert/) or via
See also [VictoriaMetrics Monitoring](https://victoriametrics.com/blog/victoriametrics-monitoring/)
and [troubleshooting docs](https://docs.victoriametrics.com/victoriametrics/troubleshooting/).
> [VictoriaMetrics Cloud](https://console.victoriametrics.cloud/signUp?utm_source=website&utm_campaign=docs_vm_single_monitoring)
> provides built-in monitoring dashboards and automatic alerts when resource consumption is high or configured limits are approached,
> so you get notified before issues impact your workload. See the [VictoriaMetrics Cloud documentation](https://docs.victoriametrics.com/victoriametrics-cloud/) to get started.
> VictoriaMetrics components do not expose metadata `TYPE` and `HELP` fields on `/metrics` page.
> Services like Google Cloud Managed Prometheus could require metadata to be present for scraping. In this case, pass `-metrics.exposeMetadata`
command-line to them. See [these docs](https://cloud.google.com/stackdriver/docs/managed-prometheus/troubleshooting#missing-metric-type) for details.

View File

@@ -39,7 +39,7 @@ Released at 2026-03-27
* FEATURE: [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/): allow specifying `basic_auth` in scrape configs without `username`. Previously this resulted in a config error. Now a warning is logged instead. See [#6956](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6956). Thanks to @andriibeee for the contribution.
* FEATURE: [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/) and [vmsingle](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/): add support for negative buckets in [OpenTelemetry](https://docs.victoriametrics.com/victoriametrics/integrations/opentelemetry/) `ExponentialHistogram` during ingestion. See [#9896-comment](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/9896#issuecomment-4037522985).
* FEATURE: [vmrestore](https://docs.victoriametrics.com/victoriametrics/vmrestore/): Improve restore speed on linux systems by pre-allocating files and optimizing write path. This behavior can be disabled with `-skipFilePreallocation`. See [#10661](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/10661/). Thanks to @BenNF for the contribution.
* FEATURE: [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/): [yandexcloud_sd_configs](https://docs.victoriametrics.com/victoriametrics/sd_configs/#yandexcloud_sd_configs) now supports `folder_ids` for limiting discovery to specific folders. See [#10623](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/10623). Thanks to @Br1an67 for the contribution.
* BUGFIX: `vmselect` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/): retry RPC by dialing a new connection instead of reusing a pooled one when the previous attempt fails with `io.EOF`, `broken pipe` or `reset by peer`. This reduces query failures caused by stale connections to restarted vmstorage nodes. See [#10314](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/10314)
* BUGFIX: [vmui](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#vmui): fix autocomplete dropdown not closing on the Raw Query page. See [#10665](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/10665)
@@ -62,7 +62,6 @@ Released at 2026-03-13
* FEATURE: [vmalert](https://docs.victoriametrics.com/victoriametrics/vmalert/): support negative values for the group `eval_offset` option, which allows starting group evaluation at `groupInterval-abs(eval_offset)` within `[0...groupInterval]`. See [#10424](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/10424).
* FEATURE: [vmsingle](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/) and `vmselect` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/): Disable `/graphite/tags/tagSeries` and `/graphite/tags/tagMultiSeries` for Graphite tag registration since it is unlikely it is used in context of VictoriaMetrics. See [10544](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/10544).
* FEATURE: [vmui](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#vmui): rename debug tools buttons for clarity. See [#10453](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/10453).
* FEATURE: [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/): [yandexcloud_sd_configs](https://docs.victoriametrics.com/victoriametrics/sd_configs/#yandexcloud_sd_configs) now supports `folder_ids` for limiting discovery to specific folders. See [#10623](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/10623). Thanks to @Br1an67 for the contribution.
* BUGFIX: all VictoriaMetrics components: replace `histogram` with `untyped` metric metadata type for [VictoriaMetrics histograms](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#histogram) when `-metrics.exposeMetadata` is set. See [#82](https://github.com/VictoriaMetrics/metrics/issues/82).
* BUGFIX: [vmauth](https://docs.victoriametrics.com/victoriametrics/vmauth/): properly route requests to `default_url`. Previously, `request_path` query arg could be set incorrectly during concurrent requests. See [#10626](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/10626).
@@ -104,6 +103,20 @@ It enables back `Discovered targets` debug UI by default.
* BUGFIX: `vmstorage` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/): properly apply `extra_filters[]` filter when querying `vm_account_id` or `vm_project_id` labels via [multitenant](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#multitenancy) request for `/api/v1/label/…/values` API. Before, `extra_filters` was ignored. See [#10503](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/10503).
* BUGFIX: [vmsingle](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/) and `vmselect` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/): revert the use of rollup result cache for [instant queries](https://docs.victoriametrics.com/keyConcepts.html#instant-query) that contain [`rate`](https://docs.victoriametrics.com/MetricsQL.html#rate) function with a lookbehind window larger than `-search.minWindowForInstantRollupOptimization`. The cache usage was removed since [v1.132.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.132.0). See [#10098](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/10098#issuecomment-3895011084) for more details.
## [v1.136.3](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.136.3)
Released at 2026-03-27
**v1.136.x is a line of [LTS releases](https://docs.victoriametrics.com/victoriametrics/lts-releases/). It contains important up-to-date bugfixes for [VictoriaMetrics enterprise](https://docs.victoriametrics.com/victoriametrics/enterprise/).
All these fixes are also included in [the latest community release](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest).
The v1.136.x line will be supported for at least 12 months since [v1.136.0](https://docs.victoriametrics.com/victoriametrics/changelog/#v11360) release**
* BUGFIX: `vmselect` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/): retry RPC by dialing a new connection instead of reusing a pooled one when the previous attempt fails with `io.EOF`, `broken pipe` or `reset by peer`. This reduces query failures caused by stale connections to restarted vmstorage nodes. See [#10314](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/10314)
* BUGFIX: [vmui](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#vmui): fix autocomplete dropdown not closing on the Raw Query page. See [#10665](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/10665)
* BUGFIX: [vmsingle](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/) and `vmselect` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/): enforce `datasource_type=prometheus` when [proxying](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#vmalert) Grafana requests to [vmalert](https://docs.victoriametrics.com/victoriametrics/vmalert/). Grafana supports only `prometheus` and `loki` alerts. Without this fix, Grafana shows `Error loading alerts` when non-Prometheus alert types are returned. See [victoriametrics-datasource#329](https://github.com/VictoriaMetrics/victoriametrics-datasource/issues/329).
* BUGFIX: [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/): stop logging `error`-level messages when scraping targets that expose OpenMetrics `info`, `gaugehistogram`, `stateset`, or `unknown` metric types. These are valid [OpenMetrics](https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md) types and should be parsed without error. See [#10685](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/10685). Thanks to @tsarna for the contribution.
* BUGFIX: [vmsingle](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/) and `vmstorage` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/): prevent panic during directory deletion on `NFS`-based mounts. The bug was introduced in [83da33d8](https://github.com/VictoriaMetrics/VictoriaMetrics/commit/83da33d8cfe8352fd0022d05a8b6346ebb48420d) and included in [v1.123.0](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/docs/victoriametrics/changelog/CHANGELOG_2025.md#v11230). See [#9842](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/9842).
## [v1.136.2](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.136.2)
Released at 2026-03-13
@@ -293,6 +306,17 @@ See changes [here](https://docs.victoriametrics.com/victoriametrics/changelog/ch
See changes [here](https://docs.victoriametrics.com/victoriametrics/changelog/changelog_2025/#v11230)
## [v1.122.18](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.122.18)
Released at 2026-03-27
**v1.122.x is a line of [LTS releases](https://docs.victoriametrics.com/victoriametrics/lts-releases/). It contains important up-to-date bugfixes for [VictoriaMetrics enterprise](https://docs.victoriametrics.com/victoriametrics/enterprise/).
All these fixes are also included in [the latest community release](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest).
The v1.122.x line will be supported for at least 12 months since [v1.122.0](https://docs.victoriametrics.com/victoriametrics/changelog/#v11220) release**
* BUGFIX: `vmselect` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/): retry RPC by dialing a new connection instead of reusing a pooled one when the previous attempt fails with `io.EOF`, `broken pipe` or `reset by peer`. This reduces query failures caused by stale connections to restarted vmstorage nodes. See [#10314](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/10314)
* BUGFIX: [vmsingle](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/) and `vmselect` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/): enforce `datasource_type=prometheus` when [proxying](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#vmalert) Grafana requests to [vmalert](https://docs.victoriametrics.com/victoriametrics/vmalert/). Grafana supports only `prometheus` and `loki` alerts. Without this fix, Grafana shows `Error loading alerts` when non-Prometheus alert types are returned. See [victoriametrics-datasource#329](https://github.com/VictoriaMetrics/victoriametrics-datasource/issues/329).
## [v1.122.17](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.122.17)
Released at 2026-03-13
@@ -1418,4 +1442,4 @@ See changes [here](https://docs.victoriametrics.com/victoriametrics/changelog/ch
## Previous releases
See [releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases).
See [releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases).

View File

@@ -36,6 +36,10 @@ Pass `-help` to `vmauth` in order to see all the supported command-line flags wi
Feel free to [contact us](mailto:info@victoriametrics.com) if you need a customized auth proxy for VictoriaMetrics with the support of LDAP, SSO, RBAC, SAML, accounting, and rate limiting, such as [vmgateway](https://docs.victoriametrics.com/victoriametrics/vmgateway/).
> [VictoriaMetrics Cloud](https://console.victoriametrics.cloud/signUp?utm_source=website&utm_campaign=docs_vm_vmauth_access)
> provides built-in access control with organization-level roles, and lets you create and revoke read, write, or read/write access tokens with a couple of clicks from the UI, without the need of additional proxies.
> See the [VictoriaMetrics Cloud documentation](https://docs.victoriametrics.com/victoriametrics-cloud/) to get started.
## Use cases
* [Simple HTTP proxy](#simple-http-proxy)

View File

@@ -26,6 +26,10 @@ This tool simplifies the creation of hourly, daily, weekly and monthly backups.
Enterprise binaries can be downloaded and evaluated for free from [the releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest).
See how to request a [free trial license](https://victoriametrics.com/products/enterprise/trial/).
> If you'd prefer not to manage backups at all, [VictoriaMetrics Cloud](https://console.victoriametrics.cloud/signUp?utm_source=website&utm_campaign=docs_vm_vmbackup_intro)
> handles backups and restores automatically without the need of extra configurations.
> See the [VictoriaMetrics Cloud documentation](https://docs.victoriametrics.com/victoriametrics-cloud/) to get started.
### Single node backup
To make a complete backup for VictoriaMetrics single node run the following command:
```sh

View File

@@ -16,6 +16,10 @@ aliases:
Restore process can be interrupted at any time. It is automatically resumed when restarting `vmrestore` with the same args. If file preallocation is enabled{{% available_from "v1.139.0" %}}, it resumes from the last complete file, if file preallocation is disabled via `-skipFilePreallocation` then it resumes from the interruption point mid file.
> If you'd prefer not to manage backups/restores at all, [VictoriaMetrics Cloud](https://console.victoriametrics.cloud/signUp?utm_source=website&utm_campaign=docs_vm_vmbackup_intro)
> handles backups and restores automatically without the need of extra configurations.
> See the [VictoriaMetrics Cloud documentation](https://docs.victoriametrics.com/victoriametrics-cloud/) to get started.
## Usage
VictoriaMetrics must be stopped during the restore process.