VictoriaMetrics/deployment/docker/README.md

Docker compose environment for VictoriaMetrics

Docker compose environment for VictoriaMetrics includes VictoriaMetrics and VictoriaLogs components, Alertmanager and Grafana.

For starting the docker-compose environment ensure that you have docker installed and running, and that you have access to the Internet. All commands should be executed from the root directory of the VictoriaMetrics repo.

• Metrics:
  • VictoriaMetrics single server
  • VictoriaMetrics cluster
  • vmagent
• Logs:
  • VictoriaLogs single server
  • VictoriaLogs cluster
• Common
  • vmauth
  • vmalert
  • alertmanager
  • Grafana
• Alerts

VictoriaMetrics single server

To spin-up environment with VictoriaMetrics single server run the following command:

make docker-vm-single-up

See compose-vm-single.yml

VictoriaMetrics will be accessible on the following ports:

• --graphiteListenAddr=:2003
• --opentsdbListenAddr=:4242
• --httpListenAddr=:8428

The communication scheme between components is the following:

• vmagent sends scraped metrics to VictoriaMetrics single-node;
• grafana is configured with datasource pointing to VictoriaMetrics single-node;
• vmalert is configured to query VictoriaMetrics single-node, and send alerts state and recording rules results back to vmagent;
• alertmanager is configured to receive notifications from vmalert.

To access Grafana use link http://localhost:3000.

To access vmui use link http://localhost:8428/vmui.

To access vmalert use link http://localhost:8428/vmalert.

To shutdown environment run:

make docker-vm-single-down

VictoriaMetrics cluster

To spin-up environment with VictoriaMetrics cluster run the following command:

make docker-vm-cluster-up

See compose-vm-cluster.yml

VictoriaMetrics cluster environment consists of vminsert, vmstorage and vmselect components. vminsert exposes port :8480 for ingestion. Access to vmselect for reads goes through vmauth on port :8427, and the rest of components are available only inside the environment.

The communication scheme between components is the following:

• vmagent sends scraped metrics to vminsert;
• vminsert shards and forwards data to vmstorage;
• vmselects are connected to vmstorage for querying data;
• vmauth balances incoming read requests among vmselects;
• grafana is configured with datasource pointing to vmauth;
• vmalert is configured to query vmselects via vmauth and send alerts state and recording rules to vmagent;
• alertmanager is configured to receive notifications from vmalert.

To access Grafana use link http://localhost:3000.

To access vmui use link http://localhost:8427/select/0/prometheus/vmui/.

To access vmalert use link http://localhost:8427/select/0/prometheus/vmalert/.

To shutdown environment execute the following command:

make docker-vm-cluster-down

vmagent

vmagent is used for scraping and pushing time series to VictoriaMetrics instance. It accepts Prometheus-compatible configuration with listed targets for scraping:

• scraping VictoriaMetrics single-node services;
• scraping VictoriaMetrics cluster services;
• scraping VictoriaLogs single-node services;
• scraping VictoriaLogs cluster services;

Web interface link is http://localhost:8429/.

VictoriaLogs server

To spin-up environment with VictoriaLogs run the following command:

make docker-vl-single-up

See compose-vl-single.yml

VictoriaLogs will be accessible on the --httpListenAddr=:9428 port. In addition to VictoriaLogs server, the docker compose contains the following components:

• vector service for collecting docker logs and sending them to VictoriaLogs;
• VictoriaMetrics single-node to collect metrics from all the components;
• Grafana is configured with VictoriaLogs datasource.
• vmalert is configured to query VictoriaLogs single-node, and send alerts state and recording rules results to VictoriaMetrics single-node;
• alertmanager is configured to receive notifications from vmalert.

To access Grafana use link http://localhost:3000.

To access VictoriaLogs UI use link http://localhost:8427/select/vmui.

Please, also see how to monitor VictoriaLogs installations.

To shutdown environment execute the following command:

make docker-vl-single-down

VictoriaLogs cluster

To spin-up environment with VictoriaLogs cluster run the following command:

make docker-vl-cluster-up

See compose-vl-cluster.yml

VictoriaLogs cluster environment consists of vlinsert, vlstorage and vlselect components. vlinsert and vlselect are available through vmauth on port :8427. For example, vector pushes logs via http://vmauth:8427/insert/elasticsearch/ path, and Grafana queries http://vmauth:8427 for datasource queries.

The rest of components are available only inside the environment.

In addition to VictoriaLogs cluster, the docker compose contains the following components:

• vector service for collecting docker logs and sending them to vlinsert;
• Grafana is configured with VictoriaLogs datasource and pointing to vmauth.
• VictoriaMetrics single-node to collect metrics from all the components;
• vlinsert forwards ingested data to vlstorage
• vlselects are connected to vlstorage for querying data;
• vmauth balances incoming read and write requests among vlselects and vlinserts;
• vmalert is configured to query vlselects, and send alerts state and recording rules results to VictoriaMetrics single-node;
• alertmanager is configured to receive notifications from vmalert.

To access Grafana use link http://localhost:3000.

To access VictoriaLogs UI use link http://localhost:8427/select/vmui.

Please, also see how to monitor VictoriaLogs installations.

To shutdown environment execute the following command:

make docker-vl-cluster-down

Please see more examples on integration of VictoriaLogs with other log shippers below:

• filebeat
• fluentbit
• logstash
• promtail
• vector
• datadog-agent
• journald
• opentelemetry-collector
• telegraf
• fluentd
• datadog-serverless

Common components

vmauth

vmauth acts as a load balancer to spread the load across vmselect's or vlselect's. Grafana and vmalert use vmauth for read queries. vmauth routes read queries to VictoriaMetrics or VictoriaLogs depending on requested path. vmauth config is available here for VictoriaMetrics, VictoriaLogs single-server, VictoriaLogs cluster.

vmalert

vmalert evaluates various alerting rules. It is connected with AlertManager for firing alerts, and with VictoriaMetrics or VictoriaLogs for executing queries and storing alert's state.

Web interface link http://localhost:8880/.

alertmanager

AlertManager accepts notifications from vmalert and fires alerts. All notifications are blackholed according to alertmanager.yml config.

Web interface link http://localhost:9093/.

Grafana

Web interface link http://localhost:3000.

Default credentials:

• login: admin
• password: admin

Grafana is provisioned with default dashboards and datasources.

Alerts

See below a list of recommended alerting rules for various VictoriaMetrics components for running in production. Some alerting rules thresholds are just recommendations and could require an adjustment. The list of alerting rules is the following:

• alerts-health.yml: alerting rules related to all VictoriaMetrics components for tracking their "health" state;
• alerts.yml: alerting rules related to single-server VictoriaMetrics installation;
• alerts-cluster.yml: alerting rules related to cluster version of VictoriaMetrics;
• alerts-vmagent.yml: alerting rules related to vmagent component;
• alerts-vmalert.yml: alerting rules related to vmalert component;
• alerts-vmauth.yml: alerting rules related to vmauth component;
• alerts-vlogs.yml: alerting rules related to VictoriaLogs;
• alerts-vmanomaly.yml: alerting rules related to VictoriaMetrics Anomaly Detection;

Please, also see how to monitor VictoriaMetrics installations and how to monitor VictoriaLogs installations.