mirror of
https://github.com/VictoriaMetrics/VictoriaMetrics.git
synced 2026-05-18 17:26:31 +03:00
Compare commits
5 Commits
debug-grou
...
docs-add-r
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
d9c8814231 | ||
|
|
3abc327752 | ||
|
|
f5a5637e53 | ||
|
|
c3078246c8 | ||
|
|
a84636e8c2 |
@@ -1,7 +1,10 @@
|
||||
PLATFORM := $(shell uname -m)
|
||||
DOCKER_PLATFORM := "linux/$(if $(findstring $(PLATFORM),arm64),arm64,amd64)"
|
||||
|
||||
# These commands must be run from the VictoriaMetrics repository root
|
||||
|
||||
# Converts images at docs folder to webp format
|
||||
# See https://docs.victoriametrics.com/single-server-victoriametrics/#images-in-documentation
|
||||
# See https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#images-in-documentation
|
||||
docs-image:
|
||||
if [ ! -d vmdocs ]; then \
|
||||
git clone --depth 1 git@github.com:VictoriaMetrics/vmdocs vmdocs; \
|
||||
@@ -12,12 +15,14 @@ docs-image:
|
||||
cd .. && \
|
||||
docker build \
|
||||
-t vmdocs-docker-package \
|
||||
--platform $(DOCKER_PLATFORM) \
|
||||
vmdocs
|
||||
|
||||
docs-debug: docs docs-image
|
||||
docker run \
|
||||
--rm \
|
||||
--name vmdocs-docker-container \
|
||||
--platform $(DOCKER_PLATFORM) \
|
||||
-p 1313:1313 \
|
||||
$(foreach dir,$(wildcard ./docs/$(dir)/*), -v ./docs/$(notdir $(dir)):/opt/docs/content/$(notdir $(dir))) \
|
||||
vmdocs-docker-package
|
||||
@@ -27,6 +32,7 @@ docs-update-version: docs-image
|
||||
docker run \
|
||||
--rm \
|
||||
--entrypoint /usr/bin/find \
|
||||
--platform $(DOCKER_PLATFORM) \
|
||||
--name vmdocs-docker-container \
|
||||
-v ./docs:/opt/docs/content/victoriametrics vmdocs-docker-package \
|
||||
content \
|
||||
@@ -37,6 +43,7 @@ docs-update-version: docs-image
|
||||
docs-images-to-webp: docs-image
|
||||
docker run \
|
||||
--rm \
|
||||
--platform $(DOCKER_PLATFORM) \
|
||||
--entrypoint /usr/bin/find \
|
||||
--name vmdocs-docker-container \
|
||||
-v ./docs:/opt/docs/content/victoriametrics vmdocs-docker-package \
|
||||
|
||||
@@ -74,7 +74,7 @@ Released: 2025-01-20
|
||||
|
||||
> **Note**: This release contains a bug introduced in [v1.18.7](#v1187) - [`PeriodicScheduler`](https://docs.victoriametrics.com/anomaly-detection/components/scheduler/#periodic-scheduler) where configurations with `fit_every` > `fit_window` could cause inference to be skipped for |fit_every - fit_window| time, until the next `fit_every` call happens. For `fit_every` > `fit_window` configurations we recommend upgrading to [v1.20.1](#v1201), which resolves this issue.
|
||||
|
||||
- FEATURE: Added support for per-query `tenant_id` in the [`VmReader`](https://docs.victoriametrics.com/anomaly-detection/components/reader/#vm-reader). This allows overriding the reader-level `tenant_id` within a single global `vmanomaly` configuration on a *per-query* basis, enabling isolation of data for different tenants in separate queries when querying the [VictoriaMetrics cluster version](https://docs.victoriametrics.com/cluster-victoriametrics/). For details, see the [documentation](https://docs.victoriametrics.com/anomaly-detection/components/reader/?highlight=tenant_id#per-query-parameters).
|
||||
- FEATURE: Added support for per-query `tenant_id` in the [`VmReader`](https://docs.victoriametrics.com/anomaly-detection/components/reader/#vm-reader). This allows overriding the reader-level `tenant_id` within a single global `vmanomaly` configuration on a *per-query* basis, enabling isolation of data for different tenants in separate queries when querying the [VictoriaMetrics cluster version](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/). For details, see the [documentation](https://docs.victoriametrics.com/anomaly-detection/components/reader/?highlight=tenant_id#per-query-parameters).
|
||||
- IMPROVEMEMT: Speedup the model infer stage on multicore systems.
|
||||
- IMPROVEMEMT: Speedup the model fitting stage by 1.25–3x, depending on configuration complexity.
|
||||
- IMPROVEMENT: Reduced service RAM usage by 5-10%, depending on configuration complexity.
|
||||
@@ -169,7 +169,7 @@ Released: 2024-10-28
|
||||
Released: 2024-10-22
|
||||
|
||||
- IMPROVEMENT: Added `vmanomaly_version_info` (service) and `vmanomaly_ui_version_info` (vmui) gauges to self-monitoring metrics.
|
||||
- IMPROVEMENT: Added `instance` and `job` labels to [pushed](https://docs.victoriametrics.com/keyconcepts/#push-model) metrics so they have the same labels as vmanomaly metrics that are [pulled](https://docs.victoriametrics.com/keyconcepts/#pull-model)/scraped. Metric labels can be customized via the [`extra_labels` argument](https://docs.victoriametrics.com/anomaly-detection/components/monitoring/?highlight=extra_labels#push-config-parameters). By default job label will be `vmanomaly` and the instance label will be `f'{hostname}:{vmanomaly_port}`. See [monitoring.push](https://docs.victoriametrics.com/anomaly-detection/components/monitoring/#push-config-parameters) for examples and details.
|
||||
- IMPROVEMENT: Added `instance` and `job` labels to [pushed](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#push-model) metrics so they have the same labels as vmanomaly metrics that are [pulled](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#pull-model)/scraped. Metric labels can be customized via the [`extra_labels` argument](https://docs.victoriametrics.com/anomaly-detection/components/monitoring/?highlight=extra_labels#push-config-parameters). By default job label will be `vmanomaly` and the instance label will be `f'{hostname}:{vmanomaly_port}`. See [monitoring.push](https://docs.victoriametrics.com/anomaly-detection/components/monitoring/#push-config-parameters) for examples and details.
|
||||
- IMPROVEMENT: Added a subsection to [monitoring](https://docs.victoriametrics.com/anomaly-detection/components/monitoring/#logs-generated-by-vmanomaly) page with detailed per-component service logs, including reader and writer logs, error handling, metrics updates, and multi-tenancy warnings.
|
||||
- IMPROVEMENT: Added a new [Command-line arguments](https://docs.victoriametrics.com/anomaly-detection/quickstart/#command-line-arguments) subsection to the [Quickstart guide](https://docs.victoriametrics.com/anomaly-detection/quickstart/), providing details on available options for configuring `vmanomaly`.
|
||||
|
||||
@@ -208,7 +208,7 @@ Released: 2024-10-08
|
||||
|
||||
## v1.16.2
|
||||
Released: 2024-10-06
|
||||
- FEATURE: Added support for `multitenant` value in `tenant_id` arg to enable querying across multiple tenants in [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/) (option available from [v1.104.0](https://docs.victoriametrics.com/cluster-victoriametrics/#multitenancy-via-labels)):
|
||||
- FEATURE: Added support for `multitenant` value in `tenant_id` arg to enable querying across multiple tenants in [VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/) (option available from [v1.104.0](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#multitenancy-via-labels)):
|
||||
- Applied when reading input data from `vmselect` via the [VmReader](https://docs.victoriametrics.com/anomaly-detection/components/reader#vm-reader).
|
||||
- Applied when writing generated results through `vminsert` via the [VmWriter](https://docs.victoriametrics.com/anomaly-detection/components/writer#vm-writer).
|
||||
- For more details, refer to the `tenant_id` arg description in the documentation of the mentioned components.
|
||||
@@ -227,7 +227,7 @@ Released: 2024-10-01
|
||||
- FEATURE: Introduced data dumps to a host filesystem for [VmReader](https://docs.victoriametrics.com/anomaly-detection/components/reader#vm-reader). Resource-intensive setups (multiple queries returning many metrics, bigger `fit_window` arg) will have RAM consumption reduced during fit calls.
|
||||
- IMPROVEMENT: Added a `groupby` argument for logical grouping in [multivariate models](https://docs.victoriametrics.com/anomaly-detection/components/models#multivariate-models). When specified, a separate multivariate model is trained for each unique combination of label values in the `groupby` columns. For example, to perform multivariate anomaly detection on metrics at the machine level without cross-entity interference, you can use `groupby: [host]` or `groupby: [instance]`, ensuring one model per entity being trained (e.g., per host). Please find more details [here](https://docs.victoriametrics.com/anomaly-detection/components/models/#group-by).
|
||||
- IMPROVEMENT: Improved performance of [VmReader](https://docs.victoriametrics.com/anomaly-detection/components/reader#vm-reader) on multicore instances for reading and data processing.
|
||||
- IMPROVEMENT: Introduced new CLI argument aliases to enhance compatibility with [Helm charts](https://github.com/VictoriaMetrics/helm-charts/blob/master/charts/victoria-metrics-anomaly/README.md) (i.e. using secrets) and better align with [VictoriaMetrics flags](https://docs.victoriametrics.com/#list-of-command-line-flags):
|
||||
- IMPROVEMENT: Introduced new CLI argument aliases to enhance compatibility with [Helm charts](https://github.com/VictoriaMetrics/helm-charts/blob/master/charts/victoria-metrics-anomaly/README.md) (i.e. using secrets) and better align with [VictoriaMetrics flags](https://docs.victoriametrics.com/victoriametrics#list-of-command-line-flags):
|
||||
- `--licenseFile` as an alias for `--license-file`
|
||||
- `--license.forceOffline` as an alias for `--license-verify-offline`
|
||||
- `--loggerLevel` as an alias for `--log-level`
|
||||
|
||||
@@ -14,7 +14,7 @@ VictoriaMetrics Anomaly Detection, also known as `vmanomaly`, is a service for d
|
||||
|
||||
Please refer to [our QuickStart section](https://docs.victoriametrics.com/anomaly-detection/#practical-guides-and-installation) to find out more.
|
||||
|
||||
> **Note: `vmanomaly` is a part of [enterprise package](https://docs.victoriametrics.com/enterprise/). You need to get a [free trial license](https://victoriametrics.com/products/enterprise/trial/) for evaluation.**
|
||||
> **Note: `vmanomaly` is a part of [enterprise package](https://docs.victoriametrics.com/victoriametrics/enterprise/). You need to get a [free trial license](https://victoriametrics.com/products/enterprise/trial/) for evaluation.**
|
||||
|
||||
## What is anomaly score?
|
||||
Among the metrics produced by `vmanomaly` (as detailed in [vmanomaly output metrics](https://docs.victoriametrics.com/anomaly-detection/components/models#vmanomaly-output)), `anomaly_score` is a pivotal one. It is **a continuous score > 0**, calculated in such a way that **scores ranging from 0.0 to 1.0 usually represent normal data**, while **scores exceeding 1.0 are typically classified as anomalous**. However, it's important to note that the threshold for anomaly detection can be customized in the alert configuration settings.
|
||||
@@ -47,12 +47,12 @@ Please see example graph illustrating this logic below:
|
||||
|
||||
|
||||
## What data does vmanomaly operate on?
|
||||
`vmanomaly` operates on data fetched from VictoriaMetrics, where you can leverage full power of [MetricsQL](https://docs.victoriametrics.com/metricsql/) for data selection, sampling, and processing. Users can also [apply global filters](https://docs.victoriametrics.com/#prometheus-querying-api-enhancements) for more targeted data analysis, enhancing scope limitation and tenant visibility.
|
||||
`vmanomaly` operates on data fetched from VictoriaMetrics, where you can leverage full power of [MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/) for data selection, sampling, and processing. Users can also [apply global filters](https://docs.victoriametrics.com/victoriametrics#prometheus-querying-api-enhancements) for more targeted data analysis, enhancing scope limitation and tenant visibility.
|
||||
|
||||
Respective config is defined in a [`reader`](https://docs.victoriametrics.com/anomaly-detection/components/reader#vm-reader) section.
|
||||
|
||||
## Handling noisy input data
|
||||
`vmanomaly` operates on data fetched from VictoriaMetrics using [MetricsQL](https://docs.victoriametrics.com/metricsql/) queries, so the initial data quality can be fine-tuned with aggregation, grouping, and filtering to reduce noise and improve anomaly detection accuracy.
|
||||
`vmanomaly` operates on data fetched from VictoriaMetrics using [MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/) queries, so the initial data quality can be fine-tuned with aggregation, grouping, and filtering to reduce noise and improve anomaly detection accuracy.
|
||||
|
||||
## Handling timezones
|
||||
|
||||
@@ -158,12 +158,12 @@ models:
|
||||
```
|
||||
|
||||
## Alert generation in vmanomaly
|
||||
While `vmanomaly` detects anomalies and produces scores, it *does not directly generate alerts*. The anomaly scores are written back to VictoriaMetrics, where respective alerting tool, like [`vmalert`](https://docs.victoriametrics.com/vmalert), can be used to create alerts based on these scores for integrating it with your alerting management system. See an example diagram of how `vmanomaly` integrates into observability pipeline for anomaly detection on `node_exporter` metrics:
|
||||
While `vmanomaly` detects anomalies and produces scores, it *does not directly generate alerts*. The anomaly scores are written back to VictoriaMetrics, where respective alerting tool, like [`vmalert`](https://docs.victoriametrics.com/victoriametrics/vmalert), can be used to create alerts based on these scores for integrating it with your alerting management system. See an example diagram of how `vmanomaly` integrates into observability pipeline for anomaly detection on `node_exporter` metrics:
|
||||
|
||||
<img src="https://docs.victoriametrics.com/anomaly-detection/guides/guide-vmanomaly-vmalert/guide-vmanomaly-vmalert_overview.webp" alt="node_exporter_example_diagram" style="width:60%"/>
|
||||
|
||||
## Preventing alert fatigue
|
||||
Produced anomaly scores are designed in such a way that values from 0.0 to 1.0 indicate non-anomalous data, while a value greater than 1.0 is generally classified as an anomaly. However, there are no perfect models for anomaly detection, that's why reasonable defaults expressions like `anomaly_score > 1` may not work 100% of the time. However, anomaly scores, produced by `vmanomaly` are written back as metrics to VictoriaMetrics, where tools like [`vmalert`](https://docs.victoriametrics.com/vmalert) can use [MetricsQL](https://docs.victoriametrics.com/metricsql/) expressions to fine-tune alerting thresholds and conditions, balancing between avoiding [false negatives](https://victoriametrics.com/blog/victoriametrics-anomaly-detection-handbook-chapter-1/#false-negative) and reducing [false positives](https://victoriametrics.com/blog/victoriametrics-anomaly-detection-handbook-chapter-1/#false-positive).
|
||||
Produced anomaly scores are designed in such a way that values from 0.0 to 1.0 indicate non-anomalous data, while a value greater than 1.0 is generally classified as an anomaly. However, there are no perfect models for anomaly detection, that's why reasonable defaults expressions like `anomaly_score > 1` may not work 100% of the time. However, anomaly scores, produced by `vmanomaly` are written back as metrics to VictoriaMetrics, where tools like [`vmalert`](https://docs.victoriametrics.com/victoriametrics/vmalert) can use [MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/) expressions to fine-tune alerting thresholds and conditions, balancing between avoiding [false negatives](https://victoriametrics.com/blog/victoriametrics-anomaly-detection-handbook-chapter-1/#false-negative) and reducing [false positives](https://victoriametrics.com/blog/victoriametrics-anomaly-detection-handbook-chapter-1/#false-positive).
|
||||
|
||||
## How to backtest particular configuration on historical data?
|
||||
|
||||
@@ -392,7 +392,7 @@ Please note that this approach may not fully resolve the issue if subqueries are
|
||||
|
||||
`vmanomaly` supports **vertical** scalability, benefiting from additional CPU cores (resulting in faster processing times) and increased RAM (allowing more models to be trained and larger volumes of timeseries data to be processed efficiently).
|
||||
|
||||
For **horizontal** scalability, `vmanomaly` can be deployed as multiple independent instances, each configured with its own [MetricsQL](https://docs.victoriametrics.com/metricsql/) queries and [configurations](https://docs.victoriametrics.com/anomaly-detection/components/):
|
||||
For **horizontal** scalability, `vmanomaly` can be deployed as multiple independent instances, each configured with its own [MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/) queries and [configurations](https://docs.victoriametrics.com/anomaly-detection/components/):
|
||||
|
||||
- Splitting by **queries** [defined in the reader section](https://docs.victoriametrics.com/anomaly-detection/components/reader#vm-reader) and assigning each subset to a separate service instance should be used when having *a single query returning a large number of timeseries*. This can be further split by applying global MetricsQL filters using the `extra_filters` [parameter in the reader](https://docs.victoriametrics.com/anomaly-detection/components/reader?highlight=extra_filters#vm-reader). See example below.
|
||||
|
||||
|
||||
@@ -50,7 +50,7 @@ Although this mode is designed to be as flexible as possible, it includes a pre-
|
||||
|
||||
### Grafana Dashboard
|
||||
|
||||
> Note: [For additional benefits](https://docs.victoriametrics.com/victoriametrics-datasource/#motivation), this dashboard is based on [VictoriaMetrics datasource](https://docs.victoriametrics.com/victoriametrics-datasource/) rather than on `Prometheus` datasource. Please follow [these instructions](https://docs.victoriametrics.com/victoriametrics-datasource/#installation) to enable datasource in Grafana.
|
||||
> Note: [For additional benefits](https://docs.victoriametrics.com/victoriametrics/victoriametrics-datasource/#motivation), this dashboard is based on [VictoriaMetrics datasource](https://docs.victoriametrics.com/victoriametrics/victoriametrics-datasource/) rather than on `Prometheus` datasource. Please follow [these instructions](https://docs.victoriametrics.com/victoriametrics/victoriametrics-datasource/#installation) to enable datasource in Grafana.
|
||||
|
||||
You can find the Grafana dashboard `.json` file either [here](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker/vmanomaly/vmanomaly-default/dashboard.json) or, for already running `vmanomaly` instance: `http://localhost:8490/presets/dashboard.json`.
|
||||
|
||||
@@ -191,7 +191,7 @@ The Node-Exporter preset simplifies the monitoring and anomaly detection of key
|
||||
|
||||
> **Note: Node-Exporter preset assets can be also found [here](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker/vmanomaly/vmanomaly-node-exporter-preset/)**
|
||||
|
||||
For enabling Node-Exporter in config file set the `preset` arg accordingly. Also, include at least `datasource_url`-s (and `tenant_id` if using [cluster version of VictoriaMetrics](https://docs.victoriametrics.com/cluster-victoriametrics/)) in `reader` and `writer` sections, like that:
|
||||
For enabling Node-Exporter in config file set the `preset` arg accordingly. Also, include at least `datasource_url`-s (and `tenant_id` if using [cluster version of VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/)) in `reader` and `writer` sections, like that:
|
||||
|
||||
```yaml
|
||||
preset: "node-exporter"
|
||||
@@ -206,7 +206,7 @@ writer:
|
||||
Run a service using such config file with one of the [available options](https://docs.victoriametrics.com/anomaly-detection/quickstart/#how-to-install-and-run-vmanomaly).
|
||||
|
||||
### Generated anomaly scores
|
||||
Machine learning models will be fit for each timeseries, returned by underlying [MetricsQL](https://docs.victoriametrics.com/metricsql/) queries.
|
||||
Machine learning models will be fit for each timeseries, returned by underlying [MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/) queries.
|
||||
In addition to preset label, anomaly score metric labels will also contain [model](https://docs.victoriametrics.com/anomaly-detection/components/models/) and [scheduler](https://docs.victoriametrics.com/anomaly-detection/components/scheduler/) aliases for labelset uniqueness (`preset`, `model_alias`, `scheduler_alias` labels, respectively).
|
||||
|
||||
Here's an example of produced metrics:
|
||||
|
||||
@@ -68,7 +68,7 @@ vm_license_expires_at 1.6963776e+09
|
||||
vm_license_expires_in_seconds 4.886608e+06
|
||||
```
|
||||
|
||||
Example alerts for [vmalert](https://docs.victoriametrics.com/vmalert/):
|
||||
Example alerts for [vmalert](https://docs.victoriametrics.com/victoriametrics/vmalert/):
|
||||
|
||||
```yaml
|
||||
groups:
|
||||
@@ -220,7 +220,7 @@ reader:
|
||||
datasource_url: "http://victoriametrics:8428/" # [YOUR_DATASOURCE_URL]
|
||||
sampling_period: "1m"
|
||||
queries:
|
||||
# define your queries with MetricsQL - https://docs.victoriametrics.com/metricsql/
|
||||
# define your queries with MetricsQL - https://docs.victoriametrics.com/victoriametrics/metricsql/
|
||||
cache: "sum(rate(vm_cache_entries))"
|
||||
|
||||
writer:
|
||||
@@ -236,12 +236,12 @@ writer:
|
||||
- For example, if receiving **alerts every 15 minutes** is sufficient (when `anomaly_score > 1`), set `infer_every` to match `reader.sampling_period` or override it per query via `reader.queries.query_xxx.step` for an optimal setup.
|
||||
|
||||
**Reader**:
|
||||
- Setup the datasource to read data from in the [reader](https://docs.victoriametrics.com/anomaly-detection/components/reader/) section. Include tenant ID if using a [cluster version of VictoriaMetrics](https://docs.victoriametrics.com/cluster-victoriametrics/) (`multitenant` value {{% available_from "v1.16.2" anomaly %}} can be also used here).
|
||||
- Define queries for input data using [MetricsQL](https://docs.victoriametrics.com/metricsql/) under `reader.queries` section. Note, it's possible to override reader-level arguments at query level for increased flexibility, e.g. specifying per-query timezone, data frequency, data range, etc.
|
||||
- Setup the datasource to read data from in the [reader](https://docs.victoriametrics.com/anomaly-detection/components/reader/) section. Include tenant ID if using a [cluster version of VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/) (`multitenant` value {{% available_from "v1.16.2" anomaly %}} can be also used here).
|
||||
- Define queries for input data using [MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/) under `reader.queries` section. Note, it's possible to override reader-level arguments at query level for increased flexibility, e.g. specifying per-query timezone, data frequency, data range, etc.
|
||||
|
||||
**Writer**:
|
||||
- Specify where and how to store anomaly detection metrics in the [writer](https://docs.victoriametrics.com/anomaly-detection/components/writer/) section.
|
||||
- Include tenant ID if using a [cluster version of VictoriaMetrics](https://docs.victoriametrics.com/cluster-victoriametrics/) for writing the results.
|
||||
- Include tenant ID if using a [cluster version of VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/) for writing the results.
|
||||
- Adding `for` label to `metric_format` argument is recommended for smoother visual experience in the [anomaly score dashboard](https://docs.victoriametrics.com/anomaly-detection/presets/#default). Please refer to `metric_format` argument description [here](https://docs.victoriametrics.com/anomaly-detection/components/writer/?highlight=metric_format#config-parameters).
|
||||
|
||||
**Models**:
|
||||
|
||||
@@ -79,7 +79,7 @@ Additionally, a replication factor `R ≥ 1` ensures [high availability](#high-a
|
||||
|
||||
<img src="../vmanomaly-sharding-ha-diagram.webp" alt="vmanomaly-sharding-ha-diagram" width="800px"/>
|
||||
|
||||
> Please [refer to deployment options section](#deployment-options) for the examples (Docker, Docker Compose, Helm). To avoid duplicate metrics being reported from each vmanomaly service used in sharded mode, make sure that [deduplication](https://docs.victoriametrics.com/#deduplication) is configured on vmsingle or vmselect and vmstorage for the VictoriaMetrics instance used in the [writer section of the configuration](https://docs.victoriametrics.com/anomaly-detection/components/writer).
|
||||
> Please [refer to deployment options section](#deployment-options) for the examples (Docker, Docker Compose, Helm). To avoid duplicate metrics being reported from each vmanomaly service used in sharded mode, make sure that [deduplication](https://docs.victoriametrics.com/victoriametrics#deduplication) is configured on vmsingle or vmselect and vmstorage for the VictoriaMetrics instance used in the [writer section of the configuration](https://docs.victoriametrics.com/anomaly-detection/components/writer).
|
||||
|
||||
Sharding configuration can be controlled by using the following environment variables:
|
||||
|
||||
@@ -131,7 +131,7 @@ When `VMANOMALY_REPLICATION_FACTOR` > 1, each [sub-config](#sub-configuration) `
|
||||
|
||||
<img src="../vmanomaly-sharding-ha-diagram.webp" alt="vmanomaly-sharding-ha-diagram" width="800px"/>
|
||||
|
||||
> Please [refer to deployment options section](#deployment-options) for the examples (Docker, Docker Compose, Helm). To avoid duplicate metrics being reported from each vmanomaly service used in sharded mode, make sure that [deduplication](https://docs.victoriametrics.com/#deduplication) is configured on vmsingle or vmselect and vmstorage for the VictoriaMetrics instance used in the [writer section of the configuration](https://docs.victoriametrics.com/anomaly-detection/components/writer).
|
||||
> Please [refer to deployment options section](#deployment-options) for the examples (Docker, Docker Compose, Helm). To avoid duplicate metrics being reported from each vmanomaly service used in sharded mode, make sure that [deduplication](https://docs.victoriametrics.com/victoriametrics#deduplication) is configured on vmsingle or vmselect and vmstorage for the VictoriaMetrics instance used in the [writer section of the configuration](https://docs.victoriametrics.com/anomaly-detection/components/writer).
|
||||
|
||||
### Example
|
||||
|
||||
@@ -242,4 +242,4 @@ To deploy `vmanomaly` with `N > 1` shards using Helm, ensure **chart version [1.
|
||||
- Set shard count **(`.Values.shardsCount`)**: Defines the number of shards (`N > 1`) to enable horizontal scaling. Configure it [here](https://github.com/VictoriaMetrics/helm-charts/blob/207dd54f90919f0159b950dbd29bf2f57abf38f1/charts/victoria-metrics-anomaly/values.yaml#L187).
|
||||
- *(Optional)* Enable [high availability](#high-availability) **(`.Values.replicationFactor`)**: If `R > 1`, each [sub-config](#sub-configuration) is assigned to exactly `R` shards. Configure it [here](https://github.com/VictoriaMetrics/helm-charts/blob/207dd54f90919f0159b950dbd29bf2f57abf38f1/charts/victoria-metrics-anomaly/values.yaml#L189).
|
||||
|
||||
With **StatefulSet enabled**, `vmanomaly` automatically extracts shard numbers from pod names. For example, if the pod is named `vmanomaly-node-exporter-0`, then `VMANOMALY_MEMBER_NUM=0` is assigned automatically.
|
||||
With **StatefulSet enabled**, `vmanomaly` automatically extracts shard numbers from pod names. For example, if the pod is named `vmanomaly-node-exporter-0`, then `VMANOMALY_MEMBER_NUM=0` is assigned automatically.
|
||||
|
||||
@@ -16,7 +16,7 @@ Self-monitoring refers to the ability of the service to track and report its own
|
||||
|
||||
VictoriaMetrics Anomaly Detection (`vmanomaly`) supports self-monitoring by generating metrics related to different operational aspects of the service, including model execution, reader behavior, writer behavior, and overall service health. These metrics provide insights into system bottlenecks, errors, or unusual behavior, and allow for data-driven decisions on tuning performance and reliability.
|
||||
|
||||
Self-monitoring metrics are available in both the [push](https://docs.victoriametrics.com/keyconcepts/#push-model) and [pull](https://docs.victoriametrics.com/keyconcepts/#pull-model) models, providing flexibility to fit into different monitoring environments. By specifying relevant parameters in the `monitoring` section of the configuration, `vmanomaly` components can seamlessly integrate with VictoriaMetrics, Prometheus or other monitoring solutions, enabling centralized visibility into both service and anomaly detection outcomes.
|
||||
Self-monitoring metrics are available in both the [push](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#push-model) and [pull](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#pull-model) models, providing flexibility to fit into different monitoring environments. By specifying relevant parameters in the `monitoring` section of the configuration, `vmanomaly` components can seamlessly integrate with VictoriaMetrics, Prometheus or other monitoring solutions, enabling centralized visibility into both service and anomaly detection outcomes.
|
||||
|
||||
> **Note**: For the detailed overview of self-monitoring metrics that are produced by `vmanomaly` and how to enable their tracking for push/pull models, please refer to [monitoring](https://docs.victoriametrics.com/anomaly-detection/components/monitoring) section docs.
|
||||
|
||||
@@ -100,7 +100,7 @@ This global panel holds resource utilization (CPU, RAM, File Descriptors) on bot
|
||||
<img src="../vmanomaly-dashboard-3-global-panel-resources.webp" alt="vmanomaly-dashboard-3-global-resources" width="800px"/>
|
||||
|
||||
**Healthy scenario**:
|
||||
- **Timeseries graphs**: Should appear stable over time, without significant spikes or drops. An absence of upward trends (e.g., trends in RAM usage may indicate a [high churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate) in your input data).
|
||||
- **Timeseries graphs**: Should appear stable over time, without significant spikes or drops. An absence of upward trends (e.g., trends in RAM usage may indicate a [high churn rate](https://docs.victoriametrics.com/victoriametrics/faq/#what-is-high-churn-rate) in your input data).
|
||||
|
||||
### Model Statistics
|
||||
|
||||
@@ -111,7 +111,7 @@ These panels contain repeated blocks for each unique `model_alias` (a distinct e
|
||||
**Healthy scenario**:
|
||||
- **Erroneous Runs**: There should be zero erroneous runs. Any errors suggest potential issues with the service or uncaught corner cases that need immediate attention.
|
||||
- **Skipped Runs**: Should be minimal, ideally none. A significant number of skipped runs may indicate missing data, configuration problems, high churn rate (resulting in no models trained yet for new time series), or other underlying issues.
|
||||
- **Timeseries graphs**: Should remain stable over time, without significant spikes or drops. An absence of upward trends (e.g., in RAM usage) could indicate normal operation without excessive [churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate).
|
||||
- **Timeseries graphs**: Should remain stable over time, without significant spikes or drops. An absence of upward trends (e.g., in RAM usage) could indicate normal operation without excessive [churn rate](https://docs.victoriametrics.com/victoriametrics/faq/#what-is-high-churn-rate).
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -14,7 +14,7 @@ This chapter describes different components, that correspond to respective secti
|
||||
|
||||
Below, you will find an example illustrating how the components of `vmanomaly` interact with each other and with a single-node VictoriaMetrics setup.
|
||||
|
||||
> **Note**: [Reader](https://docs.victoriametrics.com/anomaly-detection/components/reader/#vm-reader) and [Writer](https://docs.victoriametrics.com/anomaly-detection/components/writer/#vm-writer) also support [multitenancy](https://docs.victoriametrics.com/cluster-victoriametrics/#multitenancy), so you can read/write from/to different locations - see `tenant_id` param description.
|
||||
> **Note**: [Reader](https://docs.victoriametrics.com/anomaly-detection/components/reader/#vm-reader) and [Writer](https://docs.victoriametrics.com/anomaly-detection/components/writer/#vm-writer) also support [multitenancy](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#multitenancy), so you can read/write from/to different locations - see `tenant_id` param description.
|
||||
|
||||

|
||||
|
||||
|
||||
@@ -411,7 +411,7 @@ Moreover, {{% available_from "v1.15.0" anomaly %}}, there exist **[online (incre
|
||||
|
||||
For a univariate type, **one separate model** is fit/used for inference per **each time series**, defined in its [queries](#queries) arg.
|
||||
|
||||
For example, if you have some **univariate** model, defined to use 3 [MetricQL queries](https://docs.victoriametrics.com/metricsql/), each returning 5 time series, there will be 3*5=15 models created in total. Each such model produce **individual [output](#vmanomaly-output)** for each of time series.
|
||||
For example, if you have some **univariate** model, defined to use 3 [MetricQL queries](https://docs.victoriametrics.com/victoriametrics/metricsql/), each returning 5 time series, there will be 3*5=15 models created in total. Each such model produce **individual [output](#vmanomaly-output)** for each of time series.
|
||||
|
||||
If during an inference, you got a series having **new labelset** (not present in any of fitted models), the inference will be skipped until you get a model, trained particularly for such labelset during forthcoming re-fit step.
|
||||
|
||||
@@ -426,7 +426,7 @@ If during an inference, you got a series having **new labelset** (not present in
|
||||
|
||||
For a multivariate type, **one shared model** is fit/used for inference on **all time series** simultaneously, defined in its [queries](#queries) arg.
|
||||
|
||||
For example, if you have some **multivariate** model to use 3 [MetricQL queries](https://docs.victoriametrics.com/metricsql/), each returning 5 time series, there will be one shared model created in total. Once fit, this model will expect **exactly 15 time series with exact same labelsets as an input**. This model will produce **one shared [output](#vmanomaly-output)**.
|
||||
For example, if you have some **multivariate** model to use 3 [MetricQL queries](https://docs.victoriametrics.com/victoriametrics/metricsql/), each returning 5 time series, there will be one shared model created in total. Once fit, this model will expect **exactly 15 time series with exact same labelsets as an input**. This model will produce **one shared [output](#vmanomaly-output)**.
|
||||
|
||||
> **Note:** {{% available_from "v1.16.0" anomaly %}}, N models — one for each unique combination of label values specified in the `groupby` [common argument](#group-by) — can be trained. This allows for context separation (e.g., one model per host, region, or other relevant grouping label), leading to improved accuracy and faster training. See an example [here](#group-by).
|
||||
|
||||
|
||||
@@ -9,7 +9,7 @@ menu:
|
||||
aliases:
|
||||
- ./monitoring.html
|
||||
---
|
||||
There are 2 models to monitor VictoriaMetrics Anomaly Detection behavior - [push](https://docs.victoriametrics.com/keyconcepts/#push-model) and [pull](https://docs.victoriametrics.com/keyconcepts/#pull-model). Parameters for each of them should be specified in the config file, `monitoring` section.
|
||||
There are 2 models to monitor VictoriaMetrics Anomaly Detection behavior - [push](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#push-model) and [pull](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#pull-model). Parameters for each of them should be specified in the config file, `monitoring` section.
|
||||
|
||||
> **Note**: there was an enhancement of [self-monitoring](https://docs.victoriametrics.com/anomaly-detection/components/monitoring/#metrics-generated-by-vmanomaly) metrics for consistency across the components ([v.1.17.0](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1170)). Documentation was updated accordingly. Key changes included:
|
||||
- Converting several [self-monitoring](https://docs.victoriametrics.com/anomaly-detection/components/monitoring/#metrics-generated-by-vmanomaly) metrics from `Summary` to `Histogram` to enable quantile calculation. This addresses the limitation of the `prometheus_client`'s [Summary](https://prometheus.github.io/client_python/instrumenting/summary/) implementation, which does not support quantiles. The change ensures metrics are more informative for performance analysis. Affected metrics are:
|
||||
@@ -246,7 +246,7 @@ monitoring:
|
||||
|
||||
## mTLS protection
|
||||
|
||||
`vmanomaly` components such as [VmWriter](https://docs.victoriametrics.com/anomaly-detection/components/writer/#vm-writer) support [mTLS](https://en.wikipedia.org/wiki/Mutual_authentication){{% available_from "v1.16.3" anomaly %}} to ensure secure communication with [VictoriaMetrics Enterprise, configured with mTLS](https://docs.victoriametrics.com/#mtls-protection).
|
||||
`vmanomaly` components such as [VmWriter](https://docs.victoriametrics.com/anomaly-detection/components/writer/#vm-writer) support [mTLS](https://en.wikipedia.org/wiki/Mutual_authentication){{% available_from "v1.16.3" anomaly %}} to ensure secure communication with [VictoriaMetrics Enterprise, configured with mTLS](https://docs.victoriametrics.com/victoriametrics#mtls-protection).
|
||||
|
||||
For detailed guidance on configuring mTLS parameters such as `verify_tls`, `tls_cert_file`, and `tls_key_file`, please refer to the [mTLS protection section](https://docs.victoriametrics.com/anomaly-detection/components/reader/#mtls-protection) in the [Reader](https://docs.victoriametrics.com/anomaly-detection/components/reader/#vm-reader) documentation. The configuration principles apply consistently across all these `vmanomaly` components.
|
||||
|
||||
|
||||
@@ -9,7 +9,7 @@ aliases:
|
||||
- /anomaly-detection/components/reader.html
|
||||
---
|
||||
|
||||
VictoriaMetrics Anomaly Detection (`vmanomaly`) primarily uses [VmReader](#vm-reader) to ingest data. This reader focuses on fetching time-series data directly from VictoriaMetrics with the help of powerful [MetricsQL](https://docs.victoriametrics.com/metricsql/) expressions for aggregating, filtering and grouping your data, ensuring seamless integration and efficient data handling.
|
||||
VictoriaMetrics Anomaly Detection (`vmanomaly`) primarily uses [VmReader](#vm-reader) to ingest data. This reader focuses on fetching time-series data directly from VictoriaMetrics with the help of powerful [MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/) expressions for aggregating, filtering and grouping your data, ensuring seamless integration and efficient data handling.
|
||||
|
||||
Future updates will introduce additional readers, expanding the range of data sources `vmanomaly` can work with.
|
||||
|
||||
@@ -74,9 +74,9 @@ There is change{{% available_from "v1.13.0" anomaly %}} of [`queries`](https://d
|
||||
- `tenant_id` {{% available_from "v1.19.0" anomaly %}} (string): this optional argument enables tenant-level separation for queries (e.g. `query1` to get the data from tenant "0:0", `query2` - from tenant "1:0"). It works as follows:
|
||||
- if *not set, inherits* reader-level `tenant_id`
|
||||
- if *set, overrides* reader-level `tenant_id`
|
||||
- *raises config validation error*, if *reader-level is not set* and *query-level is found* (mixing of VictoriaMetrics [single-node](https://docs.victoriametrics.com/single-server-victoriametrics/) and [cluster](https://docs.victoriametrics.com/cluster-victoriametrics/) is prohibited in a single config)
|
||||
- *raises config validation warning*, if `writer.tenant_id` is not explicitly set to `multitenant` when reader uses tenants, meaning [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/) will be used for data querying.
|
||||
- also *raises config validation error* if a set of `reader.queries` for [multivariate models](https://docs.victoriametrics.com/anomaly-detection/components/models/#multivariate-models) has *different* tenant_ids (meaning tenant data is mixed, and special labels like `vm_project_id`, `vm_account_id` will have [ambiguous values](https://docs.victoriametrics.com/cluster-victoriametrics/#multitenancy-via-labels))
|
||||
- *raises config validation error*, if *reader-level is not set* and *query-level is found* (mixing of VictoriaMetrics [single-node](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/) and [cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/) is prohibited in a single config)
|
||||
- *raises config validation warning*, if `writer.tenant_id` is not explicitly set to `multitenant` when reader uses tenants, meaning [VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/) will be used for data querying.
|
||||
- also *raises config validation error* if a set of `reader.queries` for [multivariate models](https://docs.victoriametrics.com/anomaly-detection/components/models/#multivariate-models) has *different* tenant_ids (meaning tenant data is mixed, and special labels like `vm_project_id`, `vm_account_id` will have [ambiguous values](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#multitenancy-via-labels))
|
||||
|
||||
> **Note:** The recommended approach for using per-query `tenant_id`s is to set both `reader.tenant_id` and `writer.tenant_id` to `multitenant`. See [this section](https://docs.victoriametrics.com/anomaly-detection/components/writer/#multitenancy-support) for more details. Configurations where `reader.tenant_id` equals `writer.tenant_id` and is not `multitenant` are also considered safe, provided there is a single, DISTINCT `tenant_id` defined in the reader (either at the reader level or the query level, if set).
|
||||
|
||||
@@ -165,7 +165,7 @@ Datasource URL address
|
||||
`0:0`, `multitenant`
|
||||
</td>
|
||||
<td>
|
||||
For VictoriaMetrics Cluster version only, tenants are identified by `accountID` or `accountID:projectID`. Starting from [v1.16.2](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1162), `multitenant` [endpoint](https://docs.victoriametrics.com/cluster-victoriametrics/?highlight=reads#multitenancy-via-labels) is supported, to execute queries over multiple [tenants](https://docs.victoriametrics.com/cluster-victoriametrics/#multitenancy). See VictoriaMetrics Cluster [multitenancy docs](https://docs.victoriametrics.com/cluster-victoriametrics/#multitenancy)
|
||||
For VictoriaMetrics Cluster version only, tenants are identified by `accountID` or `accountID:projectID`. Starting from [v1.16.2](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1162), `multitenant` [endpoint](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/?highlight=reads#multitenancy-via-labels) is supported, to execute queries over multiple [tenants](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#multitenancy). See VictoriaMetrics Cluster [multitenancy docs](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#multitenancy)
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
@@ -322,7 +322,7 @@ Path to a file, which contains token, that is passed in the standard format with
|
||||
`[]`
|
||||
</td>
|
||||
<td>
|
||||
List of strings with series selector. See: [Prometheus querying API enhancements](https://docs.victoriametrics.com/##prometheus-querying-api-enhancements)
|
||||
List of strings with series selector. See: [Prometheus querying API enhancements](https://docs.victoriametrics.com/victoriametrics##prometheus-querying-api-enhancements)
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
@@ -416,7 +416,7 @@ reader:
|
||||
|
||||
### mTLS protection
|
||||
|
||||
`vmanomaly` supports [mutual TLS (mTLS)](https://en.wikipedia.org/wiki/Mutual_authentication){{% available_from "v1.16.3" anomaly %}} for secure communication across its components, including [VmReader](https://docs.victoriametrics.com/anomaly-detection/components/reader/#vm-reader), [VmWriter](https://docs.victoriametrics.com/anomaly-detection/components/writer/#vm-writer), and [Monitoring/Push](https://docs.victoriametrics.com/anomaly-detection/components/monitoring/#push-config-parameters). This allows for mutual authentication between the client and server when querying or writing data to [VictoriaMetrics Enterprise, configured for mTLS](https://docs.victoriametrics.com/#mtls-protection).
|
||||
`vmanomaly` supports [mutual TLS (mTLS)](https://en.wikipedia.org/wiki/Mutual_authentication){{% available_from "v1.16.3" anomaly %}} for secure communication across its components, including [VmReader](https://docs.victoriametrics.com/anomaly-detection/components/reader/#vm-reader), [VmWriter](https://docs.victoriametrics.com/anomaly-detection/components/writer/#vm-writer), and [Monitoring/Push](https://docs.victoriametrics.com/anomaly-detection/components/monitoring/#push-config-parameters). This allows for mutual authentication between the client and server when querying or writing data to [VictoriaMetrics Enterprise, configured for mTLS](https://docs.victoriametrics.com/victoriametrics#mtls-protection).
|
||||
|
||||
mTLS ensures that both the client and server verify each other's identity using certificates, which enhances security by preventing unauthorized access.
|
||||
|
||||
|
||||
@@ -67,7 +67,7 @@ Datasource URL address
|
||||
</td>
|
||||
<td>
|
||||
|
||||
For VictoriaMetrics Cluster version only, tenants are identified by `accountID` or `accountID:projectID`. multitenant` [endpoint](https://docs.victoriametrics.com/cluster-victoriametrics/?highlight=writes#multitenancy-via-labels){{% available_from "v1.16.2" anomaly %}} is supported, to write data to multiple [tenants](https://docs.victoriametrics.com/cluster-victoriametrics/#multitenancy). See VictoriaMetrics Cluster [multitenancy docs](https://docs.victoriametrics.com/cluster-victoriametrics/#multitenancy)
|
||||
For VictoriaMetrics Cluster version only, tenants are identified by `accountID` or `accountID:projectID`. multitenant` [endpoint](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/?highlight=writes#multitenancy-via-labels){{% available_from "v1.16.2" anomaly %}} is supported, to write data to multiple [tenants](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#multitenancy). See VictoriaMetrics Cluster [multitenancy docs](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#multitenancy)
|
||||
</td>
|
||||
</tr>
|
||||
<!-- Additional rows for metric_format -->
|
||||
@@ -277,7 +277,7 @@ writer:
|
||||
|
||||
### Multitenancy support
|
||||
|
||||
> This feature applies to the VictoriaMetrics Cluster version only. Tenants are identified by either `accountID` or `accountID:projectID`. `multitenant` [endpoint](https://docs.victoriametrics.com/cluster-victoriametrics/?highlight=writes#multitenancy-via-labels){{% available_from "v1.15.9" anomaly %}} is supported for writing data across multiple [tenants](https://docs.victoriametrics.com/cluster-victoriametrics/#multitenancy). For more details, refer to the VictoriaMetrics Cluster [multitenancy documentation](https://docs.victoriametrics.com/cluster-victoriametrics/#multitenancy).
|
||||
> This feature applies to the VictoriaMetrics Cluster version only. Tenants are identified by either `accountID` or `accountID:projectID`. `multitenant` [endpoint](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/?highlight=writes#multitenancy-via-labels){{% available_from "v1.15.9" anomaly %}} is supported for writing data across multiple [tenants](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#multitenancy). For more details, refer to the VictoriaMetrics Cluster [multitenancy documentation](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#multitenancy).
|
||||
|
||||
Please note the different behaviors depending on the `tenant_id` value:
|
||||
|
||||
@@ -309,7 +309,7 @@ Please note the different behaviors depending on the `tenant_id` value:
|
||||
|
||||
### mTLS protection
|
||||
|
||||
`vmanomaly` components such as [VmWriter](https://docs.victoriametrics.com/anomaly-detection/components/writer/#vm-writer) support [mTLS](https://en.wikipedia.org/wiki/Mutual_authentication) to ensure secure communication with [VictoriaMetrics Enterprise, configured with mTLS](https://docs.victoriametrics.com/#mtls-protection){{% available_from "v1.16.3" anomaly %}}.
|
||||
`vmanomaly` components such as [VmWriter](https://docs.victoriametrics.com/anomaly-detection/components/writer/#vm-writer) support [mTLS](https://en.wikipedia.org/wiki/Mutual_authentication) to ensure secure communication with [VictoriaMetrics Enterprise, configured with mTLS](https://docs.victoriametrics.com/victoriametrics#mtls-protection){{% available_from "v1.16.3" anomaly %}}.
|
||||
|
||||
For detailed guidance on configuring mTLS parameters such as `verify_tls`, `tls_cert_file`, and `tls_key_file`, please refer to the [mTLS protection section](https://docs.victoriametrics.com/anomaly-detection/components/reader/#mtls-protection) in the [Reader](https://docs.victoriametrics.com/anomaly-detection/components/reader/#vm-reader) documentation. The configuration principles apply consistently across all these `vmanomaly` components.
|
||||
|
||||
|
||||
@@ -2,9 +2,9 @@
|
||||
|
||||
- To use *vmanomaly*, part of the enterprise package, a license key is required. Obtain your key [here](https://victoriametrics.com/products/enterprise/trial/) for this tutorial or for enterprise use.
|
||||
- In the tutorial, we'll be using the following VictoriaMetrics components:
|
||||
- [VictoriaMetrics Single-Node](https://docs.victoriametrics.com/single-server-victoriametrics) (v1.115.0)
|
||||
- [vmalert](https://docs.victoriametrics.com/vmalert/) (v1.115.0)
|
||||
- [vmagent](https://docs.victoriametrics.com/vmagent/) (v1.115.0)
|
||||
- [VictoriaMetrics Single-Node](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics) (v1.115.0)
|
||||
- [vmalert](https://docs.victoriametrics.com/victoriametrics/vmalert/) (v1.115.0)
|
||||
- [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/) (v1.115.0)
|
||||
- [Grafana](https://grafana.com/) (v.10.2.1)
|
||||
- [Docker](https://docs.docker.com/get-docker/) and [Docker Compose](https://docs.docker.com/compose/)
|
||||
- [Node exporter](https://github.com/prometheus/node_exporter#node-exporter) (v1.7.0) and [Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/) (v0.27.0)
|
||||
@@ -44,9 +44,9 @@ Then, users can enable alerting rules based on the **anomaly score** with [vmale
|
||||
|
||||
## 2. What is vmalert?
|
||||
|
||||
[vmalert](https://docs.victoriametrics.com/vmalert/) is an alerting tool for VictoriaMetrics. It executes a list of the given alerting or recording rules against configured `-datasource.url`.
|
||||
[vmalert](https://docs.victoriametrics.com/victoriametrics/vmalert/) is an alerting tool for VictoriaMetrics. It executes a list of the given alerting or recording rules against configured `-datasource.url`.
|
||||
|
||||
[Alerting rules](https://docs.victoriametrics.com/vmalert/#alerting-rules) allow you to define conditions that, when met, will notify the user. The alerting condition is defined in a form of a query expression via [MetricsQL query language](https://docs.victoriametrics.com/metricsql/). For example, in our case, the expression `anomaly_score > 1.0` will notify a user when the calculated anomaly score exceeds a threshold of `1.0`.
|
||||
[Alerting rules](https://docs.victoriametrics.com/victoriametrics/vmalert/#alerting-rules) allow you to define conditions that, when met, will notify the user. The alerting condition is defined in a form of a query expression via [MetricsQL query language](https://docs.victoriametrics.com/victoriametrics/metricsql/). For example, in our case, the expression `anomaly_score > 1.0` will notify a user when the calculated anomaly score exceeds a threshold of `1.0`.
|
||||
|
||||
## 3. How does vmanomaly works with vmalert?
|
||||
|
||||
@@ -55,9 +55,9 @@ Compared to classical alerting rules, anomaly detection is more "hands-off" and
|
||||
Practical use case is to put anomaly score generated by vmanomaly into alerting rules with some threshold.
|
||||
|
||||
**In this tutorial we are going to:**
|
||||
- Configure docker-compose file with all needed services ([VictoriaMetrics Single-Node](https://docs.victoriametrics.com/single-server-victoriametrics), [vmalert](https://docs.victoriametrics.com/vmalert/), [vmagent](https://docs.victoriametrics.com/vmagent/), [Grafana](https://grafana.com/), [Node Exporter](https://prometheus.io/docs/guides/node-exporter/) and [vmanomaly](https://docs.victoriametrics.com/anomaly-detection/) ).
|
||||
- Explore configuration files for [vmanomaly](https://docs.victoriametrics.com/anomaly-detection/) and [vmalert](https://docs.victoriametrics.com/vmalert/).
|
||||
- Run our own [VictoriaMetrics](https://docs.victoriametrics.com/single-server-victoriametrics) database with data scraped from [Node Exporter](https://prometheus.io/docs/guides/node-exporter/).
|
||||
- Configure docker-compose file with all needed services ([VictoriaMetrics Single-Node](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics), [vmalert](https://docs.victoriametrics.com/victoriametrics/vmalert/), [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/), [Grafana](https://grafana.com/), [Node Exporter](https://prometheus.io/docs/guides/node-exporter/) and [vmanomaly](https://docs.victoriametrics.com/anomaly-detection/) ).
|
||||
- Explore configuration files for [vmanomaly](https://docs.victoriametrics.com/anomaly-detection/) and [vmalert](https://docs.victoriametrics.com/victoriametrics/vmalert/).
|
||||
- Run our own [VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics) database with data scraped from [Node Exporter](https://prometheus.io/docs/guides/node-exporter/).
|
||||
- Explore data for analysis in [Grafana](https://grafana.com/).
|
||||
- Explore `vmanomaly` results.
|
||||
- Explore `vmalert` alerts
|
||||
@@ -88,7 +88,7 @@ node_cpu_seconds_total{cpu="1",mode="iowait"} 51.22
|
||||
|
||||
In this context, the metric `node_cpu_seconds_total` provides a comprehensive breakdown of the time each CPU core has spent in various operational modes. These modes include: _user_, _system_, _iowait_, _idle_, _irq&softirq_, _guest_, and _steal_. Each of these eight modes is mutually exclusive, offering distinct insights into CPU activity. For instance, a predominant _iowait_ suggests disk or network bottlenecks, while elevated levels in _user_ or _system_ indicate significant CPU utilization.
|
||||
|
||||
The `node_cpu_seconds_total` metric is classified as a [counter](https://docs.victoriametrics.com/keyconcepts/#counter) type. To analyze the duration each CPU core spends in these modes, it is necessary to compute the rate of change per second using the [rate function](https://docs.victoriametrics.com/metricsql/#rate): `rate(node_cpu_seconds_total)`. For a more refined and smoother aggregation of data by mode, we apply the sum function. The resulting query is formulated as follows: `sum(rate(node_cpu_seconds_total[5m])) by (mode, instance, job)`.
|
||||
The `node_cpu_seconds_total` metric is classified as a [counter](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#counter) type. To analyze the duration each CPU core spends in these modes, it is necessary to compute the rate of change per second using the [rate function](https://docs.victoriametrics.com/victoriametrics/metricsql/#rate): `rate(node_cpu_seconds_total)`. For a more refined and smoother aggregation of data by mode, we apply the sum function. The resulting query is formulated as follows: `sum(rate(node_cpu_seconds_total[5m])) by (mode, instance, job)`.
|
||||
|
||||
Below is an illustrative example of how this query might be visualized in Grafana:
|
||||

|
||||
@@ -202,9 +202,9 @@ Selecting an appropriate threshold for the anomaly score depends on your specifi
|
||||
|
||||
## 8. Docker Compose configuration
|
||||
|
||||
You can find the `docker-compose.yml` and all configs in this [folder](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker/vmanomaly/vmanomaly-integration/)
|
||||
You can find the `compose.yml` and all configs in this [folder](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker/vmanomaly/vmanomaly-integration/)
|
||||
|
||||
Now we are going to configure the `docker-compose.yml` file to run all needed services.
|
||||
Now we are going to configure the `compose.yml` file to run all needed services.
|
||||
Here are all services we are going to run:
|
||||
|
||||
* vmanomaly - VictoriaMetrics Anomaly Detection service.
|
||||
@@ -308,7 +308,7 @@ receivers:
|
||||
|
||||
### Docker-compose
|
||||
|
||||
Let's wrap it all up together into the `docker-compose.yml` file.
|
||||
Let's wrap it all up together into the `compose.yml` file.
|
||||
|
||||
|
||||
``` yaml
|
||||
|
||||
@@ -8,5 +8,7 @@ menu:
|
||||
weight: 20
|
||||
identifier: guides
|
||||
pageRef: /guides/
|
||||
params:
|
||||
columns: 2
|
||||
---
|
||||
{{% section %}}
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
VictoriaMetrics and VictoriaLogs support ingestion [metrics](https://docs.victoriametrics.com/single-server-victoriametrics/#sending-data-via-opentelemetry)
|
||||
VictoriaMetrics and VictoriaLogs support ingestion [metrics](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#sending-data-via-opentelemetry)
|
||||
and [logs](https://docs.victoriametrics.com/victorialogs/data-ingestion/opentelemetry/) in OpenTelemetry format.
|
||||
This guide covers examples of using [opentelemetry-collector](https://opentelemetry.io/docs/collector/) and direct pushing of metrics and logs from the Go application.
|
||||
|
||||
@@ -44,7 +44,7 @@ VictoriaMetrics helm chart provides the following URL for writing data:
|
||||
Write URL inside the kubernetes cluster:
|
||||
http://victoria-metrics-victoria-metrics-single-server.default.svc.cluster.local.:8428/<protocol-specific-write-endpoint>
|
||||
|
||||
All supported write endpoints can be found at https://docs.victoriametrics.com/single-server-victoriametrics/#how-to-import-time-series-data.
|
||||
All supported write endpoints can be found at https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#how-to-import-time-series-data.
|
||||
```
|
||||
|
||||
For OpenTelemetry VictoriaMetrics write endpoint is:
|
||||
@@ -119,7 +119,7 @@ config:
|
||||
compression: gzip
|
||||
encoding: proto
|
||||
# Setting below will work for sending data to VictoriaMetrics single-node version.
|
||||
# Cluster version of VictoriaMetrics will require a different URL - https://docs.victoriametrics.com/cluster-victoriametrics/#url-format
|
||||
# Cluster version of VictoriaMetrics will require a different URL - https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#url-format
|
||||
metrics_endpoint: http://victoria-metrics-victoria-metrics-single-server.default.svc.cluster.local:8428/opentelemetry/v1/metrics
|
||||
logs_endpoint: http://victoria-logs-victoria-logs-single-server.default.svc.cluster.local:9428/insert/opentelemetry/v1/logs
|
||||
tls:
|
||||
@@ -154,7 +154,7 @@ kubectl port-forward svc/victoria-metrics-victoria-metrics-single-server 8428
|
||||
```
|
||||
|
||||
Visit [http://localhost:8428/vmui/#/?g0.expr=k8s_container_ready](http://localhost:8428/vmui/#/?g0.expr=k8s_container_ready) to check if metric `k8s_container_ready` is present.
|
||||
Check other available metrics by visiting [cardinality explorer](https://docs.victoriametrics.com/#cardinality-explorer) page.
|
||||
Check other available metrics by visiting [cardinality explorer](https://docs.victoriametrics.com/victoriametrics#cardinality-explorer) page.
|
||||
|
||||
Forward VictoriaLogs port to local machine to explore logs ingested by the collector:
|
||||
```sh
|
||||
@@ -244,7 +244,7 @@ After a few seconds you should start seeing metrics sent to VictoriaMetrics by v
|
||||
|
||||

|
||||
|
||||
Check other available metrics by visiting [cardinality explorer](https://docs.victoriametrics.com/#cardinality-explorer) page.
|
||||
Check other available metrics by visiting [cardinality explorer](https://docs.victoriametrics.com/victoriametrics#cardinality-explorer) page.
|
||||
|
||||
Logs should be available by visiting [http://localhost:9428/select/vmui](http://localhost:9428/select/vmui)
|
||||
using query `service.name: unknown_service:otel`.
|
||||
|
||||
@@ -1,8 +1,8 @@
|
||||
**The guide covers:**
|
||||
|
||||
* The setup of a [VM Operator](https://github.com/VictoriaMetrics/helm-charts/tree/master/charts/victoria-metrics-operator) via Helm in [Kubernetes](https://kubernetes.io/) with Helm charts.
|
||||
* The setup of a [VictoriaMetrics Cluster](https://docs.victoriametrics.com/cluster-victoriametrics/) via [VM Operator](https://github.com/VictoriaMetrics/helm-charts/tree/master/charts/victoria-metrics-operator).
|
||||
* How to add CRD for a [VictoriaMetrics Cluster](https://docs.victoriametrics.com/cluster-victoriametrics/) via [VM Operator](https://github.com/VictoriaMetrics/helm-charts/tree/master/charts/victoria-metrics-operator).
|
||||
* The setup of a [VictoriaMetrics Cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/) via [VM Operator](https://github.com/VictoriaMetrics/helm-charts/tree/master/charts/victoria-metrics-operator).
|
||||
* How to add CRD for a [VictoriaMetrics Cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/) via [VM Operator](https://github.com/VictoriaMetrics/helm-charts/tree/master/charts/victoria-metrics-operator).
|
||||
* How to visualize stored data
|
||||
* How to store metrics in [VictoriaMetrics](https://victoriametrics.com)
|
||||
|
||||
@@ -58,7 +58,7 @@ vmoperator-victoria-metrics-operator-67cff44cd6-s47n6 1/1 Running 0
|
||||
|
||||
> For this example we will use default value for `name: example-vmcluster-persistent`. Change it value up to your needs.
|
||||
|
||||
Run the following command to install [VictoriaMetrics Cluster](https://docs.victoriametrics.com/cluster-victoriametrics/) via [VM Operator](https://github.com/VictoriaMetrics/helm-charts/tree/master/charts/victoria-metrics-operator):
|
||||
Run the following command to install [VictoriaMetrics Cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/) via [VM Operator](https://github.com/VictoriaMetrics/helm-charts/tree/master/charts/victoria-metrics-operator):
|
||||
|
||||
<p id="example-cluster-config"></p>
|
||||
|
||||
@@ -86,8 +86,8 @@ The expected output:
|
||||
vmcluster.operator.victoriametrics.com/example-vmcluster-persistent created
|
||||
```
|
||||
|
||||
* By applying this CRD we install the [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/) to the default [namespace](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) of your k8s cluster with following params:
|
||||
* `retentionPeriod: "12"` defines the [retention](https://docs.victoriametrics.com/single-server-victoriametrics/#retention) to 12 months.
|
||||
* By applying this CRD we install the [VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/) to the default [namespace](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) of your k8s cluster with following params:
|
||||
* `retentionPeriod: "12"` defines the [retention](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#retention) to 12 months.
|
||||
* `replicaCount: 2` creates two replicas of vmselect, vminsert and vmstorage.
|
||||
|
||||
Please note that it may take some time for the pods to start. To check that the pods are started, run the following command:
|
||||
@@ -120,7 +120,7 @@ NAME INSERT COUNT STORAGE COUNT SELECT COUNT AGE
|
||||
example-vmcluster-persistent 2 2 2 5m53s operational
|
||||
```
|
||||
|
||||
Internet traffic goes through the Kubernetes Load balancer which use the set of Pods targeted by a [Kubernetes Service](https://kubernetes.io/docs/concepts/services-networking/service/). The service in [VictoriaMetrics Cluster architecture](https://docs.victoriametrics.com/cluster-victoriametrics/#architecture-overview) which accepts the ingested data named `vminsert` and in Kubernetes it is a `vminsert ` service. So we need to use it for remote_write url.
|
||||
Internet traffic goes through the Kubernetes Load balancer which use the set of Pods targeted by a [Kubernetes Service](https://kubernetes.io/docs/concepts/services-networking/service/). The service in [VictoriaMetrics Cluster architecture](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#architecture-overview) which accepts the ingested data named `vminsert` and in Kubernetes it is a `vminsert ` service. So we need to use it for remote_write url.
|
||||
|
||||
To get the name of `vminsert` services, please run the following command:
|
||||
|
||||
@@ -135,7 +135,7 @@ The expected output:
|
||||
vminsert-example-vmcluster-persistent ClusterIP 10.107.47.136 <none> 8480/TCP 5m58s
|
||||
```
|
||||
|
||||
To scrape metrics from Kubernetes with a VictoriaMetrics Cluster we will need to install [VMAgent](https://docs.victoriametrics.com/vmagent/) with some additional configurations.
|
||||
To scrape metrics from Kubernetes with a VictoriaMetrics Cluster we will need to install [VMAgent](https://docs.victoriametrics.com/victoriametrics/vmagent/) with some additional configurations.
|
||||
Copy `vminsert-example-vmcluster-persistent` (or whatever user put into metadata.name field [https://docs.victoriametrics.com/guides/getting-started-with-vm-operator.html#example-cluster-config](https://docs.victoriametrics.com/guides/getting-started-with-vm-operator.html#example-cluster-config)) service name and add it to the `remoteWrite` URL from [quick-start example](https://github.com/VictoriaMetrics/operator/blob/master/docs/quick-start.MD#vmagent).
|
||||
Here is an example of the full configuration that we need to apply:
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
Using [Grafana](https://grafana.com/) with [vmgateway](https://docs.victoriametrics.com/vmgateway/) is a great way to provide [multi-tenant](https://docs.victoriametrics.com/cluster-victoriametrics/#multitenancy) access to your metrics.
|
||||
Using [Grafana](https://grafana.com/) with [vmgateway](https://docs.victoriametrics.com/victoriametrics/vmgateway/) is a great way to provide [multi-tenant](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#multitenancy) access to your metrics.
|
||||
vmgateway provides a way to authenticate users using [JWT tokens](https://en.wikipedia.org/wiki/JSON_Web_Token) issued by an external identity provider.
|
||||
Those tokens can include information about the user and the tenant they belong to, which can be used
|
||||
to restrict access to metrics to only those that belong to the tenant.
|
||||
@@ -8,7 +8,7 @@ to restrict access to metrics to only those that belong to the tenant.
|
||||
* Identity service that can issue [JWT tokens](https://en.wikipedia.org/wiki/JSON_Web_Token)
|
||||
* [Grafana](https://grafana.com/)
|
||||
* VictoriaMetrics single-node or cluster version
|
||||
* [vmgateway](https://docs.victoriametrics.com/vmgateway/)
|
||||
* [vmgateway](https://docs.victoriametrics.com/victoriametrics/vmgateway/)
|
||||
* An active license key. You can obtain a trial license key [here](https://victoriametrics.com/products/enterprise/trial/).
|
||||
|
||||
## Configure identity service
|
||||
@@ -26,7 +26,7 @@ The identity service must be able to issue JWT tokens with the following `vm_acc
|
||||
}
|
||||
```
|
||||
|
||||
See details about all supported options in the [vmgateway documentation](https://docs.victoriametrics.com/vmgateway/#access-control).
|
||||
See details about all supported options in the [vmgateway documentation](https://docs.victoriametrics.com/victoriametrics/vmgateway/#access-control).
|
||||
|
||||
### Configuration example for Keycloak
|
||||
|
||||
@@ -138,7 +138,7 @@ or manually managing access at another proxy level.
|
||||
|
||||
In order to use multi-tenant access with single-node VictoriaMetrics, you can use token claims such as `extra_labels`
|
||||
or `extra_filters` filled dynamically by using Identity Provider's user information.
|
||||
vmgateway uses those claims and [enhanced Prometheus querying API](https://docs.victoriametrics.com/single-server-victoriametrics/#prometheus-querying-api-enhancements)
|
||||
vmgateway uses those claims and [enhanced Prometheus querying API](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#prometheus-querying-api-enhancements)
|
||||
to provide additional filtering capabilities.
|
||||
|
||||
For example, the following claims can be used to restrict user access to specific metrics:
|
||||
@@ -164,7 +164,7 @@ So when user will try to query `vm_http_requests_total` query will be transforme
|
||||
|
||||
### Token signature verification
|
||||
|
||||
It is also possible to enable [JWT token signature verification](https://docs.victoriametrics.com/vmgateway/#jwt-signature-verification) at
|
||||
It is also possible to enable [JWT token signature verification](https://docs.victoriametrics.com/victoriametrics/vmgateway/#jwt-signature-verification) at
|
||||
vmgateway.
|
||||
To do this by using OpenID Connect discovery endpoint you need to specify the `-auth.oidcDiscoveryEndpoints` flag. For example:
|
||||
|
||||
@@ -186,7 +186,7 @@ Now vmgateway will print the following message on startup:
|
||||
|
||||
That means that vmgateway has successfully fetched the public keys from the OpenID Connect discovery endpoint.
|
||||
|
||||
It is also possible to provide the public keys directly via the `-auth.publicKeys` flag. See the [vmgateway documentation](https://docs.victoriametrics.com/vmgateway/#jwt-signature-verification) for details.
|
||||
It is also possible to provide the public keys directly via the `-auth.publicKeys` flag. See the [vmgateway documentation](https://docs.victoriametrics.com/victoriametrics/vmgateway/#jwt-signature-verification) for details.
|
||||
|
||||
## Use Grafana to query metrics
|
||||
|
||||
@@ -200,7 +200,7 @@ In the "Type and version" section it is recommended to set the type to "Promethe
|
||||
This allows Grafana to use a more efficient API to get label values.
|
||||
|
||||
You can also use VictoriaMetrics [Grafana datasource](https://github.com/VictoriaMetrics/victoriametrics-datasource) plugin.
|
||||
See installation instructions [here](https://docs.victoriametrics.com/victoriametrics-datasource/#installation).
|
||||
See installation instructions [here](https://docs.victoriametrics.com/victoriametrics/victoriametrics-datasource/#installation).
|
||||
|
||||
Enable `Forward OAuth identity` flag.<br>
|
||||

|
||||
|
||||
@@ -1,10 +1,10 @@
|
||||
Data deletion is an operation people expect a database to have. [VictoriaMetrics](https://victoriametrics.com) supports
|
||||
[delete operation](https://docs.victoriametrics.com/single-server-victoriametrics/#how-to-delete-time-series) but to a limited extent. Due to implementation details, VictoriaMetrics remains an [append-only database](https://en.wikipedia.org/wiki/Append-only), which perfectly fits the case for storing time series data. But the drawback of such architecture is that it is extremely expensive to mutate the data. Hence, `delete` or `update` operations support is very limited. In this guide, we'll walk through the possible workarounds for deleting or changing already written data in VictoriaMetrics.
|
||||
[delete operation](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#how-to-delete-time-series) but to a limited extent. Due to implementation details, VictoriaMetrics remains an [append-only database](https://en.wikipedia.org/wiki/Append-only), which perfectly fits the case for storing time series data. But the drawback of such architecture is that it is extremely expensive to mutate the data. Hence, `delete` or `update` operations support is very limited. In this guide, we'll walk through the possible workarounds for deleting or changing already written data in VictoriaMetrics.
|
||||
|
||||
### Precondition
|
||||
|
||||
- [Single-node VictoriaMetrics](https://docs.victoriametrics.com/single-server-victoriametrics/);
|
||||
- [Cluster version of VictoriaMetrics](https://docs.victoriametrics.com/cluster-victoriametrics/);
|
||||
- [Single-node VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/);
|
||||
- [Cluster version of VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/);
|
||||
- [curl](https://curl.se/docs/manual.html)
|
||||
- [jq tool](https://stedolan.github.io/jq/)
|
||||
|
||||
@@ -12,7 +12,7 @@ Data deletion is an operation people expect a database to have. [VictoriaMetrics
|
||||
|
||||
_Warning: time series deletion is not recommended to use on a regular basis. Each call to delete API could have a performance penalty. The API was provided for one-off operations to deleting malformed data or to satisfy GDPR compliance._
|
||||
|
||||
[Delete API](https://docs.victoriametrics.com/single-server-victoriametrics/#how-to-delete-time-series) expects from user to specify [time series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors). So the first thing to do before the deletion is to verify whether the selector matches the correct series.
|
||||
[Delete API](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#how-to-delete-time-series) expects from user to specify [time series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors). So the first thing to do before the deletion is to verify whether the selector matches the correct series.
|
||||
|
||||
To check that metrics are present in **VictoriaMetrics Cluster** run the following command:
|
||||
|
||||
@@ -22,7 +22,7 @@ _Warning: response can return many metrics, so be careful with series selector._
|
||||
curl -s 'http://vmselect:8481/select/0/prometheus/api/v1/series?match[]=process_cpu_cores_available' | jq
|
||||
```
|
||||
|
||||
_See URL example for single-node [here](https://docs.victoriametrics.com/url-examples/#apiv1series)._
|
||||
_See URL example for single-node [here](https://docs.victoriametrics.com/victoriametrics/url-examples/#apiv1series)._
|
||||
|
||||
The expected output:
|
||||
|
||||
@@ -50,17 +50,17 @@ The expected output:
|
||||
}
|
||||
```
|
||||
|
||||
When you're sure [time series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors) is correct, send a POST request to [delete API](https://docs.victoriametrics.com/url-examples/#apiv1admintsdbdelete_series) with [`match[]=<time-series-selector>`](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors) argument. For example:
|
||||
When you're sure [time series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors) is correct, send a POST request to [delete API](https://docs.victoriametrics.com/victoriametrics/url-examples/#apiv1admintsdbdelete_series) with [`match[]=<time-series-selector>`](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors) argument. For example:
|
||||
|
||||
```sh
|
||||
curl -s 'http://vmselect:8481/delete/0/prometheus/api/v1/admin/tsdb/delete_series?match[]=process_cpu_cores_available'
|
||||
```
|
||||
|
||||
_See URL example for single-node [here](https://docs.victoriametrics.com/url-examples/#apiv1admintsdbdelete_series)._
|
||||
_See URL example for single-node [here](https://docs.victoriametrics.com/victoriametrics/url-examples/#apiv1admintsdbdelete_series)._
|
||||
|
||||
If operation was successful, the deleted series will stop being [queryable](https://docs.victoriametrics.com/keyconcepts/#query-data). Storage space for the deleted time series isn't freed instantly - it is freed during subsequent [background merges of data files](https://medium.com/@valyala/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282). The background merges may never occur for data from previous months, so storage space won't be freed for historical data. In this case [forced merge](https://docs.victoriametrics.com/single-server-victoriametrics/#forced-merge) may help freeing up storage space.
|
||||
If operation was successful, the deleted series will stop being [queryable](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#query-data). Storage space for the deleted time series isn't freed instantly - it is freed during subsequent [background merges of data files](https://medium.com/@valyala/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282). The background merges may never occur for data from previous months, so storage space won't be freed for historical data. In this case [forced merge](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#forced-merge) may help freeing up storage space.
|
||||
|
||||
To trigger [forced merge](https://docs.victoriametrics.com/single-server-victoriametrics/#forced-merge) on VictoriaMetrics Cluster run the following command:
|
||||
To trigger [forced merge](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#forced-merge) on VictoriaMetrics Cluster run the following command:
|
||||
|
||||
```sh
|
||||
curl -v -X POST http://vmstorage:8482/internal/force_merge
|
||||
@@ -72,10 +72,10 @@ After the merge is complete, the data will be permanently deleted from the disk.
|
||||
|
||||
By default, VictoriaMetrics doesn't provide a mechanism for replacing or updating data. As a workaround, take the following actions:
|
||||
|
||||
- [export time series to a file](https://docs.victoriametrics.com/url-examples/#apiv1export);
|
||||
- [export time series to a file](https://docs.victoriametrics.com/victoriametrics/url-examples/#apiv1export);
|
||||
- change the values of time series in the file and save it;
|
||||
- [delete time series from a database](https://docs.victoriametrics.com/url-examples/#apiv1admintsdbdelete_series);
|
||||
- [import saved file to VictoriaMetrics](https://docs.victoriametrics.com/url-examples/#apiv1import).
|
||||
- [delete time series from a database](https://docs.victoriametrics.com/victoriametrics/url-examples/#apiv1admintsdbdelete_series);
|
||||
- [import saved file to VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics/url-examples/#apiv1import).
|
||||
|
||||
### Export metrics
|
||||
|
||||
@@ -85,7 +85,7 @@ For example, let's export metric for `node_memory_MemTotal_bytes` with labels `i
|
||||
curl -X POST -g http://vmselect:8481/select/0/prometheus/api/v1/export -d 'match[]=node_memory_MemTotal_bytes{instance="node-exporter:9100", job="hostname.com"}' > data.jsonl
|
||||
```
|
||||
|
||||
_See URL example for single-node [here](https://docs.victoriametrics.com/url-examples/#apiv1export)._
|
||||
_See URL example for single-node [here](https://docs.victoriametrics.com/victoriametrics/url-examples/#apiv1export)._
|
||||
|
||||
To check that exported file contains time series we can use [cat](https://man7.org/linux/man-pages/man1/cat.1.html) and [jq](https://stedolan.github.io/jq/download/):
|
||||
|
||||
@@ -160,7 +160,7 @@ See [How-to-delete-metrics](https://docs.victoriametrics.com/guides/guide-delete
|
||||
|
||||
### Import metrics
|
||||
|
||||
VictoriaMetrics supports a lot of [ingestion protocols](https://docs.victoriametrics.com/single-server-victoriametrics/#how-to-import-time-series-data) and we will use [import from JSON line format](https://docs.victoriametrics.com/single-server-victoriametrics/#how-to-import-data-in-json-line-format).
|
||||
VictoriaMetrics supports a lot of [ingestion protocols](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#how-to-import-time-series-data) and we will use [import from JSON line format](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#how-to-import-data-in-json-line-format).
|
||||
|
||||
The next command will import metrics from `data.jsonl` to VictoriaMetrics:
|
||||
|
||||
@@ -168,9 +168,9 @@ The next command will import metrics from `data.jsonl` to VictoriaMetrics:
|
||||
curl -v -X POST http://vminsert:8480/insert/0/prometheus/api/v1/import -T data.jsonl
|
||||
```
|
||||
|
||||
_See URL example for single-node [here](https://docs.victoriametrics.com/url-examples/#apiv1import)._
|
||||
_See URL example for single-node [here](https://docs.victoriametrics.com/victoriametrics/url-examples/#apiv1import)._
|
||||
|
||||
Please note, importing data with old timestamps is called **backfilling** and may require resetting caches as described [here](https://docs.victoriametrics.com/#backfilling).
|
||||
Please note, importing data with old timestamps is called **backfilling** and may require resetting caches as described [here](https://docs.victoriametrics.com/victoriametrics#backfilling).
|
||||
|
||||
### Check imported metrics
|
||||
|
||||
|
||||
@@ -4,21 +4,21 @@ Setup Victoria Metrics Cluster with support of multiple retention periods within
|
||||
|
||||
**Enterprise Solution**
|
||||
|
||||
[VictoriaMetrics enterprise](https://docs.victoriametrics.com/enterprise/) supports specifying multiple retentions
|
||||
for distinct sets of time series and [tenants](https://docs.victoriametrics.com/cluster-victoriametrics/#multitenancy)
|
||||
via [retention filters](https://docs.victoriametrics.com/cluster-victoriametrics/#retention-filters).
|
||||
[VictoriaMetrics enterprise](https://docs.victoriametrics.com/victoriametrics/enterprise/) supports specifying multiple retentions
|
||||
for distinct sets of time series and [tenants](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#multitenancy)
|
||||
via [retention filters](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#retention-filters).
|
||||
|
||||
**Open Source Solution**
|
||||
|
||||
Community version of VictoriaMetrics supports only one retention period per `vmstorage` node via [-retentionPeriod](https://docs.victoriametrics.com/#retention) command-line flag.
|
||||
Community version of VictoriaMetrics supports only one retention period per `vmstorage` node via [-retentionPeriod](https://docs.victoriametrics.com/victoriametrics#retention) command-line flag.
|
||||
|
||||
A multi-retention setup can be implemented by dividing a [victoriametrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/) into logical groups with different retentions.
|
||||
A multi-retention setup can be implemented by dividing a [victoriametrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/) into logical groups with different retentions.
|
||||
|
||||
Example:
|
||||
Setup should handle 3 different retention groups 3months, 1year and 3 years.
|
||||
Solution contains 3 groups of vmstorages + vminserts and one group of vmselects. Routing is done by [vmagent](https://docs.victoriametrics.com/vmagent/)
|
||||
by [splitting data streams](https://docs.victoriametrics.com/vmagent/#splitting-data-streams-among-multiple-systems).
|
||||
The [-retentionPeriod](https://docs.victoriametrics.com/#retention) sets how long to keep the metrics.
|
||||
Solution contains 3 groups of vmstorages + vminserts and one group of vmselects. Routing is done by [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/)
|
||||
by [splitting data streams](https://docs.victoriametrics.com/victoriametrics/vmagent/#splitting-data-streams-among-multiple-systems).
|
||||
The [-retentionPeriod](https://docs.victoriametrics.com/victoriametrics#retention) sets how long to keep the metrics.
|
||||
|
||||
The diagram below shows a proposed solution
|
||||
|
||||
@@ -26,12 +26,12 @@ The diagram below shows a proposed solution
|
||||
|
||||
**Implementation Details**
|
||||
|
||||
1. Groups of vminserts A know about only vmstorages A and this is explicitly specified via `-storageNode` [configuration](https://docs.victoriametrics.com/cluster-victoriametrics/#cluster-setup).
|
||||
1. Groups of vminserts B know about only vmstorages B and this is explicitly specified via `-storageNode` [configuration](https://docs.victoriametrics.com/cluster-victoriametrics/#cluster-setup).
|
||||
1. Groups of vminserts C know about only vmstorages C and this is explicitly specified via `-storageNode` [configuration](https://docs.victoriametrics.com/cluster-victoriametrics/#cluster-setup).
|
||||
1. vmselect reads data from all vmstorage nodes via `-storageNode` [configuration](https://docs.victoriametrics.com/cluster-victoriametrics/#cluster-setup)
|
||||
with [deduplication](https://docs.victoriametrics.com/cluster-victoriametrics/#deduplication) setting equal to vmagent's scrape interval or minimum interval between collected samples.
|
||||
1. vmagent routes incoming metrics to the given set of `vminsert` nodes using relabeling rules specified at `-remoteWrite.urlRelabelConfig` [configuration](https://docs.victoriametrics.com/vmagent/#relabeling).
|
||||
1. Groups of vminserts A know about only vmstorages A and this is explicitly specified via `-storageNode` [configuration](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#cluster-setup).
|
||||
1. Groups of vminserts B know about only vmstorages B and this is explicitly specified via `-storageNode` [configuration](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#cluster-setup).
|
||||
1. Groups of vminserts C know about only vmstorages C and this is explicitly specified via `-storageNode` [configuration](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#cluster-setup).
|
||||
1. vmselect reads data from all vmstorage nodes via `-storageNode` [configuration](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#cluster-setup)
|
||||
with [deduplication](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#deduplication) setting equal to vmagent's scrape interval or minimum interval between collected samples.
|
||||
1. vmagent routes incoming metrics to the given set of `vminsert` nodes using relabeling rules specified at `-remoteWrite.urlRelabelConfig` [configuration](https://docs.victoriametrics.com/victoriametrics/vmagent/#relabeling).
|
||||
|
||||
**Multi-Tenant Setup**
|
||||
|
||||
@@ -39,4 +39,4 @@ Every group of vmstorages can handle one tenant or multiple one. Different group
|
||||
|
||||
**Additional Enhancements**
|
||||
|
||||
You can set up [vmauth](https://docs.victoriametrics.com/vmauth/) for routing data to the given vminsert group depending on the needed retention.
|
||||
You can set up [vmauth](https://docs.victoriametrics.com/victoriametrics/vmauth/) for routing data to the given vminsert group depending on the needed retention.
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
**The guide covers:**
|
||||
|
||||
* High availability monitoring via [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/) in [Kubernetes](https://kubernetes.io/) with Helm charts
|
||||
* High availability monitoring via [VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/) in [Kubernetes](https://kubernetes.io/) with Helm charts
|
||||
* How to store metrics
|
||||
* How to scrape metrics from k8s components using a service discovery
|
||||
* How to visualize stored data
|
||||
@@ -50,8 +50,8 @@ vmstorage:
|
||||
EOF
|
||||
```
|
||||
|
||||
* The `Helm install vmcluster vm/victoria-metrics-cluster` command installs [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/) to the default [namespace](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/).
|
||||
* `dedup.minScrapeInterval: 1ms` configures [de-duplication](https://docs.victoriametrics.com/#deduplication) for the cluster that de-duplicates data points in the same time series if they fall within the same discrete 1ms bucket. The earliest data point will be kept. In the case of equal timestamps, an arbitrary data point will be kept.
|
||||
* The `Helm install vmcluster vm/victoria-metrics-cluster` command installs [VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/) to the default [namespace](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/).
|
||||
* `dedup.minScrapeInterval: 1ms` configures [de-duplication](https://docs.victoriametrics.com/victoriametrics#deduplication) for the cluster that de-duplicates data points in the same time series if they fall within the same discrete 1ms bucket. The earliest data point will be kept. In the case of equal timestamps, an arbitrary data point will be kept.
|
||||
* `replicationFactor: 2` Replication factor for the ingested data, i.e. how many copies should be made among distinct `-storageNode` instances. If the replication factor is greater than one, the deduplication must be enabled on the remote storage side.
|
||||
* `podAnnotations: prometheus.io/scrape: "true"` enables the scraping of metrics from the vmselect, vminsert and vmstorage pods.
|
||||
* `podAnnotations:prometheus.io/port: "some_port" ` enables the scraping of metrics from the vmselect, vminsert and vmstorage pods from corresponding ports.
|
||||
@@ -135,7 +135,7 @@ vmcluster-victoria-metrics-cluster-vmstorage-2 1/1 Running
|
||||
|
||||
## 3. Install vmagent from the Helm chart
|
||||
|
||||
To scrape metrics from Kubernetes with a VictoriaMetrics Cluster we will need to install [vmagent](https://docs.victoriametrics.com/vmagent/) with some additional configurations. To do so, please run the following command:
|
||||
To scrape metrics from Kubernetes with a VictoriaMetrics Cluster we will need to install [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/) with some additional configurations. To do so, please run the following command:
|
||||
|
||||
```yaml
|
||||
helm install vmagent vm/victoria-metrics-agent -f https://docs.victoriametrics.com/guides/examples/guide-vmcluster-vmagent-values.yaml
|
||||
@@ -222,7 +222,7 @@ scrape_configs:
|
||||
target_label: systemd_service_name
|
||||
replacement: '${1}'
|
||||
```
|
||||
* By updating `remoteWrite` we configuring [vmagent](https://docs.victoriametrics.com/vmagent/) to write scraped metrics into the `vminsert` service.
|
||||
* By updating `remoteWrite` we configuring [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/) to write scraped metrics into the `vminsert` service.
|
||||
* The `metric_relabel_configs` section allows you to process Kubernetes metrics for the Grafana dashboard.
|
||||
|
||||
|
||||
@@ -337,7 +337,7 @@ The expected output is:
|
||||
}
|
||||
```
|
||||
|
||||
* Query `http://127.0.0.1:8481/select/0/prometheus/api/v1/query_range` uses [VictoriaMetrics querying API](https://docs.victoriametrics.com/cluster-victoriametrics/#url-format) to fetch previously stored data points;
|
||||
* Query `http://127.0.0.1:8481/select/0/prometheus/api/v1/query_range` uses [VictoriaMetrics querying API](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#url-format) to fetch previously stored data points;
|
||||
* Argument `query=count(up{kubernetes_pod_name=~".*vmselect.*"})` specifies the query we want to execute. Specifically, we calculate the number of `vmselect` pods.
|
||||
* Additional arguments `start=-10m&step=1m'` set requested time range from -10 minutes (10 minutes ago) to now (default value if `end` argument is omitted) and step (the distance between returned data points) of 1 minute;
|
||||
* By adding `| jq` we pass the output to the jq utility which outputs information in json format
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
**This guide covers:**
|
||||
|
||||
* The setup of a [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/) in [Kubernetes](https://kubernetes.io/) via Helm charts
|
||||
* The setup of a [VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/) in [Kubernetes](https://kubernetes.io/) via Helm charts
|
||||
* How to scrape metrics from k8s components using service discovery
|
||||
* How to visualize stored data
|
||||
* How to store metrics in [VictoriaMetrics](https://victoriametrics.com) tsdb
|
||||
@@ -17,7 +17,7 @@ We will use:
|
||||
|
||||
## 1. VictoriaMetrics Helm repository
|
||||
|
||||
You need to add the VictoriaMetrics Helm repository to install VictoriaMetrics components. We’re going to use [VictoriaMetrics Cluster](https://docs.victoriametrics.com/cluster-victoriametrics/). You can do this by running the following command:
|
||||
You need to add the VictoriaMetrics Helm repository to install VictoriaMetrics components. We’re going to use [VictoriaMetrics Cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/). You can do this by running the following command:
|
||||
|
||||
```shell
|
||||
helm repo add vm https://victoriametrics.github.io/helm-charts/
|
||||
@@ -76,7 +76,7 @@ vmstorage:
|
||||
EOF
|
||||
```
|
||||
|
||||
* By running `Helm install vmcluster vm/victoria-metrics-cluster` we install [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/) to default [namespace](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) inside your cluster.
|
||||
* By running `Helm install vmcluster vm/victoria-metrics-cluster` we install [VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/) to default [namespace](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) inside your cluster.
|
||||
* By adding `podAnnotations: prometheus.io/scrape: "true"` we enable the scraping of metrics from the vmselect, vminsert and vmstorage pods.
|
||||
* By adding `podAnnotations:prometheus.io/port: "some_port" ` we enable the scraping of metrics from the vmselect, vminsert and vmstorage pods from their ports as well.
|
||||
|
||||
@@ -135,7 +135,7 @@ for example - inside the Kubernetes cluster:
|
||||
|
||||
For us it’s important to remember the url for the datasource (copy lines from the output).
|
||||
|
||||
Verify that [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/) pods are up and running by executing the following command:
|
||||
Verify that [VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/) pods are up and running by executing the following command:
|
||||
|
||||
|
||||
```sh
|
||||
@@ -156,7 +156,7 @@ vmcluster-victoria-metrics-cluster-vmstorage-1 1/1 Running
|
||||
|
||||
## 3. Install vmagent from the Helm chart
|
||||
|
||||
To scrape metrics from Kubernetes with a [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/) we need to install [vmagent](https://docs.victoriametrics.com/vmagent/) with additional configuration. To do so, please run these commands in your terminal:
|
||||
To scrape metrics from Kubernetes with a [VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/) we need to install [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/) with additional configuration. To do so, please run these commands in your terminal:
|
||||
|
||||
|
||||
```shell
|
||||
@@ -390,7 +390,7 @@ config:
|
||||
target_label: kubernetes_pod_name
|
||||
```
|
||||
|
||||
* By updating `remoteWrite` we configuring [vmagent](https://docs.victoriametrics.com/vmagent/) to write scraped metrics into the `vminsert` service.
|
||||
* By updating `remoteWrite` we configuring [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/) to write scraped metrics into the `vminsert` service.
|
||||
* The second part of this yaml file is needed to add the `metric_relabel_configs` section that helps us to show Kubernetes metrics on the Grafana dashboard.
|
||||
|
||||
|
||||
@@ -471,8 +471,8 @@ EOF
|
||||
By running this command we:
|
||||
* Install Grafana from the Helm repository.
|
||||
* Provision a VictoriaMetrics data source with the url from the output above which we remembered.
|
||||
* Add [this dashboard](https://grafana.com/grafana/dashboards/11176) for [VictoriaMetrics Cluster](https://docs.victoriametrics.com/cluster-victoriametrics/).
|
||||
* Add [this dashboard](https://grafana.com/grafana/dashboards/12683) for [VictoriaMetrics Agent](https://docs.victoriametrics.com/vmagent/).
|
||||
* Add [this dashboard](https://grafana.com/grafana/dashboards/11176) for [VictoriaMetrics Cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/).
|
||||
* Add [this dashboard](https://grafana.com/grafana/dashboards/12683) for [VictoriaMetrics Agent](https://docs.victoriametrics.com/victoriametrics/vmagent/).
|
||||
* Add [this dashboard](https://grafana.com/grafana/dashboards/14205) dashboard to see Kubernetes cluster metrics.
|
||||
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
**This guide covers:**
|
||||
|
||||
* The setup of a [VictoriaMetrics Single](https://docs.victoriametrics.com/single-server-victoriametrics/) in [Kubernetes](https://kubernetes.io/) via Helm charts
|
||||
* The setup of a [VictoriaMetrics Single](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/) in [Kubernetes](https://kubernetes.io/) via Helm charts
|
||||
* How to scrape metrics from k8s components using service discovery
|
||||
* How to visualize stored data
|
||||
* How to store metrics in [VictoriaMetrics](https://victoriametrics.com) tsdb
|
||||
@@ -17,7 +17,7 @@ We will use:
|
||||
|
||||
## 1. VictoriaMetrics Helm repository
|
||||
|
||||
You need to add the VictoriaMetrics Helm repository to install VictoriaMetrics components. We’re going to use [VictoriaMetrics Single](https://docs.victoriametrics.com/single-server-victoriametrics/). You can do this by running the following command:
|
||||
You need to add the VictoriaMetrics Helm repository to install VictoriaMetrics components. We’re going to use [VictoriaMetrics Single](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/). You can do this by running the following command:
|
||||
|
||||
|
||||
```shell
|
||||
@@ -56,7 +56,7 @@ vm/victoria-metrics-single 0.15.1 v1.113.0 Victoria Metrics Singl
|
||||
```
|
||||
|
||||
|
||||
## 2. Install [VictoriaMetrics Single](https://docs.victoriametrics.com/single-server-victoriametrics/) from Helm Chart
|
||||
## 2. Install [VictoriaMetrics Single](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/) from Helm Chart
|
||||
|
||||
Run this command in your terminal:
|
||||
|
||||
@@ -151,8 +151,8 @@ server:
|
||||
```
|
||||
|
||||
|
||||
* By running `helm install vmsingle vm/victoria-metrics-single` we install [VictoriaMetrics Single](https://docs.victoriametrics.com/single-server-victoriametrics/) to default [namespace](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) inside your cluster
|
||||
* By adding `scrape: enabled: true` we add and enable autodiscovery scraping from kubernetes cluster to [VictoriaMetrics Single](https://docs.victoriametrics.com/single-server-victoriametrics/)
|
||||
* By running `helm install vmsingle vm/victoria-metrics-single` we install [VictoriaMetrics Single](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/) to default [namespace](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) inside your cluster
|
||||
* By adding `scrape: enabled: true` we add and enable autodiscovery scraping from kubernetes cluster to [VictoriaMetrics Single](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/)
|
||||
* On line 67 from [https://docs.victoriametrics.com/guides/examples/guide-vmsingle-values.yaml](https://docs.victoriametrics.com/guides/examples/guide-vmsingle-values.yaml) we added `metric_relabel_configs` section that will help us to show Kubernetes metrics on Grafana dashboard.
|
||||
|
||||
|
||||
@@ -177,7 +177,7 @@ Metrics Ingestion:
|
||||
Write URL inside the kubernetes cluster:
|
||||
http://vmsingle-victoria-metrics-single-server.default.svc.cluster.local.:8428/<protocol-specific-write-endpoint>
|
||||
|
||||
All supported write endpoints can be found at https://docs.victoriametrics.com/single-server-victoriametrics/#how-to-import-time-series-data
|
||||
All supported write endpoints can be found at https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#how-to-import-time-series-data
|
||||
|
||||
E.g: for Prometheus:
|
||||
http://vmsingle-victoria-metrics-single-server.default.svc.cluster.local.:8428/api/v1/write
|
||||
|
||||
@@ -5,8 +5,8 @@ analytics. It has its query language, unique data model, and rich tooling for co
|
||||
VictoriaMetrics is a high-performance opensource time series database specifically designed to deal with huge volumes of
|
||||
monitoring data while remaining cost-efficient at the same time. Many companies are choosing to migrate from InfluxDB to
|
||||
VictoriaMetrics specifically for performance and scalability reasons. Along them see case studies provided by
|
||||
[ARNES](https://docs.victoriametrics.com/casestudies/#arnes)
|
||||
and [Brandwatch](https://docs.victoriametrics.com/casestudies/#brandwatch).
|
||||
[ARNES](https://docs.victoriametrics.com/victoriametrics/casestudies/#arnes)
|
||||
and [Brandwatch](https://docs.victoriametrics.com/victoriametrics/casestudies/#brandwatch).
|
||||
|
||||
This guide covers the differences between two solutions, most commonly asked questions, and approaches for migrating
|
||||
from InfluxDB to VictoriaMetrics.
|
||||
@@ -14,17 +14,17 @@ from InfluxDB to VictoriaMetrics.
|
||||
## Data model differences
|
||||
|
||||
Readers are familiar with [InfluxDB key concepts](https://docs.influxdata.com/influxdb/v2.2/reference/key-concepts/).
|
||||
The data model of VictoriaMetrics is [explained here](https://docs.victoriametrics.com/keyconcepts/#data-model).
|
||||
The data model of VictoriaMetrics is [explained here](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#data-model).
|
||||
Let's list similarities and differences:
|
||||
|
||||
* both databases are **schemaless**, which means there is no need to define metrics or their tags in advance;
|
||||
* multidimensional data support is implemented
|
||||
via [tags](https://docs.influxdata.com/influxdb/v2.2/reference/key-concepts/data-elements/#tags)
|
||||
in InfluxDB and via [labels](https://docs.victoriametrics.com/keyconcepts/#structure-of-a-metric) in
|
||||
VictoriaMetrics. However, labels in VictoriaMetrics are always [strings](https://docs.victoriametrics.com/keyconcepts/#labels),
|
||||
in InfluxDB and via [labels](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#structure-of-a-metric) in
|
||||
VictoriaMetrics. However, labels in VictoriaMetrics are always [strings](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#labels),
|
||||
while InfluxDB supports many data types;
|
||||
* timestamps are stored with **nanosecond** resolution in InfluxDB, while in VictoriaMetrics it is **milliseconds**;
|
||||
* in VictoriaMetrics metric value is always [float64](https://docs.victoriametrics.com/keyconcepts/#raw-samples),
|
||||
* in VictoriaMetrics metric value is always [float64](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#raw-samples),
|
||||
while InfluxDB supports multiple data types.
|
||||
* there are
|
||||
no [measurements](https://docs.influxdata.com/influxdb/v2.2/reference/key-concepts/data-elements/#measurement)
|
||||
@@ -34,8 +34,8 @@ Let's list similarities and differences:
|
||||
* there are no [databases](https://docs.influxdata.com/influxdb/v1.8/concepts/glossary/#database),
|
||||
[buckets](https://docs.influxdata.com/influxdb/v2.2/reference/key-concepts/data-elements/#bucket)
|
||||
or [organizations](https://docs.influxdata.com/influxdb/v2.2/reference/key-concepts/data-elements/#organization).
|
||||
All data in VictoriaMetrics is stored in a global namespace or within a [tenant](https://docs.victoriametrics.com/keyconcepts/#multi-tenancy).
|
||||
* VictoriaMetrics query language is [MetricsQL](https://docs.victoriametrics.com/metricsql/). Influx has multiple versions
|
||||
All data in VictoriaMetrics is stored in a global namespace or within a [tenant](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#multi-tenancy).
|
||||
* VictoriaMetrics query language is [MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/). Influx has multiple versions
|
||||
of query languages. VictoriaMetrics doesn't support any of them.
|
||||
|
||||
Let's consider the following [sample data](https://docs.influxdata.com/influxdb/v2.2/reference/key-concepts/data-elements/#sample-data)
|
||||
@@ -63,7 +63,7 @@ by VictoriaMetrics, so lookups by names or labels have the same query speed.
|
||||
|
||||
## Write data
|
||||
|
||||
VictoriaMetrics supports [InfluxDB line protocol](https://docs.victoriametrics.com/#how-to-send-data-from-influxdb-compatible-agents-such-as-telegraf)
|
||||
VictoriaMetrics supports [InfluxDB line protocol](https://docs.victoriametrics.com/victoriametrics#how-to-send-data-from-influxdb-compatible-agents-such-as-telegraf)
|
||||
for data ingestion. For example, to write a measurement to VictoriaMetrics we need to send an HTTP POST request with
|
||||
payload in a line protocol format:
|
||||
```sh
|
||||
@@ -95,7 +95,7 @@ The expected response is the following:
|
||||
}
|
||||
```
|
||||
|
||||
_VictoriaMetrics performs additional [data mapping](https://docs.victoriametrics.com/#how-to-send-data-from-influxdb-compatible-agents-such-as-telegraf)
|
||||
_VictoriaMetrics performs additional [data mapping](https://docs.victoriametrics.com/victoriametrics#how-to-send-data-from-influxdb-compatible-agents-such-as-telegraf)
|
||||
to the data ingested via InfluxDB line protocol._
|
||||
|
||||
VictoriaMetrics is also compatible with [Telegraf](https://github.com/influxdata/telegraf).
|
||||
@@ -107,18 +107,18 @@ To make Telegraf sending data, simply add `http://<victoriametric-addr>:8428` UR
|
||||
```
|
||||
|
||||
In addition to InfluxDB line protocol, VictoriaMetrics supports many other ways for
|
||||
[metrics collection](https://docs.victoriametrics.com/keyconcepts/#write-data).
|
||||
[metrics collection](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#write-data).
|
||||
|
||||
## Query data
|
||||
|
||||
VictoriaMetrics does not have a command-line interface (CLI). Instead, it provides
|
||||
[VMUI](https://docs.victoriametrics.com/single-server-victoriametrics/#vmui) - a graphical User Interface for
|
||||
[VMUI](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#vmui) - a graphical User Interface for
|
||||
querying and visualizing metrics:
|
||||
|
||||

|
||||
|
||||
VictoriaMetrics can be configured as [Grafana datasource](https://docs.victoriametrics.com/single-server-victoriametrics/#grafana-setup).
|
||||
See more about [how to query data in VictoriaMetrics](https://docs.victoriametrics.com/keyconcepts/#query-data).
|
||||
VictoriaMetrics can be configured as [Grafana datasource](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#grafana-setup).
|
||||
See more about [how to query data in VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#query-data).
|
||||
|
||||
### Basic concepts
|
||||
|
||||
@@ -154,23 +154,23 @@ WHERE ("instance" = 'localhost')
|
||||
GROUP BY time (1m)
|
||||
```
|
||||
|
||||
Now, let's [import](https://docs.victoriametrics.com/#how-to-send-data-from-influxdb-compatible-agents-such-as-telegraf)
|
||||
Now, let's [import](https://docs.victoriametrics.com/victoriametrics#how-to-send-data-from-influxdb-compatible-agents-such-as-telegraf)
|
||||
the same data sample in VictoriaMetrics and plot it in Grafana. To understand how the InfluxQL query might be translated
|
||||
to [MetricsQL](https://docs.victoriametrics.com/metricsql/) let's break it into components first:
|
||||
to [MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/) let's break it into components first:
|
||||
|
||||
* `SELECT last("bar") FROM "foo"` - all requests to [instant](https://docs.victoriametrics.com/keyconcepts/#instant-query)
|
||||
or [range](https://docs.victoriametrics.com/keyconcepts/#range-query) VictoriaMetrics APIs are reads, so no need
|
||||
* `SELECT last("bar") FROM "foo"` - all requests to [instant](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#instant-query)
|
||||
or [range](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#range-query) VictoriaMetrics APIs are reads, so no need
|
||||
to specify the `SELECT` statement. There are no `measurements` or `fields` in VictoriaMetrics, so the whole expression
|
||||
can be replaced with `foo_bar` in MetricsQL;
|
||||
* `WHERE ("instance" = 'localhost')`- [filtering by labels](https://docs.victoriametrics.com/keyconcepts/#filtering)
|
||||
* `WHERE ("instance" = 'localhost')`- [filtering by labels](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#filtering)
|
||||
in MetricsQL requires specifying the filter in curly braces next to the metric name. So in MetricsQL filter expression
|
||||
will be translated to `{instance="localhost"}`;
|
||||
* `WHERE $timeFilter` - filtering by time is done via request params sent along with query, so in MetricsQL no need to
|
||||
specify this filter;
|
||||
* `GROUP BY time(1m)` - grouping by time is done by default
|
||||
in [range](https://docs.victoriametrics.com/keyconcepts/#range-query) API according to specified `step` param.
|
||||
in [range](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#range-query) API according to specified `step` param.
|
||||
This param is also a part of params sent along with request. See how to perform additional
|
||||
[aggregations and grouping via MetricsQL](https://docs.victoriametrics.com/keyconcepts/#aggregation-and-grouping-functions).
|
||||
[aggregations and grouping via MetricsQL](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#aggregation-and-grouping-functions).
|
||||
|
||||
In result, the MetricsQL expression will have the following form: `foo_bar{instance="localhost"}`.
|
||||
See its result executed with `step=1m` for the same set of data in Grafana below:
|
||||
@@ -178,13 +178,13 @@ See its result executed with `step=1m` for the same set of data in Grafana below
|
||||

|
||||
|
||||
Visualizations from both databases are a bit different - VictoriaMetrics shows some extra points filling the gaps in the graph.
|
||||
This behavior is described in more detail [here](https://docs.victoriametrics.com/keyconcepts/#range-query).
|
||||
This behavior is described in more detail [here](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#range-query).
|
||||
In InfluxDB, we can achieve a similar behavior by adding `fill(previous)` to the query.
|
||||
|
||||
VictoriaMetrics fills the gaps on the graph assuming time series are always continuous and not discrete.
|
||||
To limit the interval on which VictoriaMetrics will try to fill the gaps, set `-search.setLookbackToStep`
|
||||
command-line flag. This limits the gap filling to a single `step` interval passed to
|
||||
[/api/v1/query_range](https://docs.victoriametrics.com/keyconcepts/#range-query).
|
||||
[/api/v1/query_range](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#range-query).
|
||||
|
||||
### Advanced usage
|
||||
|
||||
@@ -193,52 +193,52 @@ Let's consider one of the most popular Grafana dashboards [Node Exporter Full](h
|
||||
It has almost 15 million downloads and about 230 queries in it! But a closer look at those queries shows the following:
|
||||
|
||||
* ~120 queries are just selecting a metric with label filters, e.g. `node_textfile_scrape_error{instance="$node",job="$job"}`;
|
||||
* ~80 queries are using [rate](https://docs.victoriametrics.com/metricsql/#rate) function for selected metric,
|
||||
* ~80 queries are using [rate](https://docs.victoriametrics.com/victoriametrics/metricsql/#rate) function for selected metric,
|
||||
e.g. `rate(node_netstat_Tcp_InSegs{instance="$node",job="$job"}[5m])`
|
||||
* and the rest are [aggregation functions](https://docs.victoriametrics.com/keyconcepts/#aggregation-and-grouping-functions)
|
||||
like [sum](https://docs.victoriametrics.com/metricsql/#sum) or [count](https://docs.victoriametrics.com/metricsql/#count).
|
||||
* and the rest are [aggregation functions](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#aggregation-and-grouping-functions)
|
||||
like [sum](https://docs.victoriametrics.com/victoriametrics/metricsql/#sum) or [count](https://docs.victoriametrics.com/victoriametrics/metricsql/#count).
|
||||
|
||||
To get a better understanding of how MetricsQL works, see the following resources:
|
||||
* [MetricsQL concepts](https://docs.victoriametrics.com/keyconcepts/#metricsql);
|
||||
* [MetricsQL functions](https://docs.victoriametrics.com/metricsql/);
|
||||
* [MetricsQL concepts](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#metricsql);
|
||||
* [MetricsQL functions](https://docs.victoriametrics.com/victoriametrics/metricsql/);
|
||||
* [PromQL tutorial for beginners](https://valyala.medium.com/promql-tutorial-for-beginners-9ab455142085).
|
||||
|
||||
## How to migrate current data from InfluxDB to VictoriaMetrics
|
||||
|
||||
Migrating data from other databases to VictoriaMetrics is as simple as importing data via any of
|
||||
[supported ingestion formats](https://docs.victoriametrics.com/keyconcepts/#push-model).
|
||||
[supported ingestion formats](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#push-model).
|
||||
|
||||
But migration from InfluxDB might get easier with [vmctl](https://docs.victoriametrics.com/vmctl/). See more about
|
||||
migrating [from InfluxDB v1.x versions](https://docs.victoriametrics.com/vmctl/#migrating-data-from-influxdb-1x).
|
||||
Migrating data from InfluxDB v2.x is not supported. But there is a useful [3rd party solution](https://docs.victoriametrics.com/vmctl/#migrating-data-from-influxdb-2x)
|
||||
But migration from InfluxDB might get easier with [vmctl](https://docs.victoriametrics.com/victoriametrics/vmctl/). See more about
|
||||
migrating [from InfluxDB v1.x versions](https://docs.victoriametrics.com/victoriametrics/vmctl/#migrating-data-from-influxdb-1x).
|
||||
Migrating data from InfluxDB v2.x is not supported. But there is a useful [3rd party solution](https://docs.victoriametrics.com/victoriametrics/vmctl/#migrating-data-from-influxdb-2x)
|
||||
for this.
|
||||
|
||||
Please note, data migration is a backfilling process, so read about [backfilling tips](https://docs.victoriametrics.com/single-server-victoriametrics/#backfilling).
|
||||
Please note, data migration is a backfilling process, so read about [backfilling tips](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#backfilling).
|
||||
|
||||
## Frequently asked questions
|
||||
|
||||
* How does VictoriaMetrics compare to InfluxDB?
|
||||
* _[VictoriaMetrics is way more resource efficient](https://docs.victoriametrics.com/faq/#how-does-victoriametrics-compare-to-influxdb)._
|
||||
* _[VictoriaMetrics is way more resource efficient](https://docs.victoriametrics.com/victoriametrics/faq/#how-does-victoriametrics-compare-to-influxdb)._
|
||||
* Why don't VictoriaMetrics support Remote Read API, so I don't need to learn MetricsQL?
|
||||
* _[Remote Read API has very high performance overhead](https://docs.victoriametrics.com/faq/#why-doesnrsquot-victoriametrics-support-the-prometheus-remote-read-api)._
|
||||
* _[Remote Read API has very high performance overhead](https://docs.victoriametrics.com/victoriametrics/faq/#why-doesnrsquot-victoriametrics-support-the-prometheus-remote-read-api)._
|
||||
* PromQL and MetricsQL are often mentioned together - why is that?
|
||||
* _MetricsQL - query language inspired by PromQL. MetricsQL is backward-compatible with PromQL, so Grafana
|
||||
dashboards backed by Prometheus datasource should work the same after switching from Prometheus to
|
||||
VictoriaMetrics. Both languages share the same concepts with slight differences._
|
||||
* Query returns more data points than expected - why?
|
||||
* _VictoriaMetrics may return non-existing data points if `step` param is lower than the actual data resolution. See
|
||||
more about this [here](https://docs.victoriametrics.com/keyconcepts/#range-query)._
|
||||
more about this [here](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#range-query)._
|
||||
* How do I get the `real` last data point?
|
||||
* _[last_over_time](https://docs.victoriametrics.com/metricsql/#last_over_time) function returns last value on
|
||||
* _[last_over_time](https://docs.victoriametrics.com/victoriametrics/metricsql/#last_over_time) function returns last value on
|
||||
the given look-behind window. For example, `last_over_time(metric[10s])` would return
|
||||
sample values only if the real samples are located closer than 10 seconds to the calculated timestamps
|
||||
according to `start`, `end` and `step` query args passed
|
||||
to [range query](https://docs.victoriametrics.com/keyconcepts/#range-query)._
|
||||
* _[tlast_over_time](https://docs.victoriametrics.com/metricsql/#tlast_over_time) function returns last timestamp on
|
||||
the given look-behind window, similarly to [last_over_time](https://docs.victoriametrics.com/metricsql/#last_over_time)._
|
||||
to [range query](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#range-query)._
|
||||
* _[tlast_over_time](https://docs.victoriametrics.com/victoriametrics/metricsql/#tlast_over_time) function returns last timestamp on
|
||||
the given look-behind window, similarly to [last_over_time](https://docs.victoriametrics.com/victoriametrics/metricsql/#last_over_time)._
|
||||
* How do I get raw data points with MetricsQL?
|
||||
* _For getting raw data points specify the interval at which you want them in square brackets and send
|
||||
as [instant query](https://docs.victoriametrics.com/keyconcepts/#instant-query). For
|
||||
as [instant query](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#instant-query). For
|
||||
example, `GET api/v1/query?query=my_metric[5m]&time=<time>` will return raw samples for `my_metric` in interval
|
||||
from `<time>` to `<time>-5m`._
|
||||
* Can you have multiple aggregators in a MetricsQL query, e.g. `SELECT MAX(field), MIN(field) ...`?
|
||||
|
||||
@@ -29,20 +29,20 @@ Using this schema, you can achieve:
|
||||
|
||||
* If you scrape data from Prometheus-compatible targets, then please specify `-promscrape.config` parameter as well.
|
||||
|
||||
Here is a Quickstart guide for [vmagent](https://docs.victoriametrics.com/vmagent/#quick-start)
|
||||
Here is a Quickstart guide for [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/#quick-start)
|
||||
|
||||
### How to read the data from Ground Control regions
|
||||
|
||||
You can use one of the following options:
|
||||
|
||||
1. Multi-level [vmselect setup](https://docs.victoriametrics.com/cluster-victoriametrics/#multi-level-cluster-setup) in cluster setup, top-level vmselect(s) reads data from cluster-level vmselects
|
||||
1. Multi-level [vmselect setup](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#multi-level-cluster-setup) in cluster setup, top-level vmselect(s) reads data from cluster-level vmselects
|
||||
* Returns data in one of the clusters is unavailable
|
||||
* Merges data from both sources. You need to turn on [deduplication](https://docs.victoriametrics.com/cluster-victoriametrics/#deduplication) to remove duplicates
|
||||
* Merges data from both sources. You need to turn on [deduplication](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#deduplication) to remove duplicates
|
||||
1. Regional endpoints - use one regional endpoint as default and switch to another if there is an issue.
|
||||
1. Load balancer - that sends queries to a particular region. The benefit and disadvantage of this setup is that it's simple.
|
||||
1. Promxy - proxy that reads data from multiple Prometheus-like sources. It allows reading data more intelligently to cover the region's unavailability out of the box. It doesn't support MetricsQL yet (please check this issue).
|
||||
1. Global vmselect in cluster setup - you can set up an additional subset of vmselects that knows about all storages in all regions.
|
||||
* The [deduplication](https://docs.victoriametrics.com/cluster-victoriametrics/#deduplication) in 1ms on the vmselect side must be turned on. This setup allows you to query data using MetricsQL.
|
||||
* The [deduplication](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#deduplication) in 1ms on the vmselect side must be turned on. This setup allows you to query data using MetricsQL.
|
||||
* The downside is that vmselect waits for a response from all storages in all regions.
|
||||
|
||||
|
||||
@@ -68,8 +68,8 @@ An additional VictoriaMetrics single can be set up in every region, scraping met
|
||||
You also may evaluate the option to send these metrics to the neighbour region to achieve HA.
|
||||
|
||||
Additional context
|
||||
* VictoriaMetrics Single - [https://docs.victoriametrics.com/single-server-victoriametrics/#monitoring](https://docs.victoriametrics.com/single-server-victoriametrics/#monitoring)
|
||||
* VictoriaMetrics Cluster - [https://docs.victoriametrics.com/cluster-victoriametrics/#monitoring](https://docs.victoriametrics.com/cluster-victoriametrics/#monitoring)
|
||||
* VictoriaMetrics Single - [https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#monitoring](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#monitoring)
|
||||
* VictoriaMetrics Cluster - [https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#monitoring](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#monitoring)
|
||||
|
||||
|
||||
### What more can we do?
|
||||
|
||||
@@ -1,20 +1,20 @@
|
||||
This guide is based on capacity planning for [Single-Node](https://docs.victoriametrics.com/single-server-victoriametrics/#capacity-planning),
|
||||
[Cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#capacity-planning)
|
||||
This guide is based on capacity planning for [Single-Node](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#capacity-planning),
|
||||
[Cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#capacity-planning)
|
||||
and [VictoriaMetrics Cloud](https://docs.victoriametrics.com/victoriametrics-cloud/) docs.
|
||||
|
||||
## Terminology
|
||||
|
||||
- [Active Time Series](https://docs.victoriametrics.com/faq/#what-is-an-active-time-series) - a [time series](https://docs.victoriametrics.com/keyconcepts/#time-series)
|
||||
- [Active Time Series](https://docs.victoriametrics.com/victoriametrics/faq/#what-is-an-active-time-series) - a [time series](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#time-series)
|
||||
that was update at least one time during the last hour;
|
||||
- Ingestion Rate - how many [samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) are ingest into the database per second;
|
||||
- [Churn Rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate) - how frequently a new [time series](https://docs.victoriametrics.com/keyconcepts/#time-series)
|
||||
- Ingestion Rate - how many [samples](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#raw-samples) are ingest into the database per second;
|
||||
- [Churn Rate](https://docs.victoriametrics.com/victoriametrics/faq/#what-is-high-churn-rate) - how frequently a new [time series](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#time-series)
|
||||
is created. For example, changing pod name in Kubernetes is a common source of time series churn;
|
||||
- Queries per Second - how many [read queries](https://docs.victoriametrics.com/keyconcepts/#query-data) are executed per second;
|
||||
- [Retention Period](https://docs.victoriametrics.com/#retention) - for how long data is stored in the database.
|
||||
- Queries per Second - how many [read queries](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#query-data) are executed per second;
|
||||
- [Retention Period](https://docs.victoriametrics.com/victoriametrics#retention) - for how long data is stored in the database.
|
||||
|
||||
### Active Time Series
|
||||
|
||||
Time series [exposed](https://docs.victoriametrics.com/keyconcepts/#push-model) by applications on `/metrics` page
|
||||
Time series [exposed](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#push-model) by applications on `/metrics` page
|
||||
during the last 1h are considered as Active Time Series. For example, [Node exporter](https://prometheus.io/docs/guides/node-exporter/)
|
||||
exposes **1000** time series per instance. Therefore, if you collect metrics from 50 node exporters, the approximate
|
||||
amount of Active Time Series is **1000 * 50 = 50,000** series.
|
||||
@@ -31,11 +31,11 @@ sum(max_over_time(vm_cache_entries{type="storage/hour_metric_ids"}[24h]))
|
||||
|
||||
_Note: if you have more than one Prometheus, you need to run this query across all of them and summarise the results._
|
||||
|
||||
For [pushed](https://docs.victoriametrics.com/keyconcepts/#push-model) metrics the math is the same. Find the average
|
||||
amount of unique [time series](https://docs.victoriametrics.com/keyconcepts/#time-series) pushed by one application
|
||||
For [pushed](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#push-model) metrics the math is the same. Find the average
|
||||
amount of unique [time series](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#time-series) pushed by one application
|
||||
and multiply it by the number of applications.
|
||||
|
||||
Applying [replication Factor](https://docs.victoriametrics.com/cluster-victoriametrics/#replication-and-data-safety)
|
||||
Applying [replication Factor](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#replication-and-data-safety)
|
||||
multiplies the number of Active Time Series since each series will be stored ReplicationFactor times.
|
||||
|
||||
### Churn Rate
|
||||
@@ -55,13 +55,13 @@ To see the Churn Rate in VictoriaMetrics over last 24h use the following query:
|
||||
sum(increase(vm_new_timeseries_created_total[24h]))
|
||||
```
|
||||
|
||||
The metrics with the highest number of time series can be tracked via VictoriaMetrics [Cardinality Explorer](https://docs.victoriametrics.com/#cardinality-explorer).
|
||||
The metrics with the highest number of time series can be tracked via VictoriaMetrics [Cardinality Explorer](https://docs.victoriametrics.com/victoriametrics#cardinality-explorer).
|
||||
|
||||
### Ingestion Rate
|
||||
|
||||
Ingestion rate is how many time series are pulled (scraped) or pushed per second into the database. For example,
|
||||
if you scrape a service that exposes **1000** time series with an interval of **15s**, the Ingestion Rate would be
|
||||
**1000/15 = 66** [samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) per second. The more services are
|
||||
**1000/15 = 66** [samples](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#raw-samples) per second. The more services are
|
||||
scraped or the lower scrape interval is, the higher would be the Ingestion Rate.
|
||||
|
||||
For Prometheus, get the Ingestion Rate by running the following query:
|
||||
@@ -90,7 +90,7 @@ There are two types of queries **light** and **heavy**:
|
||||
|
||||
The larger the time range and the more series are needed to be scanned - the more heavy and expensive query is.
|
||||
|
||||
To scale VictoriaMetrics cluster for high RPS, consider deploying more [vmselect](https://docs.victoriametrics.com/cluster-victoriametrics/#architecture-overview)
|
||||
To scale VictoriaMetrics cluster for high RPS, consider deploying more [vmselect](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#architecture-overview)
|
||||
replicas (scale horizontally).
|
||||
To improve latency of **heavy** queries, consider giving more compute resources to vmselects (scale vertically).
|
||||
|
||||
@@ -100,8 +100,8 @@ It is hard to predict the amount of compute resources (CPU, Mem) or cluster size
|
||||
Active Time Series. The much better approach is to run tests for your type of load (ingestion and reads) and extrapolate
|
||||
from there.
|
||||
|
||||
For example, if you already run [Prometheus](https://docs.victoriametrics.com/#prometheus-setup)
|
||||
or [Telegraf](https://docs.victoriametrics.com/#how-to-send-data-from-influxdb-compatible-agents-such-as-telegraf)
|
||||
For example, if you already run [Prometheus](https://docs.victoriametrics.com/victoriametrics#prometheus-setup)
|
||||
or [Telegraf](https://docs.victoriametrics.com/victoriametrics#how-to-send-data-from-influxdb-compatible-agents-such-as-telegraf)
|
||||
for metrics collection then just configure them (or a part of them) to replicate data to VictoriaMetrics. In this way,
|
||||
you'd have the most precise simulation of your production environment.
|
||||
|
||||
@@ -140,13 +140,13 @@ A Kubernetes environment that produces 5k time series per second with 1-year of
|
||||
VictoriaMetrics requires additional disk space for the index. The lower Churn Rate, the lower is disk space usage for the index.
|
||||
Usually, index takes about **20%** of the disk space for storing data. High cardinality setups may use **>50%** of storage size for index.
|
||||
|
||||
You can significantly reduce the amount of disk usage by using [Downsampling](https://docs.victoriametrics.com/#downsampling)
|
||||
and [Retention Filters](https://docs.victoriametrics.com/#retention-filters). These settings are available in VictoriaMetrics Cloud and Enterprise.
|
||||
You can significantly reduce the amount of disk usage by using [Downsampling](https://docs.victoriametrics.com/victoriametrics#downsampling)
|
||||
and [Retention Filters](https://docs.victoriametrics.com/victoriametrics#retention-filters). These settings are available in VictoriaMetrics Cloud and Enterprise.
|
||||
See a blog post about [reducing expenses on monitoring](https://victoriametrics.com/blog/reducing-costs-p2/) for more techniques.
|
||||
|
||||
### Cluster size
|
||||
|
||||
It is [recommended](https://docs.victoriametrics.com/cluster-victoriametrics/#cluster-setup) to run many small vmstorage
|
||||
It is [recommended](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#cluster-setup) to run many small vmstorage
|
||||
nodes over a few big vmstorage nodes. This reduces the workload increase on the remaining vmstorage nodes when some of
|
||||
vmstorage nodes become temporarily unavailable. Prefer giving at least 2 vCPU per each vmstorage node.
|
||||
|
||||
@@ -174,6 +174,6 @@ You can collect metrics from
|
||||
|
||||
### On-Premise
|
||||
|
||||
Please follow these capacity planning documents ([Single-Node](https://docs.victoriametrics.com/single-server-victoriametrics/#capacity-planning),
|
||||
[Cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#capacity-planning)). It contains the number of CPUs
|
||||
Please follow these capacity planning documents ([Single-Node](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#capacity-planning),
|
||||
[Cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#capacity-planning)). It contains the number of CPUs
|
||||
and Memory required to handle the Ingestion Rate, Active Time Series, Churn Rate, QPS and Retention Period.
|
||||
|
||||
@@ -8,6 +8,8 @@ menu:
|
||||
weight: 7
|
||||
title: CHANGELOG
|
||||
aliases:
|
||||
- /VictoriaLogs/changelog
|
||||
- /VictoriaLogs/CHANGELOG.html
|
||||
- /victorialogs/CHANGELOG.html
|
||||
---
|
||||
|
||||
@@ -87,7 +89,7 @@ Released at 2025-02-27
|
||||
|
||||
* FEATURE: [`pack_json` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#pack_json-pipe): allow packing fields, which start with the given prefixes. For example, `pack_json fields (foo.*, bar.*)` creates a JSON containing all the fields, which start with either `foo.` or `bar.`.
|
||||
* FEATURE: [`pack_logfmt` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#pack_logfmt-pipe): allow packing fields, which start with the given prefixes. For example, `pack_logfmt fields (foo.*, bar.*)` creates [logfmt](https://brandur.org/logfmt) message containing all the fields, which start with either `foo.` or `bar.`.
|
||||
* FEATURE: expose `vl_http_request_duration_seconds` [summaries](https://docs.victoriametrics.com/keyconcepts/#summary) for [select APIs](https://docs.victoriametrics.com/victorialogs/querying/#http-api) at the [/metrics](https://docs.victoriametrics.com/victorialogs/#monitoring) page.
|
||||
* FEATURE: expose `vl_http_request_duration_seconds` [summaries](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#summary) for [select APIs](https://docs.victoriametrics.com/victorialogs/querying/#http-api) at the [/metrics](https://docs.victoriametrics.com/victorialogs/#monitoring) page.
|
||||
* FEATURE: allow passing `*` as a subquery inside [`in(*)`, `contains_any(*)` and `contains_all(*)` filters](https://docs.victoriametrics.com/victorialogs/logsql/#subquery-filter). Such filters are treated as `match all` aka `*`. This is going to be used by [Grafana plugin for VictoriaLogs](https://docs.victoriametrics.com/victorialogs/victorialogs-datasource/). See [this issue](https://github.com/VictoriaMetrics/victorialogs-datasource/issues/238#issuecomment-2685447673).
|
||||
* FEATURE: [victorialogs dashboard](https://grafana.com/grafana/dashboards/22084): add panels to display amount of ingested logs in bytes, latency of [select APIs](https://docs.victoriametrics.com/victorialogs/querying/#http-api) calls, troubleshooting panels.
|
||||
* FEATURE: provide alternative registry for all VictoriaLogs components at [Quay.io](https://quay.io/organization/victoriametrics): [VictoriaLogs](https://quay.io/repository/victoriametrics/victoria-logs?tab=tags) and [vlogscli](https://quay.io/repository/victoriametrics/vlogscli?tab=tags).
|
||||
@@ -309,11 +311,11 @@ Released at 2024-12-05
|
||||
* FEATURE: [web UI](https://docs.victoriametrics.com/victorialogs/querying/#web-ui): reduce memory usage across all tabs for improved performance and stability. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7185).
|
||||
* FEATURE: [Grafana Loki data ingestion](https://docs.victoriametrics.com/victorialogs/data-ingestion/promtail/): use Loki [stream labels](https://grafana.com/docs/loki/latest/get-started/labels/) as VictoriaLogs [stream fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields) by default. The set of stream fields can be overridden via `_stream_fields` query arg or via `VL-Stream-Fields` header as described [here](https://docs.victoriametrics.com/victorialogs/data-ingestion/#http-parameters).
|
||||
* FEATURE: [OpenTelemetry data ingestion](https://docs.victoriametrics.com/victorialogs/data-ingestion/opentelemetry/): use [resource labels](https://opentelemetry.io/docs/concepts/resources/) as VictoriaLogs [stream fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields) by default. The set of stream fields can be overridden via `_stream_fields` query arg or via `VL-Stream-Fields` header as described [here](https://docs.victoriametrics.com/victorialogs/data-ingestion/#http-parameters).
|
||||
* FEATURE: [data ingestion](https://docs.victoriametrics.com/victorialogs/data-ingestion/): expose `vl_bytes_ingested_total` [counter](https://docs.victoriametrics.com/keyconcepts/#counter) at `/metrics` page. This counter tracks an estimated number of bytes processed when parsing the ingested logs. This counter is exposed individually per every [supported data ingestion protocol](https://docs.victoriametrics.com/victorialogs/data-ingestion/) - the protocol name is exposed in the `type` label. For example, `vl_bytes_ingested_total{type="jsonline"}` tracks an estimated number of bytes processed when reading the ingested logs via [json line protocol](https://docs.victoriametrics.com/victorialogs/data-ingestion/#json-stream-api). Thanks to @tenmozes for the idea and [the initial implementation](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7682).
|
||||
* FEATURE: [data ingestion](https://docs.victoriametrics.com/victorialogs/data-ingestion/): expose `vl_too_long_lines_skipped_total` [counter](https://docs.victoriametrics.com/keyconcepts/#counter) at `/metrics` page. This counter tracks the number of the ingested lines with the length bigger than the value of `-insert.maxLineSizeBytes` command-line flag. Such lines are ignored.
|
||||
* FEATURE: [data ingestion](https://docs.victoriametrics.com/victorialogs/data-ingestion/): expose `vl_bytes_ingested_total` [counter](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#counter) at `/metrics` page. This counter tracks an estimated number of bytes processed when parsing the ingested logs. This counter is exposed individually per every [supported data ingestion protocol](https://docs.victoriametrics.com/victorialogs/data-ingestion/) - the protocol name is exposed in the `type` label. For example, `vl_bytes_ingested_total{type="jsonline"}` tracks an estimated number of bytes processed when reading the ingested logs via [json line protocol](https://docs.victoriametrics.com/victorialogs/data-ingestion/#json-stream-api). Thanks to @tenmozes for the idea and [the initial implementation](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7682).
|
||||
* FEATURE: [data ingestion](https://docs.victoriametrics.com/victorialogs/data-ingestion/): expose `vl_too_long_lines_skipped_total` [counter](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#counter) at `/metrics` page. This counter tracks the number of the ingested lines with the length bigger than the value of `-insert.maxLineSizeBytes` command-line flag. Such lines are ignored.
|
||||
|
||||
* BUGFIX: [`/select/logsql/stats_query_range` API](https://docs.victoriametrics.com/victorialogs/querying/#querying-log-range-stats): properly handle [`limit` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#limit-pipe) after [`sort` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#sort-pipe). Previously the `limit` was applied globally across all the calculated stats, while it must be applied individually per each `step` on the `start ... end` time range. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7699).
|
||||
* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): fix for `showLegend` and `alias` flags in predefined panels. [See this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7565)
|
||||
* BUGFIX: [vmui](https://docs.victoriametrics.com/victoriametrics#vmui): fix for `showLegend` and `alias` flags in predefined panels. [See this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7565)
|
||||
* BUGFIX: fix `too big number of columns detected in the block` panic when the ingested logs contain more than 2000 fields with different names per every [log stream](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7568) for details.
|
||||
* BUGFIX: properly parse lines after too long [JSON lines](https://docs.victoriametrics.com/victorialogs/data-ingestion/#json-stream-api) and [Elasticsearch lines](https://docs.victoriametrics.com/victorialogs/data-ingestion/#elasticsearch-bulk-api) with the length exceeding `-insert.maxLineSizeBytes`. Previously all the lines after the too long line in the stream were ignored.
|
||||
|
||||
@@ -414,7 +416,7 @@ Released at 2024-10-16
|
||||
* BUGFIX: avoid possible panic when logs for a new day are ingested during execution of concurrent queries.
|
||||
* BUGFIX: avoid panic at `lib/logstorage.(*blockResultColumn).forEachDictValue()` when the query contains [stats with additional filters](https://docs.victoriametrics.com/victorialogs/logsql/#stats-with-additional-filters). The panic has been introduced in [v0.33.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.33.0-victorialogs) in [this commit](https://github.com/VictoriaMetrics/VictoriaMetrics/commit/a350be48b68330ee1a487e1fb09b002d3be45163).
|
||||
* BUGFIX: add more checks for [stats query APIs](https://docs.victoriametrics.com/victorialogs/querying/#querying-log-stats) to avoid invalid results.
|
||||
* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): fix error messages rendering from overflowing the screen with long messages. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7207).
|
||||
* BUGFIX: [vmui](https://docs.victoriametrics.com/victoriametrics#vmui): fix error messages rendering from overflowing the screen with long messages. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7207).
|
||||
|
||||
## [v0.35.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.35.0-victorialogs)
|
||||
|
||||
@@ -494,7 +496,7 @@ Released at 2024-09-27
|
||||
|
||||
Released at 2024-09-08
|
||||
|
||||
* FEATURE: add [`/select/logsql/stats_query` HTTP API](https://docs.victoriametrics.com/victorialogs/querying/#querying-log-stats), which is going to be used by [vmalert](https://docs.victoriametrics.com/vmalert/) for executing alerting and recording rules against VictoriaLogs. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6942) for details.
|
||||
* FEATURE: add [`/select/logsql/stats_query` HTTP API](https://docs.victoriametrics.com/victorialogs/querying/#querying-log-stats), which is going to be used by [vmalert](https://docs.victoriametrics.com/victoriametrics/vmalert/) for executing alerting and recording rules against VictoriaLogs. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6942) for details.
|
||||
* FEATURE: add [`/select/logsql/stats_query_range` HTTP API](https://docs.victoriametrics.com/victorialogs/querying/#querying-log-range-stats), which is going to be used by [VictoriaLogs plugin for Grafana](https://docs.victoriametrics.com/victorialogs/victorialogs-datasource/) for building time series panels. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6943) for details.
|
||||
* FEATURE: optimize [multi-exact queries](https://docs.victoriametrics.com/victorialogs/logsql/#multi-exact-filter) with many phrases to search. For example, `ip:in(path:="/foo/bar" | keep ip)` when there are many unique values for `ip` field among log entries with `/foo/bar` path.
|
||||
* FEATURE: [web UI](https://docs.victoriametrics.com/victorialogs/querying/#web-ui): add support for displaying the top 5 log streams in the hits graph. The remaining log streams are grouped into an "other" label. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6545).
|
||||
@@ -510,7 +512,7 @@ Released at 2024-09-08
|
||||
* FEATURE: [vlinsert](https://docs.victoriametrics.com/victorialogs/): added OpenTelemetry logs ingestion support. See this [PR](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6218) for details.
|
||||
|
||||
* BUGFIX: properly handle Logstash requests for Elasticsearch configuration when using `outputs.elasticsearch` in Logstash pipelines. Previously, the requests could be rejected with `400 Bad Request` response. Updates [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4750).
|
||||
* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): fix `not found index.js` error when loading vmui in VictoriaLogs. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6764). Thanks to @yincongcyincong for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6770).
|
||||
* BUGFIX: [vmui](https://docs.victoriametrics.com/victoriametrics#vmui): fix `not found index.js` error when loading vmui in VictoriaLogs. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6764). Thanks to @yincongcyincong for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6770).
|
||||
* BUGFIX: properly execute queries with `OR` [filters](https://docs.victoriametrics.com/victorialogs/logsql/#logical-filter) for distinct [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). For example, `field1:foo OR field2:bar`. Previously logs matching these filters may be skipped during querying. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6554) for details. Thanks to @yincongcyincong for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6556).
|
||||
|
||||
## [v0.28.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.28.0-victorialogs)
|
||||
|
||||
@@ -8,6 +8,10 @@ menu:
|
||||
weight: 6
|
||||
title: FAQ
|
||||
aliases:
|
||||
- /VictoriaLogs/FAQ.html
|
||||
- /VictoriaLogs/faq.html
|
||||
- /VictoriaLogs/FAQ
|
||||
- /VictoriaLogs/faq
|
||||
- /victorialogs/FAQ.html
|
||||
- /victorialogs/faq.html
|
||||
---
|
||||
|
||||
@@ -6,6 +6,8 @@ menu:
|
||||
parent: "victorialogs"
|
||||
weight: 5
|
||||
aliases:
|
||||
- /VictoriaLogs/logsql
|
||||
- /VictoriaLogs/LogsQL.html
|
||||
- /victorialogs/LogsQL.html
|
||||
---
|
||||
LogsQL is a simple yet powerful query language for [VictoriaLogs](https://docs.victoriametrics.com/victorialogs/).
|
||||
@@ -314,10 +316,10 @@ The following formats are supported for `_time` filter:
|
||||
- `_time:<max_time` - matches logs with timestamps smaller than the `max_time`.
|
||||
- `_time:<=max_time` - matches logs with timestamps smaller or equal to the `max_time`.
|
||||
- `_time:[min_time, max_time]` - matches logs on the time range `[min_time, max_time]`, including both `min_time` and `max_time`.
|
||||
The `min_time` and `max_time` can contain any format specified [here](https://docs.victoriametrics.com/#timestamp-formats).
|
||||
The `min_time` and `max_time` can contain any format specified [here](https://docs.victoriametrics.com/victoriametrics#timestamp-formats).
|
||||
For example, `_time:[2023-04-01Z, 2023-04-30Z]` matches logs for the whole April, 2023 by UTC, e.g. it is equivalent to `_time:2023-04Z`.
|
||||
- `_time:[min_time, max_time)` - matches logs on the time range `[min_time, max_time)`, not including `max_time`.
|
||||
The `min_time` and `max_time` can contain any format specified [here](https://docs.victoriametrics.com/#timestamp-formats).
|
||||
The `min_time` and `max_time` can contain any format specified [here](https://docs.victoriametrics.com/victoriametrics#timestamp-formats).
|
||||
For example, `_time:[2023-02-01Z, 2023-03-01Z)` matches logs for the whole February, 2023 by UTC, e.g. it is equivalent to `_time:2023-02Z`.
|
||||
|
||||
It is possible to specify time zone offset for all the absolute time formats by appending `+hh:mm` or `-hh:mm` suffix.
|
||||
@@ -443,7 +445,7 @@ See also:
|
||||
|
||||
VictoriaLogs provides an optimized way to select logs, which belong to particular [log streams](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields).
|
||||
This can be done via `{...}` filter, which may contain arbitrary
|
||||
[Prometheus-compatible label selector](https://docs.victoriametrics.com/keyconcepts/#filtering)
|
||||
[Prometheus-compatible label selector](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#filtering)
|
||||
over fields associated with [log streams](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields).
|
||||
For example, the following query selects [log entries](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model)
|
||||
with `app` field equal to `nginx`:
|
||||
|
||||
@@ -8,6 +8,9 @@ menu:
|
||||
weight: 1
|
||||
title: Quick Start
|
||||
aliases:
|
||||
- /VictoriaLogs/quick-start.html
|
||||
- /VictoriaLogs/quick-start/
|
||||
- /VictoriaLogs/QuickStart.html
|
||||
- /victorialogs/QuickStart.html
|
||||
- /victorialogs/quick-start.html
|
||||
- /victorialogs/quick-start/
|
||||
|
||||
@@ -55,14 +55,14 @@ If you want playing with [LogsQL](https://docs.victoriametrics.com/victorialogs/
|
||||
|
||||
VictoriaLogs exposes internal metrics in Prometheus exposition format at `http://localhost:9428/metrics` page.
|
||||
It is recommended to set up monitoring of these metrics via VictoriaMetrics
|
||||
(see [these docs](https://docs.victoriametrics.com/#how-to-scrape-prometheus-exporters-such-as-node-exporter)),
|
||||
vmagent (see [these docs](https://docs.victoriametrics.com/vmagent/#how-to-collect-metrics-in-prometheus-format)) or via Prometheus.
|
||||
(see [these docs](https://docs.victoriametrics.com/victoriametrics#how-to-scrape-prometheus-exporters-such-as-node-exporter)),
|
||||
vmagent (see [these docs](https://docs.victoriametrics.com/victoriametrics/vmagent/#how-to-collect-metrics-in-prometheus-format)) or via Prometheus.
|
||||
|
||||
We recommend installing Grafana dashboard for [VictoriaLogs single-node](https://grafana.com/grafana/dashboards/22084)
|
||||
or [cluster](https://grafana.com/grafana/dashboards/23274) (if link is unavailable use [source link](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/dashboards/victorialogs-cluster.json)).
|
||||
|
||||
We recommend setting up [alerts](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/rules/alerts-vlogs.yml)
|
||||
via [vmalert](https://docs.victoriametrics.com/vmalert/) or via Prometheus.
|
||||
via [vmalert](https://docs.victoriametrics.com/victoriametrics/vmalert/) or via Prometheus.
|
||||
|
||||
VictoriaLogs emits its own logs to stdout. It is recommended to investigate these logs during troubleshooting.
|
||||
|
||||
@@ -102,7 +102,7 @@ It automatically drops partition directories outside the configured retention.
|
||||
VictoriaLogs automatically drops logs at [data ingestion](https://docs.victoriametrics.com/victorialogs/data-ingestion/) stage
|
||||
if they have timestamps outside the configured retention. A sample of dropped logs is logged with `WARN` message in order to simplify troubleshooting.
|
||||
The `vl_rows_dropped_total` [metric](#monitoring) is incremented each time an ingested log entry is dropped because of timestamp outside the retention.
|
||||
It is recommended to set up the following alerting rule at [vmalert](https://docs.victoriametrics.com/vmalert/) in order to be notified
|
||||
It is recommended to set up the following alerting rule at [vmalert](https://docs.victoriametrics.com/victoriametrics/vmalert/) in order to be notified
|
||||
when logs with wrong timestamps are ingested into VictoriaLogs:
|
||||
|
||||
```metricsql
|
||||
@@ -192,7 +192,7 @@ This schema outlines how to configure a High Availability (HA) setup using Victo
|
||||
|
||||
- **VictoriaLogs Single-Node Instances**: Use two or more instances to achieve HA.
|
||||
|
||||
- **[vmauth](https://docs.victoriametrics.com/vmauth/#load-balancing) or Load Balancer**: Used for reading data from one of the replicas to ensure balanced and redundant access.
|
||||
- **[vmauth](https://docs.victoriametrics.com/victoriametrics/vmauth/#load-balancing) or Load Balancer**: Used for reading data from one of the replicas to ensure balanced and redundant access.
|
||||
|
||||

|
||||
|
||||
@@ -261,11 +261,11 @@ If `AccountID` and/or `ProjectID` request headers aren't set, then the default `
|
||||
|
||||
VictoriaLogs has very low overhead for per-tenant management, so it is OK to have thousands of tenants in a single VictoriaLogs instance.
|
||||
|
||||
VictoriaLogs doesn't perform per-tenant authorization. Use [vmauth](https://docs.victoriametrics.com/vmauth/) or similar tools for per-tenant authorization.
|
||||
VictoriaLogs doesn't perform per-tenant authorization. Use [vmauth](https://docs.victoriametrics.com/victoriametrics/vmauth/) or similar tools for per-tenant authorization.
|
||||
|
||||
### Multitenancy access control
|
||||
|
||||
Enforce access control for tenants by using [vmauth](https://docs.victoriametrics.com/vmauth/). Access control can be configured for each tenant by setting up the following rules:
|
||||
Enforce access control for tenants by using [vmauth](https://docs.victoriametrics.com/victoriametrics/vmauth/). Access control can be configured for each tenant by setting up the following rules:
|
||||
|
||||
```yaml
|
||||
users:
|
||||
@@ -298,7 +298,7 @@ This configuration allows `foo` to use the `/select/.*` and `/insert/.*` endpoin
|
||||
|
||||
It is expected that VictoriaLogs runs in a protected environment, which is unreachable from the Internet without proper authorization.
|
||||
It is recommended providing access to VictoriaLogs [data ingestion APIs](https://docs.victoriametrics.com/victorialogs/data-ingestion/)
|
||||
and [querying APIs](https://docs.victoriametrics.com/victorialogs/querying/#http-api) via [vmauth](https://docs.victoriametrics.com/vmauth/)
|
||||
and [querying APIs](https://docs.victoriametrics.com/victorialogs/querying/#http-api) via [vmauth](https://docs.victoriametrics.com/victoriametrics/vmauth/)
|
||||
or similar authorization proxies.
|
||||
|
||||
## Benchmarks
|
||||
@@ -315,7 +315,7 @@ It is recommended [setting up VictoriaLogs](https://docs.victoriametrics.com/vic
|
||||
log management systems and comparing resource usage + query performance between VictoriaLogs and your system such as Elasticsearch or Grafana Loki.
|
||||
|
||||
Please share benchmark results and ideas on how to improve benchmarks / VictoriaLogs
|
||||
via [VictoriaMetrics community channels](https://docs.victoriametrics.com/#community-and-contributions).
|
||||
via [VictoriaMetrics community channels](https://docs.victoriametrics.com/victoriametrics#community-and-contributions).
|
||||
|
||||
## Profiling
|
||||
|
||||
@@ -367,7 +367,7 @@ Pass `-help` to VictoriaLogs in order to see the list of supported command-line
|
||||
-enableTCP6
|
||||
Whether to enable IPv6 for listening and dialing. By default, only IPv4 TCP and UDP are used
|
||||
-envflag.enable
|
||||
Whether to enable reading flags from environment variables in addition to the command line. Command line flag values have priority over values from environment vars. Flags are read only from the command line if this flag isn't set. See https://docs.victoriametrics.com/#environment-variables for more details
|
||||
Whether to enable reading flags from environment variables in addition to the command line. Command line flag values have priority over values from environment vars. Flags are read only from the command line if this flag isn't set. See https://docs.victoriametrics.com/victoriametrics#environment-variables for more details
|
||||
-envflag.prefix string
|
||||
Prefix for environment variables if -envflag.enable is set
|
||||
-filestream.disableFadvise
|
||||
@@ -497,11 +497,11 @@ Pass `-help` to VictoriaLogs in order to see the list of supported command-line
|
||||
Auth key for /metrics endpoint. It must be passed via authKey query arg. It overrides -httpAuth.*
|
||||
Flag value can be read from the given file when using -metricsAuthKey=file:///abs/path/to/file or -metricsAuthKey=file://./relative/path/to/file . Flag value can be read from the given http/https url when using -metricsAuthKey=http://host/path or -metricsAuthKey=https://host/path
|
||||
-mtls array
|
||||
Whether to require valid client certificate for https requests to the corresponding -httpListenAddr . This flag works only if -tls flag is set. See also -mtlsCAFile . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/
|
||||
Whether to require valid client certificate for https requests to the corresponding -httpListenAddr . This flag works only if -tls flag is set. See also -mtlsCAFile . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/victoriametrics/enterprise/
|
||||
Supports array of values separated by comma or specified via multiple flags.
|
||||
Empty values are set to false.
|
||||
-mtlsCAFile array
|
||||
Optional path to TLS Root CA for verifying client certificates at the corresponding -httpListenAddr when -mtls is enabled. By default the host system TLS Root CA is used for client certificate verification. This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/
|
||||
Optional path to TLS Root CA for verifying client certificates at the corresponding -httpListenAddr when -mtls is enabled. By default the host system TLS Root CA is used for client certificate verification. This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/victoriametrics/enterprise/
|
||||
Supports an array of values separated by comma or specified via multiple flags.
|
||||
Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces.
|
||||
-opentelemetry.maxRequestSize size
|
||||
@@ -523,7 +523,7 @@ Pass `-help` to VictoriaLogs in order to see the list of supported command-line
|
||||
-pushmetrics.interval duration
|
||||
Interval for pushing metrics to every -pushmetrics.url (default 10s)
|
||||
-pushmetrics.url array
|
||||
Optional URL to push metrics exposed at /metrics page. See https://docs.victoriametrics.com/#push-metrics . By default, metrics exposed at /metrics page aren't pushed to any remote storage
|
||||
Optional URL to push metrics exposed at /metrics page. See https://docs.victoriametrics.com/victoriametrics#push-metrics . By default, metrics exposed at /metrics page aren't pushed to any remote storage
|
||||
Supports an array of values separated by comma or specified via multiple flags.
|
||||
Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces.
|
||||
-retention.maxDiskSpaceUsageBytes size
|
||||
@@ -674,11 +674,11 @@ Pass `-help` to VictoriaLogs in order to see the list of supported command-line
|
||||
Supports array of values separated by comma or specified via multiple flags.
|
||||
Empty values are set to false.
|
||||
-tlsAutocertCacheDir string
|
||||
Directory to store TLS certificates issued via Let's Encrypt. Certificates are lost on restarts if this flag isn't set. This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/
|
||||
Directory to store TLS certificates issued via Let's Encrypt. Certificates are lost on restarts if this flag isn't set. This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/victoriametrics/enterprise/
|
||||
-tlsAutocertEmail string
|
||||
Contact email for the issued Let's Encrypt TLS certificates. See also -tlsAutocertHosts and -tlsAutocertCacheDir .This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/
|
||||
Contact email for the issued Let's Encrypt TLS certificates. See also -tlsAutocertHosts and -tlsAutocertCacheDir .This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/victoriametrics/enterprise/
|
||||
-tlsAutocertHosts array
|
||||
Optional hostnames for automatic issuing of Let's Encrypt TLS certificates. These hostnames must be reachable at -httpListenAddr . The -httpListenAddr must listen tcp port 443 . The -tlsAutocertHosts overrides -tlsCertFile and -tlsKeyFile . See also -tlsAutocertEmail and -tlsAutocertCacheDir . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/
|
||||
Optional hostnames for automatic issuing of Let's Encrypt TLS certificates. These hostnames must be reachable at -httpListenAddr . The -httpListenAddr must listen tcp port 443 . The -tlsAutocertHosts overrides -tlsCertFile and -tlsKeyFile . See also -tlsAutocertEmail and -tlsAutocertCacheDir . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/victoriametrics/enterprise/
|
||||
Supports an array of values separated by comma or specified via multiple flags.
|
||||
Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces.
|
||||
-tlsCertFile array
|
||||
|
||||
@@ -8,12 +8,14 @@ menu:
|
||||
weight: 8
|
||||
title: Roadmap
|
||||
aliases:
|
||||
- /VictoriaLogs/Roadmap.html
|
||||
- /VictoriaLogs/roadmap/
|
||||
- /victorialogs/Roadmap.html
|
||||
---
|
||||
|
||||
The following functionality is planned in the future versions of VictoriaLogs:
|
||||
|
||||
- [ ] Ability to make instant snapshots and backups in the way [similar to VictoriaMetrics](https://docs.victoriametrics.com/#how-to-work-with-snapshots).
|
||||
- [ ] Ability to make instant snapshots and backups in the way [similar to VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics#how-to-work-with-snapshots).
|
||||
- [ ] Ability to store data to object storage (such as S3, GCS, Minio).
|
||||
- [ ] Data migration tool from Grafana Loki to VictoriaLogs (similar to [vmctl](https://docs.victoriametrics.com/vmctl/)).
|
||||
- [ ] Retention filters based on tenant and stream fields similar to [Victoriametrics](https://docs.victoriametrics.com/#retention-filters) (Enterprise only)
|
||||
- [ ] Data migration tool from Grafana Loki to VictoriaLogs (similar to [vmctl](https://docs.victoriametrics.com/victoriametrics/vmctl/)).
|
||||
- [ ] Retention filters based on tenant and stream fields similar to [Victoriametrics](https://docs.victoriametrics.com/victoriametrics#retention-filters) (Enterprise only)
|
||||
|
||||
@@ -7,6 +7,8 @@ menu:
|
||||
identifier: victorialogs
|
||||
pageRef: /victorialogs/
|
||||
aliases:
|
||||
- /VictoriaLogs/
|
||||
- /VictoriaLogs/index.html
|
||||
- /victorialogs/
|
||||
- /victorialogs/index.html
|
||||
---
|
||||
|
||||
@@ -59,7 +59,7 @@ and vice versa.
|
||||
- `vlselect` sends requests to HTTP endpoints at `vlstorage` starting with `/internal/select/`.
|
||||
|
||||
This allows using various http proxies for authorization, routing and encryption of requests between these components.
|
||||
It is recommended to use [vmauth](https://docs.victoriametrics.com/vmauth/).
|
||||
It is recommended to use [vmauth](https://docs.victoriametrics.com/victoriametrics/vmauth/).
|
||||
|
||||
See also [multi-level cluster setup](#multi-level-cluster-setup).
|
||||
|
||||
@@ -91,7 +91,7 @@ See [security docs](#security) on how to protect communications between multiple
|
||||
## Security
|
||||
|
||||
All the VictoriaLogs cluster components must run in protected internal network without direct access from the Internet.
|
||||
`vlstorage` must have no access from the Internet. HTTP authorization proxies such as [vmauth](https://docs.victoriametrics.com/vmauth/)
|
||||
`vlstorage` must have no access from the Internet. HTTP authorization proxies such as [vmauth](https://docs.victoriametrics.com/victoriametrics/vmauth/)
|
||||
must be used in front of `vlinsert` and `vlselect` for authorizing access to these components from the Internet.
|
||||
|
||||
By default `vlinsert` and `vlselect` communicate with `vlstorage` via unencrypted HTTP. This is OK if all these components are located
|
||||
@@ -124,7 +124,7 @@ It is also recommended authorizing HTTPS requests to `vlstorage` via Basic Auth:
|
||||
./victoria-logs-prod -storageNode=... -storageNode.tls -storageNode.username=... -storageNode.password=...
|
||||
```
|
||||
|
||||
Another option is to use third-party HTTP proxies such as [vmauth](https://docs.victoriametrics.com/vmauth/), `nginx`, etc. for authorizing and encrypting communications
|
||||
Another option is to use third-party HTTP proxies such as [vmauth](https://docs.victoriametrics.com/victoriametrics/vmauth/), `nginx`, etc. for authorizing and encrypting communications
|
||||
between VictoriaLogs cluster components over untrusted networks.
|
||||
|
||||
|
||||
|
||||
@@ -8,13 +8,15 @@ menu:
|
||||
weight: 5
|
||||
url: /victorialogs/data-ingestion/datadog-agent/
|
||||
aliases:
|
||||
- /VictoriaLogs/data-ingestion/DataDogAgent.html
|
||||
- /VictoriaLogs/data-ingestion/datadog-agent/
|
||||
- /victorialogs/data-ingestion/DataDogAgent.html
|
||||
---
|
||||
|
||||
Datadog Agent doesn't support custom path prefix, so for this reason it's required to use [VMAuth](https://docs.victoriametrics.com/vmauth/) or any other
|
||||
Datadog Agent doesn't support custom path prefix, so for this reason it's required to use [VMAuth](https://docs.victoriametrics.com/victoriametrics/vmauth/) or any other
|
||||
reverse proxy to append `/insert/datadog` path prefix to all Datadog API logs requests.
|
||||
|
||||
In case of [VMAuth](https://docs.victoriametrics.com/vmauth/) your config should look like:
|
||||
In case of [VMAuth](https://docs.victoriametrics.com/victoriametrics/vmauth/) your config should look like:
|
||||
|
||||
```yaml
|
||||
unauthorized_user:
|
||||
|
||||
@@ -7,6 +7,9 @@ menu:
|
||||
parent: "victorialogs-data-ingestion"
|
||||
weight: 1
|
||||
aliases:
|
||||
- /VictoriaLogs/data-ingestion/filebeat/
|
||||
- /VictoriaLogs/data-ingestion/Filebeat.html
|
||||
- /VictoriaLogs/data-ingestion/filebeat.html
|
||||
- /victorialogs/data-ingestion/Filebeat.html
|
||||
- /victorialogs/data-ingestion/filebeat.html
|
||||
---
|
||||
|
||||
@@ -7,6 +7,9 @@ menu:
|
||||
parent: "victorialogs-data-ingestion"
|
||||
weight: 2
|
||||
aliases:
|
||||
- /VictoriaLogs/data-ingestion/fluentbit/
|
||||
- /VictoriaLogs/data-ingestion/fluentbit.html
|
||||
- /VictoriaLogs/data-ingestion/Fluentbit.html
|
||||
- /victorialogs/data-ingestion/fluentbit.html
|
||||
- /victorialogs/data-ingestion/Fluentbit.html
|
||||
---
|
||||
|
||||
@@ -7,6 +7,9 @@ menu:
|
||||
parent: "victorialogs-data-ingestion"
|
||||
weight: 2
|
||||
aliases:
|
||||
- /VictoriaLogs/data-ingestion/fluentd/
|
||||
- /VictoriaLogs/data-ingestion/fluentd.html
|
||||
- /VictoriaLogs/data-ingestion/Fluentd.html
|
||||
- /victorialogs/data-ingestion/fluentd.html
|
||||
- /victorialogs/data-ingestion/Fluentd.html
|
||||
---
|
||||
|
||||
@@ -7,6 +7,8 @@ menu:
|
||||
parent: "victorialogs-data-ingestion"
|
||||
weight: 10
|
||||
aliases:
|
||||
- /VictoriaLogs/data-ingestion/journald/
|
||||
- /VictoriaLogs/data-ingestion/Journald.html
|
||||
- /victorialogs/data-ingestion/Journald.html
|
||||
---
|
||||
On a client site which should already have journald please install additionally [systemd-journal-upload](https://www.freedesktop.org/software/systemd/man/latest/systemd-journal-upload.service.html) and edit `/etc/systemd/journal-upload.conf` and set `URL` to VictoriaLogs endpoint:
|
||||
|
||||
@@ -7,6 +7,9 @@ menu:
|
||||
parent: "victorialogs-data-ingestion"
|
||||
weight: 3
|
||||
aliases:
|
||||
- /VictoriaLogs/data-ingestion/logstash/
|
||||
- /VictoriaLogs/data-ingestion/logstash.html
|
||||
- /VictoriaLogs/data-ingestion/Logstash.html
|
||||
- /victorialogs/data-ingestion/logstash.html
|
||||
- /victorialogs/data-ingestion/Logstash.html
|
||||
---
|
||||
|
||||
@@ -7,6 +7,9 @@ menu:
|
||||
parent: "victorialogs-data-ingestion"
|
||||
weight: 4
|
||||
aliases:
|
||||
- /VictoriaLogs/data-ingestion/promtail/
|
||||
- /VictoriaLogs/data-ingestion/Promtail.html
|
||||
- /VictoriaLogs/data-ingestion/promtail.html
|
||||
- /victorialogs/data-ingestion/Promtail.html
|
||||
- /victorialogs/data-ingestion/promtail.html
|
||||
---
|
||||
|
||||
@@ -7,6 +7,8 @@ menu:
|
||||
parent: "victorialogs-data-ingestion"
|
||||
weight: 5
|
||||
aliases:
|
||||
- /VictoriaLogs/data-ingestion/Telegraf.html
|
||||
- /VictoriaLogs/data-ingestion/telegraf/
|
||||
- /victorialogs/data-ingestion/Telegraf.html
|
||||
---
|
||||
VictoriaLogs supports given below Telegraf outputs:
|
||||
|
||||
@@ -7,6 +7,8 @@ menu:
|
||||
parent: "victorialogs-data-ingestion"
|
||||
weight: 20
|
||||
aliases:
|
||||
- /VictoriaLogs/data-ingestion/vector.html
|
||||
- /VictoriaLogs/data-ingestion/vector/
|
||||
- /victorialogs/data-ingestion/Vector.html
|
||||
- /victorialogs/data-ingestion/vector.html
|
||||
---
|
||||
|
||||
@@ -7,6 +7,8 @@ menu:
|
||||
parent: "victorialogs-data-ingestion"
|
||||
weight: 4
|
||||
aliases:
|
||||
- /VictoriaLogs/data-ingestion/OpenTelemetry.html
|
||||
- /VictoriaLogs/data-ingestion/open-telemetry/
|
||||
- /victorialogs/data-ingestion/OpenTelemetry.html
|
||||
---
|
||||
VictoriaLogs supports both client open-telemetry [SDK](https://opentelemetry.io/docs/languages/) and [collector](https://opentelemetry.io/docs/collector/).
|
||||
|
||||
@@ -8,6 +8,8 @@ menu:
|
||||
weight: 2
|
||||
title: Key concepts
|
||||
aliases:
|
||||
- /VictoriaLogs/keyConcepts/
|
||||
- /VictoriaLogs/keyConcepts.html
|
||||
- /victorialogs/keyConcepts.html
|
||||
---
|
||||
## Data model
|
||||
@@ -189,7 +191,7 @@ Every ingested log entry is associated with a log stream. Every log stream consi
|
||||
- `_stream_id` - this is an unique identifier for the log stream. All the logs for the particular stream can be selected
|
||||
via [`_stream_id:...` filter](https://docs.victoriametrics.com/victorialogs/logsql/#_stream_id-filter).
|
||||
|
||||
- `_stream` - this field contains stream labels in the format similar to [labels in Prometheus metrics](https://docs.victoriametrics.com/keyconcepts/#labels):
|
||||
- `_stream` - this field contains stream labels in the format similar to [labels in Prometheus metrics](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#labels):
|
||||
```
|
||||
{field1="value1", ..., fieldN="valueN"}
|
||||
```
|
||||
|
||||
@@ -66,7 +66,7 @@ By default the `/select/logsql/query` returns all the log entries matching the g
|
||||
curl http://localhost:9428/select/logsql/query -d 'query=error | limit 10'
|
||||
```
|
||||
- By adding [`_time` filter](https://docs.victoriametrics.com/victorialogs/logsql/#time-filter). The time range for the query can be specified via optional
|
||||
`start` and `end` query args formatted according to [these docs](https://docs.victoriametrics.com/single-server-victoriametrics/#timestamp-formats).
|
||||
`start` and `end` query args formatted according to [these docs](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#timestamp-formats).
|
||||
- By adding more specific [filters](https://docs.victoriametrics.com/victorialogs/logsql/#filters) to the query, which select lower number of logs.
|
||||
|
||||
The `/select/logsql/query` endpoint returns [a stream of JSON lines](https://jsonlines.org/),
|
||||
@@ -204,7 +204,7 @@ VictoriaLogs provides `/select/logsql/hits?query=<query>&start=<start>&end=<end>
|
||||
of matching log entries for the given [`<query>`](https://docs.victoriametrics.com/victorialogs/logsql/) on the given `[<start> ... <end>]`
|
||||
time range grouped by `<step>` buckets. The returned results are sorted by time.
|
||||
|
||||
The `<start>` and `<end>` args can contain values in [any supported format](https://docs.victoriametrics.com/#timestamp-formats).
|
||||
The `<start>` and `<end>` args can contain values in [any supported format](https://docs.victoriametrics.com/victoriametrics#timestamp-formats).
|
||||
If `<start>` is missing, then it equals to the minimum timestamp across logs stored in VictoriaLogs.
|
||||
If `<end>` is missing, then it equals to the maximum timestamp across logs stored in VictoriaLogs.
|
||||
|
||||
@@ -326,7 +326,7 @@ VictoriaLogs provides `/select/logsql/facets?query=<query>&start=<start>&end=<en
|
||||
per each [log field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) seen in the logs returned
|
||||
by the given [`<query>`](https://docs.victoriametrics.com/victorialogs/logsql/) on the given `[<start> ... <end>]` time range.
|
||||
|
||||
The `<start>` and `<end>` args can contain values in [any supported format](https://docs.victoriametrics.com/#timestamp-formats).
|
||||
The `<start>` and `<end>` args can contain values in [any supported format](https://docs.victoriametrics.com/victoriametrics#timestamp-formats).
|
||||
If `<start>` is missing, then it equals to the minimum timestamp across logs stored in VictoriaLogs.
|
||||
If `<end>` is missing, then it equals to the maximum timestamp across logs stored in VictoriaLogs.
|
||||
|
||||
@@ -419,7 +419,7 @@ in the format compatible with [Prometheus querying API](https://prometheus.io/do
|
||||
The `<query>` must contain [`stats` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#stats-pipe). The calculated stats is converted into metrics
|
||||
with labels from `by(...)` clause of the `| stats by(...)` pipe.
|
||||
|
||||
The `<t>` arg can contain values in [any supported format](https://docs.victoriametrics.com/#timestamp-formats).
|
||||
The `<t>` arg can contain values in [any supported format](https://docs.victoriametrics.com/victoriametrics#timestamp-formats).
|
||||
If `<t>` is missing, then it equals to the current time.
|
||||
|
||||
For example, the following command returns the number of logs per each `level` [field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model)
|
||||
@@ -491,7 +491,7 @@ The stats is returned in the format compatible with [Prometheus querying API](ht
|
||||
The `<query>` must contain [`stats` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#stats-pipe). The calculated stats is converted into metrics
|
||||
with labels from `by(...)` clause of the `| stats by(...)` pipe.
|
||||
|
||||
The `<start>` and `<end>` args can contain values in [any supported format](https://docs.victoriametrics.com/#timestamp-formats).
|
||||
The `<start>` and `<end>` args can contain values in [any supported format](https://docs.victoriametrics.com/victoriametrics#timestamp-formats).
|
||||
If `<start>` is missing, then it equals to the minimum timestamp across logs stored in VictoriaLogs.
|
||||
If `<end>` is missing, then it equals to the maximum timestamp across logs stored in VictoriaLogs.
|
||||
|
||||
@@ -583,7 +583,7 @@ for the [log streams](https://docs.victoriametrics.com/victorialogs/keyconcepts/
|
||||
of the given [`<query>`](https://docs.victoriametrics.com/victorialogs/logsql/) on the given `[<start> ... <end>]` time range.
|
||||
The response also contains the number of log results per every `_stream_id`.
|
||||
|
||||
The `<start>` and `<end>` args can contain values in [any supported format](https://docs.victoriametrics.com/#timestamp-formats).
|
||||
The `<start>` and `<end>` args can contain values in [any supported format](https://docs.victoriametrics.com/victoriametrics#timestamp-formats).
|
||||
If `<start>` is missing, then it equals to the minimum timestamp across logs stored in VictoriaLogs.
|
||||
If `<end>` is missing, then it equals to the maximum timestamp across logs stored in VictoriaLogs.
|
||||
|
||||
@@ -641,7 +641,7 @@ VictoriaLogs provides `/select/logsql/streams?query=<query>&start=<start>&end=<e
|
||||
from results of the given [`<query>`](https://docs.victoriametrics.com/victorialogs/logsql/) on the given `[<start> ... <end>]` time range.
|
||||
The response also contains the number of log results per every `_stream`.
|
||||
|
||||
The `<start>` and `<end>` args can contain values in [any supported format](https://docs.victoriametrics.com/#timestamp-formats).
|
||||
The `<start>` and `<end>` args can contain values in [any supported format](https://docs.victoriametrics.com/victoriametrics#timestamp-formats).
|
||||
If `<start>` is missing, then it equals to the minimum timestamp across logs stored in VictoriaLogs.
|
||||
If `<end>` is missing, then it equals to the maximum timestamp across logs stored in VictoriaLogs.
|
||||
|
||||
@@ -700,7 +700,7 @@ VictoriaLogs provides `/select/logsql/stream_field_names?query=<query>&start=<st
|
||||
of the given [`<query>`](https://docs.victoriametrics.com/victorialogs/logsql/) on the given `[<start> ... <end>]` time range.
|
||||
The response also contains the number of log results per every field name.
|
||||
|
||||
The `<start>` and `<end>` args can contain values in [any supported format](https://docs.victoriametrics.com/#timestamp-formats).
|
||||
The `<start>` and `<end>` args can contain values in [any supported format](https://docs.victoriametrics.com/victoriametrics#timestamp-formats).
|
||||
If `<start>` is missing, then it equals to the minimum timestamp across logs stored in VictoriaLogs.
|
||||
If `<end>` is missing, then it equals to the maximum timestamp across logs stored in VictoriaLogs.
|
||||
|
||||
@@ -755,7 +755,7 @@ which returns [log stream](https://docs.victoriametrics.com/victorialogs/keyconc
|
||||
from results of the given [`<query>`](https://docs.victoriametrics.com/victorialogs/logsql/) on the given `[<start> ... <end>]` time range.
|
||||
The response also contains the number of log results per every field value.
|
||||
|
||||
The `<start>` and `<end>` args can contain values in [any supported format](https://docs.victoriametrics.com/#timestamp-formats).
|
||||
The `<start>` and `<end>` args can contain values in [any supported format](https://docs.victoriametrics.com/victoriametrics#timestamp-formats).
|
||||
If `<start>` is missing, then it equals to the minimum timestamp across logs stored in VictoriaLogs.
|
||||
If `<end>` is missing, then it equals to the maximum timestamp across logs stored in VictoriaLogs.
|
||||
|
||||
@@ -808,7 +808,7 @@ VictoriaLogs provides `/select/logsql/field_names?query=<query>&start=<start>&en
|
||||
from results of the given [`<query>`](https://docs.victoriametrics.com/victorialogs/logsql/) on the given `[<start> ... <end>]` time range.
|
||||
The response also contains the number of log results per every field name.
|
||||
|
||||
The `<start>` and `<end>` args can contain values in [any supported format](https://docs.victoriametrics.com/#timestamp-formats).
|
||||
The `<start>` and `<end>` args can contain values in [any supported format](https://docs.victoriametrics.com/victoriametrics#timestamp-formats).
|
||||
If `<start>` is missing, then it equals to the minimum timestamp across logs stored in VictoriaLogs.
|
||||
If `<end>` is missing, then it equals to the maximum timestamp across logs stored in VictoriaLogs.
|
||||
|
||||
@@ -863,7 +863,7 @@ unique values for the given `<fieldName>` [field](https://docs.victoriametrics.c
|
||||
from results of the given [`<query>`](https://docs.victoriametrics.com/victorialogs/logsql/) on the given `[<start> ... <end>]` time range.
|
||||
The response also contains the number of log results per every field value.
|
||||
|
||||
The `<start>` and `<end>` args can contain values in [any supported format](https://docs.victoriametrics.com/#timestamp-formats).
|
||||
The `<start>` and `<end>` args can contain values in [any supported format](https://docs.victoriametrics.com/victoriametrics#timestamp-formats).
|
||||
If `<start>` is missing, then it equals to the minimum timestamp across logs stored in VictoriaLogs.
|
||||
If `<end>` is missing, then it equals to the maximum timestamp across logs stored in VictoriaLogs.
|
||||
|
||||
|
||||
@@ -7,6 +7,8 @@ menu:
|
||||
parent: "victorialogs"
|
||||
weight: 4
|
||||
aliases:
|
||||
- /VictoriaLogs/querying/
|
||||
- /VictoriaLogs/querying/index.html
|
||||
- /victorialogs/querying/
|
||||
- /victorialogs/querying/index.html
|
||||
---
|
||||
|
||||
@@ -1,7 +1,6 @@
|
||||
---
|
||||
weight: 9
|
||||
title: Grafana datasource
|
||||
editLink: https://github.com/VictoriaMetrics/victorialogs-datasource/blob/main/README.md
|
||||
menu:
|
||||
docs:
|
||||
identifier: victorialogs-grafana-datasource
|
||||
|
||||
@@ -7,15 +7,17 @@ menu:
|
||||
weight: 10
|
||||
identifier: "victorialogs-vmalert"
|
||||
aliases:
|
||||
- /VictoriaLogs/vmalert/
|
||||
- /VictoriaLogs/vmalert.html
|
||||
- /victorialogs/vmalert.html
|
||||
---
|
||||
|
||||
[vmalert](https://docs.victoriametrics.com/vmalert/){{% available_from "v1.106.0" %}} integrates with VictoriaLogs {{% available_from "v0.36.0" "logs" %}} via stats APIs [`/select/logsql/stats_query`](https://docs.victoriametrics.com/victorialogs/querying/#querying-log-stats)
|
||||
[vmalert](https://docs.victoriametrics.com/victoriametrics/vmalert/){{% available_from "v1.106.0" %}} integrates with VictoriaLogs {{% available_from "v0.36.0" "logs" %}} via stats APIs [`/select/logsql/stats_query`](https://docs.victoriametrics.com/victorialogs/querying/#querying-log-stats)
|
||||
and [`/select/logsql/stats_query_range`](https://docs.victoriametrics.com/victorialogs/querying/#querying-log-range-stats).
|
||||
These endpoints return the log stats in a format compatible with [Prometheus querying API](https://prometheus.io/docs/prometheus/latest/querying/api/#instant-queries).
|
||||
It allows using VictoriaLogs as the datasource in vmalert, creating alerting and recording rules via [LogsQL](https://docs.victoriametrics.com/victorialogs/logsql/).
|
||||
|
||||
_Note: This page provides only integration instructions for vmalert and VictoriaLogs. See the full textbook for vmalert [here](https://docs.victoriametrics.com/vmalert)._
|
||||
_Note: This page provides only integration instructions for vmalert and VictoriaLogs. See the full textbook for vmalert [here](https://docs.victoriametrics.com/victoriametrics/vmalert)._
|
||||
|
||||
## Quick Start
|
||||
|
||||
@@ -32,7 +34,7 @@ Run vmalert with the following settings:
|
||||
> For rules in [LogsQL](https://docs.victoriametrics.com/victorialogs/logsql/) specify `type: vlogs` on [Group level](#groups).
|
||||
> Or set `-rule.defaultRuleType=vlogs` cmd-line flag to apply `type: vlogs` to all configured groups.
|
||||
|
||||
Each `-rule` file may contain arbitrary number of [groups](https://docs.victoriametrics.com/vmalert/#groups).
|
||||
Each `-rule` file may contain arbitrary number of [groups](https://docs.victoriametrics.com/victoriametrics/vmalert/#groups).
|
||||
See examples in [Groups](#groups) section. See the full list of configuration flags and their descriptions in [configuration](#configuration) section.
|
||||
|
||||
With configuration example above, vmalert will perform the following interactions:
|
||||
@@ -42,14 +44,14 @@ With configuration example above, vmalert will perform the following interaction
|
||||
1. Triggered alerting notifications are sent to [Alertmanager](https://github.com/prometheus/alertmanager) service configured via `-notifier.url`;
|
||||
1. Results of recording rules expressions and alerts state are persisted to Prometheus-compatible remote-write endpoint
|
||||
(i.e. VictoriaMetrics) configured via `-remoteWrite.url`;
|
||||
1. On vmalert restarts, alerts state [can be restored](https://docs.victoriametrics.com/vmalert/#alerts-state-on-restarts)
|
||||
1. On vmalert restarts, alerts state [can be restored](https://docs.victoriametrics.com/victoriametrics/vmalert/#alerts-state-on-restarts)
|
||||
by querying Prometheus-compatible HTTP API endpoint (i.e. VictoriaMetrics) configured via `-remoteRead.url`.
|
||||
|
||||
## Configuration
|
||||
|
||||
### Flags
|
||||
|
||||
For a complete list of command-line flags, visit https://docs.victoriametrics.com/vmalert/#flags or execute `./vmalert --help` command.
|
||||
For a complete list of command-line flags, visit https://docs.victoriametrics.com/victoriametrics/vmalert/#flags or execute `./vmalert --help` command.
|
||||
The following are key flags related to integration with VictoriaLogs:
|
||||
|
||||
```shellhelp
|
||||
@@ -86,11 +88,11 @@ The following are key flags related to integration with VictoriaLogs:
|
||||
Since there is no intentional search delay in VictoriaLogs, `-rule.evalDelay` can be reduced to a few seconds to accommodate network and ingestion time.
|
||||
```
|
||||
|
||||
See full list of configuration options [here](https://docs.victoriametrics.com/vmalert/#configuration).
|
||||
See full list of configuration options [here](https://docs.victoriametrics.com/victoriametrics/vmalert/#configuration).
|
||||
|
||||
### Groups
|
||||
|
||||
Check the complete group attributes [here](https://docs.victoriametrics.com/vmalert/#groups).
|
||||
Check the complete group attributes [here](https://docs.victoriametrics.com/victoriametrics/vmalert/#groups).
|
||||
|
||||
#### Alerting rules
|
||||
|
||||
@@ -180,7 +182,7 @@ vmalert supports alerting and recording rules backfilling (aka replay) against V
|
||||
-replay.timeTo=2021-05-29T18:40:43Z # to finish replay by, is optional
|
||||
```
|
||||
|
||||
See more details about backfilling [here](https://docs.victoriametrics.com/vmalert/#rules-backfilling).
|
||||
See more details about backfilling [here](https://docs.victoriametrics.com/victoriametrics/vmalert/#rules-backfilling).
|
||||
|
||||
## Performance tip
|
||||
|
||||
@@ -223,8 +225,8 @@ For additional tips on writing LogsQL, refer to this [doc](https://docs.victoria
|
||||
|
||||
### How to use [multitenancy](https://docs.victoriametrics.com/victorialogs/#multitenancy) in rules?
|
||||
|
||||
vmalert doesn't support multi-tenancy for VictoriaLogs in the same way as it [supports it for VictoriaMetrics in ENT version](https://docs.victoriametrics.com/vmalert/#multitenancy).
|
||||
However, it is possible to specify the queried tenant from VictoriaLogs datasource via `headers` param in [Group config](https://docs.victoriametrics.com/vmalert/#groups).
|
||||
vmalert doesn't support multi-tenancy for VictoriaLogs in the same way as it [supports it for VictoriaMetrics in ENT version](https://docs.victoriametrics.com/victoriametrics/vmalert/#multitenancy).
|
||||
However, it is possible to specify the queried tenant from VictoriaLogs datasource via `headers` param in [Group config](https://docs.victoriametrics.com/victoriametrics/vmalert/#groups).
|
||||
For example, the following config will execute all the rules within the group against tenant with `AccountID=1` and `ProjectID=2`:
|
||||
```yaml
|
||||
groups:
|
||||
@@ -236,7 +238,7 @@ For example, the following config will execute all the rules within the group ag
|
||||
```
|
||||
By default, vmalert persists all results to the specific tenant in VictoriaMetrics that specified by `-remotewrite.url`. For example, if the `-remotewrite.url=http://vminsert:8480/insert/0/prometheus/`, all data goes to tenant `0`.
|
||||
To persist different rule results to different tenants in VictoriaMetrics, there are following approaches:
|
||||
1. To use the [multitenant endpoint of vminsert](https://docs.victoriametrics.com/cluster-victoriametrics/#multitenancy-via-labels) as the `-remoteWrite.url`, and add tenant labels under the group configuration.
|
||||
1. To use the [multitenant endpoint of vminsert](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#multitenancy-via-labels) as the `-remoteWrite.url`, and add tenant labels under the group configuration.
|
||||
|
||||
For example, run vmalert with:
|
||||
```
|
||||
@@ -266,7 +268,7 @@ To persist different rule results to different tenants in VictoriaMetrics, there
|
||||
expr: 'tags.path:/var/log/httpd OR tags.path:/var/log/nginx | stats by (tags.host) count() requests'
|
||||
```
|
||||
|
||||
2. To run [enterprise version of vmalert](https://docs.victoriametrics.com/enterprise/) with `-clusterMode` enabled, and specify tenant parameter per each group.
|
||||
2. To run [enterprise version of vmalert](https://docs.victoriametrics.com/victoriametrics/enterprise/) with `-clusterMode` enabled, and specify tenant parameter per each group.
|
||||
|
||||
For example, run vmalert with:
|
||||
```
|
||||
@@ -299,7 +301,7 @@ We recommend running separate instances of vmalert for VictoriaMetrics and Victo
|
||||
However, vmalert allows having many groups with different rule types (`vlogs`, `prometheus`, `graphite`).
|
||||
But only one `-datasource.url` cmd-line flag can be specified, so it can't be configured with more than 1 datasource.
|
||||
VictoriaMetrics and VictoriaLogs datasources have different query path prefixes, so it is possible to use
|
||||
[vmauth](https://docs.victoriametrics.com/vmauth/) to route requests of different types between datasources.
|
||||
[vmauth](https://docs.victoriametrics.com/victoriametrics/vmauth/) to route requests of different types between datasources.
|
||||
See example of vmauth config for such routing below:
|
||||
```yaml
|
||||
unauthorized_user:
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
VictoriaMetrics Cloud is a managed, easy to use monitoring solution that integrates seamlessly with
|
||||
other tools and frameworks in the Observability ecosystem such as OpenTelemetry, Grafana, Prometheus, Graphite,
|
||||
InfluxDB, OpenTSDB and DataDog - see [these docs](https://docs.victoriametrics.com/#how-to-import-time-series-data)
|
||||
InfluxDB, OpenTSDB and DataDog - see [these docs](https://docs.victoriametrics.com/victoriametrics#how-to-import-time-series-data)
|
||||
for further details about importing time series data into VictoriaMetrics.
|
||||
|
||||
<br>
|
||||
@@ -26,4 +26,4 @@ Discover VictoriaMetrics Cloud Features and Benefits [here](/victoriametrics-clo
|
||||
## Learn more
|
||||
* [VictoriaMetrics Cloud announcement](https://victoriametrics.com/blog/introduction-to-managed-monitoring/).
|
||||
* [Pricing comparison for Managed Prometheus](https://victoriametrics.com/blog/managed-prometheus-pricing/).
|
||||
* [Monitoring Proxmox VE via VictoriaMetrics Cloud and vmagent](https://victoriametrics.com/blog/proxmox-monitoring-with-dbaas/).
|
||||
* [Monitoring Proxmox VE via VictoriaMetrics Cloud and vmagent](https://victoriametrics.com/blog/proxmox-monitoring-with-dbaas/).
|
||||
|
||||
@@ -6,6 +6,8 @@ menu:
|
||||
weight: 40
|
||||
identifier: cloud
|
||||
pageRef: /victoriametrics-cloud/
|
||||
params:
|
||||
columns: 2
|
||||
aliases:
|
||||
- /victoriametrics-cloud/index.html
|
||||
- /managed-victoriametrics/index.html
|
||||
|
||||
@@ -16,7 +16,7 @@ This guide explains the different ways in which you can use vmalert in conjuncti
|
||||
|
||||
## Preconditions
|
||||
|
||||
* [vmalert](https://docs.victoriametrics.com/vmalert/) is installed. You can obtain it by building it from [source](https://docs.victoriametrics.com/vmalert/#quickstart), downloading it from the [GitHub releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest), or using the docker image [Docker Hub](https://hub.docker.com/r/victoriametrics/vmalert) or [Quay](https://quay.io/repository/victoriametrics/vmalert?tab=tags) for the container ecosystem (such as docker, k8s, etc.).
|
||||
* [vmalert](https://docs.victoriametrics.com/victoriametrics/vmalert/) is installed. You can obtain it by building it from [source](https://docs.victoriametrics.com/victoriametrics/vmalert/#quickstart), downloading it from the [GitHub releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest), or using the docker image [Docker Hub](https://hub.docker.com/r/victoriametrics/vmalert) or [Quay](https://quay.io/repository/victoriametrics/vmalert?tab=tags) for the container ecosystem (such as docker, k8s, etc.).
|
||||
* [Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/) is installed.
|
||||
* You have a [single or cluster](https://docs.victoriametrics.com/victoriametrics-cloud/quickstart/#creating-deployment) deployment in [VictoriaMetrics Cloud](https://docs.victoriametrics.com/victoriametrics-cloud/overview/).
|
||||
* If you are using helm, add the [VictoriaMetrics helm chart](https://docs.victoriametrics.com/helm/victoriametrics-alert#how-to-install) repository to your helm repositories. This step is optional.
|
||||
|
||||
@@ -108,7 +108,7 @@ Although usage is metered hourly, billing is conducted monthly. The billing date
|
||||
|
||||
### Can you help reduce my costs?
|
||||
|
||||
We recommend using Enterprise features such as [downsampling](https://docs.victoriametrics.com/#downsampling) and [retention filters](https://docs.victoriametrics.com/#retention-filters) for cost optimization. Contact [VictoriaMetrics Cloud Support](https://docs.victoriametrics.com/victoriametrics-cloud/support/) for assistance.
|
||||
We recommend using Enterprise features such as [downsampling](https://docs.victoriametrics.com/victoriametrics#downsampling) and [retention filters](https://docs.victoriametrics.com/victoriametrics#retention-filters) for cost optimization. Contact [VictoriaMetrics Cloud Support](https://docs.victoriametrics.com/victoriametrics-cloud/support/) for assistance.
|
||||
|
||||
### I want to extend my trial or get more credits. What should I do?
|
||||
|
||||
|
||||
@@ -39,7 +39,7 @@ In summary, PrivateLink can be set up manually on an individual basis, upon requ
|
||||
In any case, it’s important to note that connecting via public access is always secured via TLS with all endpoints.
|
||||
|
||||
## How does VictoriaMetrics Cloud ensure data integrity and consistency?
|
||||
We use the VictoriaMetrics Open Source project. To learn more, visit the [Open Source documentation](https://docs.victoriametrics.com/cluster-victoriametrics/#architecture-overview).
|
||||
We use the VictoriaMetrics Open Source project. To learn more, visit the [Open Source documentation](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#architecture-overview).
|
||||
|
||||
## How does VictoriaMetrics Cloud handle scalability within AWS resources?
|
||||
VictoriaMetrics Cloud deployments run in isolated environments, so scaling can be done freely. We have processes that ensure zero-downtime in cluster setups and very low downtime for single setups.
|
||||
@@ -65,11 +65,11 @@ Let us know if you need more integrations at support-cloud@victoriametrics.com.
|
||||
In case of deployment issues, users are notified with alerts, which have recommendations for possible fixes. Instance logs are also available under the Logs tab (log messages also usually contains recommendations) and for instance metrics available in the Monitoring tab of each deployment.
|
||||
|
||||
Apart from that, there are other mechanisms:
|
||||
* Cardinality explorer: https://docs.victoriametrics.com/#cardinality-explorer
|
||||
* Query tracing: https://docs.victoriametrics.com/#query-tracing
|
||||
* Top queries: https://docs.victoriametrics.com/#top-queries
|
||||
* Active queries: https://docs.victoriametrics.com/#active-queries
|
||||
* And other tools (https://docs.victoriametrics.com/#vmui) like Metric relabel debugger, Downsampling filters debugger, Retention filters debugger, Raw query view, etc…
|
||||
* Cardinality explorer: https://docs.victoriametrics.com/victoriametrics#cardinality-explorer
|
||||
* Query tracing: https://docs.victoriametrics.com/victoriametrics#query-tracing
|
||||
* Top queries: https://docs.victoriametrics.com/victoriametrics#top-queries
|
||||
* Active queries: https://docs.victoriametrics.com/victoriametrics#active-queries
|
||||
* And other tools (https://docs.victoriametrics.com/victoriametrics#vmui) like Metric relabel debugger, Downsampling filters debugger, Retention filters debugger, Raw query view, etc…
|
||||
|
||||
Also, in case of problems, support is always available to help you at support-cloud@victoriametrics.com.
|
||||
|
||||
@@ -85,7 +85,7 @@ Yes. We charge $0.09 per GB for external traffic, which matches AWS’ rate. Est
|
||||
VictoriaMetrics Cloud does not charge extra for API calls.
|
||||
Regarding storage, the price is $1.46 for 10 Gb per Month. Since VictoriaMetrics Cloud is easy to
|
||||
scale, we recommend users to expand storage resources with consumption, instead of allocating all storage space from the beginning.
|
||||
We also offer deduplication and [cardinality explorer](https://docs.victoriametrics.com/#cardinality-explorer) mechanisms
|
||||
We also offer deduplication and [cardinality explorer](https://docs.victoriametrics.com/victoriametrics#cardinality-explorer) mechanisms
|
||||
to help reducing costs.
|
||||
|
||||
## Can VictoriaMetrics Cloud expenses be consolidated into my AWS bill?
|
||||
|
||||
@@ -10,7 +10,7 @@ aliases:
|
||||
- /managed-victoriametrics/quickstart/features.html
|
||||
---
|
||||
|
||||
VictoriaMetrics Cloud helps optimizing your data and maximizing its value in the most reliable way. It can be used as an **Enterprise-level Managed Prometheus**: just configure Prometheus, [vmagent](https://docs.victoriametrics.com/vmagent/), an OpenTelemetry Collector or any agent to write data to VictoriaMetrics Cloud, and point Grafana to VictoriaMetrics Cloud by configuring it as a Prometheus datasource.
|
||||
VictoriaMetrics Cloud helps optimizing your data and maximizing its value in the most reliable way. It can be used as an **Enterprise-level Managed Prometheus**: just configure Prometheus, [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/), an OpenTelemetry Collector or any agent to write data to VictoriaMetrics Cloud, and point Grafana to VictoriaMetrics Cloud by configuring it as a Prometheus datasource.
|
||||
|
||||
## Features
|
||||
VictoriaMetrics Cloud offers a robust suite of features designed to optimize your cloud experience. Seamless integrations, scalability and cost-saving measures, and comprehensive operational tools ensure that VictoriaMetrics Cloud can support your business needs.
|
||||
|
||||
@@ -71,7 +71,7 @@ You'll also be notified via email once your deployment is ready to use.
|
||||
* **Data point sizes** are approximated to 0.8 bytes, based on our own experience managing VictoriaMetrics Cloud. This magnitude is increases with **cardinality**. For high cardinality data, more storage is expected.
|
||||
* **Long time retention**: for 6 months or more retention times, we recommend to start with a smaller storage size and increase it over time.
|
||||
* **Storage size can be increased**, however, you cannot reduce it due to AWS limitations.
|
||||
* **Enterprise features** like [downsampling](https://docs.victoriametrics.com/#downsampling) and [retention filters](https://docs.victoriametrics.com/#retention-filters) may dramatically help to optimize disk space.
|
||||
* **Enterprise features** like [downsampling](https://docs.victoriametrics.com/victoriametrics#downsampling) and [retention filters](https://docs.victoriametrics.com/victoriametrics#retention-filters) may dramatically help to optimize disk space.
|
||||
* The **formula** we use for calculating the recommended storage can be found [here](https://docs.victoriametrics.com/guides/understand-your-setup-size/#retention-perioddisk-space).
|
||||
|
||||
> Feel free to adjust your deployment based on these recommendations.
|
||||
@@ -136,9 +136,9 @@ deployment's page. It is important to know that downgrade for clusters is curren
|
||||
|
||||
Additional configuration options may be found under `Advanced Settings` where the following additional parameters can be set:
|
||||
|
||||
* [`Deduplication`](https://docs.victoriametrics.com/cluster-victoriametrics/#deduplication) defines interval when deployment leaves a single raw sample with the biggest timestamp per each discrete interval;
|
||||
* [`Deduplication`](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#deduplication) defines interval when deployment leaves a single raw sample with the biggest timestamp per each discrete interval;
|
||||
* `Maintenance Window` when deployment should start an upgrade process if needed;
|
||||
* `Settings` to define VictoriaMetrics deployment flags, depending on your deployment type: [Cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#list-of-command-line-flags) or [Single-node](https://docs.victoriametrics.com/single-server-victoriametrics/#list-of-command-line-flags).
|
||||
* `Settings` to define VictoriaMetrics deployment flags, depending on your deployment type: [Cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#list-of-command-line-flags) or [Single-node](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#list-of-command-line-flags).
|
||||
|
||||
> These updates require a deployment restart and may result in a short downtime for **single-node** deployments.
|
||||
|
||||
|
||||
@@ -14,10 +14,10 @@ The tier parameters are derived from testing in typical monitoring environments,
|
||||
|
||||
| **Parameter** | **Maximum Value** | **Description** |
|
||||
|-------------------------------------------|-----------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
||||
| **Data Ingestion Rate** | Per Tier Limits | Number of [time series](https://docs.victoriametrics.com/keyconcepts/#time-series) ingested per second. |
|
||||
| **Active Time Series Count** | Per Tier Limits | Number of [active time series](https://docs.victoriametrics.com/faq/#what-is-an-active-time-series) that received at least one data point in the last hour. |
|
||||
| **Data Ingestion Rate** | Per Tier Limits | Number of [time series](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#time-series) ingested per second. |
|
||||
| **Active Time Series Count** | Per Tier Limits | Number of [active time series](https://docs.victoriametrics.com/victoriametrics/faq/#what-is-an-active-time-series) that received at least one data point in the last hour. |
|
||||
| **Read Rate** | Per Tier Limits | Number of datapoints retrieved from the database per second. |
|
||||
| **New Series Over 24 Hours** (churn rate) | `<= Active Time Series Count` | Number of new series created in 24 hours. High [churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate) leads to higher resource consumption. |
|
||||
| **New Series Over 24 Hours** (churn rate) | `<= Active Time Series Count` | Number of new series created in 24 hours. High [churn rate](https://docs.victoriametrics.com/victoriametrics/faq/#what-is-high-churn-rate) leads to higher resource consumption. |
|
||||
| **Concurrent Requests per Token** | `<= 600` | Maximum concurrent requests per access token. It is recommended to create separate tokens for different clients and environments. This can be adjusted via [support](mailto:support-cloud@victoriametrics.com). |
|
||||
|
||||
For a detailed explanation of each parameter, visit the guide on [Understanding Your Setup Size](https://docs.victoriametrics.com/guides/understand-your-setup-size.html).
|
||||
@@ -27,13 +27,13 @@ For a detailed explanation of each parameter, visit the guide on [Understanding
|
||||
| **Flag** | **Default Value** | **Description** |
|
||||
|-----------------------------------|---------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
||||
| **Max Label Value Length** | `<= 1kb` (Default: `4kb`) | Maximum length of label values. Time series with longer values are dropped. Large label values can lead to high RAM consumption. This can be adjusted via [support](mailto:support-cloud@victoriametrics.com). |
|
||||
| **Max Labels per Time Series** | `<= 30` | Maximum number of labels per time series. Time series with excess labels are dropped. Higher values can increase [cardinality](https://docs.victoriametrics.com/keyconcepts/#cardinality) and resource usage. This can be configured in [deployment settings](https://docs.victoriametrics.com/victoriametrics-cloud/quickstart/#modifying-an-existing-deployment). |
|
||||
| **Max Labels per Time Series** | `<= 30` | Maximum number of labels per time series. Time series with excess labels are dropped. Higher values can increase [cardinality](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#cardinality) and resource usage. This can be configured in [deployment settings](https://docs.victoriametrics.com/victoriametrics-cloud/quickstart/#modifying-an-existing-deployment). |
|
||||
|
||||
|
||||
## Terms and definitions:
|
||||
|
||||
- [Time series](https://docs.victoriametrics.com/keyconcepts/#time-series)
|
||||
- [Labels](https://docs.victoriametrics.com/keyconcepts/#labels)
|
||||
- [Active time series](https://docs.victoriametrics.com/faq/#what-is-an-active-time-series)
|
||||
- [Churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate)
|
||||
- [Cardinality](https://docs.victoriametrics.com/keyconcepts/#cardinality)
|
||||
- [Time series](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#time-series)
|
||||
- [Labels](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#labels)
|
||||
- [Active time series](https://docs.victoriametrics.com/victoriametrics/faq/#what-is-an-active-time-series)
|
||||
- [Churn rate](https://docs.victoriametrics.com/victoriametrics/faq/#what-is-high-churn-rate)
|
||||
- [Cardinality](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#cardinality)
|
||||
|
||||
@@ -6,11 +6,12 @@ menu:
|
||||
parent: 'victoriametrics'
|
||||
weight: 29
|
||||
aliases:
|
||||
- /articles
|
||||
- /Articles.html
|
||||
- /articles/index.html
|
||||
- /articles/
|
||||
---
|
||||
See also [case studies](https://docs.victoriametrics.com/casestudies/).
|
||||
See also [case studies](https://docs.victoriametrics.com/victoriametrics/casestudies/).
|
||||
|
||||
## Third-party articles and slides about VictoriaMetrics
|
||||
|
||||
|
||||
@@ -57,7 +57,7 @@ and [kubernetes operator](https://docs.victoriametrics.com/operator/). See how t
|
||||
|
||||
Common recommendations:
|
||||
1. Prefer setting [requests equal to limits](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#requests-and-limits)
|
||||
for stateful components like [vmstorage](https://docs.victoriametrics.com/cluster-victoriametrics/#architecture-overview) to avoid unnecessary
|
||||
for stateful components like [vmstorage](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#architecture-overview) to avoid unnecessary
|
||||
component restarts.
|
||||
|
||||
1. Avoid using [fractional CPU units](https://kubernetes.io/docs/tasks/configure-pod-container/assign-cpu-resource/#cpu-units)
|
||||
@@ -81,7 +81,7 @@ The following steps must be performed during the upgrade / downgrade procedure:
|
||||
|
||||
## Backup Recommendations
|
||||
|
||||
VictoriaMetrics supports backups via [vmbackup](https://docs.victoriametrics.com/vmbackup/) and [vmrestore](https://docs.victoriametrics.com/vmrestore/) tools. There is also [vmbackupmanager](https://docs.victoriametrics.com/vmbackupmanager/), which simplifies backup automation.
|
||||
VictoriaMetrics supports backups via [vmbackup](https://docs.victoriametrics.com/victoriametrics/vmbackup/) and [vmrestore](https://docs.victoriametrics.com/victoriametrics/vmrestore/) tools. There is also [vmbackupmanager](https://docs.victoriametrics.com/victoriametrics/vmbackupmanager/), which simplifies backup automation.
|
||||
|
||||
## Technical Support and Services
|
||||
|
||||
@@ -91,4 +91,4 @@ There are the following channels for providing technical support for VictoriaMet
|
||||
* [Slack Inviter](https://slack.victoriametrics.com/) and [Slack channel](https://victoriametrics.slack.com/)
|
||||
* [Telegram channel](https://t.me/VictoriaMetrics_en)
|
||||
|
||||
We also provide [Enterprise support](https://docs.victoriametrics.com/enterprise/).
|
||||
We also provide [Enterprise support](https://docs.victoriametrics.com/victoriametrics/enterprise/).
|
||||
|
||||
@@ -7,6 +7,7 @@ menu:
|
||||
parent: victoriametrics
|
||||
weight: 400
|
||||
aliases:
|
||||
- /contributing
|
||||
- /CONTRIBUTING.html
|
||||
- /contributing/index.html
|
||||
- /contributing/
|
||||
@@ -16,13 +17,13 @@ If you like VictoriaMetrics and want to contribute, then it would be great:
|
||||
- Joining VictoriaMetrics community Slack ([Slack inviter](https://slack.victoriametrics.com/) and [Slack channel](https://victoriametrics.slack.com/))
|
||||
and helping other community members there.
|
||||
- Filing issues, feature requests and questions [at VictoriaMetrics GitHub](https://github.com/VictoriaMetrics/VictoriaMetrics/issues).
|
||||
- Improving [VictoriaMetrics docs](https://docs.victoriametrics.com/). See how to update docs [here](https://docs.victoriametrics.com/#documentation).
|
||||
- Improving [VictoriaMetrics docs](https://docs.victoriametrics.com/). See how to update docs [here](https://docs.victoriametrics.com/victoriametrics#documentation).
|
||||
- Spreading the word about VictoriaMetrics via various channels:
|
||||
- conference talks
|
||||
- blogposts, articles and [case studies](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/docs/CaseStudies.md)
|
||||
- comments at Hacker News, Twitter, LinkedIn, Reddit, Facebook, etc.
|
||||
- experience sharing with colleagues.
|
||||
- Convincing your management to sign [Enterprise contract](https://docs.victoriametrics.com/enterprise/) with VictoriaMetrics.
|
||||
- Convincing your management to sign [Enterprise contract](https://docs.victoriametrics.com/victoriametrics/enterprise/) with VictoriaMetrics.
|
||||
|
||||
## Issues
|
||||
|
||||
@@ -35,7 +36,7 @@ instead of [enhancements](https://github.com/VictoriaMetrics/VictoriaMetrics/iss
|
||||
Helping other people with their [questions](https://github.com/VictoriaMetrics/VictoriaMetrics/issues?q=is%3Aopen+is%3Aissue+label%3Aquestion) is also a contribution.
|
||||
|
||||
If you'd like to contribute to [documentation](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/docs) please
|
||||
read the [guideline](https://docs.victoriametrics.com/#documentation).
|
||||
read the [guideline](https://docs.victoriametrics.com/victoriametrics#documentation).
|
||||
|
||||
### Labels
|
||||
|
||||
@@ -50,7 +51,7 @@ to categorize GitHub issues. We have the following labels:
|
||||
to understand that this issue wasn't forgotten but waits for the feedback from user.
|
||||
1. `waiting for release`, assigned to issues that required code changes and those changes were merged to upstream, but not released yet.
|
||||
Once a release is made, maintainers go through all labeled issues, leave a comment about the new release, remove the label, and close the issue.
|
||||
1. `vmui`, assigned to issues related to https://docs.victoriametrics.com/#vmui or https://docs.victoriametrics.com/victorialogs/querying/#web-ui
|
||||
1. `vmui`, assigned to issues related to https://docs.victoriametrics.com/victoriametrics#vmui or https://docs.victoriametrics.com/victorialogs/querying/#web-ui
|
||||
|
||||
## Pull request checklist
|
||||
|
||||
@@ -68,7 +69,7 @@ A pull request should contain the following attributes:
|
||||
1. Documentation update, if needed. For example, adding a new flag or changing behavior of existing flags or features
|
||||
requires reflecting these changes in the documentation. For new features add `{{%/* available_from "#" */%}}` shortcode
|
||||
to the documentation. It will be later automatically replaced with an actual release version.
|
||||
1. A line in the [changelog](https://docs.victoriametrics.com/changelog/#tip) mentioning the change and related issue in a way
|
||||
1. A line in the [changelog](https://docs.victoriametrics.com/victoriametrics/changelog/#tip) mentioning the change and related issue in a way
|
||||
that would be clear to other readers even if they don't have the full context. Use the same guidelines as for commit message.
|
||||
1. Reviewers who you think have the best expertise on the matter.
|
||||
|
||||
@@ -89,7 +90,7 @@ We are open to third-party pull requests provided they follow [KISS design princ
|
||||
- Prefer simple code and architecture.
|
||||
- Avoid complex abstractions.
|
||||
- Avoid magic code and fancy algorithms.
|
||||
- Apply optimizations, which make the code harder to understand, only if [profiling](https://docs.victoriametrics.com/#profiling)
|
||||
- Apply optimizations, which make the code harder to understand, only if [profiling](https://docs.victoriametrics.com/victoriametrics#profiling)
|
||||
shows significant improvements in performance and scalability or significant reduction in RAM usage.
|
||||
Profiling must be performed on [Go benchmarks](https://pkg.go.dev/testing#hdr-Benchmarks) and on production workload.
|
||||
- Avoid [big external dependencies](https://medium.com/@valyala/stripping-dependency-bloat-in-victoriametrics-docker-image-983fb5912b0d).
|
||||
@@ -98,11 +99,11 @@ We are open to third-party pull requests provided they follow [KISS design princ
|
||||
|
||||
Adhering `KISS` principle simplifies the resulting code and architecture, so it can be reviewed, understood and debugged by wider audience.
|
||||
|
||||
Due to `KISS`, [cluster version of VictoriaMetrics](https://docs.victoriametrics.com/cluster-victoriametrics/) has none of the following "features" popular in distributed computing world:
|
||||
Due to `KISS`, [cluster version of VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/) has none of the following "features" popular in distributed computing world:
|
||||
|
||||
- Fragile gossip protocols. See [failed attempt in Thanos](https://github.com/improbable-eng/thanos/blob/030bc345c12c446962225221795f4973848caab5/docs/proposals/completed/201809_gossip-removal.md).
|
||||
- Hard-to-understand-and-implement-properly [Paxos protocols](https://www.quora.com/In-distributed-systems-what-is-a-simple-explanation-of-the-Paxos-algorithm).
|
||||
- Complex replication schemes, which may go nuts in unforeseen edge cases. See [replication docs](https://docs.victoriametrics.com/cluster-victoriametrics/#replication-and-data-safety) for details.
|
||||
- Complex replication schemes, which may go nuts in unforeseen edge cases. See [replication docs](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#replication-and-data-safety) for details.
|
||||
- Automatic data reshuffling between storage nodes, which may hurt cluster performance and availability.
|
||||
- Automatic cluster resizing, which may cost you a lot of money if improperly configured.
|
||||
- Automatic discovering and addition of new nodes in the cluster, which may mix data between dev and prod clusters :)
|
||||
|
||||
@@ -6,6 +6,8 @@ menu:
|
||||
parent: "victoriametrics"
|
||||
weight: 25
|
||||
aliases:
|
||||
- /casestudies
|
||||
- /CaseStudies
|
||||
- /CaseStudies.html
|
||||
- /casestudies/index.html
|
||||
- /casestudies/
|
||||
@@ -14,39 +16,40 @@ aliases:
|
||||
Below please find public case studies and talks from VictoriaMetrics users. You can also join our [community Slack channel](https://slack.victoriametrics.com/)
|
||||
where you can chat with VictoriaMetrics users to get additional references, reviews and case studies.
|
||||
|
||||
- [Case studies and talks](#case-studies-and-talks)
|
||||
- [AbiosGaming](#abiosgaming)
|
||||
- [adidas](#adidas)
|
||||
- [Adsterra](#adsterra)
|
||||
- [ARNES](#arnes)
|
||||
- [Brandwatch](#brandwatch)
|
||||
- [CERN](#cern)
|
||||
- [COLOPL](#colopl)
|
||||
- [Criteo](#criteo)
|
||||
- [Dig Security](#dig-security)
|
||||
- [Fly.io](#flyio)
|
||||
- [German Research Center for Artificial Intelligence](#german-research-center-for-artificial-intelligence)
|
||||
- [Grammarly](#grammarly)
|
||||
- [Groove X](#groove-x)
|
||||
- [Idealo.de](#idealode)
|
||||
- [MHI Vestas Offshore Wind](#mhi-vestas-offshore-wind)
|
||||
- [Naver](#naver)
|
||||
- [NetEase Cloud Music](#netease-cloud-music)
|
||||
- [Percona](#percona)
|
||||
- [Razorpay](#razorpay)
|
||||
- [RELEX Solutions](#relex-solutions)
|
||||
- [Roblox](#roblox)
|
||||
- [Sensedia](#sensedia)
|
||||
- [Smarkets](#smarkets)
|
||||
- [Synthesio](#synthesio)
|
||||
- [Wedos.com](#wedoscom)
|
||||
- [Wix.com](#wixcom)
|
||||
- [Xiaohongshu](#xiaohongshu)
|
||||
- [Zerodha](#zerodha)
|
||||
- [zhihu](#zhihu)
|
||||
- [Zomato](#zomato)
|
||||
# Case studies and talks
|
||||
|
||||
You can also read [articles about VictoriaMetrics from our users](https://docs.victoriametrics.com/articles/#third-party-articles-and-slides-about-victoriametrics).
|
||||
- [AbiosGaming](#abiosgaming)
|
||||
- [adidas](#adidas)
|
||||
- [Adsterra](#adsterra)
|
||||
- [ARNES](#arnes)
|
||||
- [Brandwatch](#brandwatch)
|
||||
- [CERN](#cern)
|
||||
- [COLOPL](#colopl)
|
||||
- [Criteo](#criteo)
|
||||
- [Dig Security](#dig-security)
|
||||
- [Fly.io](#flyio)
|
||||
- [German Research Center for Artificial Intelligence](#german-research-center-for-artificial-intelligence)
|
||||
- [Grammarly](#grammarly)
|
||||
- [Groove X](#groove-x)
|
||||
- [Idealo.de](#idealode)
|
||||
- [MHI Vestas Offshore Wind](#mhi-vestas-offshore-wind)
|
||||
- [Naver](#naver)
|
||||
- [NetEase Cloud Music](#netease-cloud-music)
|
||||
- [Percona](#percona)
|
||||
- [Razorpay](#razorpay)
|
||||
- [RELEX Solutions](#relex-solutions)
|
||||
- [Roblox](#roblox)
|
||||
- [Sensedia](#sensedia)
|
||||
- [Smarkets](#smarkets)
|
||||
- [Synthesio](#synthesio)
|
||||
- [Wedos.com](#wedoscom)
|
||||
- [Wix.com](#wixcom)
|
||||
- [Xiaohongshu](#xiaohongshu)
|
||||
- [Zerodha](#zerodha)
|
||||
- [zhihu](#zhihu)
|
||||
- [Zomato](#zomato)
|
||||
|
||||
You can also read [articles about VictoriaMetrics from our users](https://docs.victoriametrics.com/victoriametrics/articles/#third-party-articles-and-slides-about-victoriametrics).
|
||||
|
||||
## AbiosGaming
|
||||
|
||||
@@ -89,12 +92,12 @@ We ended up with the following configuration:
|
||||
|
||||
We learned that remote write protocol generated too much traffic and connections so after 8 months we started looking for alternatives.
|
||||
|
||||
Around the same time, VictoriaMetrics released [vmagent](https://docs.victoriametrics.com/vmagent/).
|
||||
Around the same time, VictoriaMetrics released [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/).
|
||||
We tried to scrape all the metrics via a single instance of vmagent but that didn't work because vmagent wasn't able to catch up with writes
|
||||
into VictoriaMetrics. We tested different options and end up with the following scheme:
|
||||
|
||||
- We removed Prometheus from our setup.
|
||||
- VictoriaMetrics [can scrape targets](https://docs.victoriametrics.com/single-server-victoriametrics/#how-to-scrape-prometheus-exporters-such-as-node-exporter) as well
|
||||
- VictoriaMetrics [can scrape targets](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#how-to-scrape-prometheus-exporters-such-as-node-exporter) as well
|
||||
so we removed vmagent. Now, VictoriaMetrics scrapes all the metrics from 110 jobs and 5531 targets.
|
||||
- We use [Promxy](https://github.com/jacksontj/promxy) for alerting.
|
||||
|
||||
@@ -105,7 +108,7 @@ Such a scheme has generated the following benefits compared with Prometheus:
|
||||
|
||||
Cons are the following:
|
||||
|
||||
- VictoriaMetrics didn't support replication (it [supports replication now](https://docs.victoriametrics.com/cluster-victoriametrics/#replication-and-data-safety)) - we run an extra instance of VictoriaMetrics and Promxy in front of a VictoriaMetrics pair for high availability.
|
||||
- VictoriaMetrics didn't support replication (it [supports replication now](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#replication-and-data-safety)) - we run an extra instance of VictoriaMetrics and Promxy in front of a VictoriaMetrics pair for high availability.
|
||||
- VictoriaMetrics stores 1 extra month for defined retention (if retention is set to N months, then VM stores N+1 months of data), but this is still better than other solutions.
|
||||
|
||||
Here are some numbers from our single-node VictoriaMetrics setup:
|
||||
@@ -573,7 +576,7 @@ Numbers:
|
||||
|
||||
## Wedos.com
|
||||
|
||||
> [Wedos](https://www.wedos.com/) is the biggest hosting provider in the Czech Republic. We have two our own private data centers that hold our servers and technologies, such as cooling the servers in bath oils. We started using [cluster VictoriaMetrics](https://docs.victoriametrics.com/cluster-victoriametrics/) to store Prometheus metrics from all our infrastructure after receiving positive references from people who had successfully used VictoriaMetrics. We're using it throughout our services, including the new WEDOS Global Protection.
|
||||
> [Wedos](https://www.wedos.com/) is the biggest hosting provider in the Czech Republic. We have two our own private data centers that hold our servers and technologies, such as cooling the servers in bath oils. We started using [cluster VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/) to store Prometheus metrics from all our infrastructure after receiving positive references from people who had successfully used VictoriaMetrics. We're using it throughout our services, including the new WEDOS Global Protection.
|
||||
|
||||
Numbers:
|
||||
|
||||
@@ -588,7 +591,7 @@ Numbers:
|
||||
|
||||
[Wix.com](https://en.wikipedia.org/wiki/Wix.com) is the leading web development platform.
|
||||
|
||||
> We needed to redesign our metrics infrastructure from the ground up after the move to Kubernetes. We had tried out a few different options before landing on this solution which is working great. We have a Prometheus instance in every datacenter with 2 hours retention for local storage and remote write into [HA pair of single-node VictoriaMetrics instances](https://docs.victoriametrics.com/single-server-victoriametrics/#high-availability).
|
||||
> We needed to redesign our metrics infrastructure from the ground up after the move to Kubernetes. We had tried out a few different options before landing on this solution which is working great. We have a Prometheus instance in every datacenter with 2 hours retention for local storage and remote write into [HA pair of single-node VictoriaMetrics instances](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#high-availability).
|
||||
|
||||
Numbers:
|
||||
|
||||
@@ -611,7 +614,7 @@ Numbers:
|
||||
- Enough headroom/scaling capacity for future growth which is planned to be up to 100M active time series.
|
||||
- Ability to split DB replicas per workload. Alert queries go to one replica and user queries go to another (speed for users, effective cache).
|
||||
|
||||
> Optimizing for those points and our specific workload, VictoriaMetrics proved to be the best option. As icing on the cake we’ve got [PromQL extensions](https://docs.victoriametrics.com/metricsql/) - `default 0` and `histogram` are my favorite ones. We really like having a lot of tsdb params easily available via config options which makes tsdb easy to tune for each specific use case. We've also found a great community in [Slack channel](https://slack.victoriametrics.com/) and responsive and helpful maintainer support.
|
||||
> Optimizing for those points and our specific workload, VictoriaMetrics proved to be the best option. As icing on the cake we’ve got [PromQL extensions](https://docs.victoriametrics.com/victoriametrics/metricsql/) - `default 0` and `histogram` are my favorite ones. We really like having a lot of tsdb params easily available via config options which makes tsdb easy to tune for each specific use case. We've also found a great community in [Slack channel](https://slack.victoriametrics.com/) and responsive and helpful maintainer support.
|
||||
|
||||
Alex Ulstein, Head of Monitoring, Wix.com
|
||||
|
||||
@@ -647,7 +650,7 @@ Thanos, Cortex and VictoriaMetrics were evaluated as a long-term storage for Pro
|
||||
- Blazingly fast benchmarks for a single node setup.
|
||||
- Single binary mode. Easy to scale vertically with far fewer operational headaches.
|
||||
- Considerable [improvements on creating Histograms](https://medium.com/@valyala/improving-histogram-usability-for-prometheus-and-grafana-bc7e5df0e350).
|
||||
- [MetricsQL](https://docs.victoriametrics.com/metricsql/) gives us the ability to extend PromQL with more aggregation operators.
|
||||
- [MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/) gives us the ability to extend PromQL with more aggregation operators.
|
||||
- The API is compatible with Prometheus and nearly all standard PromQL queries work well out of the box.
|
||||
- Handles storage well, with periodic compaction which makes it easy to take snapshots.
|
||||
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -4,4 +4,4 @@ weight: 100
|
||||
|
||||
# MetricsQL old page
|
||||
|
||||
The page has been moved to [MetricsQL](https://docs.victoriametrics.com/metricsql/).
|
||||
The page has been moved to [MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/).
|
||||
|
||||
@@ -6,6 +6,8 @@ menu:
|
||||
parent: 'victoriametrics'
|
||||
weight: 24
|
||||
aliases:
|
||||
- /FAQ
|
||||
- /faq
|
||||
- /FAQ.html
|
||||
- /faq.html
|
||||
- /faq/index.html
|
||||
@@ -17,15 +19,15 @@ To provide the best monitoring solution.
|
||||
|
||||
## Who uses VictoriaMetrics?
|
||||
|
||||
See [case studies](https://docs.victoriametrics.com/casestudies/).
|
||||
See [case studies](https://docs.victoriametrics.com/victoriametrics/casestudies/).
|
||||
|
||||
## Which features does VictoriaMetrics have?
|
||||
|
||||
See [these docs](https://docs.victoriametrics.com/single-server-victoriametrics/#prominent-features).
|
||||
See [these docs](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#prominent-features).
|
||||
|
||||
## Are there performance comparisons with other solutions?
|
||||
|
||||
Yes. See [these benchmarks](https://docs.victoriametrics.com/articles/#benchmarks).
|
||||
Yes. See [these benchmarks](https://docs.victoriametrics.com/victoriametrics/articles/#benchmarks).
|
||||
|
||||
## How to start using VictoriaMetrics?
|
||||
|
||||
@@ -33,67 +35,67 @@ See [these docs](https://docs.victoriametrics.com/quick-start/).
|
||||
|
||||
## How to contribute to VictoriaMetrics?
|
||||
|
||||
See [these docs](https://docs.victoriametrics.com/contributing/).
|
||||
See [these docs](https://docs.victoriametrics.com/victoriametrics/contributing/).
|
||||
|
||||
## Does VictoriaMetrics support replication?
|
||||
|
||||
Yes. See [these docs](https://docs.victoriametrics.com/cluster-victoriametrics/#replication-and-data-safety) for details.
|
||||
Yes. See [these docs](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#replication-and-data-safety) for details.
|
||||
|
||||
## Can I use VictoriaMetrics instead of Prometheus?
|
||||
|
||||
Yes in most cases. VictoriaMetrics can substitute Prometheus in the following aspects:
|
||||
|
||||
* Prometheus-compatible service discovery and target scraping can be done with [vmagent](https://docs.victoriametrics.com/vmagent/) and with single-node VictoriaMetrics. See [these docs](https://docs.victoriametrics.com/#how-to-scrape-prometheus-exporters-such-as-node-exporter).
|
||||
* Prometheus-compatible alerting rules and recording rules can be processed with [vmalert](https://docs.victoriametrics.com/vmalert/).
|
||||
* Prometheus-compatible querying in Grafana is supported by VictoriaMetrics. See [these docs](https://docs.victoriametrics.com/#grafana-setup).
|
||||
* Prometheus-compatible service discovery and target scraping can be done with [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/) and with single-node VictoriaMetrics. See [these docs](https://docs.victoriametrics.com/victoriametrics#how-to-scrape-prometheus-exporters-such-as-node-exporter).
|
||||
* Prometheus-compatible alerting rules and recording rules can be processed with [vmalert](https://docs.victoriametrics.com/victoriametrics/vmalert/).
|
||||
* Prometheus-compatible querying in Grafana is supported by VictoriaMetrics. See [these docs](https://docs.victoriametrics.com/victoriametrics#grafana-setup).
|
||||
|
||||
## What is the difference between vmagent and Prometheus?
|
||||
|
||||
While both [vmagent](https://docs.victoriametrics.com/vmagent/) and Prometheus may scrape Prometheus targets (aka `/metrics` pages)
|
||||
according to the provided Prometheus-compatible [scrape configs](https://docs.victoriametrics.com/sd_configs/#scrape_configs)
|
||||
While both [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/) and Prometheus may scrape Prometheus targets (aka `/metrics` pages)
|
||||
according to the provided Prometheus-compatible [scrape configs](https://docs.victoriametrics.com/victoriametrics/sd_configs/#scrape_configs)
|
||||
and send data to multiple remote storage systems, vmagent has the following additional features:
|
||||
|
||||
* vmagent usually requires lower amounts of CPU, RAM and disk IO compared to Prometheus when scraping an enormous number of targets (more than 1000)
|
||||
or targets with a great number of exposed metrics.
|
||||
* vmagent provides independent [disk-backed buffers](https://docs.victoriametrics.com/vmagent/#calculating-disk-space-for-persistence-queue) for each configured remote storage (see `-remoteWrite.url`). This means that slow or temporarily unavailable storage
|
||||
* vmagent provides independent [disk-backed buffers](https://docs.victoriametrics.com/victoriametrics/vmagent/#calculating-disk-space-for-persistence-queue) for each configured remote storage (see `-remoteWrite.url`). This means that slow or temporarily unavailable storage
|
||||
doesn't prevent it from sending data to healthy storage in parallel. Prometheus uses a single shared buffer for all the configured remote storage systems (see `remote_write->url`)
|
||||
with a hardcoded retention of 2 hours.
|
||||
* vmagent may accept, relabel and filter data obtained via multiple data ingestion protocols in addition to data scraped from Prometheus targets.
|
||||
That means it supports both `pull` and `push` protocols for data ingestion.
|
||||
See [these docs](https://docs.victoriametrics.com/vmagent/#features) for details.
|
||||
* vmagent may be used in different [use cases](https://docs.victoriametrics.com/vmagent/#use-cases):
|
||||
* [IoT and edge monitoring](https://docs.victoriametrics.com/vmagent/#iot-and-edge-monitoring)
|
||||
* [Drop-in replacement for Prometheus](https://docs.victoriametrics.com/vmagent/#drop-in-replacement-for-prometheus)
|
||||
* [Statsd alternative](https://docs.victoriametrics.com/vmagent/#statsd-alternative)
|
||||
* [Flexible metrics relay](https://docs.victoriametrics.com/vmagent/#flexible-metrics-relay)
|
||||
* [Replication and high availability](https://docs.victoriametrics.com/vmagent/#replication-and-high-availability)
|
||||
* [Sharding among remote storages](https://docs.victoriametrics.com/vmagent/#sharding-among-remote-storages)
|
||||
* [Relabeling and filtering](https://docs.victoriametrics.com/vmagent/#relabeling-and-filtering)
|
||||
* [Splitting data streams among multiple systems](https://docs.victoriametrics.com/vmagent/#splitting-data-streams-among-multiple-systems)
|
||||
* [Prometheus remote_write proxy](https://docs.victoriametrics.com/vmagent/#prometheus-remote_write-proxy)
|
||||
* [remote_write for clustered version](https://docs.victoriametrics.com/vmagent/#remote_write-for-clustered-version)
|
||||
See [these docs](https://docs.victoriametrics.com/victoriametrics/vmagent/#features) for details.
|
||||
* vmagent may be used in different [use cases](https://docs.victoriametrics.com/victoriametrics/vmagent/#use-cases):
|
||||
* [IoT and edge monitoring](https://docs.victoriametrics.com/victoriametrics/vmagent/#iot-and-edge-monitoring)
|
||||
* [Drop-in replacement for Prometheus](https://docs.victoriametrics.com/victoriametrics/vmagent/#drop-in-replacement-for-prometheus)
|
||||
* [Statsd alternative](https://docs.victoriametrics.com/victoriametrics/vmagent/#statsd-alternative)
|
||||
* [Flexible metrics relay](https://docs.victoriametrics.com/victoriametrics/vmagent/#flexible-metrics-relay)
|
||||
* [Replication and high availability](https://docs.victoriametrics.com/victoriametrics/vmagent/#replication-and-high-availability)
|
||||
* [Sharding among remote storages](https://docs.victoriametrics.com/victoriametrics/vmagent/#sharding-among-remote-storages)
|
||||
* [Relabeling and filtering](https://docs.victoriametrics.com/victoriametrics/vmagent/#relabeling-and-filtering)
|
||||
* [Splitting data streams among multiple systems](https://docs.victoriametrics.com/victoriametrics/vmagent/#splitting-data-streams-among-multiple-systems)
|
||||
* [Prometheus remote_write proxy](https://docs.victoriametrics.com/victoriametrics/vmagent/#prometheus-remote_write-proxy)
|
||||
* [remote_write for clustered version](https://docs.victoriametrics.com/victoriametrics/vmagent/#remote_write-for-clustered-version)
|
||||
|
||||
## What is the difference between vmagent and Prometheus agent?
|
||||
|
||||
Both [vmagent](https://docs.victoriametrics.com/vmagent/) and [Prometheus agent](https://prometheus.io/blog/2021/11/16/agent/) serve the same purpose – to efficiently scrape Prometheus-compatible targets at the edge. They have the following differences:
|
||||
Both [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/) and [Prometheus agent](https://prometheus.io/blog/2021/11/16/agent/) serve the same purpose – to efficiently scrape Prometheus-compatible targets at the edge. They have the following differences:
|
||||
|
||||
* vmagent usually requires lower amounts of CPU, RAM and disk IO compared to the Prometheus agent.
|
||||
* vmagent supports both pull and push data collection – it can accept data via many popular data ingestion protocols such as InfluxDB line protocol, Graphite protocol, OpenTSDB protocol, DataDog protocol, Prometheus protocol, OpenTelemetry metrics protocol, CSV and JSON – see [these docs](https://docs.victoriametrics.com/vmagent/#features).
|
||||
* vmagent supports both pull and push data collection – it can accept data via many popular data ingestion protocols such as InfluxDB line protocol, Graphite protocol, OpenTSDB protocol, DataDog protocol, Prometheus protocol, OpenTelemetry metrics protocol, CSV and JSON – see [these docs](https://docs.victoriametrics.com/victoriametrics/vmagent/#features).
|
||||
* vmagent doesn't have limitations on backfilling of historical data.
|
||||
* vmagent can easily scale horizontally to multiple instances for scraping a big number of targets – see [these docs](https://docs.victoriametrics.com/vmagent/#scraping-big-number-of-targets).
|
||||
* vmagent supports [improved relabeling](https://docs.victoriametrics.com/vmagent/#relabeling).
|
||||
* vmagent can limit the number of scraped metrics per target – see [these docs](https://docs.victoriametrics.com/vmagent/#cardinality-limiter).
|
||||
* vmagent supports loading scrape configs from multiple files – see [these docs](https://docs.victoriametrics.com/vmagent/#loading-scrape-configs-from-multiple-files).
|
||||
* vmagent supports data reading and data writing from/to Kafka – see [these docs](https://docs.victoriametrics.com/vmagent/#kafka-integration).
|
||||
* vmagent can easily scale horizontally to multiple instances for scraping a big number of targets – see [these docs](https://docs.victoriametrics.com/victoriametrics/vmagent/#scraping-big-number-of-targets).
|
||||
* vmagent supports [improved relabeling](https://docs.victoriametrics.com/victoriametrics/vmagent/#relabeling).
|
||||
* vmagent can limit the number of scraped metrics per target – see [these docs](https://docs.victoriametrics.com/victoriametrics/vmagent/#cardinality-limiter).
|
||||
* vmagent supports loading scrape configs from multiple files – see [these docs](https://docs.victoriametrics.com/victoriametrics/vmagent/#loading-scrape-configs-from-multiple-files).
|
||||
* vmagent supports data reading and data writing from/to Kafka – see [these docs](https://docs.victoriametrics.com/victoriametrics/vmagent/#kafka-integration).
|
||||
* vmagent can read and update scrape configs from http and https URLs, while the Prometheus agent can read them only from the local file system.
|
||||
* vmagent supports [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/) feature for performing aggregates on collected or received samples before sending them to remote storage.
|
||||
* vmagent supports [stream aggregation](https://docs.victoriametrics.com/victoriametrics/stream-aggregation/) feature for performing aggregates on collected or received samples before sending them to remote storage.
|
||||
|
||||
## Is it safe to enable [remote write](https://prometheus.io/docs/operating/integrations/#remote-endpoints-and-storage) in Prometheus?
|
||||
|
||||
Yes. Prometheus continues writing data to local storage after enabling remote write, so all the existing local storage data
|
||||
and new data is available for querying via Prometheus as usual.
|
||||
|
||||
It is recommended using [vmagent](https://docs.victoriametrics.com/vmagent/) for scraping Prometheus targets
|
||||
It is recommended using [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/) for scraping Prometheus targets
|
||||
and writing data to VictoriaMetrics.
|
||||
|
||||
## How does VictoriaMetrics compare to other remote storage solutions for Prometheus such as [M3DB](https://github.com/m3db/m3), [Thanos](https://github.com/thanos-io/thanos), [Cortex](https://github.com/cortexproject/cortex), [Mimir](https://github.com/grafana/mimir), etc.?
|
||||
@@ -116,13 +118,13 @@ VictoriaMetrics also [uses less RAM than Thanos components](https://github.com/t
|
||||
## What is the difference between VictoriaMetrics and [QuestDB](https://questdb.io/)?
|
||||
|
||||
* QuestDB needs more than 20x storage space than VictoriaMetrics. This translates to higher storage costs and slower queries over historical data, which must be read from the disk.
|
||||
* QuestDB is much harder to set up and operate than VictoriaMetrics. Compare [setup instructions for QuestDB](https://questdb.io/docs/get-started/binaries) to [setup instructions for VictoriaMetrics](https://docs.victoriametrics.com/#how-to-start-victoriametrics).
|
||||
* VictoriaMetrics provides the [MetricsQL](https://docs.victoriametrics.com/metricsql/) query language, which is better suited for typical queries over time series data than the SQL-like query language provided by QuestDB. See [this article](https://valyala.medium.com/promql-tutorial-for-beginners-9ab455142085) for details.
|
||||
* VictoriaMetrics can be queried via the [Prometheus querying API](https://docs.victoriametrics.com/#prometheus-querying-api-usage) and via [Graphite's API](https://docs.victoriametrics.com/#graphite-api-usage).
|
||||
* Thanks to PromQL support, VictoriaMetrics [can be used as a drop-in replacement for Prometheus in Grafana](https://docs.victoriametrics.com/#grafana-setup), while QuestDB needs a full rewrite of existing dashboards in Grafana.
|
||||
* Thanks to Prometheus' remote_write API support, VictoriaMetrics can be used as a long-term storage for Prometheus or for [vmagent](https://docs.victoriametrics.com/vmagent/), while QuestDB has no integration with Prometheus.
|
||||
* QuestDB [supports a smaller range of popular data ingestion protocols](https://questdb.io/docs/develop/insert-data) compared to VictoriaMetrics (compare to [the list of supported data ingestion protocols for VictoriaMetrics](https://docs.victoriametrics.com/#how-to-import-time-series-data)).
|
||||
* [VictoriaMetrics supports backfilling (e.g. storing historical data) out of the box](https://docs.victoriametrics.com/#backfilling), while QuestDB provides [very limited support for backfilling](https://questdb.io/blog/2021/05/10/questdb-release-6-0-tsbs-benchmark#the-problem-with-out-of-order-data).
|
||||
* QuestDB is much harder to set up and operate than VictoriaMetrics. Compare [setup instructions for QuestDB](https://questdb.io/docs/get-started/binaries) to [setup instructions for VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics#how-to-start-victoriametrics).
|
||||
* VictoriaMetrics provides the [MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/) query language, which is better suited for typical queries over time series data than the SQL-like query language provided by QuestDB. See [this article](https://valyala.medium.com/promql-tutorial-for-beginners-9ab455142085) for details.
|
||||
* VictoriaMetrics can be queried via the [Prometheus querying API](https://docs.victoriametrics.com/victoriametrics#prometheus-querying-api-usage) and via [Graphite's API](https://docs.victoriametrics.com/victoriametrics#graphite-api-usage).
|
||||
* Thanks to PromQL support, VictoriaMetrics [can be used as a drop-in replacement for Prometheus in Grafana](https://docs.victoriametrics.com/victoriametrics#grafana-setup), while QuestDB needs a full rewrite of existing dashboards in Grafana.
|
||||
* Thanks to Prometheus' remote_write API support, VictoriaMetrics can be used as a long-term storage for Prometheus or for [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/), while QuestDB has no integration with Prometheus.
|
||||
* QuestDB [supports a smaller range of popular data ingestion protocols](https://questdb.io/docs/develop/insert-data) compared to VictoriaMetrics (compare to [the list of supported data ingestion protocols for VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics#how-to-import-time-series-data)).
|
||||
* [VictoriaMetrics supports backfilling (e.g. storing historical data) out of the box](https://docs.victoriametrics.com/victoriametrics#backfilling), while QuestDB provides [very limited support for backfilling](https://questdb.io/blog/2021/05/10/questdb-release-6-0-tsbs-benchmark#the-problem-with-out-of-order-data).
|
||||
|
||||
## What is the difference between VictoriaMetrics and [Grafana Mimir](https://github.com/grafana/mimir)?
|
||||
|
||||
@@ -135,13 +137,13 @@ See also [Grafana Mimir vs VictoriaMetrics benchmark](https://victoriametrics.co
|
||||
|
||||
VictoriaMetrics is similar to Cortex in the following aspects:
|
||||
|
||||
* Both systems accept data from [vmagent](https://docs.victoriametrics.com/vmagent/) or Prometheus
|
||||
* Both systems accept data from [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/) or Prometheus
|
||||
via the standard [remote_write API](https://prometheus.io/docs/practices/remote_write/), so there is no need for running sidecars
|
||||
unlike in [Thanos](https://github.com/thanos-io/thanos)' case.
|
||||
* Both systems support multi-tenancy out of the box. See [the corresponding docs for VictoriaMetrics](https://docs.victoriametrics.com/cluster-victoriametrics/#multitenancy).
|
||||
* Both systems support data replication. See [replication in Cortex](https://github.com/cortexproject/cortex/blob/fe56f1420099aa1bf1ce09316c186e05bddee879/docs/architecture.md#hashing) and [replication in VictoriaMetrics](https://docs.victoriametrics.com/cluster-victoriametrics/#replication-and-data-safety).
|
||||
* Both systems scale horizontally to multiple nodes. See [these docs](https://docs.victoriametrics.com/cluster-victoriametrics/#cluster-resizing-and-scalability) for details.
|
||||
* Both systems support alerting and recording rules via the corresponding tools such as [vmalert](https://docs.victoriametrics.com/vmalert/).
|
||||
* Both systems support multi-tenancy out of the box. See [the corresponding docs for VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#multitenancy).
|
||||
* Both systems support data replication. See [replication in Cortex](https://github.com/cortexproject/cortex/blob/fe56f1420099aa1bf1ce09316c186e05bddee879/docs/architecture.md#hashing) and [replication in VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#replication-and-data-safety).
|
||||
* Both systems scale horizontally to multiple nodes. See [these docs](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#cluster-resizing-and-scalability) for details.
|
||||
* Both systems support alerting and recording rules via the corresponding tools such as [vmalert](https://docs.victoriametrics.com/victoriametrics/vmalert/).
|
||||
* Both systems can be queried via the [Prometheus querying API](https://prometheus.io/docs/prometheus/latest/querying/api/) and integrate perfectly with Grafana.
|
||||
|
||||
The main differences between Cortex and VictoriaMetrics:
|
||||
@@ -150,17 +152,17 @@ The main differences between Cortex and VictoriaMetrics:
|
||||
* Cortex heavily relies on third-party services such as Consul, Memcache, DynamoDB, BigTable, Cassandra, etc.
|
||||
This may increase operational complexity and reduce system reliability compared to VictoriaMetrics' case,
|
||||
which doesn't use any external services. Compare [Cortex' Architecture](https://github.com/cortexproject/cortex/blob/master/docs/architecture.md)
|
||||
to [VictoriaMetrics' architecture](https://docs.victoriametrics.com/cluster-victoriametrics/#architecture-overview).
|
||||
* VictoriaMetrics provides [production-ready single-node solution](https://docs.victoriametrics.com/single-server-victoriametrics/),
|
||||
to [VictoriaMetrics' architecture](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#architecture-overview).
|
||||
* VictoriaMetrics provides [production-ready single-node solution](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/),
|
||||
which is much easier to set up and operate than a Cortex cluster.
|
||||
* Cortex may lose up to 12 hours of recent data on Ingestor failure – see [the corresponding docs](https://github.com/cortexproject/cortex/blob/fe56f1420099aa1bf1ce09316c186e05bddee879/docs/architecture.md#ingesters-failure-and-data-loss).
|
||||
VictoriaMetrics may lose only a few seconds of recent data, which isn't synced to persistent storage yet.
|
||||
See [this article for details](https://medium.com/@valyala/wal-usage-looks-broken-in-modern-time-series-databases-b62a627ab704).
|
||||
* Cortex is usually slower and requires more CPU and RAM than VictoriaMetrics. See [this talk from adidas at PromCon 2019](https://promcon.io/2019-munich/talks/remote-write-storage-wars/) and [other case studies](https://docs.victoriametrics.com/casestudies/).
|
||||
* Cortex is usually slower and requires more CPU and RAM than VictoriaMetrics. See [this talk from adidas at PromCon 2019](https://promcon.io/2019-munich/talks/remote-write-storage-wars/) and [other case studies](https://docs.victoriametrics.com/victoriametrics/casestudies/).
|
||||
* VictoriaMetrics accepts data in multiple popular data ingestion protocols additionally to Prometheus remote_write protocol – InfluxDB, OpenTSDB, Graphite, CSV, JSON, native binary.
|
||||
See [these docs](https://docs.victoriametrics.com/single-server-victoriametrics/#how-to-import-time-series-data) for details.
|
||||
* VictoriaMetrics provides the [MetricsQL](https://docs.victoriametrics.com/metricsql/) query language, while Cortex provides the [PromQL](https://prometheus.io/docs/prometheus/latest/querying/basics/) query language.
|
||||
* VictoriaMetrics can be queried via [Graphite's API](https://docs.victoriametrics.com/#graphite-api-usage).
|
||||
See [these docs](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#how-to-import-time-series-data) for details.
|
||||
* VictoriaMetrics provides the [MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/) query language, while Cortex provides the [PromQL](https://prometheus.io/docs/prometheus/latest/querying/basics/) query language.
|
||||
* VictoriaMetrics can be queried via [Graphite's API](https://docs.victoriametrics.com/victoriametrics#graphite-api-usage).
|
||||
|
||||
## What is the difference between VictoriaMetrics and [Thanos](https://github.com/thanos-io/thanos)?
|
||||
|
||||
@@ -174,33 +176,33 @@ The main differences between Cortex and VictoriaMetrics:
|
||||
VictoriaMetrics works perfectly with HDD-based block storage – there is no need for using more expensive SSD or NVMe disks in most cases.
|
||||
* Thanos may lose up to 2 hours of recent data, which wasn't uploaded yet to object storage. VictoriaMetrics may lose only a few seconds of recent data,
|
||||
which hasn't been synced to persistent storage yet. See [this article for details](https://medium.com/@valyala/wal-usage-looks-broken-in-modern-time-series-databases-b62a627ab704).
|
||||
* VictoriaMetrics provides a [production-ready single-node solution](https://docs.victoriametrics.com/single-server-victoriametrics/),
|
||||
* VictoriaMetrics provides a [production-ready single-node solution](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/),
|
||||
which is much easier to set up and operate than Thanos components.
|
||||
* Thanos may be harder to set up and operate compared to VictoriaMetrics, since it has more moving parts, which can be connected with fewer reliable networks.
|
||||
See [this article for details](https://medium.com/faun/comparing-thanos-to-victoriametrics-cluster-b193bea1683).
|
||||
* Thanos is usually slower and requires more CPU and RAM than VictoriaMetrics. See [this talk from adidas at PromCon 2019](https://promcon.io/2019-munich/talks/remote-write-storage-wars/).
|
||||
* VictoriaMetrics accepts data via multiple popular data ingestion protocols in addition to the Prometheus remote_write protocol – InfluxDB, OpenTSDB, Graphite, CSV, JSON, native binary.
|
||||
See [these docs](https://docs.victoriametrics.com/single-server-victoriametrics/#how-to-import-time-series-data) for details.
|
||||
* VictoriaMetrics provides the [MetricsQL](https://docs.victoriametrics.com/metricsql/) query language, while Thanos provides the [PromQL](https://prometheus.io/docs/prometheus/latest/querying/basics/) query language.
|
||||
* VictoriaMetrics can be queried via [Graphite's API](https://docs.victoriametrics.com/#graphite-api-usage).
|
||||
See [these docs](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#how-to-import-time-series-data) for details.
|
||||
* VictoriaMetrics provides the [MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/) query language, while Thanos provides the [PromQL](https://prometheus.io/docs/prometheus/latest/querying/basics/) query language.
|
||||
* VictoriaMetrics can be queried via [Graphite's API](https://docs.victoriametrics.com/victoriametrics#graphite-api-usage).
|
||||
|
||||
## How does VictoriaMetrics compare to [InfluxDB](https://www.influxdata.com/time-series-platform/influxdb/)?
|
||||
|
||||
* VictoriaMetrics requires [10x less RAM](https://medium.com/@valyala/insert-benchmarks-with-inch-influxdb-vs-victoriametrics-e31a41ae2893) and it [works faster](https://medium.com/@valyala/measuring-vertical-scalability-for-time-series-databases-in-google-cloud-92550d78d8ae).
|
||||
* VictoriaMetrics needs lower amounts of storage space than InfluxDB for production data.
|
||||
* VictoriaMetrics doesn't support InfluxQL or Flux but provides a better query language – [MetricsQL](https://docs.victoriametrics.com/metricsql/). See [this tutorial](https://medium.com/@valyala/promql-tutorial-for-beginners-9ab455142085) for details.
|
||||
* VictoriaMetrics doesn't support InfluxQL or Flux but provides a better query language – [MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/). See [this tutorial](https://medium.com/@valyala/promql-tutorial-for-beginners-9ab455142085) for details.
|
||||
* VictoriaMetrics accepts data in multiple popular data ingestion protocols in addition to InfluxDB – Prometheus remote_write, OpenTSDB, Graphite, CSV, JSON, native binary.
|
||||
See [these docs](https://docs.victoriametrics.com/single-server-victoriametrics/#how-to-import-time-series-data) for details.
|
||||
* VictoriaMetrics can be queried via [Graphite's API](https://docs.victoriametrics.com/#graphite-api-usage).
|
||||
See [these docs](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#how-to-import-time-series-data) for details.
|
||||
* VictoriaMetrics can be queried via [Graphite's API](https://docs.victoriametrics.com/victoriametrics#graphite-api-usage).
|
||||
|
||||
## How does VictoriaMetrics compare to [TimescaleDB](https://www.timescale.com/)?
|
||||
|
||||
* TimescaleDB insists on using SQL as a query language. While SQL is more powerful than PromQL, this power is rarely required during typical usages of a TSDB. Real-world queries usually [look clearer and simpler when written in PromQL than in SQL](https://medium.com/@valyala/promql-tutorial-for-beginners-9ab455142085).
|
||||
* VictoriaMetrics requires [up to 70x less storage space compared to TimescaleDB](https://medium.com/@valyala/when-size-matters-benchmarking-victoriametrics-vs-timescale-and-influxdb-6035811952d4) for storing the same amount of time series data. The gap in storage space usage can be lowered from 70x to 3x if [compression in TimescaleDB is properly configured](https://docs.timescale.com/use-timescale/latest/compression/) (it isn't an easy task in general :)).
|
||||
* VictoriaMetrics requires up to 10x less CPU and RAM resources than TimescaleDB for processing production data. See [this article](https://abiosgaming.com/press/high-cardinality-aggregations/) for details.
|
||||
* TimescaleDB is [harder to set up, configure and operate](https://docs.timescale.com/timescaledb/latest/how-to-guides/install-timescaledb/self-hosted/ubuntu/installation-apt-ubuntu/) than VictoriaMetrics (see [how to run VictoriaMetrics](https://docs.victoriametrics.com/#how-to-start-victoriametrics)).
|
||||
* TimescaleDB is [harder to set up, configure and operate](https://docs.timescale.com/timescaledb/latest/how-to-guides/install-timescaledb/self-hosted/ubuntu/installation-apt-ubuntu/) than VictoriaMetrics (see [how to run VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics#how-to-start-victoriametrics)).
|
||||
* VictoriaMetrics accepts data in multiple popular data ingestion protocols – InfluxDB, OpenTSDB, Graphite, CSV – while TimescaleDB supports only SQL inserts.
|
||||
* VictoriaMetrics can be queried via [Graphite's API](https://docs.victoriametrics.com/#graphite-api-usage).
|
||||
* VictoriaMetrics can be queried via [Graphite's API](https://docs.victoriametrics.com/victoriametrics#graphite-api-usage).
|
||||
|
||||
## Does VictoriaMetrics use Prometheus technologies like other clustered TSDBs built on top of Prometheus such as [Thanos](https://github.com/thanos-io/thanos) or [Cortex](https://github.com/cortexproject/cortex)?
|
||||
|
||||
@@ -211,7 +213,7 @@ The architecture is [optimized for storing and querying large amounts of time se
|
||||
|
||||
The following versions are open source and free:
|
||||
|
||||
* [Single-node version](https://docs.victoriametrics.com/single-server-victoriametrics/).
|
||||
* [Single-node version](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/).
|
||||
* [Cluster version](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/cluster).
|
||||
|
||||
We provide commercial support for both versions. [Contact us](mailto:info@victoriametrics.com) for the pricing.
|
||||
@@ -233,12 +235,12 @@ if a query covers 1000 metrics with 10K values each, then the remote read API ha
|
||||
This is slow and expensive.
|
||||
Prometheus' remote read API isn't intended for querying foreign data – aka `global query view`. See [this issue](https://github.com/prometheus/prometheus/issues/4456) for details.
|
||||
|
||||
So just query VictoriaMetrics directly via [vmui](https://docs.victoriametrics.com/#vmui), the [Prometheus Querying API](https://docs.victoriametrics.com/#prometheus-querying-api-usage)
|
||||
or via [Prometheus datasource in Grafana](https://docs.victoriametrics.com/#grafana-setup).
|
||||
So just query VictoriaMetrics directly via [vmui](https://docs.victoriametrics.com/victoriametrics#vmui), the [Prometheus Querying API](https://docs.victoriametrics.com/victoriametrics#prometheus-querying-api-usage)
|
||||
or via [Prometheus datasource in Grafana](https://docs.victoriametrics.com/victoriametrics#grafana-setup).
|
||||
|
||||
## Does VictoriaMetrics deduplicate data from Prometheus instances scraping the same targets (aka `HA pairs`)?
|
||||
|
||||
Yes. See [these docs](https://docs.victoriametrics.com/single-server-victoriametrics/#deduplication) for details.
|
||||
Yes. See [these docs](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#deduplication) for details.
|
||||
|
||||
## Where is the source code of VictoriaMetrics?
|
||||
|
||||
@@ -256,21 +258,21 @@ and scales horizontally to multiple nodes.
|
||||
|
||||
## What is the difference between single-node and cluster versions of VictoriaMetrics?
|
||||
|
||||
Both [single-node](https://docs.victoriametrics.com/single-server-victoriametrics/) and
|
||||
[cluster](https://docs.victoriametrics.com/cluster-victoriametrics/) versions of VictoriaMetrics
|
||||
Both [single-node](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/) and
|
||||
[cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/) versions of VictoriaMetrics
|
||||
share the core source code, so they have many common features. They have the following differences though:
|
||||
|
||||
* [Single-node VictoriaMetrics](https://docs.victoriametrics.com/single-server-victoriametrics/) runs on a single host,
|
||||
while [cluster version of VictoriaMetrics](https://docs.victoriametrics.com/cluster-victoriametrics/) can scale to many hosts.
|
||||
* [Single-node VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/) runs on a single host,
|
||||
while [cluster version of VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/) can scale to many hosts.
|
||||
Single-node VictoriaMetrics scales vertically though, e.g. its capacity and performance scales almost linearly when increasing
|
||||
available CPU, RAM, disk IO and disk space. See [an article about vertical scalability of a single-node VictoriaMetrics](https://valyala.medium.com/measuring-vertical-scalability-for-time-series-databases-in-google-cloud-92550d78d8ae).
|
||||
|
||||
* Cluster version of VictoriaMetrics supports [multitenancy](https://docs.victoriametrics.com/cluster-victoriametrics/#multitenancy),
|
||||
* Cluster version of VictoriaMetrics supports [multitenancy](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#multitenancy),
|
||||
while single-node VictoriaMetrics doesn't support it.
|
||||
|
||||
* Cluster version of VictoriaMetrics supports data replication, while single-node VictoriaMetrics relies on the durability
|
||||
of the persistent storage pointed by `-storageDataPath` command-line flag.
|
||||
See [these docs](https://docs.victoriametrics.com/cluster-victoriametrics/#replication-and-data-safety) for details.
|
||||
See [these docs](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#replication-and-data-safety) for details.
|
||||
|
||||
* Single-node VictoriaMetrics provides higher capacity and performance comparing to cluster version of VictoriaMetrics
|
||||
when running on the same hardware with the same amounts of CPU and RAM, since it has no overhead on data transfer
|
||||
@@ -285,7 +287,7 @@ Questions about VictoriaMetrics can be asked via the following channels:
|
||||
* [Slack Inviter](https://slack.victoriametrics.com/) and [Slack channel](https://victoriametrics.slack.com/)
|
||||
* [Telegram channel](https://t.me/VictoriaMetrics_en)
|
||||
|
||||
See the full list of [community channels](https://docs.victoriametrics.com/#community-and-contributions).
|
||||
See the full list of [community channels](https://docs.victoriametrics.com/victoriametrics#community-and-contributions).
|
||||
|
||||
## Where can I file bugs and feature requests regarding VictoriaMetrics?
|
||||
|
||||
@@ -293,7 +295,7 @@ File bugs and feature requests [here](https://github.com/VictoriaMetrics/Victori
|
||||
|
||||
## Where can I find information about multi-tenancy?
|
||||
|
||||
See [these docs](https://docs.victoriametrics.com/cluster-victoriametrics/#multitenancy). Multitenancy is supported only by the [cluster version](https://docs.victoriametrics.com/cluster-victoriametrics/) of VictoriaMetrics.
|
||||
See [these docs](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#multitenancy). Multitenancy is supported only by the [cluster version](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/) of VictoriaMetrics.
|
||||
|
||||
## How to set a memory limit for VictoriaMetrics components?
|
||||
|
||||
@@ -301,10 +303,10 @@ All the VictoriaMetrics components provide command-line flags to control the siz
|
||||
|
||||
Memory usage for VictoriaMetrics components can be tuned according to the following docs:
|
||||
|
||||
* [Resource usage limits for single-node VictoriaMetrics](https://docs.victoriametrics.com/#resource-usage-limits)
|
||||
* [Resource usage limits for cluster VictoriaMetrics](https://docs.victoriametrics.com/cluster-victoriametrics/#resource-usage-limits)
|
||||
* [Troubleshooting for vmagent](https://docs.victoriametrics.com/vmagent/#troubleshooting)
|
||||
* [Troubleshooting for single-node VictoriaMetrics](https://docs.victoriametrics.com/#troubleshooting)
|
||||
* [Resource usage limits for single-node VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics#resource-usage-limits)
|
||||
* [Resource usage limits for cluster VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#resource-usage-limits)
|
||||
* [Troubleshooting for vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/#troubleshooting)
|
||||
* [Troubleshooting for single-node VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics#troubleshooting)
|
||||
|
||||
## How can I run VictoriaMetrics on FreeBSD/OpenBSD?
|
||||
|
||||
@@ -314,13 +316,13 @@ or use pre-built binaries from [releases page](https://github.com/VictoriaMetric
|
||||
|
||||
## Does VictoriaMetrics support the Graphite query language?
|
||||
|
||||
Yes. See [these docs](https://docs.victoriametrics.com/#graphite-api-usage).
|
||||
Yes. See [these docs](https://docs.victoriametrics.com/victoriametrics#graphite-api-usage).
|
||||
|
||||
## What is an active time series?
|
||||
|
||||
A time series is uniquely identified by its name plus a set of its labels. For example, `temperature{city="NY",country="US"}` and `temperature{city="SF",country="US"}`
|
||||
are two distinct series, since they differ by the `city` label. A time series is considered active if it received at least a single sample during the last hour or it has been touched by queries during the last hour.
|
||||
The number of active time series is displayed on the official Grafana dashboard for VictoriaMetrics - see [these docs](https://docs.victoriametrics.com/#monitoring) for details.
|
||||
The number of active time series is displayed on the official Grafana dashboard for VictoriaMetrics - see [these docs](https://docs.victoriametrics.com/victoriametrics#monitoring) for details.
|
||||
|
||||
## What is high churn rate?
|
||||
|
||||
@@ -338,10 +340,10 @@ The main reason for high churn rate is a metric label with frequently changed va
|
||||
* A `hash` or `uuid` label, which changes frequently.
|
||||
|
||||
The solution against high churn rate is to identify and eliminate labels with frequently changed values.
|
||||
[Cardinality explorer](https://docs.victoriametrics.com/#cardinality-explorer) can help determining these labels. If labels can't be removed, try pre-aggregating data
|
||||
before it gets ingested into database with [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/).
|
||||
[Cardinality explorer](https://docs.victoriametrics.com/victoriametrics#cardinality-explorer) can help determining these labels. If labels can't be removed, try pre-aggregating data
|
||||
before it gets ingested into database with [stream aggregation](https://docs.victoriametrics.com/victoriametrics/stream-aggregation/).
|
||||
|
||||
The official Grafana dashboards for VictoriaMetrics contain graphs for churn rate - see [these docs](https://docs.victoriametrics.com/#monitoring) for details.
|
||||
The official Grafana dashboards for VictoriaMetrics contain graphs for churn rate - see [these docs](https://docs.victoriametrics.com/victoriametrics#monitoring) for details.
|
||||
|
||||
## What is high cardinality?
|
||||
|
||||
@@ -353,10 +355,10 @@ a large number of unique values, which presents a big share of the ingested time
|
||||
* `url`
|
||||
* `ip`
|
||||
|
||||
The solution is to identify and remove the source of high cardinality with the help of [cardinality explorer](https://docs.victoriametrics.com/#cardinality-explorer).
|
||||
The solution is to identify and remove the source of high cardinality with the help of [cardinality explorer](https://docs.victoriametrics.com/victoriametrics#cardinality-explorer).
|
||||
|
||||
The official Grafana dashboards for VictoriaMetrics contain graphs, which show the number of active time series -
|
||||
see [these docs](https://docs.victoriametrics.com/#monitoring) for details.
|
||||
see [these docs](https://docs.victoriametrics.com/victoriametrics#monitoring) for details.
|
||||
|
||||
## What is a slow insert?
|
||||
|
||||
@@ -364,34 +366,34 @@ VictoriaMetrics maintains in-memory cache for mapping of [active time series](#w
|
||||
The cache size depends on the available memory for VictoriaMetrics in the host system. If the information about all the active time series doesn't fit the cache,
|
||||
then VictoriaMetrics needs to read and unpack the information from disk on every incoming sample for time series missing in the cache.
|
||||
This operation is much slower than the cache lookup, so such an insert is named a `slow insert`.
|
||||
A high percentage of slow inserts on the [official dashboard for VictoriaMetrics](https://docs.victoriametrics.com/#monitoring) indicates
|
||||
A high percentage of slow inserts on the [official dashboard for VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics#monitoring) indicates
|
||||
a memory shortage for the current number of [active time series](#what-is-an-active-time-series). Such a condition usually leads
|
||||
to a significant slowdown for data ingestion and to significantly increased disk IO and CPU usage.
|
||||
The solution is to add more memory or to reduce the number of [active time series](#what-is-an-active-time-series).
|
||||
|
||||
[Cardinality explorer](https://docs.victoriametrics.com/#cardinality-explorer) can be helpful for locating the source of high number of active time series.
|
||||
[Cardinality explorer](https://docs.victoriametrics.com/victoriametrics#cardinality-explorer) can be helpful for locating the source of high number of active time series.
|
||||
|
||||
## How to optimize MetricsQL query?
|
||||
|
||||
See [this article](https://valyala.medium.com/how-to-optimize-promql-and-metricsql-queries-85a1b75bf986).
|
||||
|
||||
VictoriaMetrics also provides [query tracer](https://docs.victoriametrics.com/#query-tracing) and [cardinality explorer](https://docs.victoriametrics.com/#cardinality-explorer),
|
||||
VictoriaMetrics also provides [query tracer](https://docs.victoriametrics.com/victoriametrics#query-tracing) and [cardinality explorer](https://docs.victoriametrics.com/victoriametrics#cardinality-explorer),
|
||||
which can help during query optimization.
|
||||
|
||||
See also [troubleshooting slow queries](https://docs.victoriametrics.com/troubleshooting/#slow-queries).
|
||||
See also [troubleshooting slow queries](https://docs.victoriametrics.com/victoriametrics/troubleshooting/#slow-queries).
|
||||
|
||||
## Which VictoriaMetrics type is recommended for use in production - single-node or cluster?
|
||||
|
||||
Both [single-node VictoriaMetrics](https://docs.victoriametrics.com/single-server-victoriametrics/) and
|
||||
[VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/) are production-ready.
|
||||
Both [single-node VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/) and
|
||||
[VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/) are production-ready.
|
||||
|
||||
Single-node VictoriaMetrics is able to handle quite big workloads in production
|
||||
with tens of millions of [active time series](https://docs.victoriametrics.com/faq/#what-is-an-active-time-series)
|
||||
at the ingestion rate of million of samples per second. See [this case study](https://docs.victoriametrics.com/casestudies/#wixcom).
|
||||
with tens of millions of [active time series](https://docs.victoriametrics.com/victoriametrics/faq/#what-is-an-active-time-series)
|
||||
at the ingestion rate of million of samples per second. See [this case study](https://docs.victoriametrics.com/victoriametrics/casestudies/#wixcom).
|
||||
|
||||
Single-node VictoriaMetrics requires lower amounts of CPU and RAM for handling the same workload comparing
|
||||
to cluster version of VictoriaMetrics, since it doesn't need to pass the encoded data over the network
|
||||
between [cluster components](https://docs.victoriametrics.com/cluster-victoriametrics/#architecture-overview).
|
||||
between [cluster components](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#architecture-overview).
|
||||
|
||||
The performance of a single-node VictoriaMetrics scales almost perfectly with the available CPU, RAM and disk IO resources on the host where it runs -
|
||||
see [this article](https://valyala.medium.com/measuring-vertical-scalability-for-time-series-databases-in-google-cloud-92550d78d8ae).
|
||||
@@ -402,40 +404,40 @@ Given the facts above **it is recommended to use single-node VictoriaMetrics in
|
||||
|
||||
Cluster version of VictoriaMetrics may be preferred over single-node VictoriaMetrics in the following relatively rare cases:
|
||||
|
||||
- If [multitenancy support](https://docs.victoriametrics.com/cluster-victoriametrics/#multitenancy) is needed,
|
||||
- If [multitenancy support](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#multitenancy) is needed,
|
||||
since single-node VictoriaMetrics doesn't support multitenancy. Though it is possible to run multiple single-node VictoriaMetrics
|
||||
instances - one per each tenant - and route incoming requests from particular tenant to the needed VictoriaMetrics instance
|
||||
via [vmauth](https://docs.victoriametrics.com/vmauth/).
|
||||
via [vmauth](https://docs.victoriametrics.com/victoriametrics/vmauth/).
|
||||
|
||||
- If the current workload cannot be handled by a single-node VictoriaMetrics. For example, if you are going to ingest hundreds of millions of active time series
|
||||
at ingestion rates exceeding a million samples per second, then it is better to use cluster version of VictoriaMetrics,
|
||||
since its capacity can [scale horizontally with the number of nodes in the cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#cluster-resizing-and-scalability).
|
||||
since its capacity can [scale horizontally with the number of nodes in the cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#cluster-resizing-and-scalability).
|
||||
|
||||
## How to migrate data from single-node VictoriaMetrics to cluster version?
|
||||
|
||||
Single-node VictoriaMetrics stores data on disk in slightly different format comparing to cluster version of VictoriaMetrics.
|
||||
So it is impossible to just copy the on-disk data from `-storageDataPath` directory from single-node VictoriaMetrics to a `vmstorage` node in VictoriaMetrics cluster.
|
||||
If you need migrating data from single-node VictoriaMetrics to cluster version, then [follow these instructions](https://docs.victoriametrics.com/vmctl/#migrating-data-from-victoriametrics).
|
||||
If you need migrating data from single-node VictoriaMetrics to cluster version, then [follow these instructions](https://docs.victoriametrics.com/victoriametrics/vmctl/#migrating-data-from-victoriametrics).
|
||||
|
||||
## Why isn't MetricsQL 100% compatible with PromQL?
|
||||
|
||||
[MetricsQL](https://docs.victoriametrics.com/metricsql/) provides better user experience than PromQL. It fixes a few annoying issues in PromQL. This prevents MetricsQL to be 100% compatible with PromQL. See [this article](https://medium.com/@romanhavronenko/victoriametrics-promql-compliance-d4318203f51e) for details.
|
||||
[MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/) provides better user experience than PromQL. It fixes a few annoying issues in PromQL. This prevents MetricsQL to be 100% compatible with PromQL. See [this article](https://medium.com/@romanhavronenko/victoriametrics-promql-compliance-d4318203f51e) for details.
|
||||
|
||||
## How to migrate data from Prometheus to VictoriaMetrics?
|
||||
|
||||
Please see [these docs](https://docs.victoriametrics.com/vmctl/#migrating-data-from-prometheus).
|
||||
Please see [these docs](https://docs.victoriametrics.com/victoriametrics/vmctl/#migrating-data-from-prometheus).
|
||||
|
||||
## How to migrate data from InfluxDB to VictoriaMetrics?
|
||||
|
||||
Please see [these docs](https://docs.victoriametrics.com/vmctl/#migrating-data-from-influxdb-1x).
|
||||
Please see [these docs](https://docs.victoriametrics.com/victoriametrics/vmctl/#migrating-data-from-influxdb-1x).
|
||||
|
||||
## How to migrate data from OpenTSDB to VictoriaMetrics?
|
||||
|
||||
Please see [these docs](https://docs.victoriametrics.com/vmctl/#migrating-data-from-opentsdb).
|
||||
Please see [these docs](https://docs.victoriametrics.com/victoriametrics/vmctl/#migrating-data-from-opentsdb).
|
||||
|
||||
## How to migrate data from Graphite to VictoriaMetrics?
|
||||
|
||||
Please use the [whisper-to-graphite](https://github.com/bzed/whisper-to-graphite) tool for reading data from Graphite and pushing them to VictoriaMetrics via [Graphite's import API](https://docs.victoriametrics.com/#how-to-send-data-from-graphite-compatible-agents-such-as-statsd).
|
||||
Please use the [whisper-to-graphite](https://github.com/bzed/whisper-to-graphite) tool for reading data from Graphite and pushing them to VictoriaMetrics via [Graphite's import API](https://docs.victoriametrics.com/victoriametrics#how-to-send-data-from-graphite-compatible-agents-such-as-statsd).
|
||||
|
||||
## Why do the same metrics have differences in VictoriaMetrics' and Prometheus' dashboards?
|
||||
|
||||
@@ -445,19 +447,19 @@ The query engine may behave differently for some functions. Please see [this art
|
||||
|
||||
## If downsampling and deduplication are enabled how will this work?
|
||||
|
||||
[Deduplication](https://docs.victoriametrics.com/#deduplication) is a special case of zero-offset [downsampling](https://docs.victoriametrics.com/#downsampling). So, if both downsampling and deduplication are enabled, then deduplication is replaced by zero-offset downsampling
|
||||
[Deduplication](https://docs.victoriametrics.com/victoriametrics#deduplication) is a special case of zero-offset [downsampling](https://docs.victoriametrics.com/victoriametrics#downsampling). So, if both downsampling and deduplication are enabled, then deduplication is replaced by zero-offset downsampling
|
||||
|
||||
## How to upgrade or downgrade VictoriaMetrics without downtime?
|
||||
|
||||
Single-node VictoriaMetrics cannot be restarted / upgraded or downgraded without downtime, since it needs to be gracefully shut down and then started again. See [how to upgrade VictoriaMetrics](https://docs.victoriametrics.com/#how-to-upgrade-victoriametrics).
|
||||
Single-node VictoriaMetrics cannot be restarted / upgraded or downgraded without downtime, since it needs to be gracefully shut down and then started again. See [how to upgrade VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics#how-to-upgrade-victoriametrics).
|
||||
|
||||
[Cluster version of VictoriaMetrics](https://docs.victoriametrics.com/cluster-victoriametrics/) can be restarted / upgraded / downgraded without downtime according to [these instructions](https://docs.victoriametrics.com/cluster-victoriametrics/#updating--reconfiguring-cluster-nodes).
|
||||
[Cluster version of VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/) can be restarted / upgraded / downgraded without downtime according to [these instructions](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#updating--reconfiguring-cluster-nodes).
|
||||
|
||||
## Why VictoriaMetrics misses automatic data re-balancing between vmstorage nodes?
|
||||
|
||||
VictoriaMetrics doesn't rebalance data between `vmstorage` nodes when new `vmstorage` nodes are added to the cluster.
|
||||
This means that newly added `vmstorage` nodes will have less data at `-storageDataPath` comparing to the old `vmstorage` nodes
|
||||
until the historical data is removed from the old `vmstorage` nodes when it goes outside the configured [retention](https://docs.victoriametrics.com/#retention).
|
||||
until the historical data is removed from the old `vmstorage` nodes when it goes outside the configured [retention](https://docs.victoriametrics.com/victoriametrics#retention).
|
||||
|
||||
The automatic rebalancing is the process of moving data between `vmstorage` nodes, so every node has the same amounts of data eventually.
|
||||
It is disabled by default because it may consume additional CPU, network bandwidth and disk IO at `vmstorage` nodes for long periods of time,
|
||||
@@ -466,22 +468,22 @@ which, in turn, can negatively impact VictoriaMetrics cluster availability.
|
||||
Additionally, it is unclear how to handle the automatic re-balancing if cluster configuration changes when the re-balancing is in progress.
|
||||
|
||||
The amounts of data stored in `vmstorage` becomes equal among old `vmstorage` nodes and new `vmstorage` nodes
|
||||
after historical data is removed from the old `vmstorage` nodes because it goes outside of configured [retention](https://docs.victoriametrics.com/#retention).
|
||||
after historical data is removed from the old `vmstorage` nodes because it goes outside of configured [retention](https://docs.victoriametrics.com/victoriametrics#retention).
|
||||
|
||||
The data ingestion load becomes even between old `vmstorage` nodes and new `vmstorage` nodes almost immediately
|
||||
after adding new `vmstorage` nodes to the cluster, since `vminsert` nodes evenly distribute incoming time series
|
||||
among the nodes specified in `-storageNode` command-line flag. The newly added `vmstorage` nodes may experience
|
||||
increased load during the first couple of minutes because they need to register [active time series](https://docs.victoriametrics.com/faq/#what-is-an-active-time-series).
|
||||
increased load during the first couple of minutes because they need to register [active time series](https://docs.victoriametrics.com/victoriametrics/faq/#what-is-an-active-time-series).
|
||||
|
||||
The query load becomes even between old `vmstorage` nodes and new `vmstorage` nodes after most of queries are executed
|
||||
over time ranges with data covered by new `vmstorage` nodes. Usually the most of queries are received
|
||||
from [alerting and recording rules](https://docs.victoriametrics.com/vmalert/), which query data on limited time ranges
|
||||
from [alerting and recording rules](https://docs.victoriametrics.com/victoriametrics/vmalert/), which query data on limited time ranges
|
||||
such as a few hours or few days at max. This means that the query load between old `vmstorage` nodes and new `vmstorage` nodes
|
||||
should become even in a few hours / days after adding new `vmstorage` nodes.
|
||||
|
||||
## Why VictoriaMetrics misses automatic recovery of replication factor?
|
||||
|
||||
VictoriaMetrics doesn't restore [replication factor](https://docs.victoriametrics.com/cluster-victoriametrics/#replication-and-data-safety)
|
||||
VictoriaMetrics doesn't restore [replication factor](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#replication-and-data-safety)
|
||||
when some of `vmstorage` nodes are removed from the cluster because of the following reasons:
|
||||
|
||||
- Automatic replication factor recovery needs copying non-trivial amounts of data between the remaining `vmstorage` nodes.
|
||||
@@ -491,5 +493,5 @@ when some of `vmstorage` nodes are removed from the cluster because of the follo
|
||||
- It is unclear when the automatic replication factor recovery must be started. How to distinguish the expected temporary
|
||||
`vmstorage` node unavailability because of maintenance, upgrade or config changes from permanent loss of data at the `vmstorage` node?
|
||||
|
||||
It is recommended reading [replication and data safety docs](https://docs.victoriametrics.com/cluster-victoriametrics/#replication-and-data-safety)
|
||||
It is recommended reading [replication and data safety docs](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#replication-and-data-safety)
|
||||
for more details.
|
||||
|
||||
@@ -6,21 +6,23 @@ menu:
|
||||
parent: 'victoriametrics'
|
||||
weight: 300
|
||||
aliases:
|
||||
- /lts-releases
|
||||
- /LTS-releases
|
||||
- /LTS-releases.html
|
||||
- /lts-releases/index.html
|
||||
- /lts-releases/
|
||||
---
|
||||
[Enterprise version of VictoriaMetrics](https://docs.victoriametrics.com/enterprise/) provides long-term support lines of releases (aka LTS releases).
|
||||
[Enterprise version of VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics/enterprise/) provides long-term support lines of releases (aka LTS releases).
|
||||
Every LTS line receives bugfixes and [security fixes](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/SECURITY.md) for 12 months after
|
||||
the initial release. New LTS lines are published every 6 months, so the latest two LTS lines are supported at any given moment. This gives up to 6 months
|
||||
for the migration to new LTS lines for [VictoriaMetrics Enterprise](https://docs.victoriametrics.com/enterprise/) users.
|
||||
for the migration to new LTS lines for [VictoriaMetrics Enterprise](https://docs.victoriametrics.com/victoriametrics/enterprise/) users.
|
||||
|
||||
LTS releases are published for [Enterprise versions of VictoriaMetrics](https://docs.victoriametrics.com/enterprise/) only.
|
||||
LTS releases are published for [Enterprise versions of VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics/enterprise/) only.
|
||||
When a new LTS line is created, the new LTS release might be publicly available for everyone until the new major OS release will be published.
|
||||
|
||||
All the bugfixes and security fixes, which are included in LTS releases, are also available in [the latest release](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest),
|
||||
so non-enterprise users are advised to regularly [upgrade](https://docs.victoriametrics.com/#how-to-upgrade-victoriametrics) VictoriaMetrics products
|
||||
to [the latest available releases](https://docs.victoriametrics.com/changelog/).
|
||||
so non-enterprise users are advised to regularly [upgrade](https://docs.victoriametrics.com/victoriametrics#how-to-upgrade-victoriametrics) VictoriaMetrics products
|
||||
to [the latest available releases](https://docs.victoriametrics.com/victoriametrics/changelog/).
|
||||
|
||||
## Currently supported LTS release lines
|
||||
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -7,6 +7,8 @@ menu:
|
||||
parent: victoriametrics
|
||||
weight: 81
|
||||
aliases:
|
||||
- /pertenantstatistic
|
||||
- /PerTenantStatistic
|
||||
- /PerTenantStatistic.html
|
||||
- /pertenantstatistic
|
||||
- /pertenantstatistic/index.html
|
||||
@@ -14,7 +16,7 @@ aliases:
|
||||
---
|
||||

|
||||
|
||||
***The per-tenant statistic is a part of [enterprise package](https://docs.victoriametrics.com/enterprise/). It is available for download and evaluation at [releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest).
|
||||
***The per-tenant statistic is a part of [enterprise package](https://docs.victoriametrics.com/victoriametrics/enterprise/). It is available for download and evaluation at [releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest).
|
||||
To get the license key you can request a [free trial license](https://victoriametrics.com/products/enterprise/trial/).***
|
||||
|
||||
VictoriaMetrics cluster for enterprise provides various metrics and statistics usage per tenant:
|
||||
@@ -80,7 +82,7 @@ Check the Billing section of [Grafana Dashboard](#visualization), it contains bi
|
||||
## Integration with vmgateway
|
||||
|
||||
`vmgateway` supports integration with Per Tenant Statistics data for rate limiting purposes.
|
||||
More information can be found [here](https://docs.victoriametrics.com/vmgateway/)
|
||||
More information can be found [here](https://docs.victoriametrics.com/victoriametrics/vmgateway/)
|
||||
|
||||
## Integration with vmalert
|
||||
|
||||
|
||||
@@ -7,6 +7,9 @@ menu:
|
||||
parent: victoriametrics
|
||||
weight: 1
|
||||
aliases:
|
||||
- /quickstart
|
||||
- /quick-start
|
||||
- /Quick-Start
|
||||
- /Quick-Start.html
|
||||
- /quick-start/index.html
|
||||
- /quick-start/
|
||||
@@ -15,10 +18,10 @@ aliases:
|
||||
|
||||
VictoriaMetrics is available in the following distributions:
|
||||
|
||||
* [Single-server-VictoriaMetrics](https://docs.victoriametrics.com/single-server-victoriametrics/) - all-in-one
|
||||
* [Single-server-VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/) - all-in-one
|
||||
binary that is easy to run and maintain. Single-server-VictoriaMetrics perfectly scales vertically and easily handles
|
||||
millions of metrics;
|
||||
* [VictoriaMetrics Cluster](https://docs.victoriametrics.com/cluster-victoriametrics/) - set of components
|
||||
* [VictoriaMetrics Cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/) - set of components
|
||||
for building horizontally scalable clusters.
|
||||
* [VictoriaMetrics Cloud](https://console.victoriametrics.cloud/signUp?utm_source=website&utm_campaign=docs_vm_quickstart_guide) -
|
||||
VictoriaMetrics installation in the cloud. Users can pick a suitable installation size and don't think of typical DevOps
|
||||
@@ -31,17 +34,17 @@ VictoriaMetrics is available as:
|
||||
* [Binary releases](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest)
|
||||
* [Ansible Roles](https://github.com/VictoriaMetrics/ansible-playbooks)
|
||||
* [Source code](https://github.com/VictoriaMetrics/VictoriaMetrics).
|
||||
See [How to build from sources](https://docs.victoriametrics.com/single-server-victoriametrics/#how-to-build-from-sources)
|
||||
See [How to build from sources](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#how-to-build-from-sources)
|
||||
* [VictoriaMetrics on Linode](https://www.linode.com/marketplace/apps/victoriametrics/victoriametrics/)
|
||||
* [VictoriaMetrics on DigitalOcean](https://marketplace.digitalocean.com/apps/victoriametrics-single)
|
||||
|
||||
Just download VictoriaMetrics and follow
|
||||
[these instructions](https://docs.victoriametrics.com/single-server-victoriametrics/#how-to-start-victoriametrics).
|
||||
Then read [Prometheus setup](https://docs.victoriametrics.com/single-server-victoriametrics/#prometheus-setup)
|
||||
and [Grafana setup](https://docs.victoriametrics.com/single-server-victoriametrics/#grafana-setup) docs.
|
||||
[these instructions](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#how-to-start-victoriametrics).
|
||||
Then read [Prometheus setup](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#prometheus-setup)
|
||||
and [Grafana setup](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#grafana-setup) docs.
|
||||
|
||||
VictoriaMetrics is developed at a fast pace, so it is recommended periodically checking the [CHANGELOG](https://docs.victoriametrics.com/changelog/)
|
||||
and performing [regular upgrades](https://docs.victoriametrics.com/#how-to-upgrade-victoriametrics).
|
||||
VictoriaMetrics is developed at a fast pace, so it is recommended periodically checking the [CHANGELOG](https://docs.victoriametrics.com/victoriametrics/changelog/)
|
||||
and performing [regular upgrades](https://docs.victoriametrics.com/victoriametrics#how-to-upgrade-victoriametrics).
|
||||
|
||||
### Starting VictoriaMetrics Single Node or Cluster on VictoriaMetrics Cloud {anchor="starting-vm-on-cloud"}
|
||||
|
||||
@@ -67,7 +70,7 @@ You should see:
|
||||
partition "2025_03" has been created
|
||||
```
|
||||
|
||||
Open `http://localhost:8428/vmui` in WEB browser to see graphical interface [vmui](https://docs.victoriametrics.com/#vmui).
|
||||
Open `http://localhost:8428/vmui` in WEB browser to see graphical interface [vmui](https://docs.victoriametrics.com/victoriametrics#vmui).
|
||||
With `--selfScrapeInterval=5s` VictoriaMetrics scrapes its own metrics, and they should become queryable 30s after start.
|
||||
Visit `http://localhost:8428/vmui/#/metrics` to explore available metrics or run an arbitrary query at
|
||||
`http://localhost:8428/vmui` (i.e. `process_cpu_cores_available`).
|
||||
@@ -100,12 +103,12 @@ Visit Grafana `http://localhost:3000/` (admin:admin) or vmui `http://localhost:8
|
||||
_Further customization is possible by editing the [compose-vm-cluster.yml](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/compose-vm-cluster.yml)
|
||||
file._
|
||||
|
||||
See more details about [cluster architecture](https://docs.victoriametrics.com/cluster-victoriametrics/#cluster-setup).
|
||||
See more details about [cluster architecture](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#cluster-setup).
|
||||
|
||||
### Starting VictoriaMetrics Single Node from a Binary {anchor="starting-vm-single-from-a-binary"}
|
||||
|
||||
1. Download the correct binary for your OS and architecture from [GitHub](https://github.com/VictoriaMetrics/VictoriaMetrics/releases).
|
||||
For Enterprise binaries see [this link](https://docs.victoriametrics.com/enterprise/#binary-releases).
|
||||
For Enterprise binaries see [this link](https://docs.victoriametrics.com/victoriametrics/enterprise/#binary-releases).
|
||||
|
||||
2. Extract the archive to /usr/local/bin by running:
|
||||
```sh
|
||||
@@ -152,9 +155,9 @@ WantedBy=multi-user.target
|
||||
END'
|
||||
```
|
||||
|
||||
Extra [command-line flags](https://docs.victoriametrics.com/#list-of-command-line-flags) can be added to `ExecStart` line.
|
||||
Extra [command-line flags](https://docs.victoriametrics.com/victoriametrics#list-of-command-line-flags) can be added to `ExecStart` line.
|
||||
|
||||
If you want to deploy VictoriaMetrics Single Node as a Windows Service review the [running as a Windows service docs](https://docs.victoriametrics.com/single-server-victoriametrics/#running-as-windows-service).
|
||||
If you want to deploy VictoriaMetrics Single Node as a Windows Service review the [running as a Windows service docs](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#running-as-windows-service).
|
||||
|
||||
> Please note, `victoriametrics` service is listening on `:8428` for HTTP connections (see `-httpListenAddr` flag).
|
||||
|
||||
@@ -170,20 +173,20 @@ sudo systemctl daemon-reload && sudo systemctl enable --now victoriametrics.serv
|
||||
sudo systemctl status victoriametrics.service
|
||||
```
|
||||
|
||||
8. After VictoriaMetrics is in `Running` state, verify [vmui](https://docs.victoriametrics.com/#vmui) is working
|
||||
8. After VictoriaMetrics is in `Running` state, verify [vmui](https://docs.victoriametrics.com/victoriametrics#vmui) is working
|
||||
by going to `http://<ip_or_hostname>:8428/vmui`.
|
||||
|
||||
|
||||
### Starting VictoriaMetrics Cluster from Binaries {anchor="starting-vm-cluster-from-binaries"}
|
||||
|
||||
VictoriaMetrics cluster consists of [3 components](https://docs.victoriametrics.com/cluster-victoriametrics/#architecture-overview).
|
||||
It is recommended to run these components in the same private network (for [security reasons](https://docs.victoriametrics.com/#security)),
|
||||
VictoriaMetrics cluster consists of [3 components](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#architecture-overview).
|
||||
It is recommended to run these components in the same private network (for [security reasons](https://docs.victoriametrics.com/victoriametrics#security)),
|
||||
but on the separate physical nodes for the best performance.
|
||||
|
||||
On all nodes you will need to do the following:
|
||||
|
||||
1. Download the correct binary for your OS and architecture with `-cluster` suffix from [GitHub](https://github.com/VictoriaMetrics/VictoriaMetrics/releases).
|
||||
For Enterprise binaries see [this link](https://docs.victoriametrics.com/enterprise/#binary-releases).
|
||||
For Enterprise binaries see [this link](https://docs.victoriametrics.com/victoriametrics/enterprise/#binary-releases).
|
||||
|
||||
2. Extract the archive to /usr/local/bin by running:
|
||||
|
||||
@@ -199,7 +202,7 @@ Replace `<victoriametrics-archive>` with the path to the archive you downloaded
|
||||
sudo useradd -s /usr/sbin/nologin victoriametrics
|
||||
```
|
||||
|
||||
See recommendations for installing each type of [cluster component](https://docs.victoriametrics.com/cluster-victoriametrics/#architecture-overview) below.
|
||||
See recommendations for installing each type of [cluster component](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#architecture-overview) below.
|
||||
|
||||
##### Installing vmstorage
|
||||
|
||||
@@ -231,7 +234,7 @@ WantedBy=multi-user.target
|
||||
END'
|
||||
```
|
||||
|
||||
Extra [command-line flags](https://docs.victoriametrics.com/cluster-victoriametrics/#list-of-command-line-flags-for-vmstorage)
|
||||
Extra [command-line flags](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#list-of-command-line-flags-for-vmstorage)
|
||||
for vmstorage can be added to `ExecStart` line.
|
||||
|
||||
> Please note, `vmstorage` service is listening on `:8400` for vminsert connections (see `-vminsertAddr` flag),
|
||||
@@ -280,7 +283,7 @@ END'
|
||||
|
||||
Replace `<list of vmstorages>` with addresses of previously configured `vmstorage` services.
|
||||
To specify multiple addresses you can repeat the flag multiple times, or separate addresses with commas
|
||||
in one flag. See more details in `-storageNode` flag description [here](https://docs.victoriametrics.com/cluster-victoriametrics/#list-of-command-line-flags-for-vminsert).
|
||||
in one flag. See more details in `-storageNode` flag description [here](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#list-of-command-line-flags-for-vminsert).
|
||||
|
||||
> Please note, `vminsert` service is listening on `:8480` for HTTP connections (see `-httpListenAddr` flag).
|
||||
|
||||
@@ -334,7 +337,7 @@ END'
|
||||
|
||||
Replace `<list of vmstorages>` with addresses of previously configured `vmstorage` services.
|
||||
To specify multiple addresses you can repeat the flag multiple times, or separate addresses with commas
|
||||
in one flag. See more details in `-storageNode` flag description [here](https://docs.victoriametrics.com/cluster-victoriametrics/#list-of-command-line-flags-for-vminsert).
|
||||
in one flag. See more details in `-storageNode` flag description [here](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#list-of-command-line-flags-for-vminsert).
|
||||
|
||||
> Please note, `vmselect` service is listening on `:8481` for HTTP connections (see `-httpListenAddr` flag).
|
||||
|
||||
@@ -351,48 +354,48 @@ sudo systemctl status vmselect.service
|
||||
```
|
||||
|
||||
5. After `vmselect` is in `Running` state, confirm the service is healthy by visiting `http://<ip_or_hostname>:8481/select/0/vmui` link.
|
||||
It should open [vmui](https://docs.victoriametrics.com/#vmui) page.
|
||||
It should open [vmui](https://docs.victoriametrics.com/victoriametrics#vmui) page.
|
||||
|
||||
## Write data
|
||||
|
||||
There are two main models in monitoring for data collection: [push](https://docs.victoriametrics.com/keyconcepts/#push-model)
|
||||
and [pull](https://docs.victoriametrics.com/keyconcepts/#pull-model). Both are used in modern monitoring and both are
|
||||
There are two main models in monitoring for data collection: [push](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#push-model)
|
||||
and [pull](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#pull-model). Both are used in modern monitoring and both are
|
||||
supported by VictoriaMetrics.
|
||||
|
||||
See more details on [writing data here](https://docs.victoriametrics.com/keyconcepts/#write-data).
|
||||
See documentation for configuring [metrics collectors](https://docs.victoriametrics.com/data-ingestion/).
|
||||
See more details on [writing data here](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#write-data).
|
||||
See documentation for configuring [metrics collectors](https://docs.victoriametrics.com/victoriametrics/data-ingestion/).
|
||||
|
||||
## Query data
|
||||
|
||||
VictoriaMetrics has built-in [vmui](https://docs.victoriametrics.com/single-server-victoriametrics/#vmui) - graphical
|
||||
User Interface for querying and visualizing metrics. [MetricsQL](https://docs.victoriametrics.com/metricsql/) - is the
|
||||
query language for executing read queries in VictoriaMetrics. See examples of MetricsQL queries [here](https://docs.victoriametrics.com/keyconcepts/#metricsql).
|
||||
VictoriaMetrics has built-in [vmui](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#vmui) - graphical
|
||||
User Interface for querying and visualizing metrics. [MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/) - is the
|
||||
query language for executing read queries in VictoriaMetrics. See examples of MetricsQL queries [here](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#metricsql).
|
||||
|
||||
VictoriaMetrics provides an [HTTP API](https://docs.victoriametrics.com/single-server-victoriametrics/#prometheus-querying-api-usage)
|
||||
for serving read queries. The API is used in various integrations such as [Grafana](https://docs.victoriametrics.com/single-server-victoriametrics/#grafana-setup).
|
||||
VictoriaMetrics provides an [HTTP API](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#prometheus-querying-api-usage)
|
||||
for serving read queries. The API is used in various integrations such as [Grafana](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#grafana-setup).
|
||||
|
||||
See more details on [querying data here](https://docs.victoriametrics.com/keyconcepts/#query-data).
|
||||
See more details on [querying data here](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#query-data).
|
||||
|
||||
## Alerting
|
||||
|
||||
To run periodic conditions checks use [vmalert](https://docs.victoriametrics.com/vmalert/).
|
||||
To run periodic conditions checks use [vmalert](https://docs.victoriametrics.com/victoriametrics/vmalert/).
|
||||
It allows creating set of conditions using MetricsQL expressions and send notifications to [Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/)
|
||||
when such conditions are met.
|
||||
|
||||
See [vmalert quick start](https://docs.victoriametrics.com/vmalert/#quickstart).
|
||||
See [vmalert quick start](https://docs.victoriametrics.com/victoriametrics/vmalert/#quickstart).
|
||||
|
||||
## Data migration
|
||||
|
||||
Migrating data from other TSDBs to VictoriaMetrics is as simple as importing data via any of
|
||||
[supported formats](https://docs.victoriametrics.com/keyconcepts/#push-model).
|
||||
[supported formats](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#push-model).
|
||||
|
||||
The migration might get easier when using [vmctl](https://docs.victoriametrics.com/vmctl/) - VictoriaMetrics
|
||||
The migration might get easier when using [vmctl](https://docs.victoriametrics.com/victoriametrics/vmctl/) - VictoriaMetrics
|
||||
command line tool. It supports the following databases for migration to VictoriaMetrics:
|
||||
* [Prometheus using snapshot API](https://docs.victoriametrics.com/vmctl/#migrating-data-from-prometheus);
|
||||
* [Thanos](https://docs.victoriametrics.com/vmctl/#migrating-data-from-thanos);
|
||||
* [InfluxDB](https://docs.victoriametrics.com/vmctl/#migrating-data-from-influxdb-1x);
|
||||
* [OpenTSDB](https://docs.victoriametrics.com/vmctl/#migrating-data-from-opentsdb);
|
||||
* [Migrate data between VictoriaMetrics single and cluster versions](https://docs.victoriametrics.com/vmctl/#migrating-data-from-victoriametrics).
|
||||
* [Prometheus using snapshot API](https://docs.victoriametrics.com/victoriametrics/vmctl/#migrating-data-from-prometheus);
|
||||
* [Thanos](https://docs.victoriametrics.com/victoriametrics/vmctl/#migrating-data-from-thanos);
|
||||
* [InfluxDB](https://docs.victoriametrics.com/victoriametrics/vmctl/#migrating-data-from-influxdb-1x);
|
||||
* [OpenTSDB](https://docs.victoriametrics.com/victoriametrics/vmctl/#migrating-data-from-opentsdb);
|
||||
* [Migrate data between VictoriaMetrics single and cluster versions](https://docs.victoriametrics.com/victoriametrics/vmctl/#migrating-data-from-victoriametrics).
|
||||
|
||||
## Productionization
|
||||
|
||||
@@ -403,7 +406,7 @@ When going to production with VictoriaMetrics we recommend following the recomme
|
||||
Each VictoriaMetrics component emits its own metrics with various details regarding performance
|
||||
and health state. Docs for the components also contain a `Monitoring` section with an explanation
|
||||
of what and how should be monitored. For example,
|
||||
[Single-server-VictoriaMetrics Monitoring](https://docs.victoriametrics.com/cluster-victoriametrics/#monitoring).
|
||||
[Single-server-VictoriaMetrics Monitoring](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#monitoring).
|
||||
|
||||
VictoriaMetrics has a list of [Grafana dashboards](https://grafana.com/orgs/victoriametrics/dashboards).
|
||||
Each dashboard contains a lot of useful information and tips. It is recommended to have these dashboards installed and up to date.
|
||||
@@ -420,30 +423,30 @@ See more details in the article [VictoriaMetrics Monitoring](https://victoriamet
|
||||
### Capacity planning
|
||||
|
||||
See capacity planning sections in [docs](https://docs.victoriametrics.com) for
|
||||
[Single-server-VictoriaMetrics](https://docs.victoriametrics.com/single-server-victoriametrics/#capacity-planning)
|
||||
and [VictoriaMetrics Cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#capacity-planning).
|
||||
[Single-server-VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#capacity-planning)
|
||||
and [VictoriaMetrics Cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#capacity-planning).
|
||||
|
||||
Capacity planning isn't possible without [monitoring](#monitoring), so consider configuring it first.
|
||||
Understanding resource usage and performance of VictoriaMetrics also requires knowing the tech terms
|
||||
[active series](https://docs.victoriametrics.com/faq/#what-is-an-active-time-series),
|
||||
[churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate),
|
||||
[cardinality](https://docs.victoriametrics.com/faq/#what-is-high-cardinality),
|
||||
[slow inserts](https://docs.victoriametrics.com/faq/#what-is-a-slow-insert).
|
||||
[active series](https://docs.victoriametrics.com/victoriametrics/faq/#what-is-an-active-time-series),
|
||||
[churn rate](https://docs.victoriametrics.com/victoriametrics/faq/#what-is-high-churn-rate),
|
||||
[cardinality](https://docs.victoriametrics.com/victoriametrics/faq/#what-is-high-cardinality),
|
||||
[slow inserts](https://docs.victoriametrics.com/victoriametrics/faq/#what-is-a-slow-insert).
|
||||
All of them are present in [Grafana dashboards](https://grafana.com/orgs/victoriametrics/dashboards).
|
||||
|
||||
### Data safety
|
||||
|
||||
It is recommended to read [Replication and data safety](https://docs.victoriametrics.com/cluster-victoriametrics/#replication-and-data-safety),
|
||||
It is recommended to read [Replication and data safety](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#replication-and-data-safety),
|
||||
[Why replication doesn’t save from disaster?](https://valyala.medium.com/speeding-up-backups-for-big-time-series-databases-533c1a927883)
|
||||
and [backups](https://docs.victoriametrics.com/single-server-victoriametrics/#backups).
|
||||
and [backups](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#backups).
|
||||
|
||||
### Configuring limits
|
||||
|
||||
To avoid excessive resource usage or performance degradation limits must be in place:
|
||||
* [Resource usage limits](https://docs.victoriametrics.com/faq/#how-to-set-a-memory-limit-for-victoriametrics-components);
|
||||
* [Cardinality limiter](https://docs.victoriametrics.com/single-server-victoriametrics/#cardinality-limiter).
|
||||
* [Resource usage limits](https://docs.victoriametrics.com/victoriametrics/faq/#how-to-set-a-memory-limit-for-victoriametrics-components);
|
||||
* [Cardinality limiter](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#cardinality-limiter).
|
||||
|
||||
### Security recommendations
|
||||
|
||||
* [Security recommendations for single-node VictoriaMetrics](https://docs.victoriametrics.com/#security)
|
||||
* [Security recommendations for cluster version of VictoriaMetrics](https://docs.victoriametrics.com/cluster-victoriametrics/#security)
|
||||
* [Security recommendations for single-node VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics#security)
|
||||
* [Security recommendations for cluster version of VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#security)
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -6,6 +6,8 @@ menu:
|
||||
parent: 'victoriametrics'
|
||||
weight: 501
|
||||
aliases:
|
||||
- /release-guide
|
||||
- /Release-Guide
|
||||
- /Release-Guide.html
|
||||
- /release-guide/index.html
|
||||
- /release-guide/
|
||||
@@ -52,7 +54,7 @@ Bumping the limits may significantly improve build speed.
|
||||
1. Run `PKG_TAG=v1.xx.y make docs-update-version` command to update version help tooltips.
|
||||
1. Cut new version in [CHANGELOG.md](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/docs/changelog/CHANGELOG.md)
|
||||
and make it merged. See example in this [commit](https://github.com/VictoriaMetrics/VictoriaMetrics/commit/b771152039d23b5ccd637a23ea748bc44a9511a7).
|
||||
1. Cherry-pick bug fixes relevant for [LTS releases](https://docs.victoriametrics.com/lts-releases/).
|
||||
1. Cherry-pick bug fixes relevant for [LTS releases](https://docs.victoriametrics.com/victoriametrics/lts-releases/).
|
||||
1. Make sure you get all changes fetched `git fetch --all`.
|
||||
1. Create the following release tags:
|
||||
* `git tag -s v1.xx.y` in `master` branch
|
||||
@@ -105,7 +107,7 @@ Bumping the limits may significantly improve build speed.
|
||||
|
||||
* Publish message in Slack at <https://victoriametrics.slack.com>
|
||||
* Post at X (Twitter) at <https://x.com/VictoriaMetrics>
|
||||
* Post in Reddit at <https://www.reddit.com/r/VictoriaMetrics/>
|
||||
* Post in Reddit at <https://reddit.com/r/VictoriaMetrics/>
|
||||
* Post in LinkedIn at <https://www.linkedin.com/company/victoriametrics/>
|
||||
* Publish message in Telegram at <https://t.me/VictoriaMetrics_en> and <https://t.me/VictoriaMetrics_ru1>
|
||||
|
||||
|
||||
@@ -7,6 +7,8 @@ menu:
|
||||
weight: 1
|
||||
title: Single-node version
|
||||
aliases:
|
||||
- /single-server-victoriametrics
|
||||
- /Single-server-VictoriaMetrics
|
||||
- /Single-server-VictoriaMetrics.html
|
||||
- /single-server-victoriametrics/index.html
|
||||
- /single-server-victoriametrics/
|
||||
|
||||
@@ -6,6 +6,8 @@ menu:
|
||||
parent: 'victoriametrics'
|
||||
weight: 35
|
||||
aliases:
|
||||
- /troubleshooting
|
||||
- /Troubleshooting
|
||||
- /Troubleshooting.html
|
||||
- /troubleshooting/index.html
|
||||
- /troubleshooting/
|
||||
@@ -27,21 +29,21 @@ If you hit some issue or have some question about VictoriaMetrics components,
|
||||
then please follow the following steps in order to quickly find the solution:
|
||||
|
||||
1. Check the version of VictoriaMetrics component, which needs to be troubleshot and compare
|
||||
it to [the latest available version](https://docs.victoriametrics.com/changelog/).
|
||||
it to [the latest available version](https://docs.victoriametrics.com/victoriametrics/changelog/).
|
||||
If the used version is lower than the latest available version, then there are high chances
|
||||
that the issue is already resolved in newer versions. Carefully read [the changelog](https://docs.victoriametrics.com/changelog/)
|
||||
that the issue is already resolved in newer versions. Carefully read [the changelog](https://docs.victoriametrics.com/victoriametrics/changelog/)
|
||||
between your version and the latest version and check whether the issue is already fixed there.
|
||||
|
||||
If the issue is already fixed in newer versions, then upgrade to the newer version and verify whether the issue is fixed:
|
||||
|
||||
- [How to upgrade single-node VictoriaMetrics](https://docs.victoriametrics.com/#how-to-upgrade-victoriametrics)
|
||||
- [How to upgrade VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#updating--reconfiguring-cluster-nodes)
|
||||
- [How to upgrade single-node VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics#how-to-upgrade-victoriametrics)
|
||||
- [How to upgrade VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#updating--reconfiguring-cluster-nodes)
|
||||
|
||||
Upgrade procedure for other VictoriaMetrics components is as simple as gracefully stopping the component
|
||||
by sending `SIGINT` signal to it and starting the new version of the component.
|
||||
|
||||
There may be breaking changes between different versions of VictoriaMetrics components in rare cases.
|
||||
These cases are documented in [the changelog](https://docs.victoriametrics.com/changelog/).
|
||||
These cases are documented in [the changelog](https://docs.victoriametrics.com/victoriametrics/changelog/).
|
||||
So please read the changelog before the upgrade.
|
||||
|
||||
1. Inspect command-line flags passed to VictoriaMetrics components and remove flags which unclear outcomes for your workload.
|
||||
@@ -122,12 +124,12 @@ If you see unexpected or unreliable query results from VictoriaMetrics, then try
|
||||
to reduce the number of returned series.
|
||||
|
||||
Sometimes the query may be improperly constructed, so it returns unexpected results.
|
||||
It is recommended reading and understanding [MetricsQL docs](https://docs.victoriametrics.com/metricsql/),
|
||||
especially [subqueries](https://docs.victoriametrics.com/metricsql/#subqueries)
|
||||
and [rollup functions](https://docs.victoriametrics.com/metricsql/#rollup-functions) sections.
|
||||
It is recommended reading and understanding [MetricsQL docs](https://docs.victoriametrics.com/victoriametrics/metricsql/),
|
||||
especially [subqueries](https://docs.victoriametrics.com/victoriametrics/metricsql/#subqueries)
|
||||
and [rollup functions](https://docs.victoriametrics.com/victoriametrics/metricsql/#rollup-functions) sections.
|
||||
|
||||
1. If the simplest query continues returning unexpected / unreliable results, then try verifying correctness
|
||||
of raw unprocessed samples for this query via [/api/v1/export](https://docs.victoriametrics.com/#how-to-export-data-in-json-line-format)
|
||||
of raw unprocessed samples for this query via [/api/v1/export](https://docs.victoriametrics.com/victoriametrics#how-to-export-data-in-json-line-format)
|
||||
on the given `[start..end]` time range and check whether they are expected:
|
||||
|
||||
```sh
|
||||
@@ -135,16 +137,16 @@ If you see unexpected or unreliable query results from VictoriaMetrics, then try
|
||||
|
||||
cluster: curl http://<vmselect>:8481/select/<tenantID>/prometheus/api/v1/export -d 'match[]=http_requests_total' -d 'start=...' -d 'end=...' -d 'reduce_mem_usage=1'
|
||||
```
|
||||
Note that responses returned from [/api/v1/query](https://docs.victoriametrics.com/keyconcepts/#instant-query)
|
||||
and from [/api/v1/query_range](https://docs.victoriametrics.com/keyconcepts/#range-query) contain **evaluated** data
|
||||
Note that responses returned from [/api/v1/query](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#instant-query)
|
||||
and from [/api/v1/query_range](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#range-query) contain **evaluated** data
|
||||
instead of raw samples stored in VictoriaMetrics. See [these docs](https://prometheus.io/docs/prometheus/latest/querying/basics/#staleness)
|
||||
for details. The raw samples can be also viewed in [vmui](https://docs.victoriametrics.com/#vmui) in `Raw Query` tab and shared via `export` button.
|
||||
for details. The raw samples can be also viewed in [vmui](https://docs.victoriametrics.com/victoriametrics#vmui) in `Raw Query` tab and shared via `export` button.
|
||||
|
||||
If you migrate from InfluxDB, then pass `-search.setLookbackToStep` command-line flag to single-node VictoriaMetrics
|
||||
or to `vmselect` in VictoriaMetrics cluster. See also [how to migrate from InfluxDB to VictoriaMetrics](https://docs.victoriametrics.com/guides/migrate-from-influx.html).
|
||||
|
||||
1. Sometimes response caching may lead to unexpected results when samples with older timestamps
|
||||
are ingested into VictoriaMetrics (aka [backfilling](https://docs.victoriametrics.com/#backfilling)).
|
||||
are ingested into VictoriaMetrics (aka [backfilling](https://docs.victoriametrics.com/victoriametrics#backfilling)).
|
||||
Try disabling response cache and see whether this helps. This can be done in the following ways:
|
||||
|
||||
- By passing `-search.disableCache` command-line flag to a single-node VictoriaMetrics
|
||||
@@ -154,10 +156,10 @@ If you see unexpected or unreliable query results from VictoriaMetrics, then try
|
||||
If you use Grafana, then this query arg can be specified in `Custom Query Parameters` field
|
||||
at Prometheus datasource settings - see [these docs](https://grafana.com/docs/grafana/latest/datasources/prometheus/) for details.
|
||||
|
||||
If the problem was in the cache, try resetting it via [resetRollupCache handler](https://docs.victoriametrics.com/url-examples/#internalresetrollupresultcache).
|
||||
If the problem was in the cache, try resetting it via [resetRollupCache handler](https://docs.victoriametrics.com/victoriametrics/url-examples/#internalresetrollupresultcache).
|
||||
|
||||
1. If you use cluster version of VictoriaMetrics, then it may return partial responses by default
|
||||
when some of `vmstorage` nodes are temporarily unavailable - see [cluster availability docs](https://docs.victoriametrics.com/cluster-victoriametrics/#cluster-availability)
|
||||
when some of `vmstorage` nodes are temporarily unavailable - see [cluster availability docs](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#cluster-availability)
|
||||
for details. If you want to prioritize query consistency over cluster availability,
|
||||
then you can pass `-search.denyPartialResponse` command-line flag to all the `vmselect` nodes.
|
||||
In this case VictoriaMetrics returns an error during querying if at least a single `vmstorage` node is unavailable.
|
||||
@@ -172,20 +174,20 @@ If you see unexpected or unreliable query results from VictoriaMetrics, then try
|
||||
1. If you observe gaps when plotting time series try simplifying your query according to p2 and follow the list.
|
||||
If problem still remains, then it is likely caused by irregular intervals for metrics collection (network delays
|
||||
or targets unavailability on scrapes, irregular pushes, irregular timestamps).
|
||||
VictoriaMetrics automatically [fills the gaps](https://docs.victoriametrics.com/keyconcepts/#range-query)
|
||||
based on median interval between [data samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples).
|
||||
VictoriaMetrics automatically [fills the gaps](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#range-query)
|
||||
based on median interval between [data samples](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#raw-samples).
|
||||
This might work incorrect for irregular data as median will be skewed. In this case it is recommended to switch
|
||||
to the static interval for gaps filling by setting `-search.minStalenessInterval=5m` cmd-line flag (`5m` is
|
||||
the static interval used by Prometheus).
|
||||
|
||||
1. If you observe recently written data is not immediately visible/queryable, then read more about
|
||||
[query latency](https://docs.victoriametrics.com/keyconcepts/#query-latency) behavior.
|
||||
[query latency](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#query-latency) behavior.
|
||||
|
||||
1. Try upgrading to the [latest available version of VictoriaMetrics](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest)
|
||||
and verifying whether the issue is fixed there.
|
||||
|
||||
1. Try executing the query with `trace=1` query arg. This enables query tracing, which may contain
|
||||
useful information on why the query returns unexpected data. See [query tracing docs](https://docs.victoriametrics.com/#query-tracing) for details.
|
||||
useful information on why the query returns unexpected data. See [query tracing docs](https://docs.victoriametrics.com/victoriametrics#query-tracing) for details.
|
||||
|
||||
1. Inspect command-line flags passed to VictoriaMetrics components. If you don't understand clearly the purpose
|
||||
or the effect of some flags, then remove them from the list of flags passed to VictoriaMetrics components,
|
||||
@@ -194,15 +196,15 @@ If you see unexpected or unreliable query results from VictoriaMetrics, then try
|
||||
|
||||
1. If the steps above didn't help identifying the root cause of unexpected query results,
|
||||
then [file a bugreport](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/new) with details on how to reproduce the issue.
|
||||
Instead of sharing screenshots in the issue, consider sharing query and [trace](https://docs.victoriametrics.com/#query-tracing)
|
||||
results in [VMUI](https://docs.victoriametrics.com/#vmui) by clicking on `Export query` button in top right corner of the graph area.
|
||||
Instead of sharing screenshots in the issue, consider sharing query and [trace](https://docs.victoriametrics.com/victoriametrics#query-tracing)
|
||||
results in [VMUI](https://docs.victoriametrics.com/victoriametrics#vmui) by clicking on `Export query` button in top right corner of the graph area.
|
||||
|
||||
|
||||
## Slow data ingestion
|
||||
|
||||
There are the following most commons reasons for slow data ingestion in VictoriaMetrics:
|
||||
|
||||
1. Memory shortage for the given amounts of [active time series](https://docs.victoriametrics.com/faq/#what-is-an-active-time-series).
|
||||
1. Memory shortage for the given amounts of [active time series](https://docs.victoriametrics.com/victoriametrics/faq/#what-is-an-active-time-series).
|
||||
|
||||
VictoriaMetrics (or `vmstorage` in cluster version of VictoriaMetrics) maintains an in-memory cache
|
||||
for quick search for internal series ids per each incoming metric.
|
||||
@@ -211,10 +213,10 @@ There are the following most commons reasons for slow data ingestion in Victoria
|
||||
for holding all the entries for active time series, then VictoriaMetrics locates the needed data on disk,
|
||||
unpacks it, re-constructs the missing entry and puts it into the cache. This takes additional CPU time and disk read IO.
|
||||
|
||||
The [official Grafana dashboards for VictoriaMetrics](https://docs.victoriametrics.com/#monitoring)
|
||||
The [official Grafana dashboards for VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics#monitoring)
|
||||
contain `Slow inserts` graph, which shows the cache miss percentage for `storage/tsid` cache
|
||||
during data ingestion. If `slow inserts` graph shows values greater than 5% for more than 10 minutes,
|
||||
then it is likely the current number of [active time series](https://docs.victoriametrics.com/faq/#what-is-an-active-time-series)
|
||||
then it is likely the current number of [active time series](https://docs.victoriametrics.com/victoriametrics/faq/#what-is-an-active-time-series)
|
||||
cannot fit the `storage/tsid` cache.
|
||||
|
||||
There are the following solutions exist for this issue:
|
||||
@@ -224,28 +226,28 @@ There are the following most commons reasons for slow data ingestion in Victoria
|
||||
memory at `vmstorage` nodes. This can be done in two ways: either to increase the available memory
|
||||
per each existing `vmstorage` node or to add more `vmstorage` nodes to the cluster.
|
||||
|
||||
- To reduce the number of active time series. The [official Grafana dashboards for VictoriaMetrics](https://docs.victoriametrics.com/#monitoring)
|
||||
- To reduce the number of active time series. The [official Grafana dashboards for VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics#monitoring)
|
||||
contain a graph showing the number of active time series. Recent versions of VictoriaMetrics
|
||||
provide [cardinality explorer](https://docs.victoriametrics.com/#cardinality-explorer),
|
||||
which can help determining and fixing the source of [high cardinality](https://docs.victoriametrics.com/faq/#what-is-high-cardinality).
|
||||
provide [cardinality explorer](https://docs.victoriametrics.com/victoriametrics#cardinality-explorer),
|
||||
which can help determining and fixing the source of [high cardinality](https://docs.victoriametrics.com/victoriametrics/faq/#what-is-high-cardinality).
|
||||
|
||||
1. [High churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate),
|
||||
1. [High churn rate](https://docs.victoriametrics.com/victoriametrics/faq/#what-is-high-churn-rate),
|
||||
e.g. when old time series are substituted with new time series at a high rate.
|
||||
When VictoriaMetrics encounters a sample for new time series, it needs to register the time series
|
||||
in the internal index (aka `indexdb`), so it can be quickly located on subsequent select queries.
|
||||
The process of registering new time series in the internal index is an order of magnitude slower
|
||||
than the process of adding new sample to already registered time series.
|
||||
So VictoriaMetrics may work slower than expected under [high churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate).
|
||||
So VictoriaMetrics may work slower than expected under [high churn rate](https://docs.victoriametrics.com/victoriametrics/faq/#what-is-high-churn-rate).
|
||||
|
||||
The [official Grafana dashboards for VictoriaMetrics](https://docs.victoriametrics.com/#monitoring)
|
||||
The [official Grafana dashboards for VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics#monitoring)
|
||||
provides `Churn rate` graph, which shows the average number of new time series registered
|
||||
during the last 24 hours. If this number exceeds the number of [active time series](https://docs.victoriametrics.com/faq/#what-is-an-active-time-series),
|
||||
then you need to identify and fix the source of [high churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate).
|
||||
during the last 24 hours. If this number exceeds the number of [active time series](https://docs.victoriametrics.com/victoriametrics/faq/#what-is-an-active-time-series),
|
||||
then you need to identify and fix the source of [high churn rate](https://docs.victoriametrics.com/victoriametrics/faq/#what-is-high-churn-rate).
|
||||
The most commons source of high churn rate is a label, which frequently changes its value. Try avoiding such labels.
|
||||
The [cardinality explorer](https://docs.victoriametrics.com/#cardinality-explorer) can help identifying
|
||||
The [cardinality explorer](https://docs.victoriametrics.com/victoriametrics#cardinality-explorer) can help identifying
|
||||
such labels.
|
||||
|
||||
1. Resource shortage. The [official Grafana dashboards for VictoriaMetrics](https://docs.victoriametrics.com/#monitoring)
|
||||
1. Resource shortage. The [official Grafana dashboards for VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics#monitoring)
|
||||
contain `resource usage` graphs, which show memory usage, CPU usage, disk IO usage and free disk size.
|
||||
Make sure VictoriaMetrics has enough free resources for graceful handling of potential spikes in workload
|
||||
according to the following recommendations:
|
||||
@@ -271,7 +273,7 @@ There are the following most commons reasons for slow data ingestion in Victoria
|
||||
|
||||
- If free disk space is lower than 20%, then VictoriaMetrics is unable to perform optimal
|
||||
background merge of the incoming data. This leads to increased number of data files on disk,
|
||||
which, in turn, slows down both data ingestion and querying. See [these docs](https://docs.victoriametrics.com/#storage) for details.
|
||||
which, in turn, slows down both data ingestion and querying. See [these docs](https://docs.victoriametrics.com/victoriametrics#storage) for details.
|
||||
|
||||
1. If you run cluster version of VictoriaMetrics, then make sure `vminsert` and `vmstorage` components
|
||||
are located in the same network with small network latency between them.
|
||||
@@ -280,7 +282,7 @@ There are the following most commons reasons for slow data ingestion in Victoria
|
||||
If the network latency between `vminsert` and `vmstorage` is high (for example, if they run in different datacenters),
|
||||
then this may become limiting factor for data ingestion speed.
|
||||
|
||||
The [official Grafana dashboard for cluster version of VictoriaMetrics](https://docs.victoriametrics.com/cluster-victoriametrics/#monitoring)
|
||||
The [official Grafana dashboard for cluster version of VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#monitoring)
|
||||
contain `connection saturation` graph for `vminsert` components. If this graph reaches 100% (1s),
|
||||
then it is likely you have issues with network latency between `vminsert` and `vmstorage`.
|
||||
Another possible issue for 100% connection saturation between `vminsert` and `vmstorage`
|
||||
@@ -289,10 +291,10 @@ There are the following most commons reasons for slow data ingestion in Victoria
|
||||
|
||||
1. Noisy neighbor. Make sure VictoriaMetrics components run in an environments without other resource-hungry apps.
|
||||
Such apps may steal RAM, CPU, disk IO and network bandwidth, which is needed for VictoriaMetrics components.
|
||||
Issues like this are very hard to catch via [official Grafana dashboard for cluster version of VictoriaMetrics](https://docs.victoriametrics.com/cluster-victoriametrics/#monitoring)
|
||||
Issues like this are very hard to catch via [official Grafana dashboard for cluster version of VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#monitoring)
|
||||
and proper diagnosis would require checking resource usage on the instances where VictoriaMetrics runs.
|
||||
|
||||
1. If you see `TooHighSlowInsertsRate` [alert](https://docs.victoriametrics.com/#monitoring) when single-node VictoriaMetrics or `vmstorage` has enough
|
||||
1. If you see `TooHighSlowInsertsRate` [alert](https://docs.victoriametrics.com/victoriametrics#monitoring) when single-node VictoriaMetrics or `vmstorage` has enough
|
||||
free CPU and RAM, then increase `-cacheExpireDuration` command-line flag at single-node VictoriaMetrics or at `vmstorage` to the value,
|
||||
which exceeds the interval between ingested samples for the same time series (aka `scrape_interval`).
|
||||
See [this comment](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3976#issuecomment-1476883183) for more details.
|
||||
@@ -309,7 +311,7 @@ Some queries may take more time and resources (CPU, RAM, network bandwidth) than
|
||||
VictoriaMetrics logs slow queries if their execution time exceeds the duration passed
|
||||
to `-search.logSlowQueryDuration` command-line flag (5s by default).
|
||||
|
||||
VictoriaMetrics provides [`top queries` page at VMUI](https://docs.victoriametrics.com/#top-queries), which shows
|
||||
VictoriaMetrics provides [`top queries` page at VMUI](https://docs.victoriametrics.com/victoriametrics#top-queries), which shows
|
||||
queries that took the most time to execute.
|
||||
|
||||
There are the following solutions exist for improving performance of slow queries:
|
||||
@@ -319,33 +321,33 @@ There are the following solutions exist for improving performance of slow querie
|
||||
with more CPU and RAM should help improving speed for slow queries. Query performance
|
||||
is always limited by resources of one `vmselect` which processes the query. For example, if 2vCPU cores on `vmselect`
|
||||
isn't enough to process query fast enough, then migrating `vmselect` to a machine with 4vCPU cores should increase heavy query performance by up to 2x.
|
||||
If the line on `concurrent select` graph form the [official Grafana dashboard for VictoriaMetrics](https://docs.victoriametrics.com/cluster-victoriametrics/#monitoring)
|
||||
If the line on `concurrent select` graph form the [official Grafana dashboard for VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#monitoring)
|
||||
is close to the limit, then prefer adding more `vmselect` nodes to the cluster.
|
||||
Sometimes adding more `vmstorage` nodes also can help improving the speed for slow queries.
|
||||
|
||||
- Rewriting slow queries, so they become faster. Unfortunately it is hard determining
|
||||
whether the given query is slow by just looking at it.
|
||||
|
||||
The main source of slow queries in practice is [alerting and recording rules](https://docs.victoriametrics.com/vmalert/#rules)
|
||||
The main source of slow queries in practice is [alerting and recording rules](https://docs.victoriametrics.com/victoriametrics/vmalert/#rules)
|
||||
with long lookbehind windows in square brackets. These queries are frequently used in SLI/SLO calculations such as [Sloth](https://github.com/slok/sloth).
|
||||
|
||||
For example, `avg_over_time(up[30d]) > 0.99` needs to read and process
|
||||
all the [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples)
|
||||
for `up` [time series](https://docs.victoriametrics.com/keyconcepts/#time-series) over the last 30 days
|
||||
all the [raw samples](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#raw-samples)
|
||||
for `up` [time series](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#time-series) over the last 30 days
|
||||
each time it executes. If this query is executed frequently, then it can take significant share of CPU, disk read IO, network bandwidth and RAM.
|
||||
Such queries can be optimized in the following ways:
|
||||
|
||||
- To reduce the lookbehind window in square brackets. For example, `avg_over_time(up[10d])` takes up to 3x less compute resources
|
||||
than `avg_over_time(up[30d])` at VictoriaMetrics.
|
||||
- To increase evaluation interval for alerting and recording rules, so they are executed less frequently.
|
||||
For example, increasing `-evaluationInterval` command-line flag value at [vmalert](https://docs.victoriametrics.com/vmalert/)
|
||||
For example, increasing `-evaluationInterval` command-line flag value at [vmalert](https://docs.victoriametrics.com/victoriametrics/vmalert/)
|
||||
from `1m` to `2m` should reduce compute resource usage at VictoriaMetrics by 2x.
|
||||
|
||||
Another source of slow queries is improper use of [subqueries](https://docs.victoriametrics.com/metricsql/#subqueries).
|
||||
Another source of slow queries is improper use of [subqueries](https://docs.victoriametrics.com/victoriametrics/metricsql/#subqueries).
|
||||
It is recommended avoiding subqueries if you don't understand clearly how they work.
|
||||
It is easy to create a subquery without knowing about it.
|
||||
For example, `rate(sum(some_metric))` is implicitly transformed into the following subquery
|
||||
according to [implicit conversion rules for MetricsQL queries](https://docs.victoriametrics.com/metricsql/#implicit-query-conversions):
|
||||
according to [implicit conversion rules for MetricsQL queries](https://docs.victoriametrics.com/victoriametrics/metricsql/#implicit-query-conversions):
|
||||
|
||||
```metricsql
|
||||
rate(
|
||||
@@ -358,7 +360,7 @@ There are the following solutions exist for improving performance of slow querie
|
||||
It is likely this query won't return the expected results. Instead, `sum(rate(some_metric))` must be used instead.
|
||||
See [this article](https://www.robustperception.io/rate-then-sum-never-sum-then-rate/) for more details.
|
||||
|
||||
VictoriaMetrics provides [query tracing](https://docs.victoriametrics.com/#query-tracing) feature,
|
||||
VictoriaMetrics provides [query tracing](https://docs.victoriametrics.com/victoriametrics#query-tracing) feature,
|
||||
which can help determining the source of slow query.
|
||||
See also [this article](https://valyala.medium.com/how-to-optimize-promql-and-metricsql-queries-85a1b75bf986),
|
||||
which explains how to determine and optimize slow queries.
|
||||
@@ -375,24 +377,24 @@ There are the following most common sources of out of memory (aka OOM) crashes i
|
||||
VictoriaMetrics is optimized for running with default flag values (e.g. when they aren't set explicitly).
|
||||
|
||||
For example, it isn't recommended tuning cache sizes in VictoriaMetrics, since it frequently leads to OOM exceptions.
|
||||
[These docs](https://docs.victoriametrics.com/#cache-tuning) refer command-line flags, which aren't
|
||||
[These docs](https://docs.victoriametrics.com/victoriametrics#cache-tuning) refer command-line flags, which aren't
|
||||
recommended to tune. If you see that VictoriaMetrics needs increasing some cache sizes for the current workload,
|
||||
then it is better migrating to a host with more memory instead of trying to tune cache sizes manually.
|
||||
|
||||
1. Unexpected heavy queries. The query is considered as heavy if it needs to select and process millions of unique time series.
|
||||
Such query may lead to OOM exception, since VictoriaMetrics needs to keep some of per-series data in memory.
|
||||
VictoriaMetrics provides [various settings](https://docs.victoriametrics.com/#resource-usage-limits),
|
||||
VictoriaMetrics provides [various settings](https://docs.victoriametrics.com/victoriametrics#resource-usage-limits),
|
||||
which can help limit resource usage.
|
||||
For more context, see [How to optimize PromQL and MetricsQL queries](https://valyala.medium.com/how-to-optimize-promql-and-metricsql-queries-85a1b75bf986).
|
||||
VictoriaMetrics also provides [query tracer](https://docs.victoriametrics.com/#query-tracing)
|
||||
VictoriaMetrics also provides [query tracer](https://docs.victoriametrics.com/victoriametrics#query-tracing)
|
||||
to help identify the source of heavy query.
|
||||
|
||||
1. Lack of free memory for processing workload spikes. If VictoriaMetrics components use almost all the available memory
|
||||
under the current workload, then it is recommended migrating to a host with bigger amounts of memory.
|
||||
This would protect from possible OOM crashes on workload spikes. It is recommended to have at least 50%
|
||||
of free memory for graceful handling of possible workload spikes.
|
||||
See [capacity planning for single-node VictoriaMetrics](https://docs.victoriametrics.com/#capacity-planning)
|
||||
and [capacity planning for cluster version of VictoriaMetrics](https://docs.victoriametrics.com/cluster-victoriametrics/#capacity-planning).
|
||||
See [capacity planning for single-node VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics#capacity-planning)
|
||||
and [capacity planning for cluster version of VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#capacity-planning).
|
||||
|
||||
|
||||
## Cluster instability
|
||||
@@ -406,7 +408,7 @@ The most common sources of cluster instability are:
|
||||
the cluster has no enough free resources for processing the increased workload,
|
||||
then it may become unstable.
|
||||
VictoriaMetrics provides various configuration settings, which can be used for limiting unexpected workload spikes.
|
||||
See [these docs](https://docs.victoriametrics.com/cluster-victoriametrics/#resource-usage-limits) for details.
|
||||
See [these docs](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#resource-usage-limits) for details.
|
||||
|
||||
- Various maintenance tasks such as rolling upgrades or rolling restarts during configuration changes.
|
||||
For example, if a cluster contains `N=3` `vmstorage` nodes and they are restarted one-by-one (aka rolling restart),
|
||||
@@ -424,10 +426,10 @@ The most common sources of cluster instability are:
|
||||
`N=11` `vmstorage` nodes, then the workload increase during rolling restart of `vmstorage` nodes
|
||||
would be `100%/(N-1)=10%`. It is recommended to have at least 8 `vmstorage` nodes in the cluster.
|
||||
The recommended number of `vmstorage` nodes should be multiplied by `-replicationFactor` if replication is enabled -
|
||||
see [replication and data safety docs](https://docs.victoriametrics.com/cluster-victoriametrics/#replication-and-data-safety)
|
||||
see [replication and data safety docs](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#replication-and-data-safety)
|
||||
for details.
|
||||
|
||||
- Time series sharding. Received time series [are consistently sharded](https://docs.victoriametrics.com/cluster-victoriametrics/#architecture-overview)
|
||||
- Time series sharding. Received time series [are consistently sharded](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#architecture-overview)
|
||||
by `vminsert` between configured `vmstorage` nodes. As a sharding key `vminsert` is using time series name and labels,
|
||||
respecting their order. If the order of labels in time series is constantly changing, this could cause wrong sharding
|
||||
calculation and result in un-even and sub-optimal time series distribution across available vmstorages. It is expected
|
||||
@@ -437,18 +439,18 @@ The most common sources of cluster instability are:
|
||||
|
||||
The obvious solution against VictoriaMetrics cluster instability is to make sure cluster components
|
||||
have enough free resources for graceful processing of the increased workload.
|
||||
See [capacity planning docs](https://docs.victoriametrics.com/cluster-victoriametrics/#capacity-planning)
|
||||
and [cluster resizing and scalability docs](https://docs.victoriametrics.com/cluster-victoriametrics/#cluster-resizing-and-scalability)
|
||||
See [capacity planning docs](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#capacity-planning)
|
||||
and [cluster resizing and scalability docs](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#cluster-resizing-and-scalability)
|
||||
for details.
|
||||
|
||||
|
||||
## Too much disk space used
|
||||
|
||||
If too much disk space is used by a [single-node VictoriaMetrics](https://docs.victoriametrics.com/) or by `vmstorage` component
|
||||
at [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/), then please check the following:
|
||||
at [VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/), then please check the following:
|
||||
|
||||
- Make sure that there are no old snapsots, since they can occupy disk space. See [how to work with snapshots](https://docs.victoriametrics.com/#how-to-work-with-snapshots)
|
||||
and [snapshot troubleshooting](https://docs.victoriametrics.com/#snapshot-troubleshooting).
|
||||
- Make sure that there are no old snapsots, since they can occupy disk space. See [how to work with snapshots](https://docs.victoriametrics.com/victoriametrics#how-to-work-with-snapshots)
|
||||
and [snapshot troubleshooting](https://docs.victoriametrics.com/victoriametrics#snapshot-troubleshooting).
|
||||
|
||||
- Under normal conditions the size of `<-storageDataPath>/indexdb` folder must be smaller than the size of `<-storageDataPath>/data` folder, where `-storageDataPath`
|
||||
is the corresponding command-line flag value. This can be checked by the following query if [VictoriaMetrics monitoring](#monitoring) is properly set up:
|
||||
@@ -457,14 +459,14 @@ at [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriame
|
||||
/
|
||||
sum(vm_data_size_bytes{type=~"(storage|indexdb)/.+"}) without(type)
|
||||
```
|
||||
If this query returns values bigger than 0.5, then it is likely there is a [high churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate) issue,
|
||||
If this query returns values bigger than 0.5, then it is likely there is a [high churn rate](https://docs.victoriametrics.com/victoriametrics/faq/#what-is-high-churn-rate) issue,
|
||||
which results in excess disk space usage for both `indexdb` and `data` folders under `-storageDataPath` folder.
|
||||
The solution is to identify and fix the source of high churn rate with [cardinality explorer](https://docs.victoriametrics.com/#cardinality-explorer).
|
||||
The solution is to identify and fix the source of high churn rate with [cardinality explorer](https://docs.victoriametrics.com/victoriametrics#cardinality-explorer).
|
||||
|
||||
|
||||
## Monitoring
|
||||
|
||||
Having proper [monitoring](https://docs.victoriametrics.com/single-server-victoriametrics/#monitoring)
|
||||
Having proper [monitoring](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#monitoring)
|
||||
would help identify and prevent most of the issues listed above.
|
||||
|
||||
[Grafana dashboards](https://grafana.com/orgs/victoriametrics/dashboards) contain panels reflecting the
|
||||
|
||||
@@ -4,6 +4,5 @@ menu:
|
||||
docs:
|
||||
weight: 10
|
||||
identifier: victoriametrics
|
||||
pageRef: "/"
|
||||
---
|
||||
{{% content "README.md" %}}
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -9,6 +9,7 @@ menu:
|
||||
parent: vm-changelog
|
||||
weight: 7
|
||||
aliases:
|
||||
- /CHANGELOG_2020
|
||||
- /CHANGELOG_2020.html
|
||||
- /changelog_2020
|
||||
- /changelog/changelog_2020/index.html
|
||||
@@ -31,7 +32,7 @@ Released at 2020-12-27
|
||||
|
||||
Released at 2020-12-19
|
||||
|
||||
* FEATURE: do not publish duplicate Docker images with `-cluster` tag suffix for [vmagent](https://docs.victoriametrics.com/vmagent/), [vmalert](https://docs.victoriametrics.com/vmalert/), [vmauth](https://docs.victoriametrics.com/vmauth/), [vmbackup](https://docs.victoriametrics.com/vmbackup/) and [vmrestore](https://docs.victoriametrics.com/vmrestore/), since they are identical to images without `-cluster` tag suffix.
|
||||
* FEATURE: do not publish duplicate Docker images with `-cluster` tag suffix for [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/), [vmalert](https://docs.victoriametrics.com/victoriametrics/vmalert/), [vmauth](https://docs.victoriametrics.com/victoriametrics/vmauth/), [vmbackup](https://docs.victoriametrics.com/victoriametrics/vmbackup/) and [vmrestore](https://docs.victoriametrics.com/victoriametrics/vmrestore/), since they are identical to images without `-cluster` tag suffix.
|
||||
|
||||
* BUGFIX: vmalert: properly populate template variables. This has been broken in v1.50.0. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/974>
|
||||
* BUGFIX: properly parse negative combined duration in MetricsQL such as `-1h3m4s`. It must be parsed as `-(1h + 3m + 4s)`. Previously it was parsed as `-1h + 3m + 4s`.
|
||||
@@ -49,7 +50,7 @@ Released at 2020-12-15
|
||||
|
||||
Released at 2020-12-15
|
||||
|
||||
* FEATURE: automatically reset response cache when samples with timestamps older than `now - search.cacheTimestampOffset` are ingested to VictoriaMetrics. This makes unnecessary disabling response cache during data backfilling or resetting it after backfilling is complete as described [in these docs](https://docs.victoriametrics.com/#backfilling). This feature applies only to single-node VictoriaMetrics. It doesn't apply to cluster version of VictoriaMetrics because `vminsert` nodes don't know about `vmselect` nodes where the response cache must be reset.
|
||||
* FEATURE: automatically reset response cache when samples with timestamps older than `now - search.cacheTimestampOffset` are ingested to VictoriaMetrics. This makes unnecessary disabling response cache during data backfilling or resetting it after backfilling is complete as described [in these docs](https://docs.victoriametrics.com/victoriametrics#backfilling). This feature applies only to single-node VictoriaMetrics. It doesn't apply to cluster version of VictoriaMetrics because `vminsert` nodes don't know about `vmselect` nodes where the response cache must be reset.
|
||||
* FEATURE: vmalert: add `query`, `first` and `value` functions to alert templates. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/539>
|
||||
* FEATURE: vmagent: return user-friendly HTML page when requesting `/targets` page from web browser. The page is returned in the old plaintext format when requesting via curl or similar tool.
|
||||
* FEATURE: allow multiple whitespace chars between measurements, fields and timestamp when parsing InfluxDB line protocol.
|
||||
@@ -76,7 +77,7 @@ Released at 2020-12-15
|
||||
Released at 2020-12-05
|
||||
|
||||
* FEATURE: optimize Consul service discovery speed when discovering big number of services. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/574>
|
||||
* FEATURE: add `label_uppercase(q, label1, ... labelN)` and `label_lowercase(q, label1, ... labelN)` function to [MetricsQL](https://docs.victoriametrics.com/metricsql/)
|
||||
* FEATURE: add `label_uppercase(q, label1, ... labelN)` and `label_lowercase(q, label1, ... labelN)` function to [MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/)
|
||||
for uppercasing and lowercasing values for the given labels. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/936>
|
||||
* FEATURE: add `count_eq_over_time(m[d], N)` and `count_ne_over_time(m[d], N)` for counting the number of samples for `m` over `d` that (equal / not equal) to `N`.
|
||||
* FEATURE: do not print usage info for all the command-line flags when incorrect command-line flag is passed. Previously it could be hard reading the error message
|
||||
@@ -105,7 +106,7 @@ Released at 2020-11-26
|
||||
* FEATURE: vmselect: add `-replicationFactor` command-line flag for reducing query duration when replication is enabled and a part of vmstorage nodes
|
||||
are temporarily slow and/or temporarily unavailable. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711>
|
||||
* FEATURE: vminsert: export `vm_rpc_vmstorage_is_reachable` metric, which can be used for monitoring reachability of vmstorage nodes from vminsert nodes.
|
||||
* FEATURE: vmagent: add [Netflix Eureka](https://github.com/Netflix/eureka) service discovery (aka [eureka_sd_config](https://docs.victoriametrics.com/sd_configs/#eureka_sd_configs)). See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/851>
|
||||
* FEATURE: vmagent: add [Netflix Eureka](https://github.com/Netflix/eureka) service discovery (aka [eureka_sd_config](https://docs.victoriametrics.com/victoriametrics/sd_configs/#eureka_sd_configs)). See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/851>
|
||||
* FEATURE: add `filters` option to `dockerswarm_sd_config` like Prometheus did in v2.23.0 - see <https://github.com/prometheus/prometheus/pull/8074>
|
||||
* FEATURE: expose `__meta_ec2_ipv6_addresses` label for `ec2_sd_config` like Prometheus will do in the next release.
|
||||
* FEATURE: add `-loggerWarnsPerSecondLimit` command-line flag for rate limiting of WARN messages in logs. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/905>
|
||||
@@ -113,7 +114,7 @@ Released at 2020-11-26
|
||||
at the rate exceeding the given limit. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/905#issuecomment-729395855>
|
||||
* FEATURE: add remoteAddr to slow query log in order to simplify identifying the client that sends slow queries to VictoriaMetrics.
|
||||
Slow query logging is controlled with `-search.logSlowQueryDuration` command-line flag.
|
||||
* FEATURE: add `/tags/delSeries` handler from Graphite Tags API. See <https://docs.victoriametrics.com/#graphite-tags-api-usage>
|
||||
* FEATURE: add `/tags/delSeries` handler from Graphite Tags API. See <https://docs.victoriametrics.com/victoriametrics#graphite-tags-api-usage>
|
||||
* FEATURE: log metric name plus all its labels when the metric timestamp is out of the configured retention. This should simplify detecting the source of metrics with unexpected timestamps.
|
||||
* FEATURE: add `-dryRun` command-line flag to single-node VictoriaMetrics in order to check config file pointed by `-promscrape.config`.
|
||||
|
||||
@@ -143,9 +144,9 @@ Released at 2020-11-16
|
||||
See <https://github.com/VictoriaMetrics/VictoriaMetrics/pull/897>
|
||||
* FEATURE: update Go builder from v1.15.4 to v1.15.5. This should fix [these issues in Go](https://github.com/golang/go/issues?q=milestone%3AGo1.15.5+label%3ACherryPickApproved).
|
||||
* FEATURE: added `/internal/force_flush` http handler for flushing recently ingested data from in-memory buffers to persistent storage.
|
||||
See [troubleshooting docs](https://docs.victoriametrics.com/#troubleshooting) for more details.
|
||||
See [troubleshooting docs](https://docs.victoriametrics.com/victoriametrics#troubleshooting) for more details.
|
||||
* FEATURE: added [Graphite Tags API](https://graphite.readthedocs.io/en/stable/tags.html) support.
|
||||
See [these docs](https://docs.victoriametrics.com/#graphite-tags-api-usage) for details.
|
||||
See [these docs](https://docs.victoriametrics.com/victoriametrics#graphite-tags-api-usage) for details.
|
||||
|
||||
* BUGFIX: do not return data points in the end of the selected time range for time series ending in the middle of the selected time range.
|
||||
See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/887> and <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/845>
|
||||
@@ -168,7 +169,7 @@ Released at 2020-11-07
|
||||
|
||||
* BUGFIX: vmagent: eliminate data race when `-promscrape.streamParse` command-line is set. Previously this mode could result in scraped metrics with garbage labels.
|
||||
See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/825#issuecomment-723198247> for details.
|
||||
* BUGFIX: properly calculate `topk_*` and `bottomk_*` functions from [MetricsQL](https://docs.victoriametrics.com/metricsql/) for time series with gaps.
|
||||
* BUGFIX: properly calculate `topk_*` and `bottomk_*` functions from [MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/) for time series with gaps.
|
||||
See <https://github.com/VictoriaMetrics/VictoriaMetrics/pull/883>
|
||||
|
||||
## [v1.45.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.45.0)
|
||||
@@ -186,9 +187,9 @@ Released at 2020-11-02
|
||||
* FEATURE: vmagent: add `stream parse` mode. This mode allows reducing memory usage when individual scrape targets expose tens of millions of metrics.
|
||||
For example, during scraping Prometheus in [federation](https://prometheus.io/docs/prometheus/latest/federation/) mode.
|
||||
See `-promscrape.streamParse` command-line option and `stream_parse: true` config option for `scrape_config` section in `-promscrape.config`.
|
||||
See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/825> and [troubleshooting docs for vmagent](https://docs.victoriametrics.com/vmagent/#troubleshooting).
|
||||
See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/825> and [troubleshooting docs for vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/#troubleshooting).
|
||||
* FEATURE: vmalert: add `-dryRun` command-line option for validating the provided config files without the need to start `vmalert` service.
|
||||
* FEATURE: accept optional third argument of string type at `topk_*` and `bottomk_*` functions. This is label name for additional time series to return with the sum of time series outside top/bottom K. See [MetricsQL docs](https://docs.victoriametrics.com/metricsql/) for more details.
|
||||
* FEATURE: accept optional third argument of string type at `topk_*` and `bottomk_*` functions. This is label name for additional time series to return with the sum of time series outside top/bottom K. See [MetricsQL docs](https://docs.victoriametrics.com/victoriametrics/metricsql/) for more details.
|
||||
* FEATURE: vmagent: expose `/api/v1/targets` page according to [the corresponding Prometheus API](https://prometheus.io/docs/prometheus/latest/querying/api/#targets).
|
||||
See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/643>
|
||||
|
||||
@@ -218,11 +219,11 @@ Released at 2020-10-13
|
||||
node_filesystem_files{ host="$host", mountpoint="/" } - node_filesystem_files_free
|
||||
```
|
||||
|
||||
* FEATURE: vmagent: add Docker Swarm service discovery (aka [dockerswarm_sd_config](https://docs.victoriametrics.com/sd_configs/#dockerswarm_sd_configs)).
|
||||
* FEATURE: vmagent: add Docker Swarm service discovery (aka [dockerswarm_sd_config](https://docs.victoriametrics.com/victoriametrics/sd_configs/#dockerswarm_sd_configs)).
|
||||
See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/656>
|
||||
* FEATURE: add ability to export data in CSV format. See [these docs](https://docs.victoriametrics.com/#how-to-export-csv-data) for details.
|
||||
* FEATURE: add ability to export data in CSV format. See [these docs](https://docs.victoriametrics.com/victoriametrics#how-to-export-csv-data) for details.
|
||||
* FEATURE: vmagent: add `-promscrape.suppressDuplicateScrapeTargetErrors` command-line flag for suppressing `duplicate scrape target` errors.
|
||||
See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/651> and <https://docs.victoriametrics.com/vmagent/#troubleshooting> .
|
||||
See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/651> and <https://docs.victoriametrics.com/victoriametrics/vmagent/#troubleshooting> .
|
||||
* FEATURE: vmagent: show original labels before relabeling is applied on `duplicate scrape target` errors. This should simplify debugging for incorrect relabeling.
|
||||
See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/651>
|
||||
* FEATURE: vmagent: `/targets` page now accepts optional `show_original_labels=1` query arg for displaying original labels for each target before relabeling is applied.
|
||||
@@ -231,9 +232,9 @@ Released at 2020-10-13
|
||||
The final merge is started after no new data is ingested into per-month partition during `-finalMergeDelay`.
|
||||
* FEATURE: add `vm_rows_added_to_storage_total` metric, which shows the total number of rows added to storage since app start.
|
||||
The `sum(rate(vm_rows_added_to_storage_total))` can be smaller than `sum(rate(vm_rows_inserted_total))` if certain metrics are dropped
|
||||
due to [relabeling](https://docs.victoriametrics.com/#relabeling). The `sum(rate(vm_rows_added_to_storage_total))` can be bigger
|
||||
than `sum(rate(vm_rows_inserted_total))` if [replication](https://docs.victoriametrics.com/cluster-victoriametrics/#replication-and-data-safety) is enabled.
|
||||
* FEATURE: keep metric name after applying [MetricsQL](https://docs.victoriametrics.com/metricsql/) functions, which don't change time series meaning.
|
||||
due to [relabeling](https://docs.victoriametrics.com/victoriametrics#relabeling). The `sum(rate(vm_rows_added_to_storage_total))` can be bigger
|
||||
than `sum(rate(vm_rows_inserted_total))` if [replication](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#replication-and-data-safety) is enabled.
|
||||
* FEATURE: keep metric name after applying [MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/) functions, which don't change time series meaning.
|
||||
The list of such functions:
|
||||
* `keep_last_value`
|
||||
* `keep_next_value`
|
||||
@@ -274,8 +275,8 @@ Released at 2020-10-13
|
||||
Released at 2020-10-06
|
||||
|
||||
* FEATURE: reduce CPU usage for repeated queries over sliding time window when no new time series are added to the database.
|
||||
Typical use cases: repeated evaluation of alerting rules in [vmalert](https://docs.victoriametrics.com/vmalert/) or dashboard auto-refresh in Grafana.
|
||||
* FEATURE: vmagent: add OpenStack service discovery aka [openstack_sd_config](https://docs.victoriametrics.com/sd_configs/#openstack_sd_configs).
|
||||
Typical use cases: repeated evaluation of alerting rules in [vmalert](https://docs.victoriametrics.com/victoriametrics/vmalert/) or dashboard auto-refresh in Grafana.
|
||||
* FEATURE: vmagent: add OpenStack service discovery aka [openstack_sd_config](https://docs.victoriametrics.com/victoriametrics/sd_configs/#openstack_sd_configs).
|
||||
See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/728> .
|
||||
* FEATURE: vmalert: make `-maxIdleConnections` configurable for datasource HTTP client. This option can be used for minimizing connection churn.
|
||||
See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/795> .
|
||||
@@ -294,10 +295,10 @@ Released at 2020-10-06
|
||||
Released at 2020-09-30
|
||||
|
||||
* FEATURE: use all the available CPU cores when accepting data via a single TCP connection
|
||||
for [all the supported protocols](https://docs.victoriametrics.com/#how-to-import-time-series-data).
|
||||
for [all the supported protocols](https://docs.victoriametrics.com/victoriametrics#how-to-import-time-series-data).
|
||||
Previously data ingested via a single TCP connection could use only a single CPU core. This could limit data ingestion performance.
|
||||
The main benefit of this feature is that data can be imported at max speed via a single connection - there is no need to open multiple concurrent
|
||||
connections to VictoriaMetrics or [vmagent](https://docs.victoriametrics.com/vmagent/) in order to achieve the maximum data ingestion speed.
|
||||
connections to VictoriaMetrics or [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/) in order to achieve the maximum data ingestion speed.
|
||||
* FEATURE: cluster: improve performance for data ingestion path from `vminsert` to `vmstorage` nodes. The maximum data ingestion performance
|
||||
for a single connection between `vminsert` and `vmstorage` node scales with the number of available CPU cores on `vmstorage` side.
|
||||
This should help with <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/791> .
|
||||
@@ -305,7 +306,7 @@ Released at 2020-09-30
|
||||
This is the most optimized approach for data migration between VictoriaMetrics instances. Both single-node and cluster instances are supported.
|
||||
See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/787#issuecomment-700632551> .
|
||||
* FEATURE: add `reduce_mem_usage` query option to `/api/v1/export` in order to reduce memory usage during data export / import.
|
||||
See [these docs](https://docs.victoriametrics.com/#how-to-export-data-in-json-line-format) for details.
|
||||
See [these docs](https://docs.victoriametrics.com/victoriametrics#how-to-export-data-in-json-line-format) for details.
|
||||
* FEATURE: improve performance for `/api/v1/series` handler when it returns big number of time series.
|
||||
* FEATURE: add `vm_merge_need_free_disk_space` metric, which can be used for estimating the number of deferred background data merges due to the lack of free disk space.
|
||||
See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/686> .
|
||||
@@ -317,7 +318,7 @@ Released at 2020-09-30
|
||||
The adjustment was increasing the resulting GOMAXPROCS by 1, which looked confusing to users.
|
||||
See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/685#issuecomment-698595309> .
|
||||
* BUGFIX: vmagent: do not show `-remoteWrite.url` in initial logs if `-remoteWrite.showURL` isn't set. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/773> .
|
||||
* BUGFIX: properly handle case when [/metrics/find](https://docs.victoriametrics.com/#graphite-metrics-api-usage) finds both a leaf and a node for the given `query=prefix.*`.
|
||||
* BUGFIX: properly handle case when [/metrics/find](https://docs.victoriametrics.com/victoriametrics#graphite-metrics-api-usage) finds both a leaf and a node for the given `query=prefix.*`.
|
||||
In this case only the node must be returned with stripped dot in the end of id as carbonapi does.
|
||||
|
||||
## Previous releases
|
||||
|
||||
@@ -9,6 +9,7 @@ menu:
|
||||
parent: vm-changelog
|
||||
weight: 6
|
||||
aliases:
|
||||
- /CHANGELOG_2021
|
||||
- /CHANGELOG_2021.html
|
||||
- /changelog_2021
|
||||
- /changelog/changelog_2021/index.html
|
||||
@@ -25,76 +26,76 @@ into increase of CPU usage until merges are finished.
|
||||
|
||||
We recommend updating in "off-peak" time when load on the VictoriaMetrics is on its minimum.
|
||||
|
||||
* FEATURE: [VictoriaMetrics enterprise](https://docs.victoriametrics.com/enterprise/): add multi-level downsampling support. See [these docs](https://docs.victoriametrics.com/#downsampling) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/36).
|
||||
* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): add ability to analyze the correlation between two queries on a single graph. Just click `+Query` button, enter the second query in the newly appeared input field and press `Ctrl+Enter`. Results for both queries should be displayed simultaneously on the same graph. Every query has its own vertical scale, which is displayed on the left and the right side of the graph. Lines for the second query are dashed. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1916).
|
||||
* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): add ability to override the interval between returned datapoints. By default it is automatically calculated depending on the selected time range and horizontal resolution of the graph. Now it is possible to override it with custom values. This may be useful during data exploration and debugging.
|
||||
* FEATURE: accept optional `extra_filters[]=series_selector` query args at Prometheus query APIs additionally to `extra_label` query args. This allows enforcing additional filters for all the Prometheus query APIs by using [vmgateway](https://docs.victoriametrics.com/vmgateway/) or [vmauth](https://docs.victoriametrics.com/vmauth/). See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1863).
|
||||
* FEATURE: [vmauth](https://docs.victoriametrics.com/vmauth/): allow specifying `http` and `https` urls in `-auth.config` command-line flag. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1898). Thanks for @TFM93 .
|
||||
* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): allow specifying `http` and `https` urls in the following command-line flags: `-promscrape.config`, `-remoteWrite.relabelConfig` and `-remoteWrite.urlRelabelConfig`.
|
||||
* FEATURE: [VictoriaMetrics enterprise](https://docs.victoriametrics.com/victoriametrics/enterprise/): add multi-level downsampling support. See [these docs](https://docs.victoriametrics.com/victoriametrics#downsampling) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/36).
|
||||
* FEATURE: [vmui](https://docs.victoriametrics.com/victoriametrics#vmui): add ability to analyze the correlation between two queries on a single graph. Just click `+Query` button, enter the second query in the newly appeared input field and press `Ctrl+Enter`. Results for both queries should be displayed simultaneously on the same graph. Every query has its own vertical scale, which is displayed on the left and the right side of the graph. Lines for the second query are dashed. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1916).
|
||||
* FEATURE: [vmui](https://docs.victoriametrics.com/victoriametrics#vmui): add ability to override the interval between returned datapoints. By default it is automatically calculated depending on the selected time range and horizontal resolution of the graph. Now it is possible to override it with custom values. This may be useful during data exploration and debugging.
|
||||
* FEATURE: accept optional `extra_filters[]=series_selector` query args at Prometheus query APIs additionally to `extra_label` query args. This allows enforcing additional filters for all the Prometheus query APIs by using [vmgateway](https://docs.victoriametrics.com/victoriametrics/vmgateway/) or [vmauth](https://docs.victoriametrics.com/victoriametrics/vmauth/). See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1863).
|
||||
* FEATURE: [vmauth](https://docs.victoriametrics.com/victoriametrics/vmauth/): allow specifying `http` and `https` urls in `-auth.config` command-line flag. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1898). Thanks for @TFM93 .
|
||||
* FEATURE: [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/): allow specifying `http` and `https` urls in the following command-line flags: `-promscrape.config`, `-remoteWrite.relabelConfig` and `-remoteWrite.urlRelabelConfig`.
|
||||
* FEATURE: vminsert: allow specifying `http` and `https` urls in `-relabelConfig` command-line flag.
|
||||
* FEATURE: vminsert: add `-maxLabelValueLen` command-line flag for the ability to configure the maximum length of label value. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1908).
|
||||
* FEATURE: preserve the order of time series passed to [limit_offset](https://docs.victoriametrics.com/metricsql/#limit_offset) function. This allows implementing series paging via `limit_offset(limit, offset, sort_by_label(...))`. See [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1920) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/951) issues.
|
||||
* FEATURE: automatically convert `(value1|...|valueN)` into `{value1,...,valueN}` inside `__graphite__` pseudo-label. This allows using [Grafana multi-value template variables](https://grafana.com/docs/grafana/latest/variables/formatting-multi-value-variables/) inside `__graphite__` pseudo-label. For example, `{__graphite__=~"foo.($bar)"}` is expanded to `{__graphite__=~"foo.{x,y}"}` if both `x` and `y` are selected for `$bar` template variable. See [these docs](https://docs.victoriametrics.com/#selecting-graphite-metrics) for details.
|
||||
* FEATURE: add [timestamp_with_name](https://docs.victoriametrics.com/metricsql/#timestamp_with_name) function. It works the same as [timestamp](https://docs.victoriametrics.com/metricsql/#timestamp), but leaves the original time series names, so it can be used in queries, which match multiple time series names: `timestamp_with_name({foo="bar"}[1h])`. See [this comment](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/949#issuecomment-995222388) for more context.
|
||||
* FEATURE: add [changes_prometheus](https://docs.victoriametrics.com/metricsql/#changes_prometheus), [increase_prometheus](https://docs.victoriametrics.com/metricsql/#increase_prometheus) and [delta_prometheus](https://docs.victoriametrics.com/metricsql/#delta_prometheus) functions, which don't take into account the previous sample before the given lookbehind window specified in square brackets. These functions may be used when the Prometheus behaviour for `changes()`, `increase()` and `delta()` functions is needed to be preserved. VictoriaMetrics uses slightly different behaviour for `changes()`, `increase()` and `delta()` functions by default - see [this article](https://medium.com/@romanhavronenko/victoriametrics-promql-compliance-d4318203f51e) for details. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1962).
|
||||
* FEATURE: preserve the order of time series passed to [limit_offset](https://docs.victoriametrics.com/victoriametrics/metricsql/#limit_offset) function. This allows implementing series paging via `limit_offset(limit, offset, sort_by_label(...))`. See [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1920) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/951) issues.
|
||||
* FEATURE: automatically convert `(value1|...|valueN)` into `{value1,...,valueN}` inside `__graphite__` pseudo-label. This allows using [Grafana multi-value template variables](https://grafana.com/docs/grafana/latest/variables/formatting-multi-value-variables/) inside `__graphite__` pseudo-label. For example, `{__graphite__=~"foo.($bar)"}` is expanded to `{__graphite__=~"foo.{x,y}"}` if both `x` and `y` are selected for `$bar` template variable. See [these docs](https://docs.victoriametrics.com/victoriametrics#selecting-graphite-metrics) for details.
|
||||
* FEATURE: add [timestamp_with_name](https://docs.victoriametrics.com/victoriametrics/metricsql/#timestamp_with_name) function. It works the same as [timestamp](https://docs.victoriametrics.com/victoriametrics/metricsql/#timestamp), but leaves the original time series names, so it can be used in queries, which match multiple time series names: `timestamp_with_name({foo="bar"}[1h])`. See [this comment](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/949#issuecomment-995222388) for more context.
|
||||
* FEATURE: add [changes_prometheus](https://docs.victoriametrics.com/victoriametrics/metricsql/#changes_prometheus), [increase_prometheus](https://docs.victoriametrics.com/victoriametrics/metricsql/#increase_prometheus) and [delta_prometheus](https://docs.victoriametrics.com/victoriametrics/metricsql/#delta_prometheus) functions, which don't take into account the previous sample before the given lookbehind window specified in square brackets. These functions may be used when the Prometheus behaviour for `changes()`, `increase()` and `delta()` functions is needed to be preserved. VictoriaMetrics uses slightly different behaviour for `changes()`, `increase()` and `delta()` functions by default - see [this article](https://medium.com/@romanhavronenko/victoriametrics-promql-compliance-d4318203f51e) for details. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1962).
|
||||
|
||||
* BUGFIX: fix `unaligned 64-bit atomic operation` panic on 32-bit architectures, which has been introduced in v1.70.0. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1944).
|
||||
* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): restore the ability to use `$labels.alertname` in labels templating. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1921).
|
||||
* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): add missing `query` caption to the input field for the query. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1900).
|
||||
* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): fix navigation over query history with `Ctrl+up/down` and fix zoom relatively to the cursor position. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1936).
|
||||
* BUGFIX: deduplicate samples more thoroughly if [deduplication](https://docs.victoriametrics.com/#deduplication) is enabled. Previously some duplicate samples may be left on disk for time series with high churn rate. This may result in bigger storage space requirements.
|
||||
* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): follow up to 5 redirects when `follow_redirects: true` is set for a particular scrape config. Previously only a single redirect was performed in this case. It is expected these redirects are performed to the original hostname. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1945).
|
||||
* BUGFIX: de-duplicate data exported via [/api/v1/export/csv](https://docs.victoriametrics.com/#how-to-export-csv-data) by default if [deduplication](https://docs.victoriametrics.com/#deduplication) is enabled. The de-duplication can be disabled by passing `reduce_mem_usage=1` query arg to `/api/v1/export/csv`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1837).
|
||||
* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): properly store [historical data](https://docs.victoriametrics.com/vmalert/#rules-backfilling) to old Prometheus versions. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1943).
|
||||
* BUGFIX: [vmalert](https://docs.victoriametrics.com/victoriametrics/vmalert/): restore the ability to use `$labels.alertname` in labels templating. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1921).
|
||||
* BUGFIX: [vmui](https://docs.victoriametrics.com/victoriametrics#vmui): add missing `query` caption to the input field for the query. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1900).
|
||||
* BUGFIX: [vmui](https://docs.victoriametrics.com/victoriametrics#vmui): fix navigation over query history with `Ctrl+up/down` and fix zoom relatively to the cursor position. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1936).
|
||||
* BUGFIX: deduplicate samples more thoroughly if [deduplication](https://docs.victoriametrics.com/victoriametrics#deduplication) is enabled. Previously some duplicate samples may be left on disk for time series with high churn rate. This may result in bigger storage space requirements.
|
||||
* BUGFIX: [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/): follow up to 5 redirects when `follow_redirects: true` is set for a particular scrape config. Previously only a single redirect was performed in this case. It is expected these redirects are performed to the original hostname. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1945).
|
||||
* BUGFIX: de-duplicate data exported via [/api/v1/export/csv](https://docs.victoriametrics.com/victoriametrics#how-to-export-csv-data) by default if [deduplication](https://docs.victoriametrics.com/victoriametrics#deduplication) is enabled. The de-duplication can be disabled by passing `reduce_mem_usage=1` query arg to `/api/v1/export/csv`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1837).
|
||||
* BUGFIX: [vmalert](https://docs.victoriametrics.com/victoriametrics/vmalert/): properly store [historical data](https://docs.victoriametrics.com/victoriametrics/vmalert/#rules-backfilling) to old Prometheus versions. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1943).
|
||||
|
||||
## [v1.70.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.70.0)
|
||||
|
||||
Released at 2021-12-02
|
||||
|
||||
* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): add ability to pass arbitrary query args to `-datasource.url` on a per-group basis via `params` option. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1892).
|
||||
* FEATURE: add `now()` function to MetricsQL. This function returns the current timestamp in seconds. See [these docs](https://docs.victoriametrics.com/metricsql/#now).
|
||||
* FEATURE: [vmalert](https://docs.victoriametrics.com/victoriametrics/vmalert/): add ability to pass arbitrary query args to `-datasource.url` on a per-group basis via `params` option. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1892).
|
||||
* FEATURE: add `now()` function to MetricsQL. This function returns the current timestamp in seconds. See [these docs](https://docs.victoriametrics.com/victoriametrics/metricsql/#now).
|
||||
* FEATURE: vmauth: allow using optional `name` field in configs. This field is then used as `username` label value for `vmauth_user_requests_total` metric. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1805).
|
||||
* FEATURE: vmagent: export `vm_persistentqueue_read_duration_seconds_total` and `vm_persistentqueue_write_duration_seconds_total` metrics, which can be used for detecting persistent queue saturation with `rate(vm_persistentqueue_write_duration_seconds_total) > 0.9` alerting rule.
|
||||
* FEATURE: export `vm_filestream_read_duration_seconds_total` and `vm_filestream_write_duration_seconds_total` metrics, which can be used for detecting persistent disk saturation with `rate(vm_filestream_read_duration_seconds_total) > 0.9` alerting rule.
|
||||
* FEATURE: export `vm_cache_size_max_bytes` metrics, which show capacity for various caches. These metrics can be used for determining caches with reach its capacity with `vm_cache_size_bytes / vm_cache_size_max_bytes > 0.9` query.
|
||||
* FEATURE: [vmbackup](https://docs.victoriametrics.com/vmbackup/), [vmrestore](https://docs.victoriametrics.com/vmrestore/): add `-s3ForcePathStyle` command-line flag, which can be used for making backups to [Aliyun OSS](https://www.aliyun.com/product/oss). See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1802).
|
||||
* FEATURE: [vmctl](https://docs.victoriametrics.com/vmctl/): improve data migration from OpenTSDB. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1809). Thanks to @johnseekins .
|
||||
* FEATURE: [vmbackup](https://docs.victoriametrics.com/victoriametrics/vmbackup/), [vmrestore](https://docs.victoriametrics.com/victoriametrics/vmrestore/): add `-s3ForcePathStyle` command-line flag, which can be used for making backups to [Aliyun OSS](https://www.aliyun.com/product/oss). See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1802).
|
||||
* FEATURE: [vmctl](https://docs.victoriametrics.com/victoriametrics/vmctl/): improve data migration from OpenTSDB. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1809). Thanks to @johnseekins .
|
||||
* FEATURE: suppress `connection reset by peer` errors when remote client resets TCP connection to VictoriaMetrics / vmagent while ingesting the data via InfluxDB line protocol, Graphite protocol or OpenTSDB protocol. This error is expected, so there is no need in logging it.
|
||||
* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): store the display type in URL, so it isn't lost when copy-pasting the URL. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1804).
|
||||
* FEATURE: [vmui](https://docs.victoriametrics.com/victoriametrics#vmui): store the display type in URL, so it isn't lost when copy-pasting the URL. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1804).
|
||||
* FEATURE: vmalert: make `-notifier.url` command-line flag optional. This flag can be omitted if `vmalert` is used solely for recording rules and doesn't evaluate alerting rules. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1870).
|
||||
* FEATURE: [vmbackup](https://docs.victoriametrics.com/vmbackup/), [vmrestore](https://docs.victoriametrics.com/vmrestore/): export internal metrics at `http://vmbackup:8420/metrics` and `http://vmrestore:8421/metrics` for better visibility of the backup/restore process.
|
||||
* FEATURE: allow trailing whitespace after the timestamp when [parsing Graphite plaintext lines](https://docs.victoriametrics.com/#how-to-send-data-from-graphite-compatible-agents-such-as-statsd). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1865).
|
||||
* FEATURE: [vmbackup](https://docs.victoriametrics.com/victoriametrics/vmbackup/), [vmrestore](https://docs.victoriametrics.com/victoriametrics/vmrestore/): export internal metrics at `http://vmbackup:8420/metrics` and `http://vmrestore:8421/metrics` for better visibility of the backup/restore process.
|
||||
* FEATURE: allow trailing whitespace after the timestamp when [parsing Graphite plaintext lines](https://docs.victoriametrics.com/victoriametrics#how-to-send-data-from-graphite-compatible-agents-such-as-statsd). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1865).
|
||||
* FEATURE: expose `/-/healthy` and `/-/ready` endpoints as Prometheus does. This is needed for improving integration with third-party solutions, which rely on these endpoints. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1833).
|
||||
|
||||
* BUGFIX: vmagent: prevent from scraping duplicate targets if `-promscrape.dropOriginalLabels` command-line flag is set. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1830). Thanks to @guidao for the fix.
|
||||
* BUGFIX: vmstorage [enterprise](https://docs.victoriametrics.com/enterprise/): added missing `vm_tenant_used_tenant_bytes` metric, which shows the approximate per-tenant disk usage. See [these docs](https://docs.victoriametrics.com/pertenantstatistic/) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1605).
|
||||
* BUGFIX: vmstorage [enterprise](https://docs.victoriametrics.com/victoriametrics/enterprise/): added missing `vm_tenant_used_tenant_bytes` metric, which shows the approximate per-tenant disk usage. See [these docs](https://docs.victoriametrics.com/pertenantstatistic/) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1605).
|
||||
* BUGFIX: vmauth: properly take into account the value passed to `-maxIdleConnsPerBackend` command-line flag. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1300).
|
||||
* BUGFIX: vmagent: fix [reading data from Kafka](https://docs.victoriametrics.com/vmagent/#reading-metrics-from-kafka).
|
||||
* BUGFIX: vmalert: fix [replay mode](https://docs.victoriametrics.com/vmalert/#rules-backfilling) in enterprise version.
|
||||
* BUGFIX: consistently return zero from [deriv()](https://docs.victoriametrics.com/metricsql/#deriv) function applied to a constant time series. Previously it could return small non-zero values in this case.
|
||||
* BUGFIX: [vmrestore](https://docs.victoriametrics.com/vmrestore/): properly resume downloading for partially downloaded big files. Previously such files were re-downloaded from the beginning after the interrupt. Now only the remaining parts of the file are downloaded. This allows saving network bandwidth. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/487).
|
||||
* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): do not store the last query across vmui page reloads. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1694).
|
||||
* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): fix `Cannot read properties of undefined` error at table view. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1797).
|
||||
* BUGFIX: vmagent: fix [reading data from Kafka](https://docs.victoriametrics.com/victoriametrics/vmagent/#reading-metrics-from-kafka).
|
||||
* BUGFIX: vmalert: fix [replay mode](https://docs.victoriametrics.com/victoriametrics/vmalert/#rules-backfilling) in enterprise version.
|
||||
* BUGFIX: consistently return zero from [deriv()](https://docs.victoriametrics.com/victoriametrics/metricsql/#deriv) function applied to a constant time series. Previously it could return small non-zero values in this case.
|
||||
* BUGFIX: [vmrestore](https://docs.victoriametrics.com/victoriametrics/vmrestore/): properly resume downloading for partially downloaded big files. Previously such files were re-downloaded from the beginning after the interrupt. Now only the remaining parts of the file are downloaded. This allows saving network bandwidth. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/487).
|
||||
* BUGFIX: [vmui](https://docs.victoriametrics.com/victoriametrics#vmui): do not store the last query across vmui page reloads. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1694).
|
||||
* BUGFIX: [vmui](https://docs.victoriametrics.com/victoriametrics#vmui): fix `Cannot read properties of undefined` error at table view. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1797).
|
||||
|
||||
## [v1.69.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.69.0)
|
||||
|
||||
Released at 2021-11-08
|
||||
|
||||
* FEATURE: vmalert: allow groups with empty rules list like Prometheus does. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1742).
|
||||
* FEATURE: vmalert: allow groups with default `tenant` in `-clusterMode`. Default `tenant` values can be specified via `-defaultTenant.prometheus` and `-defaultTenant.graphite`. See [these docs](https://docs.victoriametrics.com/vmalert/#multitenancy).
|
||||
* FEATURE: vmalert: allow groups with default `tenant` in `-clusterMode`. Default `tenant` values can be specified via `-defaultTenant.prometheus` and `-defaultTenant.graphite`. See [these docs](https://docs.victoriametrics.com/victoriametrics/vmalert/#multitenancy).
|
||||
* FEATURE: vmagent: add `collapse` and `expand` buttons per each group of targets with the same `job_name` at `http://vmagent:8429/targets` page.
|
||||
* FEATURE: automatically detect timestamp precision (ns, us, ms or s) for the data ingested into VictoriaMetrics via [InfluxDB line protocol](https://docs.victoriametrics.com/#how-to-send-data-from-influxdb-compatible-agents-such-as-telegraf).
|
||||
* FEATURE: automatically detect timestamp precision (ns, us, ms or s) for the data ingested into VictoriaMetrics via [InfluxDB line protocol](https://docs.victoriametrics.com/victoriametrics#how-to-send-data-from-influxdb-compatible-agents-such-as-telegraf).
|
||||
* FEATURE: vmagent: add ability to protect `/config` page with auth key via `-configAuthKey` command-line flag. This page may contain sensitive config information, so it may be good to restrict access to this page. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1764).
|
||||
* FEATURE: vmagent: hide passwords and auth tokens at `/config` page like Prometheus does. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1764).
|
||||
* FEATURE: vmagent: add `-promscrape.maxResponseHeadersSize` command-line flag for tuning the maximum HTTP response headers size for Prometheus scrape targets.
|
||||
* FEATURE: vmagent: send data to multiple configured remote storage systems in parallel (e.g. when multiple `-remoteWrite.url` flag values are specified). This should improve data ingestion speed.
|
||||
* FEATURE: vmagent: add `-remoteWrite.maxRowsPerBlock` command-line flag for tuning the number of samples to send to remote storage per each block. Bigger values may improve data ingestion performance at the cost of higher memory usage.
|
||||
* FEATURE: vmagent: distribute Kafka messages among all the partitions when [writing data to Kafka](https://docs.victoriametrics.com/vmagent/#writing-metrics-to-kafka).
|
||||
* FEATURE: add [label_graphite_group](https://docs.victoriametrics.com/metricsql/#label_graphite_group) function for extracting the given groups from Graphite metric names.
|
||||
* FEATURE: add [duration_over_time](https://docs.victoriametrics.com/metricsql/#duration_over_time) function for calculating the actual lifetime of the time series with possible gaps. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1780).
|
||||
* FEATURE: add [limit_offset](https://docs.victoriametrics.com/metricsql/#limit_offset) function, which can be used for implementing simple paging over big number of time series. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1778).
|
||||
* FEATURE: vmagent: distribute Kafka messages among all the partitions when [writing data to Kafka](https://docs.victoriametrics.com/victoriametrics/vmagent/#writing-metrics-to-kafka).
|
||||
* FEATURE: add [label_graphite_group](https://docs.victoriametrics.com/victoriametrics/metricsql/#label_graphite_group) function for extracting the given groups from Graphite metric names.
|
||||
* FEATURE: add [duration_over_time](https://docs.victoriametrics.com/victoriametrics/metricsql/#duration_over_time) function for calculating the actual lifetime of the time series with possible gaps. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1780).
|
||||
* FEATURE: add [limit_offset](https://docs.victoriametrics.com/victoriametrics/metricsql/#limit_offset) function, which can be used for implementing simple paging over big number of time series. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1778).
|
||||
|
||||
* BUGFIX: vmagent: reduce the increased memory usage when scraping targets with big number of metrics which periodically change. The memory usage has been increased in v1.68.0 after vmagent started generating staleness markers in [stream parse mode](https://docs.victoriametrics.com/vmagent/#stream-parsing-mode). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1745).
|
||||
* BUGFIX: vmagent: reduce the increased memory usage when scraping targets with big number of metrics which periodically change. The memory usage has been increased in v1.68.0 after vmagent started generating staleness markers in [stream parse mode](https://docs.victoriametrics.com/victoriametrics/vmagent/#stream-parsing-mode). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1745).
|
||||
* BUGFIX: vmagent: properly display `proxy_url` config option at `http://vmagent:8429/config` page. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1755).
|
||||
* BUGFIX: fix tests for Apple M1. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1653).
|
||||
|
||||
@@ -105,43 +106,43 @@ Released at 2021-10-22
|
||||
* FEATURE: vmagent: expose `-promscrape.config` contents at `/config` page as Prometheus does. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1695).
|
||||
* FEATURE: vmagent: add `show original labels` button per each scrape target displayed at `http://vmagent:8429/targets` page. This should improve debuggability for service discovery and relabeling issues similar to [this one](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1664). See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1698).
|
||||
* FEATURE: vmagent: shard targets among cluster nodes after the relabeling is applied. This should guarantee that targets with the same set of labels go to the same `vmagent` node in the cluster. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1687).
|
||||
* FEATURE: vmagent: automatically switch to [stream parsing mode](https://docs.victoriametrics.com/vmagent/#stream-parsing-mode) if the response from the given target exceeds the command-line flag value `-promscrape.minResponseSizeForStreamParse`. This should reduce memory usage when `vmagent` scrapes targets with non-uniform response sizes (this is the case in Kubernetes monitoring).
|
||||
* FEATURE: vmagent: send Prometheus-like staleness marks in [stream parsing mode](https://docs.victoriametrics.com/vmagent/#stream-parsing-mode). Previously staleness marks weren't sent in stream parsing mode. See [these docs](https://docs.victoriametrics.com/vmagent/#prometheus-staleness-markers) for details.
|
||||
* FEATURE: vmagent: properly calculate `scrape_series_added` metric for targets in [stream parsing mode](https://docs.victoriametrics.com/vmagent/#stream-parsing-mode). Previously it was set to 0 in stream parsing mode. See [more details about this metric](https://prometheus.io/docs/concepts/jobs_instances/#automatically-generated-labels-and-time-series).
|
||||
* FEATURE: vmagent: expose `promscrape_series_limit_max_series` and `promscrape_series_limit_current_series` metrics at `http://vmagent:8429/metrics` for scrape targets with the [enabled series limiter](https://docs.victoriametrics.com/vmagent/#cardinality-limiter).
|
||||
* FEATURE: vmagent: return error if `sample_limit` or `series_limit` options are set when [stream parsing mode](https://docs.victoriametrics.com/vmagent/#stream-parsing-mode) is enabled, since these limits cannot be applied in stream parsing mode.
|
||||
* FEATURE: vmagent: automatically switch to [stream parsing mode](https://docs.victoriametrics.com/victoriametrics/vmagent/#stream-parsing-mode) if the response from the given target exceeds the command-line flag value `-promscrape.minResponseSizeForStreamParse`. This should reduce memory usage when `vmagent` scrapes targets with non-uniform response sizes (this is the case in Kubernetes monitoring).
|
||||
* FEATURE: vmagent: send Prometheus-like staleness marks in [stream parsing mode](https://docs.victoriametrics.com/victoriametrics/vmagent/#stream-parsing-mode). Previously staleness marks weren't sent in stream parsing mode. See [these docs](https://docs.victoriametrics.com/victoriametrics/vmagent/#prometheus-staleness-markers) for details.
|
||||
* FEATURE: vmagent: properly calculate `scrape_series_added` metric for targets in [stream parsing mode](https://docs.victoriametrics.com/victoriametrics/vmagent/#stream-parsing-mode). Previously it was set to 0 in stream parsing mode. See [more details about this metric](https://prometheus.io/docs/concepts/jobs_instances/#automatically-generated-labels-and-time-series).
|
||||
* FEATURE: vmagent: expose `promscrape_series_limit_max_series` and `promscrape_series_limit_current_series` metrics at `http://vmagent:8429/metrics` for scrape targets with the [enabled series limiter](https://docs.victoriametrics.com/victoriametrics/vmagent/#cardinality-limiter).
|
||||
* FEATURE: vmagent: return error if `sample_limit` or `series_limit` options are set when [stream parsing mode](https://docs.victoriametrics.com/victoriametrics/vmagent/#stream-parsing-mode) is enabled, since these limits cannot be applied in stream parsing mode.
|
||||
* FEATURE: vmalert: add `-remoteRead.disablePathAppend` command-line flag, which allows specifying the full `-remoteRead.url`. If `-remoteRead.disablePathAppend` is set, then `vmalert` doesn't add `/api/v1/query` suffix to `-remoteRead.url`.
|
||||
* FEATURE: add trigonometric functions, which are going to be added in [Prometheus 2.31](https://github.com/prometheus/prometheus/pull/9239): [acosh](https://docs.victoriametrics.com/metricsql/#acosh), [asinh](https://docs.victoriametrics.com/metricsql/#asinh), [atan](https://docs.victoriametrics.com/metricsql/#atan), [atanh](https://docs.victoriametrics.com/metricsql/#atanh), [cosh](https://docs.victoriametrics.com/metricsql/#cosh), [deg](https://docs.victoriametrics.com/metricsql/#deg), [rad](https://docs.victoriametrics.com/metricsql/#rad), [sinh](https://docs.victoriametrics.com/metricsql/#sinh), [tan](https://docs.victoriametrics.com/metricsql/#tan), [tanh](https://docs.victoriametrics.com/metricsql/#tanh). Also add `atan2` binary operator. See [this pull request](https://github.com/prometheus/prometheus/pull/9248).
|
||||
* FEATURE: consistently return the same set of time series from [limitk](https://docs.victoriametrics.com/metricsql/#limitk) function. This improves the usability of periodically refreshed graphs.
|
||||
* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): various UX improvements. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1711) and [these docs](https://docs.victoriametrics.com/#vmui).
|
||||
* FEATURE: [vmauth](https://docs.victoriametrics.com/vmauth/): add ability to specify HTTP headers, which will be sent in requests to backends. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1736).
|
||||
* FEATURE: add trigonometric functions, which are going to be added in [Prometheus 2.31](https://github.com/prometheus/prometheus/pull/9239): [acosh](https://docs.victoriametrics.com/victoriametrics/metricsql/#acosh), [asinh](https://docs.victoriametrics.com/victoriametrics/metricsql/#asinh), [atan](https://docs.victoriametrics.com/victoriametrics/metricsql/#atan), [atanh](https://docs.victoriametrics.com/victoriametrics/metricsql/#atanh), [cosh](https://docs.victoriametrics.com/victoriametrics/metricsql/#cosh), [deg](https://docs.victoriametrics.com/victoriametrics/metricsql/#deg), [rad](https://docs.victoriametrics.com/victoriametrics/metricsql/#rad), [sinh](https://docs.victoriametrics.com/victoriametrics/metricsql/#sinh), [tan](https://docs.victoriametrics.com/victoriametrics/metricsql/#tan), [tanh](https://docs.victoriametrics.com/victoriametrics/metricsql/#tanh). Also add `atan2` binary operator. See [this pull request](https://github.com/prometheus/prometheus/pull/9248).
|
||||
* FEATURE: consistently return the same set of time series from [limitk](https://docs.victoriametrics.com/victoriametrics/metricsql/#limitk) function. This improves the usability of periodically refreshed graphs.
|
||||
* FEATURE: [vmui](https://docs.victoriametrics.com/victoriametrics#vmui): various UX improvements. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1711) and [these docs](https://docs.victoriametrics.com/victoriametrics#vmui).
|
||||
* FEATURE: [vmauth](https://docs.victoriametrics.com/victoriametrics/vmauth/): add ability to specify HTTP headers, which will be sent in requests to backends. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1736).
|
||||
* FEATURE: add `/flags` page to all the VictoriaMetrics components. This page contains command-line flags passed to the component.
|
||||
* FEATURE: allow using tab separators additionally to whitespace separators when [ingesting data in Graphite plaintext protocol](https://docs.victoriametrics.com/#how-to-send-data-from-graphite-compatible-agents-such-as-statsd). Such separators are [supported by Carbon-c-relay](https://github.com/grobian/carbon-c-relay/commit/f3ffe6cc2b52b07d14acbda649ad3fd6babdd528).
|
||||
* FEATURE: allow using tab separators additionally to whitespace separators when [ingesting data in Graphite plaintext protocol](https://docs.victoriametrics.com/victoriametrics#how-to-send-data-from-graphite-compatible-agents-such-as-statsd). Such separators are [supported by Carbon-c-relay](https://github.com/grobian/carbon-c-relay/commit/f3ffe6cc2b52b07d14acbda649ad3fd6babdd528).
|
||||
|
||||
* BUGFIX: vmstorage: fix `unaligned 64-bit atomic operation` panic on 32-bit architectures (arm and 386). The panic has been introduced in v1.67.0.
|
||||
* BUGFIX: vmalert, vmauth: prevent from frequent closing of TCP connections established to backends under high load. This should reduce the number of TCP sockets in `TIME_WAIT` state at `vmalert` and `vmauth` under high load. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1704).
|
||||
* BUGFIX: vmalert: correctly calculate alert ID including extra labels. Previously, ID for alert entity was generated without alertname or groupname. This led to collision, when multiple alerting rules within the same group producing same labelsets. E.g. expr: `sum(metric1) by (job) > 0` and expr: `sum(metric2) by (job) > 0` could result into same labelset `job: "job"`. The bugfix adds all extra labels right after receiving response from the datasource. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1734).
|
||||
* BUGFIX: vmalert: fix links in [Web UI](https://docs.victoriametrics.com/vmalert/#web). See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1717).
|
||||
* BUGFIX: vmagent: set `honor_timestamps: true` by default in [scrape configs](https://docs.victoriametrics.com/sd_configs/#scrape_configs) if this options isn't set explicitly. This aligns the behaviour with Prometheus.
|
||||
* BUGFIX: vmalert: fix links in [Web UI](https://docs.victoriametrics.com/victoriametrics/vmalert/#web). See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1717).
|
||||
* BUGFIX: vmagent: set `honor_timestamps: true` by default in [scrape configs](https://docs.victoriametrics.com/victoriametrics/sd_configs/#scrape_configs) if this options isn't set explicitly. This aligns the behaviour with Prometheus.
|
||||
* BUGFIX: vmagent: group scrape targets by the original job names at `http://vmagent:8429/targets` page like Prometheus does. Previously they were grouped by the job name after relabeling, which may result in unexpected empty target groups. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1707).
|
||||
* BUGFIX: [vmctl](https://docs.victoriametrics.com/vmctl/): fix importing boolean fields from InfluxDB line protocol. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1709).
|
||||
* BUGFIX: [vmctl](https://docs.victoriametrics.com/victoriametrics/vmctl/): fix importing boolean fields from InfluxDB line protocol. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1709).
|
||||
|
||||
## [v1.67.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.67.0)
|
||||
|
||||
Released at 2021-10-08
|
||||
|
||||
* FEATURE: add ability to accept metrics from [DataDog agent](https://docs.datadoghq.com/agent/) and [DogStatsD](https://docs.datadoghq.com/developers/dogstatsd/). See [these docs](https://docs.victoriametrics.com/single-server-victoriametrics/#how-to-send-data-from-datadog-agent). This option simplifies the migration path from DataDog to VictoriaMetrics. See also [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/206).
|
||||
* FEATURE: vmagent [enterprise](https://docs.victoriametrics.com/enterprise/): add support for data reading and writing from/to [Apache Kafka](https://kafka.apache.org/). See [these docs](https://docs.victoriametrics.com/vmagent/#kafka-integration).
|
||||
* FEATURE: vmui: switch to [μPlot](https://github.com/leeoniya/uPlot) and add ability to naturally scroll and zoom graphs. See [these docs](https://docs.victoriametrics.com/#vmui). Thanks to @Loori-R.
|
||||
* FEATURE: add ability to accept metrics from [DataDog agent](https://docs.datadoghq.com/agent/) and [DogStatsD](https://docs.datadoghq.com/developers/dogstatsd/). See [these docs](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#how-to-send-data-from-datadog-agent). This option simplifies the migration path from DataDog to VictoriaMetrics. See also [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/206).
|
||||
* FEATURE: vmagent [enterprise](https://docs.victoriametrics.com/victoriametrics/enterprise/): add support for data reading and writing from/to [Apache Kafka](https://kafka.apache.org/). See [these docs](https://docs.victoriametrics.com/victoriametrics/vmagent/#kafka-integration).
|
||||
* FEATURE: vmui: switch to [μPlot](https://github.com/leeoniya/uPlot) and add ability to naturally scroll and zoom graphs. See [these docs](https://docs.victoriametrics.com/victoriametrics#vmui). Thanks to @Loori-R.
|
||||
* FEATURE: vmstorage: stop accepting new data if `-storageDataPath` directory contains less than `-storage.minFreeDiskSpaceBytes` of free space. This should prevent from `out of disk space` crashes. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/269).
|
||||
* FEATURE: calculate quantiles in the same way as Prometheus does in such functions as [quantile_over_time](https://docs.victoriametrics.com/metricsql/#quantile_over_time) and [quantile](https://docs.victoriametrics.com/metricsql/#quantile). Previously results from VictoriaMetrics could be slightly different than results from Prometheus. See [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1625) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1612) issues.
|
||||
* FEATURE: add `rollup_scrape_interval(m[d])` function to [MetricsQL](https://docs.victoriametrics.com/metricsql/), which returns `min`, `max` and `avg` values for the interval between samples for `m` on the given lookbehind window `d`.
|
||||
* FEATURE: add `topk_last(k, q)` and `bottomk_last(k, q)` functions to [MetricsQL](https://docs.victoriametrics.com/metricsql/), which return up to `k` time series from `q` with the maximum / minimum last value on the graph.
|
||||
* FEATURE: calculate quantiles in the same way as Prometheus does in such functions as [quantile_over_time](https://docs.victoriametrics.com/victoriametrics/metricsql/#quantile_over_time) and [quantile](https://docs.victoriametrics.com/victoriametrics/metricsql/#quantile). Previously results from VictoriaMetrics could be slightly different than results from Prometheus. See [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1625) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1612) issues.
|
||||
* FEATURE: add `rollup_scrape_interval(m[d])` function to [MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/), which returns `min`, `max` and `avg` values for the interval between samples for `m` on the given lookbehind window `d`.
|
||||
* FEATURE: add `topk_last(k, q)` and `bottomk_last(k, q)` functions to [MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/), which return up to `k` time series from `q` with the maximum / minimum last value on the graph.
|
||||
|
||||
* BUGFIX: align behavior of the queries `a or on (labels) b`, `a and on (labels) b` and `a unless on (labels) b` where `b` has multiple time series with the given `labels` to Prometheus behavior. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1643).
|
||||
* BUGFIX: vmagent: fix `openstack_sd_config` service discovery when both `domain_name` and `project_id` config options are set. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1655).
|
||||
* BUGFIX: return proper values (zeroes) from [stddev_over_time](https://docs.victoriametrics.com/metricsql/#stddev_over_time) and [stdvar_over_time](https://docs.victoriametrics.com/metricsql/#stdvar_over_time) functions when the lookbehind window in square brackets contains only a single sample. Previously the sample value was incorrectly returned in this case.
|
||||
* BUGFIX: vminsert: fix uneven distribution of time series among storage nodes in [multi-level cluster setup](https://docs.victoriametrics.com/cluster-victoriametrics/#multi-level-cluster-setup). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1672).
|
||||
* BUGFIX: return proper values (zeroes) from [stddev_over_time](https://docs.victoriametrics.com/victoriametrics/metricsql/#stddev_over_time) and [stdvar_over_time](https://docs.victoriametrics.com/victoriametrics/metricsql/#stdvar_over_time) functions when the lookbehind window in square brackets contains only a single sample. Previously the sample value was incorrectly returned in this case.
|
||||
* BUGFIX: vminsert: fix uneven distribution of time series among storage nodes in [multi-level cluster setup](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#multi-level-cluster-setup). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1672).
|
||||
|
||||
## [v1.66.2](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.66.2)
|
||||
|
||||
@@ -158,7 +159,7 @@ Released at 2021-09-22
|
||||
|
||||
* FEATURE: add `-cluster` and/or `-enterprise` suffixes to `short_version` label at `vm_app_version` metric exposed at `/metrics` page of every VictoriaMetrics component. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1635).
|
||||
|
||||
* BUGFIX: vmselect: fix accessing [Graphite APIs](https://docs.victoriametrics.com/#graphite-api-usage). The access has been broken in v1.66.0, because `/graphite/*` path prefix accidentally clashed with `/graph*` path prefix used for VictoriaMetrics UI (aka `vmui`).
|
||||
* BUGFIX: vmselect: fix accessing [Graphite APIs](https://docs.victoriametrics.com/victoriametrics#graphite-api-usage). The access has been broken in v1.66.0, because `/graphite/*` path prefix accidentally clashed with `/graph*` path prefix used for VictoriaMetrics UI (aka `vmui`).
|
||||
* BUGFIX: fix parsing `regex: <bool_or_number>` in relabeling rules (for example, `regex: true` or `regex: 123`). The bug has been introduced in v1.66.0.
|
||||
|
||||
## [v1.66.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.66.0)
|
||||
@@ -168,65 +169,65 @@ Released at 2021-09-20
|
||||
* FEATURE: vmalert: add web UI with the list of alerting groups, alerts and alert statuses. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1602).
|
||||
* FEATURE: vmalert: add `-rule.maxResolveDuration` command-line flag, which could be used for limiting the auto-resolve duration for the alerting rule. By default it is limited to 3x evaluation interval. This could be too high for big evaluation intervals. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1586).
|
||||
* FEATURE: vmalert: add support for Bearer token authorization for `-datasource.url`, `-remoteRead.url` and `-remoteWrite.url`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1608).
|
||||
* FEATURE: vmagent: send stale markers for disappeared metrics like Prometheus does. Previously stale markers were sent only when the scrape target disappears or when it becomes temporarily unavailable. See [these docs](https://docs.victoriametrics.com/vmagent/#prometheus-staleness-markers) for details.
|
||||
* FEATURE: vmagent: add ability to set `series_limit` option for a particular scrape target via `__series_limit__` label. This allows setting the limit on the number of time series on a per-target basis. See [these docs](https://docs.victoriametrics.com/vmagent/#cardinality-limiter) for details.
|
||||
* FEATURE: vmagent: add ability to set `stream_parse` option for a particular scrape target via `__stream_parse__` label. This allows managing the stream parsing mode on a per-target basis. See [these docs](https://docs.victoriametrics.com/vmagent/#stream-parsing-mode) for details.
|
||||
* FEATURE: vmagent: send stale markers for disappeared metrics like Prometheus does. Previously stale markers were sent only when the scrape target disappears or when it becomes temporarily unavailable. See [these docs](https://docs.victoriametrics.com/victoriametrics/vmagent/#prometheus-staleness-markers) for details.
|
||||
* FEATURE: vmagent: add ability to set `series_limit` option for a particular scrape target via `__series_limit__` label. This allows setting the limit on the number of time series on a per-target basis. See [these docs](https://docs.victoriametrics.com/victoriametrics/vmagent/#cardinality-limiter) for details.
|
||||
* FEATURE: vmagent: add ability to set `stream_parse` option for a particular scrape target via `__stream_parse__` label. This allows managing the stream parsing mode on a per-target basis. See [these docs](https://docs.victoriametrics.com/victoriametrics/vmagent/#stream-parsing-mode) for details.
|
||||
* FEATURE: vmagent: add ability to set `scrape_interval` and `scrape_timeout` options for a particular target via `__scrape_interval__` and `__scrape_timeout__` labels in the same way as Prometheus 2.30 does. See [this pull request](https://github.com/prometheus/prometheus/pull/8911).
|
||||
* FEATURE: vmagent: generate `scrape_timeout_seconds` metric per each scrape target, so the target saturation could be calculated with `scrape_duration_seconds / scrape_timeout_seconds`. See the corresponding [pull request from Prometheus 2.30](https://github.com/prometheus/prometheus/pull/9247).
|
||||
* FEATURE: vmagent: reduce CPU usage when calculating the number of newly added series per scrape (this number is sent to remote storage in `scrape_series_added` metric).
|
||||
* FEATURE: vmagent: reduce CPU usage when applying `series_limit` to scrape targets with constant set of metrics. See more information about `series_limit` [here](https://docs.victoriametrics.com/vmagent/#cardinality-limiter).
|
||||
* FEATURE: vminsert: disable rerouting by default when a few of `vmstorage` nodes start accepting data at lower speed than the rest of `vmstorage` nodes. This should improve VictoriaMetrics cluster stability during rolling restarts and during spikes in [time series churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate). The rerouting can be enabled by passing `-disableRerouting=false` command-line flag to `vminsert`.
|
||||
* FEATURE: vmagent: reduce CPU usage when applying `series_limit` to scrape targets with constant set of metrics. See more information about `series_limit` [here](https://docs.victoriametrics.com/victoriametrics/vmagent/#cardinality-limiter).
|
||||
* FEATURE: vminsert: disable rerouting by default when a few of `vmstorage` nodes start accepting data at lower speed than the rest of `vmstorage` nodes. This should improve VictoriaMetrics cluster stability during rolling restarts and during spikes in [time series churn rate](https://docs.victoriametrics.com/victoriametrics/faq/#what-is-high-churn-rate). The rerouting can be enabled by passing `-disableRerouting=false` command-line flag to `vminsert`.
|
||||
* FEATURE: vmauth: do not put invalid auth tokens into log by default due to security reasons. The logging can be returned back by passing `-logInvalidAuthTokens` command-line flag to `vmauth`. Requests with invalid auth tokens are counted at `vmagent_http_request_errors_total{reason="invalid_auth_token"}` metric exposed by `vmauth` at `/metrics` page.
|
||||
* FEATURE: add new relabeling actions: `keep_metrics` and `drop_metrics`. This simplifies metrics filtering by metric names. See [these docs](https://docs.victoriametrics.com/vmagent/#relabeling) for more details.
|
||||
* FEATURE: allow splitting long `regex` in relabeling filters into an array of shorter regexps, which can be put into multiple lines for better readability and maintainability. See [these docs](https://docs.victoriametrics.com/vmagent/#relabeling) for more details.
|
||||
* FEATURE: add new relabeling actions: `keep_metrics` and `drop_metrics`. This simplifies metrics filtering by metric names. See [these docs](https://docs.victoriametrics.com/victoriametrics/vmagent/#relabeling) for more details.
|
||||
* FEATURE: allow splitting long `regex` in relabeling filters into an array of shorter regexps, which can be put into multiple lines for better readability and maintainability. See [these docs](https://docs.victoriametrics.com/victoriametrics/vmagent/#relabeling) for more details.
|
||||
* FEATURE: optimize performance for queries with regexp filters on metric name like `{__name__=~"metric1|...|metricN"}`. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1610) from @faceair.
|
||||
* FEATURE: vmui: use Prometheus-compatible query args, so `vmui` could be accessed from graph editor in Grafana. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1619). Thanks to @Loori-R.
|
||||
* FEATURE: vmselect: automatically add missing port to `-storageNode` hostnames. For example, `-storageNode=vmstorage1,vmstorage2` is automatically translated to `-storageNode=vmstorage1:8401,vmstorage2:8401`. This simplifies [manual setup of VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#cluster-setup).
|
||||
* FEATURE: vminsert: automatically add missing port to `-storageNode` hostnames. For example, `-storageNode=vmstorage1,vmstorage2` is automatically translated to `-storageNode=vmstorage1:8400,vmstorage2:8400`. This simplifies [manual setup of VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#cluster-setup).
|
||||
* FEATURE: add [mad(q)](https://docs.victoriametrics.com/metricsql/#mad) function to [MetricsQL](https://docs.victoriametrics.com/metricsql/). It calculates [Median absolute deviation](https://en.wikipedia.org/wiki/Median_absolute_deviation) for groups of points with identical timestamps across multiple time series.
|
||||
* FEATURE: add [outliers_mad(tolerance, q)](https://docs.victoriametrics.com/metricsql/#outliers_mad) function to [MetricsQL](https://docs.victoriametrics.com/metricsql/). It returns time series with peaks outside the [Median absolute deviation](https://en.wikipedia.org/wiki/Median_absolute_deviation) multiplied by `tolerance`.
|
||||
* FEATURE: add `histogram_quantiles("phiLabel", phi1, ..., phiN, buckets)` function to [MetricsQL](https://docs.victoriametrics.com/metricsql/). It calculates the given `phi*`-quantiles over the given `buckets` and returns time series per each quantile with the corresponding `{phiLabel="phi*"}` label.
|
||||
* FEATURE: add `quantiles_over_time("phiLabel", phi1, ..., phiN, series_selector[d])` function to [MetricsQL](https://docs.victoriametrics.com/metricsql/). It calculates the given `phi*`-quantiles over raw samples selected by `series_selector` on the given lookbehind window `d`. It returns time series per each quantile with the corresponding `{phiLabel="phi*"}` label.
|
||||
* FEATURE: [enterprise](https://docs.victoriametrics.com/enterprise/): do not ask for `-eula` flag if `-version` flag is passed to enterprise app. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1621).
|
||||
* FEATURE: vmselect: automatically add missing port to `-storageNode` hostnames. For example, `-storageNode=vmstorage1,vmstorage2` is automatically translated to `-storageNode=vmstorage1:8401,vmstorage2:8401`. This simplifies [manual setup of VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#cluster-setup).
|
||||
* FEATURE: vminsert: automatically add missing port to `-storageNode` hostnames. For example, `-storageNode=vmstorage1,vmstorage2` is automatically translated to `-storageNode=vmstorage1:8400,vmstorage2:8400`. This simplifies [manual setup of VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#cluster-setup).
|
||||
* FEATURE: add [mad(q)](https://docs.victoriametrics.com/victoriametrics/metricsql/#mad) function to [MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/). It calculates [Median absolute deviation](https://en.wikipedia.org/wiki/Median_absolute_deviation) for groups of points with identical timestamps across multiple time series.
|
||||
* FEATURE: add [outliers_mad(tolerance, q)](https://docs.victoriametrics.com/victoriametrics/metricsql/#outliers_mad) function to [MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/). It returns time series with peaks outside the [Median absolute deviation](https://en.wikipedia.org/wiki/Median_absolute_deviation) multiplied by `tolerance`.
|
||||
* FEATURE: add `histogram_quantiles("phiLabel", phi1, ..., phiN, buckets)` function to [MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/). It calculates the given `phi*`-quantiles over the given `buckets` and returns time series per each quantile with the corresponding `{phiLabel="phi*"}` label.
|
||||
* FEATURE: add `quantiles_over_time("phiLabel", phi1, ..., phiN, series_selector[d])` function to [MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/). It calculates the given `phi*`-quantiles over raw samples selected by `series_selector` on the given lookbehind window `d`. It returns time series per each quantile with the corresponding `{phiLabel="phi*"}` label.
|
||||
* FEATURE: [enterprise](https://docs.victoriametrics.com/victoriametrics/enterprise/): do not ask for `-eula` flag if `-version` flag is passed to enterprise app. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1621).
|
||||
|
||||
* BUGFIX: properly handle queries with multiple filters matching empty labels such as `metric{label1=~"foo|",label2="bar|"}`. This filter must match the following series: `metric`, `metric{label1="foo"}`, `metric{label2="bar"}` and `metric{label1="foo",label2="bar"}`. Previously it was matching only `metric{label1="foo",label2="bar"}`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1601).
|
||||
* BUGFIX: vmselect: reset connection timeouts after each request to `vmstorage`. This should prevent from `cannot read data in 0.000 seconds: unexpected EOF` warning in logs. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1562). Thanks to @mxlxm .
|
||||
* BUGFIX: keep metric name for time series returned from [rollup_candlestick](https://docs.victoriametrics.com/metricsql/#rollup_candlestick) function, since the returned series don't change the meaning of the original series. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1600).
|
||||
* BUGFIX: use Prometheus-compatible label value formatting for [count_values](https://docs.victoriametrics.com/metricsql/#count_values) function. Previously big values could be improperly formatted, which could break query results, which rely on label value such as `... on(label) count_values("label", ...)`.
|
||||
* BUGFIX: keep metric name for time series returned from [rollup_candlestick](https://docs.victoriametrics.com/victoriametrics/metricsql/#rollup_candlestick) function, since the returned series don't change the meaning of the original series. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1600).
|
||||
* BUGFIX: use Prometheus-compatible label value formatting for [count_values](https://docs.victoriametrics.com/victoriametrics/metricsql/#count_values) function. Previously big values could be improperly formatted, which could break query results, which rely on label value such as `... on(label) count_values("label", ...)`.
|
||||
* BUGFIX: vmagent: properly use `https` scheme for wildcard TLS certificates for `role: ingress` targets in Kubernetes service discovery. See [this issue](https://github.com/prometheus/prometheus/issues/8902).
|
||||
* BUGFIX: vmagent: support host networking mode for `docker_sd_config`. See [this issue](https://github.com/prometheus/prometheus/issues/9116).
|
||||
* BUGFIX: fix non-repeatable results from `quantile_over_time()` function when the number of input samples exceeds 1000. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1612).
|
||||
* BUGFIX: vmagent: fix EC2 zone discovery when `filters` are specified in [ec2_sc_config](https://docs.victoriametrics.com/sd_configs/#ec2_sd_configs). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1626).
|
||||
* BUGFIX: vmagent: fix EC2 zone discovery when `filters` are specified in [ec2_sc_config](https://docs.victoriametrics.com/victoriametrics/sd_configs/#ec2_sd_configs). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1626).
|
||||
|
||||
## [v1.65.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.65.0)
|
||||
|
||||
Released at 2021-09-01
|
||||
|
||||
* FEATURE: vmagent: add ability to read scrape configs from multiple files specified in `scrape_config_files` section. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1559).
|
||||
* FEATURE: vmagent: reduce memory usage and CPU usage when [Prometheus staleness tracking](https://docs.victoriametrics.com/vmagent/#prometheus-staleness-markers) is enabled for metrics exported from the deleted or disappeared scrape targets.
|
||||
* FEATURE: vmagent: add the ability to limit the number of unique time series scraped per each target. This can be done either globally via `-promscrape.seriesLimitPerTarget` command-line option or on per-target basis via `series_limit` option at `scrape_config` section. See [the updated docs on cardinality limiter](https://docs.victoriametrics.com/vmagent/#cardinality-limiter) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1561).
|
||||
* FEATURE: vmagent: discover `role: ingress` and `role: endpointslice` in [kubernetes_sd_config](https://docs.victoriametrics.com/sd_configs/#kubernetes_sd_configs) via v1 API instead of v1beta1 API if Kubernetes supports it. This fixes service discovery in Kubernetes v1.22 and newer versions. See [these docs](https://kubernetes.io/docs/reference/using-api/deprecation-guide/#ingress-v122).
|
||||
* FEATURE: vmagent: reduce memory usage and CPU usage when [Prometheus staleness tracking](https://docs.victoriametrics.com/victoriametrics/vmagent/#prometheus-staleness-markers) is enabled for metrics exported from the deleted or disappeared scrape targets.
|
||||
* FEATURE: vmagent: add the ability to limit the number of unique time series scraped per each target. This can be done either globally via `-promscrape.seriesLimitPerTarget` command-line option or on per-target basis via `series_limit` option at `scrape_config` section. See [the updated docs on cardinality limiter](https://docs.victoriametrics.com/victoriametrics/vmagent/#cardinality-limiter) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1561).
|
||||
* FEATURE: vmagent: discover `role: ingress` and `role: endpointslice` in [kubernetes_sd_config](https://docs.victoriametrics.com/victoriametrics/sd_configs/#kubernetes_sd_configs) via v1 API instead of v1beta1 API if Kubernetes supports it. This fixes service discovery in Kubernetes v1.22 and newer versions. See [these docs](https://kubernetes.io/docs/reference/using-api/deprecation-guide/#ingress-v122).
|
||||
* FEATURE: take into account failed queries in `vm_request_duration_seconds` summary at `/metrics`. Previously only successful queries were taken into account. This could result in skewed summary. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1537).
|
||||
* FEATURE: vmalert: add an official dashboard for vmalert. See [these docs](https://docs.victoriametrics.com/vmalert/#monitoring).
|
||||
* FEATURE: vmalert: add an official dashboard for vmalert. See [these docs](https://docs.victoriametrics.com/victoriametrics/vmalert/#monitoring).
|
||||
* FEATURE: vmalert: add ability to set additional labels per group via `labels` config section. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1471).
|
||||
* FEATURE: vmalert: add `-disableAlertgroupLabel` command-line flag for disabling the label with alert group name. This may be needed for proper deduplication in Alertmanager. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1532).
|
||||
* FEATURE: update Go builder from v1.16.7 to v1.17.0. This improves data ingestion and query performance by up to 5% according to benchmarks. See [the release post for Go1.17](https://go.dev/blog/go1.17).
|
||||
* FEATURE: vmagent: expose `promscrape_discovery_http_errors_total` metric, which can be used for monitoring the number of failed discovery attempts per each `http_sd` config.
|
||||
* FEATURE: do not reset response cache when a sample with old timestamp is ingested into VictoriaMetrics if `-search.disableAutoCacheReset` command-line option is set. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1570).
|
||||
* FEATURE: add `quantiles("phiLabel", phi1, ..., phiN, q)` aggregate function to [MetricsQL](https://docs.victoriametrics.com/metricsql/), which calculates the given `phi*` quantiles over time series returned by `q`. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1573).
|
||||
* FEATURE: add `quantiles("phiLabel", phi1, ..., phiN, q)` aggregate function to [MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/), which calculates the given `phi*` quantiles over time series returned by `q`. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1573).
|
||||
|
||||
* BUGFIX: rename `sign` function to `sgn` in order to be consistent with PromQL. See [this pull request from Prometheus](https://github.com/prometheus/prometheus/pull/8457).
|
||||
* BUGFIX: vmagent: add `role: endpointslice` in [kubernetes_sd_config](https://docs.victoriametrics.com/sd_configs/#kubernetes_sd_configs) in order to be consistent with Prometheus. Previously this role was supported with incorrect name: `role: endpointslices`. Now both `endpointslice` and `endpointslices` are supported. See [the corresponding code in Prometheus](https://github.com/prometheus/prometheus/blob/2ec6c7dbb82b72834021e01f1773eb90a67a371f/discovery/kubernetes/kubernetes.go#L99).
|
||||
* BUGFIX: vmagent: add `role: endpointslice` in [kubernetes_sd_config](https://docs.victoriametrics.com/victoriametrics/sd_configs/#kubernetes_sd_configs) in order to be consistent with Prometheus. Previously this role was supported with incorrect name: `role: endpointslices`. Now both `endpointslice` and `endpointslices` are supported. See [the corresponding code in Prometheus](https://github.com/prometheus/prometheus/blob/2ec6c7dbb82b72834021e01f1773eb90a67a371f/discovery/kubernetes/kubernetes.go#L99).
|
||||
* BUGFIX: improve the detection of the needed free space for background merge operation. This should prevent from possible out of disk space crashes during big merges. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1560).
|
||||
* BUGFIX: vmauth: remove trailing slash from the full url before requesting it from the backend. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1554).
|
||||
* BUGFIX: [vmbackupmanager](https://docs.victoriametrics.com/vmbackupmanager/): fix timeout error when snapshot takes longer than 10 seconds. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1571).
|
||||
* BUGFIX: [vmbackupmanager](https://docs.victoriametrics.com/victoriametrics/vmbackupmanager/): fix timeout error when snapshot takes longer than 10 seconds. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1571).
|
||||
* BUGFIX: properly parse OpenTSDB `put` messages with multiple spaces between message elements. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1574). Thanks to @envzhu for the fix.
|
||||
|
||||
## [v1.64.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.64.1)
|
||||
|
||||
Released at 2021-08-19
|
||||
|
||||
* FEATURE: add `bitmap_and(q, mask)`, `bitmap_or(q, mask)` and `bitmak_xor(q, mask)` functions to [MetricsQL](https://docs.victoriametrics.com/metricsql/). These functions allow performing bitwise operations over data points in time series. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1541).
|
||||
* FEATURE: add `bitmap_and(q, mask)`, `bitmap_or(q, mask)` and `bitmak_xor(q, mask)` functions to [MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/). These functions allow performing bitwise operations over data points in time series. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1541).
|
||||
* FEATURE: vmalert: add `-remoteWrite.disablePathAppend` command-line flag, which can be used when custom `-remoteWrite.url` must be specified. For example, `./vmalert -disablePathAppend -remoteWrite.url='http://foo.bar/a/b/c?d=e'` would write data to `http://foo.bar/a/b/c?d=e` instead of `http://foo.bar/a/b/c?d=e/api/v1/write`. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1536).
|
||||
* FEATURE: vmagent: add `-promscrape.noStaleMarkers` command-line flag for disabling sending Prometheus stale markers for metrics from disappeared scrape targets. This option may be used for reducing memory usage when scraping big number of metrics with big number of labels and when stale markers aren't needed.
|
||||
* FEATURE: vmselect: add `-search.noStaleMarkers` command-line flag for disabling stale markers handling in queries. This may save some CPU time when the queried data doesn't contain stale markers.
|
||||
@@ -243,7 +244,7 @@ Released at 2021-08-15
|
||||
* FEATURE: add support for Prometheus staleness markers. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1526).
|
||||
* FEATURE: vmagent: automatically generate Prometheus staleness markers for the scraped metrics when scrape targets disappear in the same way as Prometheus does. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1526).
|
||||
* FEATURE: add `present_over_time(m[d])` function, which returns 1 if `m` has a least a single sample over the previous duration `d`. This function has been added also to [Prometheus 2.29](https://github.com/prometheus/prometheus/releases/tag/v2.29.0).
|
||||
* FEATURE: vmagent: support multitenant writes according to [these docs](https://docs.victoriametrics.com/vmagent/#multitenancy). This allows using a single `vmagent` instance in front of VictoriaMetrics cluster for all the tenants. Thanks to @omarghader for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1505). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1491).
|
||||
* FEATURE: vmagent: support multitenant writes according to [these docs](https://docs.victoriametrics.com/victoriametrics/vmagent/#multitenancy). This allows using a single `vmagent` instance in front of VictoriaMetrics cluster for all the tenants. Thanks to @omarghader for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1505). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1491).
|
||||
* FEATURE: vmagent: add `__meta_ec2_availability_zone_id` label to discovered Amazon EC2 targets. This label is available in Prometheus [starting from v2.29](https://github.com/prometheus/prometheus/releases/tag/v2.29.0).
|
||||
* FEATURE: vmagent: add `__meta_gce_interface_ipv4_<name>` labels to discovered GCE targets. These labels are available in Prometheus [starting from v2.29](https://github.com/prometheus/prometheus/releases/tag/v2.29.0).
|
||||
* FEATURE: add `-search.maxSamplesPerSeries` command-line flag for limiting the number of raw samples a single query can process per each time series. This option can protect from out of memory errors when a query processes tens of millions of raw samples per series. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1067).
|
||||
@@ -257,7 +258,7 @@ Released at 2021-08-15
|
||||
* BUGFIX: fix corner cases for queries on time ranges exceeding 40 days. Previously some series can be missing in query results. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1486).
|
||||
* BUGFIX: vmselect: return dummy response at `/rules` page in the same way as for `/api/v1/rules` page. The `/rules` page is requested by Grafana 8. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1493) for details.
|
||||
* BUGFIX: vmbackup: automatically set default `us-east-1` S3 region if it is missing. This should simplify using S3-compatible services such as MinIO for backups. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1449).
|
||||
* BUGFIX: vmselect: prevent from possible deadlock when multiple `target` query args are passed to [Graphite Render API](https://docs.victoriametrics.com/#graphite-render-api-usage).
|
||||
* BUGFIX: vmselect: prevent from possible deadlock when multiple `target` query args are passed to [Graphite Render API](https://docs.victoriametrics.com/victoriametrics#graphite-render-api-usage).
|
||||
* BUGFIX: return series with `a op b` labels and `N` values for `(a op b) default N` if `(a op b)` returns series with all NaN values. Previously such series were removed.
|
||||
* BUGFIX: vmui: fix layout when the query selects more than 27 time series. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1497).
|
||||
* BUGFIX: vmagent: restore highlighting in red for DOWN targets at `/targets` page. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1461).
|
||||
@@ -270,8 +271,8 @@ Released at 2021-07-15
|
||||
* FEATURE: vmselect: embed [vmui](https://github.com/VictoriaMetrics/vmui) into a single-node VictoriaMetrics and into `vmselect` component of cluster version. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1413). The web interface is available at the following paths:
|
||||
* `/vmui/` for a single-node VictoriaMetrics
|
||||
* `/select/<accountID>/vmui/` for `vmselect` at cluster version of VictoriaMetrics
|
||||
* FEATURE: support durations anywhere in [MetricsQL queries](https://docs.victoriametrics.com/metricsql/). For example, `sum_over_time(m[1h]) / 1h` is a valid query, which is equivalent to `sum_over_time(m[1h]) / 3600`.
|
||||
* FEATURE: support durations without suffixes in [MetricsQL queries](https://docs.victoriametrics.com/metricsql/). For example, `rate(m[3600])` is a valid query, which is equivalent to `rate(m[1h])`.
|
||||
* FEATURE: support durations anywhere in [MetricsQL queries](https://docs.victoriametrics.com/victoriametrics/metricsql/). For example, `sum_over_time(m[1h]) / 1h` is a valid query, which is equivalent to `sum_over_time(m[1h]) / 3600`.
|
||||
* FEATURE: support durations without suffixes in [MetricsQL queries](https://docs.victoriametrics.com/victoriametrics/metricsql/). For example, `rate(m[3600])` is a valid query, which is equivalent to `rate(m[1h])`.
|
||||
* FEATURE: export `vmselect_request_duration_seconds` and `vminsert_request_duration_seconds` [VictoriaMetrics histograms](https://valyala.medium.com/improving-histogram-usability-for-prometheus-and-grafana-bc7e5df0e350) at `/metrics` page. These histograms can be used for determining latency distribution and SLI/SLO for the served requests. For example, the following query would return the percent of queries that took less than 500ms during the last hour: `histogram_share(500ms, sum(rate(vmselect_request_duration_seconds_bucket[1h])) by (vmrange))`.
|
||||
* FEATURE: vmagent: dynamically reload client TLS certificates from disk on every [mTLS connection](https://developers.cloudflare.com/cloudflare-one/identity/devices/mutual-tls-authentication). This should allow using `vmagent` with [Istio service mesh](https://istio.io/latest/about/service-mesh/). See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1420).
|
||||
* FEATURE: log http request path plus all the query args on errors during request processing. Previously only http request path was logged without query args, so it could be hard debugging such errors.
|
||||
@@ -287,15 +288,15 @@ Released at 2021-07-15
|
||||
|
||||
Released at 2021-06-25
|
||||
|
||||
* FEATURE: vmagent: add service discovery for Docker (aka [docker_sd_config](https://docs.victoriametrics.com/sd_configs/#docker_sd_configs)). See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1402).
|
||||
* FEATURE: vmagent: add service discovery for DigitalOcean (aka [digitalocean_sd_config](https://docs.victoriametrics.com/sd_configs/#digitalocean_sd_configs)). See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1367).
|
||||
* FEATURE: vmagent: add service discovery for Docker (aka [docker_sd_config](https://docs.victoriametrics.com/victoriametrics/sd_configs/#docker_sd_configs)). See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1402).
|
||||
* FEATURE: vmagent: add service discovery for DigitalOcean (aka [digitalocean_sd_config](https://docs.victoriametrics.com/victoriametrics/sd_configs/#digitalocean_sd_configs)). See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1367).
|
||||
* FEATURE: vmagent: change the default value for `-remoteWrite.queues` from 4 to `2 * numCPUs`. This should reduce scrape duration for highly loaded vmagent, which scrapes tens of thousands of targets. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1385).
|
||||
* FEATURE: vmagent: show the number of samples the target returns during the last scrape on `/targets` and `/api/v1/targets` pages. This should simplify debugging targets, which may return too big or too low number of samples. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1377).
|
||||
* FEATURE: vmagent: show jobs with zero discovered targets on `/targets` page. This should help debugging improperly configured scrape configs.
|
||||
* FEATURE: vmagent: support for http-based service discovery (aka [http_sd_config](https://docs.victoriametrics.com/sd_configs/#http_sd_configs)), which has been added since Prometheus 2.28. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1392).
|
||||
* FEATURE: vmagent: support for http-based service discovery (aka [http_sd_config](https://docs.victoriametrics.com/victoriametrics/sd_configs/#http_sd_configs)), which has been added since Prometheus 2.28. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1392).
|
||||
* FEATURE: vmagent: support namespace in Consul service discovery in the same way as Prometheus 2.28 does. See [this issue](https://github.com/prometheus/prometheus/issues/8894) for details.
|
||||
* FEATURE: vmagent: support generic auth configs in `consul_sd_configs` in the same way as Prometheus 2.28 does. See [this issue](https://github.com/prometheus/prometheus/issues/8924) for details.
|
||||
* FEATURE: [vmctl](https://docs.victoriametrics.com/vmctl/): limit the number of samples per each imported JSON line. This should limit the memory usage at VictoriaMetrics side when importing time series with big number of samples.
|
||||
* FEATURE: [vmctl](https://docs.victoriametrics.com/victoriametrics/vmctl/): limit the number of samples per each imported JSON line. This should limit the memory usage at VictoriaMetrics side when importing time series with big number of samples.
|
||||
* FEATURE: vmselect: log slow queries across all the `/api/v1/*` handlers (aka [Prometheus query API](https://prometheus.io/docs/prometheus/latest/querying/api)) if their execution duration exceeds `-search.logSlowQueryDuration`. This should simplify debugging slow requests to such handlers as `/api/v1/labels` or `/api/v1/series` additionally to `/api/v1/query` and `/api/v1/query_range`, which were logged in the previous releases.
|
||||
* FEATURE: vminsert: sort the `-storageNode` list in order to guarantee the identical `series -> vmstorage` mapping across all the `vminsert` nodes. This should reduce resource usage (RAM, CPU and disk IO) at `vmstorage` nodes if `vmstorage` addresses are passed in random order to `vminsert` nodes.
|
||||
* FEATURE: vmstorage: reduce memory usage on a system with many CPU cores under high ingestion rate.
|
||||
@@ -303,54 +304,54 @@ Released at 2021-06-25
|
||||
* BUGFIX: prevent from adding new samples to deleted time series after the rotation of the inverted index (the rotation is performed once per `-retentionPeriod`). See [this comment](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1347#issuecomment-861232136) for details.
|
||||
* BUGFIX: vmstorage: reduce high disk write IO usage on systems with big number of CPU cores. The issue has been introduced in the release [v1.59.0](#v1590). See [this commit](https://github.com/VictoriaMetrics/VictoriaMetrics/commit/aa9b56a046b6ae8083fa659df35dd5e994bf9115) and [this comment](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1338#issuecomment-863046999) for details.
|
||||
* BUGFIX: vmstorage: prevent from incorrect stats collection when multiple concurrent queries execute the same tag filter. This may help reducing CPU usage under certain workloads. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1338).
|
||||
* BUGFIX: vmselect: return the last timestamp for the max / min value from `tmax_over_time(m[d])` and `tmin_over_time(m[d])` [MetricsQL functions](https://docs.victoriametrics.com/metricsql/) as most users expect. See also [this issue](https://github.com/prometheus/prometheus/issues/8966).
|
||||
* BUGFIX: vmselect: return the expected value for `increase_pure()` [MetricsQL function](https://docs.victoriametrics.com/metricsql/) after a gap in a time series. Previously incorrect too big value could be returned after the gap from `increase_pure()`.
|
||||
* BUGFIX: vmselect: return the last timestamp for the max / min value from `tmax_over_time(m[d])` and `tmin_over_time(m[d])` [MetricsQL functions](https://docs.victoriametrics.com/victoriametrics/metricsql/) as most users expect. See also [this issue](https://github.com/prometheus/prometheus/issues/8966).
|
||||
* BUGFIX: vmselect: return the expected value for `increase_pure()` [MetricsQL function](https://docs.victoriametrics.com/victoriametrics/metricsql/) after a gap in a time series. Previously incorrect too big value could be returned after the gap from `increase_pure()`.
|
||||
|
||||
## [v1.61.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.61.1)
|
||||
|
||||
Released at 2021-06-11
|
||||
|
||||
* BUGFIX: vmalert: fix recording rules, which were broken in v1.61.0. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1369).
|
||||
* BUGFIX: reset the on-disk cache for mapping from the full metric name to an internal metric id (e.g. `metric_name{labels} -> internal_metric_id`) after deleting metrics via [delete API](https://docs.victoriametrics.com/#how-to-delete-time-series). This should prevent from possible inconsistent state after unclean shutdown. This [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1347).
|
||||
* BUGFIX: reset the on-disk cache for mapping from the full metric name to an internal metric id (e.g. `metric_name{labels} -> internal_metric_id`) after deleting metrics via [delete API](https://docs.victoriametrics.com/victoriametrics#how-to-delete-time-series). This should prevent from possible inconsistent state after unclean shutdown. This [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1347).
|
||||
|
||||
## [v1.61.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.61.0)
|
||||
|
||||
Released at 2021-06-09
|
||||
|
||||
* FEATURE: vmalert: add support for backfilling (aka replay) of recording and alerting rules. See [these docs](https://docs.victoriametrics.com/vmalert/#rules-backfilling) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/836).
|
||||
* FEATURE: vmalert: add support for backfilling (aka replay) of recording and alerting rules. See [these docs](https://docs.victoriametrics.com/victoriametrics/vmalert/#rules-backfilling) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/836).
|
||||
* FEATURE: vmalert: add a command-line flag `-rule.configCheckInterval` for automatic re-reading of `-rule` files without the need to send SIGHUP signal. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/512).
|
||||
* FEATURE: vmagent: respect the `sample_limit` and `-promscrape.maxScrapeSize` values when scraping targets in [stream parsing mode](https://docs.victoriametrics.com/vmagent/#stream-parsing-mode). See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1331).
|
||||
* FEATURE: vmauth: add ability to specify multiple `url_prefix` entries for balancing the load among multiple `vmselect` and/or `vminsert` nodes in a cluster. See [these docs](https://docs.victoriametrics.com/vmauth/#load-balancing).
|
||||
* FEATURE: vmagent: respect the `sample_limit` and `-promscrape.maxScrapeSize` values when scraping targets in [stream parsing mode](https://docs.victoriametrics.com/victoriametrics/vmagent/#stream-parsing-mode). See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1331).
|
||||
* FEATURE: vmauth: add ability to specify multiple `url_prefix` entries for balancing the load among multiple `vmselect` and/or `vminsert` nodes in a cluster. See [these docs](https://docs.victoriametrics.com/victoriametrics/vmauth/#load-balancing).
|
||||
* FEATURE: vminsert: add `-disableRerouting` command-line flag for forcibly disabling the rerouting. This should help resolving [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/791) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1054) issues.
|
||||
* FEATURE: vminsert: reduce the probability of global re-routing storm if all the vmstorage nodes cannot keep up with the given ingestion rate for some time. This should improve cluster stability in such cases. See [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/791) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1054) issues.
|
||||
* FEATURE: allow building VictoriaMetrics components for Solaris / SmartOS. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1322).
|
||||
* FEATURE: vmagent: add ability to debug relabeling rules. See [these docs](https://docs.victoriametrics.com/vmagent/#relabeling) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1343).
|
||||
* FEATURE: vmagent: add ability to debug relabeling rules. See [these docs](https://docs.victoriametrics.com/victoriametrics/vmagent/#relabeling) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1343).
|
||||
|
||||
* BUGFIX: reduce CPU usage by up to 2x during querying a database with big number of active daily time series. The issue has been introduced in `v1.59.0`.
|
||||
* BUGFIX: vmagent: properly apply auth and tls configs in `eureka_sd_configs`. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1350).
|
||||
* BUGFIX: vmauth: do not panic on aborted http requests. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1353).
|
||||
* BUGFIX: properly generate `target` property for `*Series(foo.*.bar)` responses returned from [Graphite Render API](https://docs.victoriametrics.com/#graphite-render-api-usage). Previously the `target` contained the expanded list of series for `foo.*.bar`, e.g. `sumSeries(foo.a.bar,foo.b.bar,...foo.z.bar)`. Now VictoriaMetrics returns `sumSeries(foo.*.bar)` as a target in the same way as Graphite does.
|
||||
* BUGFIX: properly generate `target` property for `*Series(foo.*.bar)` responses returned from [Graphite Render API](https://docs.victoriametrics.com/victoriametrics#graphite-render-api-usage). Previously the `target` contained the expanded list of series for `foo.*.bar`, e.g. `sumSeries(foo.a.bar,foo.b.bar,...foo.z.bar)`. Now VictoriaMetrics returns `sumSeries(foo.*.bar)` as a target in the same way as Graphite does.
|
||||
|
||||
## [v1.60.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.60.0)
|
||||
|
||||
Released at 2021-05-24
|
||||
|
||||
* FEATURE: add ability to limit the number of unique time series, which can be added to storage per hour and per day. This can help dealing with high cardinality and high churn rate issues. See [these docs](https://docs.victoriametrics.com/#cardinality-limiter).
|
||||
* FEATURE: vmagent: add ability to limit the number of unique time series, which can be sent to remote storage systems per hour and per day. This can help dealing with high cardinality and high churn rate issues. See [these docs](https://docs.victoriametrics.com/vmagent/#cardinality-limiter).
|
||||
* FEATURE: vmalert: add ability to run alerting and recording rules for multiple tenants. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/740) and [these docs](https://docs.victoriametrics.com/vmalert/#multitenancy).
|
||||
* FEATURE: vminsert: add support for data ingestion via other `vminsert` nodes. This allows building multi-level data ingestion paths in VictoriaMetrics cluster by writing data from one level of `vminsert` nodes to another level of `vminsert` nodes. See [these docs](https://docs.victoriametrics.com/cluster-victoriametrics/#multi-level-cluster-setup) and [this comment](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/541#issuecomment-835487858) for details.
|
||||
* FEATURE: add ability to limit the number of unique time series, which can be added to storage per hour and per day. This can help dealing with high cardinality and high churn rate issues. See [these docs](https://docs.victoriametrics.com/victoriametrics#cardinality-limiter).
|
||||
* FEATURE: vmagent: add ability to limit the number of unique time series, which can be sent to remote storage systems per hour and per day. This can help dealing with high cardinality and high churn rate issues. See [these docs](https://docs.victoriametrics.com/victoriametrics/vmagent/#cardinality-limiter).
|
||||
* FEATURE: vmalert: add ability to run alerting and recording rules for multiple tenants. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/740) and [these docs](https://docs.victoriametrics.com/victoriametrics/vmalert/#multitenancy).
|
||||
* FEATURE: vminsert: add support for data ingestion via other `vminsert` nodes. This allows building multi-level data ingestion paths in VictoriaMetrics cluster by writing data from one level of `vminsert` nodes to another level of `vminsert` nodes. See [these docs](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#multi-level-cluster-setup) and [this comment](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/541#issuecomment-835487858) for details.
|
||||
* FEATURE: vmagent: reload `bearer_token_file`, `credentials_file` and `password_file` contents every second. This allows dynamically changing the contents of these files during target scraping and service discovery without the need to restart `vmagent`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1297).
|
||||
* FEATURE: vmalert: add a flag to control behaviour on startup for state restore errors. Such errors were returned and logged before as well. Now user can specify whether to just log these errors (`-remoteRead.ignoreRestoreErrors=true`) or to stop the process (`-remoteRead.ignoreRestoreErrors=false`). The latter is important when VM isn't ready yet to serve queries from vmalert and it needs to wait. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1252).
|
||||
* FEATURE: vmalert: add ability to pass `round_digits` query arg to datasource via `-datasource.roundDigits` command-line flag. This can be used for limiting the number of decimal digits after the point in recording rule results. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/525).
|
||||
* FEATURE: return `X-Server-Hostname` header in http responses of all the VictoriaMetrics components. This should simplify tracing the origin server behind a load balancer or behind auth proxy during troubleshooting.
|
||||
* FEATURE: vmselect: allow to use 2x more memory for query processing at `vmselect` nodes in [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/). This should allow processing heavy queries without the need to increase RAM size at `vmselect` nodes.
|
||||
* FEATURE: add ability to filter `/api/v1/status/tsdb` output with arbitrary [time series selectors](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors) passed via `match[]` query args. See [these docs](https://docs.victoriametrics.com/#tsdb-stats) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1168) for details.
|
||||
* FEATURE: vmselect: allow to use 2x more memory for query processing at `vmselect` nodes in [VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/). This should allow processing heavy queries without the need to increase RAM size at `vmselect` nodes.
|
||||
* FEATURE: add ability to filter `/api/v1/status/tsdb` output with arbitrary [time series selectors](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors) passed via `match[]` query args. See [these docs](https://docs.victoriametrics.com/victoriametrics#tsdb-stats) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1168) for details.
|
||||
* FEATURE: automatically detect memory and cpu limits for VictoriaMetrics components running under [cgroup v2](https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html) environments such as [HashiCorp Nomad](https://www.nomadproject.io/). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1269).
|
||||
* FEATURE: vmauth: allow `-auth.config` reloading via `/-/reload` http endpoint. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1194).
|
||||
* FEATURE: add `timezone_offset(tz)` function. It returns offset in seconds for the given timezone `tz` relative to UTC. This can be useful when combining with datetime-related functions. For example, `day_of_week(time()+timezone_offset("America/Los_Angeles"))` would return weekdays for `America/Los_Angeles` time zone. Special `Local` time zone can be used for returning an offset for the time zone set on the host where VictoriaMetrics runs. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1306) and [MetricsQL docs](https://docs.victoriametrics.com/metricsql/) for more details.
|
||||
* FEATURE: add `timezone_offset(tz)` function. It returns offset in seconds for the given timezone `tz` relative to UTC. This can be useful when combining with datetime-related functions. For example, `day_of_week(time()+timezone_offset("America/Los_Angeles"))` would return weekdays for `America/Los_Angeles` time zone. Special `Local` time zone can be used for returning an offset for the time zone set on the host where VictoriaMetrics runs. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1306) and [MetricsQL docs](https://docs.victoriametrics.com/victoriametrics/metricsql/) for more details.
|
||||
* FEATURE: vmagent: add support for OAuth2 authorization for scrape targets and service discovery in the same way as Prometheus does. See [these docs](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#oauth2).
|
||||
* FEATURE: vmagent: add support for OAuth2 authorization when writing data to `-remoteWrite.url`. See `-remoteWrite.oauth2.*` config params in `/path/to/vmagent -help` output.
|
||||
* FEATURE: vmalert: add ability to set `extra_filter_labels` at alerting and recording group configs. See [these docs](https://docs.victoriametrics.com/vmalert/#groups).
|
||||
* FEATURE: vmalert: add ability to set `extra_filter_labels` at alerting and recording group configs. See [these docs](https://docs.victoriametrics.com/victoriametrics/vmalert/#groups).
|
||||
* FEATURE: vmstorage: reduce memory usage by up to 30% when ingesting big number of active time series.
|
||||
|
||||
* BUGFIX: vmagent: do not retry scraping targets, which don't support HTTP. This should reduce CPU load and network usage at `vmagent` and at scrape target. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1289).
|
||||
@@ -373,8 +374,8 @@ Released at 2021-05-01
|
||||
|
||||
* FEATURE: improved new time series registration speed on systems with many CPU cores. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1244). Thanks to @waldoweng for the idea and [draft implementation](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1243).
|
||||
* FEATURE: vmalert: use the same technique as Grafana for determining evaluation timestamps for recording rules. This should make consistent graphs for series generated by recording rules compared to graphs generated for queries from recording rules in Grafana. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1232).
|
||||
* FEATURE: vmauth: add ability to set mandatory query args in `url_prefix`. For example, `url_prefix: http://vm:8428/?extra_label=team=dev` would add `extra_label=team=dev` query arg to all the incoming requests. See [the example](https://docs.victoriametrics.com/vmauth/#auth-config) for more details.
|
||||
* FEATURE: vmctl: add OpenTSDB migration option. See more details [here](https://docs.victoriametrics.com/vmctl#migrating-data-from-opentsdb).
|
||||
* FEATURE: vmauth: add ability to set mandatory query args in `url_prefix`. For example, `url_prefix: http://vm:8428/?extra_label=team=dev` would add `extra_label=team=dev` query arg to all the incoming requests. See [the example](https://docs.victoriametrics.com/victoriametrics/vmauth/#auth-config) for more details.
|
||||
* FEATURE: vmctl: add OpenTSDB migration option. See more details [here](https://docs.victoriametrics.com/victoriametrics/vmctl#migrating-data-from-opentsdb).
|
||||
Thanks to @johnseekins!
|
||||
* FEATURE: log metrics with dropped labels if the number of labels in the ingested metric exceeds `-maxLabelsPerTimeseries`. This should simplify debugging for this case.
|
||||
* FEATURE: vmagent: list user-visible endpoints at `http://vmagent:8429/`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1251).
|
||||
@@ -389,8 +390,8 @@ Thanks to @johnseekins!
|
||||
Released at 2021-04-08
|
||||
|
||||
* FEATURE: vminsert and vmagent: add `-sortLabels` command-line flag for sorting metric labels before pushing them to `vmstorage`. This should reduce the size of `MetricName -> internal_series_id` cache (aka `vm_cache_size_bytes{type="storage/tsid"}`) when ingesting samples for the same time series with distinct order of labels. For example, `foo{k1="v1",k2="v2"}` and `foo{k2="v2",k1="v1"}` represent a single time series. Labels sorting is disabled by default, since the majority of established exporters preserve the order of labels for the exported metrics.
|
||||
* FEATURE: allow specifying label value alongside label name for the `others sum` time series returned from `topk_*` and `bottomk_*` functions from [MetricsQL](https://docs.victoriametrics.com/metricsql/). For example, `topk_avg(3, max(process_resident_memory_bytes) by (instance), "instance=other_sum")` would return top 3 series from `max(process_resident_memory_bytes) by (instance)` plus a series containing the sum of other series. The `others sum` series will have `{instance="other_sum"}` label.
|
||||
* FEATURE: do not delete `dst_label` when applying `label_copy(q, "src_label", "dst_label")` and `label_move(q, "src_label", "dst_label")` to series without `src_label` and with non-empty `dst_label`. See more details at [MetricsQL docs](https://docs.victoriametrics.com/metricsql/).
|
||||
* FEATURE: allow specifying label value alongside label name for the `others sum` time series returned from `topk_*` and `bottomk_*` functions from [MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/). For example, `topk_avg(3, max(process_resident_memory_bytes) by (instance), "instance=other_sum")` would return top 3 series from `max(process_resident_memory_bytes) by (instance)` plus a series containing the sum of other series. The `others sum` series will have `{instance="other_sum"}` label.
|
||||
* FEATURE: do not delete `dst_label` when applying `label_copy(q, "src_label", "dst_label")` and `label_move(q, "src_label", "dst_label")` to series without `src_label` and with non-empty `dst_label`. See more details at [MetricsQL docs](https://docs.victoriametrics.com/victoriametrics/metricsql/).
|
||||
* FEATURE: update Go builder from `v1.16.2` to `v1.16.3`. This should fix [these issues](https://github.com/golang/go/issues?q=milestone%3AGo1.16.3+label%3ACherryPickApproved).
|
||||
* FEATURE: vmagent: add support for `follow_redirects` option to `scrape_configs` section in the same way as [Prometheus 2.26 does](https://github.com/prometheus/prometheus/pull/8546).
|
||||
* FEATURE: vmagent: add support for `authorization` section in `-promscrape.config` in the same way as [Prometheus 2.26 does](https://github.com/prometheus/prometheus/pull/8512).
|
||||
@@ -401,11 +402,11 @@ Released at 2021-04-08
|
||||
* FEATURE: vmagent: add support for `proxy_tls_config`, `proxy_authorization`, `proxy_basic_auth`, `proxy_bearer_token` and `proxy_bearer_token_file` options in `consul_sd_config`, `dockerswarm_sd_config` and `eureka_sd_config` sections.
|
||||
* FEATURE: vmagent: pass `X-Prometheus-Scrape-Timeout-Seconds` header to scrape targets as Prometheus does. In this case scrape targets can limit the time needed for performing the scrape. See [this comment](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1179#issuecomment-813118733) for details.
|
||||
* FEATURE: vmagent: drop corrupted persistent queue files at `-remoteWrite.tmpDataPath` instead of throwing a fatal error. Corrupted files can appear after unclean shutdown of `vmagent` such as OOM kill or hardware reset. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1030).
|
||||
* FEATURE: vmauth: add support for authorization via [bearer token](https://swagger.io/docs/specification/authentication/bearer-authentication/). See [the docs](https://docs.victoriametrics.com/vmauth/#auth-config) for details.
|
||||
* FEATURE: vmauth: add support for authorization via [bearer token](https://swagger.io/docs/specification/authentication/bearer-authentication/). See [the docs](https://docs.victoriametrics.com/victoriametrics/vmauth/#auth-config) for details.
|
||||
* FEATURE: publish `arm64` and `amd64` binaries for cluster version of VictoriaMetrics at [releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases).
|
||||
|
||||
* BUGFIX: properly handle `/api/v1/labels` and `/api/v1/label/<label_name>/values` queries on big `start ... end` time range. This should fix big resource usage when VictoriaMetrics is queried with [Promxy](https://github.com/jacksontj/promxy) v0.0.62 or newer versions.
|
||||
* BUGFIX: do not break sort order for series returned from `topk*`, `bottomk*` and `outliersk` [MetricsQL](https://docs.victoriametrics.com/metricsql/) functions. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1189).
|
||||
* BUGFIX: do not break sort order for series returned from `topk*`, `bottomk*` and `outliersk` [MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/) functions. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1189).
|
||||
* BUGFIX: vmagent: properly work with simple HTTP proxies which don't support `CONNECT` method. For example, [PushProx](https://github.com/prometheus-community/PushProx). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1179).
|
||||
* BUGFIX: vmagent: properly discover targets if multiple namespace selectors are put inside `kubernetes_sd_config`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1170).
|
||||
* BUGFIX: vmagent: properly discover `role: endpoints` and `role: endpointslices` targets in `kubernetes_sd_config`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1182).
|
||||
@@ -432,37 +433,37 @@ Released at 2021-03-29
|
||||
* `process_resident_memory_shared_bytes` - RSS share for memory shared with other processes (aka shared memory). This share can be freed by the OS at any time, so it must be ignored by OOM killer.
|
||||
* `process_resident_memory_peak_bytes` - peak RSS usage for the process.
|
||||
* `process_virtual_memory_peak_bytes` - peak virtual memory usage for the process.
|
||||
* FEATURE: accept and enforce `extra_label=<label_name>=<label_value>` query arg at [Graphite APIs](https://docs.victoriametrics.com/#graphite-api-usage).
|
||||
* FEATURE: accept and enforce `extra_label=<label_name>=<label_value>` query arg at [Graphite APIs](https://docs.victoriametrics.com/victoriametrics#graphite-api-usage).
|
||||
* FEATURE: use InfluxDB field as metric name if measurement is empty and `-influxSkipSingleField` command-line is set. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1139).
|
||||
* FEATURE: vmagent: add `-promscrape.consul.waitTime` command-line flag for tuning the maximum wait time for Consul service discovery. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1144).
|
||||
* FEATURE: vmagent: add `vm_promscrape_discovery_kubernetes_stale_resource_versions_total` metric for monitoring the frequency of `too old resource version` errors during Kubernetes service discovery.
|
||||
* FEATURE: single-node VictoriaMetrics: log metrics with timestamps older than `-search.cacheTimestampOffset` compared to the current time. See [these docs](https://docs.victoriametrics.com/#backfilling) for details.
|
||||
* FEATURE: single-node VictoriaMetrics: log metrics with timestamps older than `-search.cacheTimestampOffset` compared to the current time. See [these docs](https://docs.victoriametrics.com/victoriametrics#backfilling) for details.
|
||||
|
||||
* BUGFIX: prevent from infinite loop on `{__graphite__="..."}` filters when a metric name contains `*`, `{` or `[` chars.
|
||||
* BUGFIX: prevent from infinite loop in `/metrics/find` and `/metrics/expand` [Graphite Metrics API handlers](https://docs.victoriametrics.com/#graphite-metrics-api-usage) when they match metric names or labels with `*`, `{` or `[` chars.
|
||||
* BUGFIX: prevent from infinite loop in `/metrics/find` and `/metrics/expand` [Graphite Metrics API handlers](https://docs.victoriametrics.com/victoriametrics#graphite-metrics-api-usage) when they match metric names or labels with `*`, `{` or `[` chars.
|
||||
* BUGFIX: do not merge duplicate time series during requests to `/api/v1/query`. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1141>
|
||||
* BUGFIX: vmagent: properly handle `too old resource version` error messages from Kubernetes watch API. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1150>
|
||||
* BUGFIX: vmagent: do not retry sending data blocks if remote storage returns `400 Bad Request` error. The number of dropped blocks due to such errors can be monitored with `vmagent_remotewrite_packets_dropped_total` metrics. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1149>
|
||||
* BUGFIX: properly calculate `summarize` and `*Series` functions in [Graphite Render API](https://docs.victoriametrics.com/#graphite-render-api-usage).
|
||||
* BUGFIX: properly calculate `summarize` and `*Series` functions in [Graphite Render API](https://docs.victoriametrics.com/victoriametrics#graphite-render-api-usage).
|
||||
|
||||
## [v1.56.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.56.0)
|
||||
|
||||
Released at 2021-03-17
|
||||
|
||||
* FEATURE: add the following functions to [MetricsQL](https://docs.victoriametrics.com/metricsql/):
|
||||
* FEATURE: add the following functions to [MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/):
|
||||
* `histogram_avg(buckets)` - returns the average value for the given buckets.
|
||||
* `histogram_stdvar(buckets)` - returns standard variance for the given buckets.
|
||||
* `histogram_stddev(buckets)` - returns standard deviation for the given buckets.
|
||||
* FEATURE: export `vm_available_memory_bytes` and `vm_available_cpu_cores` metrics, which show the number of available RAM and available CPU cores for VictoriaMetrics apps.
|
||||
* FEATURE: export `vm_index_search_duration_seconds` histogram, which can be used for troubleshooting time series search performance.
|
||||
* FEATURE: vmagent: add ability to replicate scrape targets among `vmagent` instances in the cluster with `-promscrape.cluster.replicationFactor` command-line flag. See [these docs](https://docs.victoriametrics.com/vmagent/#scraping-big-number-of-targets).
|
||||
* FEATURE: vmagent: accept `scrape_offset` option at `scrape_config`. This option may be useful when scrapes must start at the specified offset of every scrape interval. See [these docs](https://docs.victoriametrics.com/vmagent/#troubleshooting) for details.
|
||||
* FEATURE: vmagent: support `proxy_tls_config`, `proxy_basic_auth`, `proxy_bearer_token` and `proxy_bearer_token_file` options at `scrape_config` section for configuring proxies specified via `proxy_url`. See [these docs](https://docs.victoriametrics.com/vmagent/#scraping-targets-via-a-proxy).
|
||||
* FEATURE: vmagent: add ability to replicate scrape targets among `vmagent` instances in the cluster with `-promscrape.cluster.replicationFactor` command-line flag. See [these docs](https://docs.victoriametrics.com/victoriametrics/vmagent/#scraping-big-number-of-targets).
|
||||
* FEATURE: vmagent: accept `scrape_offset` option at `scrape_config`. This option may be useful when scrapes must start at the specified offset of every scrape interval. See [these docs](https://docs.victoriametrics.com/victoriametrics/vmagent/#troubleshooting) for details.
|
||||
* FEATURE: vmagent: support `proxy_tls_config`, `proxy_basic_auth`, `proxy_bearer_token` and `proxy_bearer_token_file` options at `scrape_config` section for configuring proxies specified via `proxy_url`. See [these docs](https://docs.victoriametrics.com/victoriametrics/vmagent/#scraping-targets-via-a-proxy).
|
||||
* FEATURE: vmauth: allow using regexp paths in `url_map`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1112) for details.
|
||||
* FEATURE: accept `round_digits` query arg at `/api/v1/query` and `/api/v1/query_range` handlers. This option can be set at Prometheus datasource in Grafana for limiting the number of digits after the decimal point in response values.
|
||||
* FEATURE: add `-influx.databaseNames` command-line flag, which can be used for accepting data from some Telegraf plugins such as [fluentd plugin](https://github.com/fangli/fluent-plugin-influxdb). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1124).
|
||||
* FEATURE: add `-logNewSeries` command-line flag, which can be used for debugging the source of time series churn rate.
|
||||
* FEATURE: publish Windows builds for [vmagent](https://docs.victoriametrics.com/vmagent/), [vmalert](https://docs.victoriametrics.com/vmalert/), [vmauth](https://docs.victoriametrics.com/vmauth/) and [vmctl](https://docs.victoriametrics.com/vmctl/) at `vmutils-windows-*.zip` archives at [releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases).
|
||||
* FEATURE: publish Windows builds for [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/), [vmalert](https://docs.victoriametrics.com/victoriametrics/vmalert/), [vmauth](https://docs.victoriametrics.com/victoriametrics/vmauth/) and [vmctl](https://docs.victoriametrics.com/victoriametrics/vmctl/) at `vmutils-windows-*.zip` archives at [releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases).
|
||||
* FEATURE: listen for IPv6 UDP if `-enableTCP6` command-line flag is passed to VictoriaMetrics. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1131).
|
||||
|
||||
* BUGFIX: vmagent: prevent from high CPU usage bug during failing scrapes with small `scrape_timeout` (less than a few seconds).
|
||||
@@ -486,8 +487,8 @@ Released at 2021-03-03
|
||||
|
||||
Released at 2021-03-02
|
||||
|
||||
* FEATURE: add `sign(q)` and `clamp(q, min, max)` functions, which are planned to be added in [the upcoming Prometheus release](https://twitter.com/roidelapluie/status/1363428376162295811) . The `last_over_time(m[d])` function is already supported in [MetricsQL](https://docs.victoriametrics.com/metricsql/).
|
||||
* FEATURE: vmagent: add `scrape_align_interval` config option, which can be used for aligning scrapes to the beginning of the configured interval. See [these docs](https://docs.victoriametrics.com/vmagent/#troubleshooting) for details.
|
||||
* FEATURE: add `sign(q)` and `clamp(q, min, max)` functions, which are planned to be added in [the upcoming Prometheus release](https://twitter.com/roidelapluie/status/1363428376162295811) . The `last_over_time(m[d])` function is already supported in [MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/).
|
||||
* FEATURE: vmagent: add `scrape_align_interval` config option, which can be used for aligning scrapes to the beginning of the configured interval. See [these docs](https://docs.victoriametrics.com/victoriametrics/vmagent/#troubleshooting) for details.
|
||||
* FEATURE: expose io-related metrics at `/metrics` page for every VictoriaMetrics component:
|
||||
* `process_io_read_bytes_total` - the number of bytes read via io syscalls such as read and pread
|
||||
* `process_io_written_bytes_total` - the number of bytes written via io syscalls such as write and pwrite
|
||||
@@ -495,12 +496,12 @@ Released at 2021-03-02
|
||||
* `process_io_write_syscalls_total` - the number of write syscalls such as write and pwrite
|
||||
* `process_io_storage_read_bytes_total` - the number of bytes read from storage layer
|
||||
* `process_io_storage_written_bytes_total` - the number of bytes written to storage layer
|
||||
* FEATURE: vmagent: add ability to spread scrape targets among multiple `vmagent` instances. See [these docs](https://docs.victoriametrics.com/vmagent/#scraping-big-number-of-targets) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1084) for details.
|
||||
* FEATURE: vmagent: add ability to spread scrape targets among multiple `vmagent` instances. See [these docs](https://docs.victoriametrics.com/victoriametrics/vmagent/#scraping-big-number-of-targets) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1084) for details.
|
||||
* FEATURE: vmagent: use watch API for Kubernetes service discovery. This should reduce load on Kubernetes API server when it tracks big number of objects (for example, 10K pods). This should also reduce the time needed for k8s targets discovery. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1057) for details.
|
||||
* FEATURE: vmagent: export `vm_promscrape_target_relabel_duration_seconds` metric, which can be used for monitoring the time spend on relabeling for discovered targets.
|
||||
* FEATURE: vmagent: optimize [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling) performance for common cases.
|
||||
* FEATURE: vmagent: optimize [relabeling](https://docs.victoriametrics.com/victoriametrics/vmagent/#relabeling) performance for common cases.
|
||||
* FEATURE: add `increase_pure(m[d])` function to MetricsQL. It works the same as `increase(m[d])` except of various edge cases. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/962) for details.
|
||||
* FEATURE: increase accuracy for `buckets_limit(limit, buckets)` results for small `limit` values. See [MetricsQL docs](https://docs.victoriametrics.com/metricsql/) for details.
|
||||
* FEATURE: increase accuracy for `buckets_limit(limit, buckets)` results for small `limit` values. See [MetricsQL docs](https://docs.victoriametrics.com/victoriametrics/metricsql/) for details.
|
||||
* FEATURE: vmagent: initial support for Windows build with `CGO_ENABLED=0 GOOS=windows go build ./app/vmagent`. See [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/70) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1036).
|
||||
* FEATURE: vmagent: support WebIdentityToken auth in EC2 service discovery. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1080) for details.
|
||||
* FEATURE: vmalert: properly process query params in `-datasource.url` and `-remoteRead.url` command-line flags. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1087) for details.
|
||||
@@ -531,14 +532,14 @@ Released at 2021-02-18
|
||||
* FEATURE: optimize searching for matching metrics for `metric{<label_filters>}` queries if `<label_filters>` contains at least a single filter. For example, the query `up{job="foobar"}` should find the matching time series much faster than previously.
|
||||
* FEATURE: reduce execution times for `q1 <binary_op> q2` queries by executing `q1` and `q2` in parallel.
|
||||
* FEATURE: switch from Go1.15 to [Go1.16](https://golang.org/doc/go1.16) for building prod binaries.
|
||||
* FEATURE: single-node VictoriaMetrics now accepts requests to handlers with `/prometheus` and `/graphite` prefixes such as `/prometheus/api/v1/query`. This improves compatibility with [handlers from VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#url-format).
|
||||
* FEATURE: single-node VictoriaMetrics now accepts requests to handlers with `/prometheus` and `/graphite` prefixes such as `/prometheus/api/v1/query`. This improves compatibility with [handlers from VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#url-format).
|
||||
* FEATURE: expose `process_open_fds` and `process_max_fds` metrics. These metrics can be used for alerting when `process_open_fds` reaches `process_max_fds`. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/402> and <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1037>
|
||||
* FEATURE: vmalert: add `-datasource.appendTypePrefix` command-line option for querying both Prometheus and Graphite datasource in cluster version of VictoriaMetrics. See [these docs](https://docs.victoriametrics.com/vmalert/#graphite) for details.
|
||||
* FEATURE: vmalert: add `-datasource.appendTypePrefix` command-line option for querying both Prometheus and Graphite datasource in cluster version of VictoriaMetrics. See [these docs](https://docs.victoriametrics.com/victoriametrics/vmalert/#graphite) for details.
|
||||
* FEATURE: vmauth: add ability to route requests from a single user to multiple destinations depending on the requested paths. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1064>
|
||||
* FEATURE: remove dependency on external programs such as `cat`, `grep` and `cut` when detecting cpu and memory limits inside Docker or LXC container.
|
||||
* FEATURE: vmagent: add `__meta_kubernetes_endpoints_label_*`, `__meta_kubernetes_endpoints_labelpresent_*`, `__meta_kubernetes_endpoints_annotation_*` and `__meta_kubernetes_endpoints_annotationpresent_*` labels for `role: endpoints` in Kubernetes service discovery. These labels where added in Prometheus 2.25.
|
||||
* FEATURE: reduce the minimum supported retention period for inverted index (aka `indexdb`) from one month to one day. This should reduce disk space usage for `<-storageDataPath>/indexdb` folder if `-retentionPeriod` is set to values smaller than one month.
|
||||
* FEATURE: vmselect: export per-tenant metrics `vm_vmselect_http_requests_total` and `vm_vmselect_http_requests_duration_ms_total` . Other per-tenant metrics are available as a part of [enterprise package](https://docs.victoriametrics.com/enterprise/). See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/932> for details.
|
||||
* FEATURE: vmselect: export per-tenant metrics `vm_vmselect_http_requests_total` and `vm_vmselect_http_requests_duration_ms_total` . Other per-tenant metrics are available as a part of [enterprise package](https://docs.victoriametrics.com/victoriametrics/enterprise/). See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/932> for details.
|
||||
|
||||
* BUGFIX: properly convert regexp tag filters containing escaped dots to non-regexp tag filters. For example, `{foo=~"bar\.baz"}` should be converted to `{foo="bar.baz"}`. Previously it was incorrectly converted to `{foo="bar\.baz"}`, which could result in missing time series for this tag filter.
|
||||
* BUGFIX: do not spam error logs when discovering Docker Swarm targets without dedicated IP. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1028> .
|
||||
@@ -556,14 +557,14 @@ Released at 2021-02-03
|
||||
|
||||
Released at 2021-02-03
|
||||
|
||||
* FEATURE: added [vmctl tool](https://docs.victoriametrics.com/vmctl/) to VictoriaMetrics release process. Now it is packaged in `vmutils-*.tar.gz` archive on [the releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases). Source code for `vmctl` tool has been moved from [github.com/VictoriaMetrics/vmctl](https://github.com/VictoriaMetrics/vmctl) to [github.com/VictoriaMetrics/VictoriaMetrics/app/vmctl](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/app/vmctl).
|
||||
* FEATURE: added [vmctl tool](https://docs.victoriametrics.com/victoriametrics/vmctl/) to VictoriaMetrics release process. Now it is packaged in `vmutils-*.tar.gz` archive on [the releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases). Source code for `vmctl` tool has been moved from [github.com/VictoriaMetrics/vmctl](https://github.com/VictoriaMetrics/vmctl) to [github.com/VictoriaMetrics/VictoriaMetrics/app/vmctl](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/app/vmctl).
|
||||
* FEATURE: added `-loggerTimezone` command-line flag for adjusting time zone for timestamps in log messages. By default UTC is used.
|
||||
* FEATURE: added `-search.maxStepForPointsAdjustment` command-line flag, which can be used for disabling adjustment for points returned by `/api/v1/query_range` handler if such points have timestamps closer than `-search.latencyOffset` to the current time. Such points may contain incomplete data, so they are substituted by the previous values for `step` query args smaller than one minute by default.
|
||||
* FEATURE: vmselect: added ability to use Graphite-compatible filters in MetricsQL via `{__graphite__="foo.*.bar"}` syntax. This expression is equivalent to `{__name__=~"foo[.][^.]*[.]bar"}`, but it works faster and it is easier to use when migrating from Graphite to VictoriaMetrics. This feature deprecates the usage of `-search.treatDotsAsIsInRegexps` command-line flag.
|
||||
* FEATURE: vmselect: added ability to set additional label filters, which must be applied during queries. Such label filters can be set via optional `extra_label` query arg, which is accepted by [querying API](https://docs.victoriametrics.com/#prometheus-querying-api-usage) handlers. For example, the request to `/api/v1/query_range?extra_label=tenant_id=123&query=<query>` adds `{tenant_id="123"}` label filter to the given `<query>`. It is expected that the `extra_label` query arg is automatically set by auth proxy sitting
|
||||
* FEATURE: vmselect: added ability to set additional label filters, which must be applied during queries. Such label filters can be set via optional `extra_label` query arg, which is accepted by [querying API](https://docs.victoriametrics.com/victoriametrics#prometheus-querying-api-usage) handlers. For example, the request to `/api/v1/query_range?extra_label=tenant_id=123&query=<query>` adds `{tenant_id="123"}` label filter to the given `<query>`. It is expected that the `extra_label` query arg is automatically set by auth proxy sitting
|
||||
in front of VictoriaMetrics. [Contact us](mailto:sales@victoriametrics.com) if you need assistance with such a proxy. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1021> .
|
||||
* FEATURE: vmalert: added `-datasource.queryStep` command-line flag for passing optional `step` query arg to `/api/v1/query` endpoint. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1025>
|
||||
* FEATURE: vmalert: added ability to query Graphite datasource when evaluating alerting and recording rules. See [these docs](https://docs.victoriametrics.com/vmalert/#graphite) for details.
|
||||
* FEATURE: vmalert: added ability to query Graphite datasource when evaluating alerting and recording rules. See [these docs](https://docs.victoriametrics.com/victoriametrics/vmalert/#graphite) for details.
|
||||
* FEATURE: vmagent: added `-remoteWrite.roundDigits` command-line option for rounding metric values to the given number of decimal digits after the point before sending the metric to the corresponding `-remoteWrite.url`. This option can be used for improving data compression on the remote storage, because values with lower number of decimal digits can be compressed better than values with bigger number of decimal digits.
|
||||
* FEATURE: vmagent: added `-remoteWrite.rateLimit` command-line flag for limiting data transfer rate to `-remoteWrite.url`. This may be useful when big amounts of buffered data is sent after temporarily unavailability of the remote storage. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1035>
|
||||
* FEATURE: vmagent: export the following additional metrics, which may be useful during troubleshooting:
|
||||
@@ -573,7 +574,7 @@ in front of VictoriaMetrics. [Contact us](mailto:sales@victoriametrics.com) if y
|
||||
* `vm_promscrape_discovery_retries_total`
|
||||
* `vm_promscrape_scrape_retries_total`
|
||||
* `vm_promscrape_service_discovery_duration_seconds`
|
||||
* FEATURE: vmselect: initial implementation for [Graphite Render API](https://docs.victoriametrics.com/#graphite-render-api-usage).
|
||||
* FEATURE: vmselect: initial implementation for [Graphite Render API](https://docs.victoriametrics.com/victoriametrics#graphite-render-api-usage).
|
||||
|
||||
* BUGFIX: vmagent: reduce HTTP reconnection rate for scrape targets. Previously vmagent could erroneously close HTTP keep-alive connections more frequently than needed.
|
||||
* BUGFIX: vmagent: retry scrape and service discovery requests when the remote server closes HTTP keep-alive connection. Previously `disable_keepalive: true` option could be used under `scrape_configs` section when working with such servers.
|
||||
@@ -584,10 +585,10 @@ Released at 2021-01-13
|
||||
|
||||
* FEATURE: provide a sample list of alerting rules for VictoriaMetrics components. It is available [here](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker#alerts).
|
||||
* FEATURE: disable final merge for data for the previous month at the beginning of new month, since it may result in high disk IO and CPU usage. Final merge can be enabled by setting `-finalMergeDelay` command-line flag to positive duration.
|
||||
* FEATURE: add `tfirst_over_time(m[d])` and `tlast_over_time(m[d])` functions to [MetricsQL](https://docs.victoriametrics.com/metricsql/) for returning timestamps for the first and the last data point in `m` over `d` duration.
|
||||
* FEATURE: add `tfirst_over_time(m[d])` and `tlast_over_time(m[d])` functions to [MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/) for returning timestamps for the first and the last data point in `m` over `d` duration.
|
||||
* FEATURE: add ability to pass multiple labels to `sort_by_label()` and `sort_by_label_desc()` functions. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/992> .
|
||||
* FEATURE: enforce at least TLS v1.2 when accepting HTTPS requests if `-tls`, `-tlsCertFile` and `-tlsKeyFile` command-line flags are set, because older TLS protocols such as v1.0 and v1.1 have been deprecated due to security vulnerabilities.
|
||||
* FEATURE: support `extra_label` query arg for all HTTP-based [data ingestion protocols](https://docs.victoriametrics.com/#how-to-import-time-series-data). This query arg can be used for specifying extra labels which should be added for the ingested data.
|
||||
* FEATURE: support `extra_label` query arg for all HTTP-based [data ingestion protocols](https://docs.victoriametrics.com/victoriametrics#how-to-import-time-series-data). This query arg can be used for specifying extra labels which should be added for the ingested data.
|
||||
* FEATURE: vmbackup: increase backup chunk size from 128MB to 1GB. This should reduce the number of Object storage API calls during backups by 8x. This may also reduce costs, since object storage API calls usually have non-zero costs. See <https://aws.amazon.com/s3/pricing/> and <https://cloud.google.com/storage/pricing#operations-pricing> .
|
||||
|
||||
* BUGFIX: properly parse escaped unicode chars in MetricsQL metric names, label names and function names. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/990>
|
||||
@@ -601,4 +602,4 @@ Released at 2021-01-13
|
||||
|
||||
## Previous releases
|
||||
|
||||
See changes for older releases [here](https://docs.victoriametrics.com/changelog_2020/).
|
||||
See changes for older releases [here](https://docs.victoriametrics.com/victoriametrics/ch_2020/).
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -9,6 +9,7 @@ menu:
|
||||
parent: vm-changelog
|
||||
weight: 2
|
||||
aliases:
|
||||
- /CHANGELOG_2025
|
||||
- /CHANGELOG_2025.html
|
||||
- /changelog_2025
|
||||
- /changelog/changelog_2025/index.html
|
||||
|
||||
@@ -7,6 +7,8 @@ menu:
|
||||
parent: victoriametrics
|
||||
weight: 100
|
||||
aliases:
|
||||
- /changelog/
|
||||
- /CHANGELOG/
|
||||
- /CHANGELOG.html
|
||||
- /changelog/index.html
|
||||
- /changelog/
|
||||
|
||||
@@ -7,6 +7,7 @@ menu:
|
||||
parent: "data-ingestion"
|
||||
weight: 3
|
||||
aliases:
|
||||
- /data-ingestion/grafana-alloy/
|
||||
- /data-ingestion/Grafana-Alloy.html
|
||||
- /data-ingestion/grafana-alloy.html
|
||||
- /data-ingestion/Grafana-Agent.html
|
||||
@@ -36,13 +37,13 @@ prometheus.remote_write "victoriametrics" {
|
||||
```
|
||||
|
||||
For pushing data to VictoriaMetrics cluster the `url` should point to vminsert and include
|
||||
the [tenantID](https://docs.victoriametrics.com/cluster-victoriametrics/#url-format):
|
||||
the [tenantID](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#url-format):
|
||||
```sh
|
||||
https://<vminsert-addr>/insert/<tenant_id>/prometheus/api/v1/write
|
||||
```
|
||||
|
||||
> Note: read more about [multitenancy](https://docs.victoriametrics.com/cluster-victoriametrics/#multitenancy)
|
||||
> or [multitenancy via labels](https://docs.victoriametrics.com/cluster-victoriametrics/#multitenancy-via-labels).
|
||||
> Note: read more about [multitenancy](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#multitenancy)
|
||||
> or [multitenancy via labels](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#multitenancy-via-labels).
|
||||
|
||||
After the configuration has been updated, Alloy must be reloaded or restarted for the change to be applied:
|
||||
- Linux: `sudo systemctl reload alloy.service`
|
||||
|
||||
@@ -7,6 +7,7 @@ menu:
|
||||
parent: "data-ingestion"
|
||||
weight: 1
|
||||
aliases:
|
||||
- /data-ingestion/prometheus/
|
||||
- /data-ingestion/prometheus.html
|
||||
- /data-ingestion/Prometheus.html
|
||||
- /data-ingestion/prometheus/index.html
|
||||
@@ -14,7 +15,7 @@ aliases:
|
||||
---
|
||||
|
||||
VictoriaMetrics supports ingesting data from Prometheus via the Prometheus remote write protocol.
|
||||
The documentation for pushing data from Prometheus to VictoriaMetrics is located [here](https://docs.victoriametrics.com/#prometheus-setup)
|
||||
The documentation for pushing data from Prometheus to VictoriaMetrics is located [here](https://docs.victoriametrics.com/victoriametrics#prometheus-setup)
|
||||
|
||||
## References
|
||||
|
||||
|
||||
@@ -7,6 +7,7 @@ menu:
|
||||
parent: "data-ingestion"
|
||||
weight: 6
|
||||
aliases:
|
||||
- /data-ingestion/proxmox/
|
||||
- /data-ingestion/proxmox.html
|
||||
- /data-ingestion/Proxmox.html
|
||||
- /data-ingestion/proxmox/index.html
|
||||
@@ -30,7 +31,7 @@ or a username and password.
|
||||
3. Set the parameters as follows:
|
||||
- Name: VictoriaMetrics (can be changed to any string)
|
||||
- Server: the hostname or IP of your VictoriaMetrics Instance
|
||||
- Port: This will vary depending on how you are sending data to VictoriaMetrics, but the defaults for all components are listed in the [data ingestion documentation](https://docs.victoriametrics.com/data-ingestion.html)
|
||||
- Port: This will vary depending on how you are sending data to VictoriaMetrics, but the defaults for all components are listed in the [data ingestion documentation](https://docs.victoriametrics.com/victoriametrics/data-ingestion/)
|
||||
- Protocol: use HTTPS if you have TLS/SSL configured otherwise use HTTP
|
||||
- Organization: leave it empty since it doesn't get used
|
||||
- Bucket: leave it empty since it doesn't get used
|
||||
@@ -54,7 +55,7 @@ You should see 1 time series per node in your PVE cluster.
|
||||
- Name: VictoriaMetrics (can be set to any string)
|
||||
- URL: http(s)://<ip_or_host>:<port>
|
||||
- set the URL to HTTPS if you have TLS enabled and HTTP if you do not
|
||||
- Port: This will vary depending on how you are sending data to VictoriaMetrics, but the defaults for all components are listed in the [data ingestion documentation](https://docs.victoriametrics.com/data-ingestion.html)
|
||||
- Port: This will vary depending on how you are sending data to VictoriaMetrics, but the defaults for all components are listed in the [data ingestion documentation](https://docs.victoriametrics.com/victoriametrics/data-ingestion/)
|
||||
- Organization: leave it empty since it doesn't get used
|
||||
- Bucket: leave it empty since it doesn't get used
|
||||
- Token: your token from vmauth or leave blank if you don't have authentication enabled
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
In This Folder you will find instructions for sending data to VictoriaMetrics from a variety of platforms.
|
||||
If your tool is not listed it is likely you can ingest your data into VictoriaMetrics using one of the protocols listed in our [Prominent features](https://docs.victoriametrics.com/#prominent-features) section.
|
||||
If your tool is not listed it is likely you can ingest your data into VictoriaMetrics using one of the protocols listed in our [Prominent features](https://docs.victoriametrics.com/victoriametrics#prominent-features) section.
|
||||
|
||||
If you are unsure what port number to use when pushing data to VictoriaMetrics single node, vminsert, vmagent, and vmauth we have listed the default ports below.
|
||||
|
||||
@@ -13,14 +13,14 @@ In the rest of the documentation we will assume you have configured your push en
|
||||
## Documented Collectors/Agents
|
||||
|
||||
|
||||
- [Telegraf](https://docs.victoriametrics.com/data-ingestion/telegraf/)
|
||||
- [Vector](https://docs.victoriametrics.com/data-ingestion/vector/)
|
||||
- [vmagent](https://docs.victoriametrics.com/data-ingestion/vmagent)
|
||||
- [Grafana Alloy](https://docs.victoriametrics.com/data-ingestion/alloy/)
|
||||
- [Prometheus](https://docs.victoriametrics.com/data-ingestion/prometheus/)
|
||||
- [Telegraf](https://docs.victoriametrics.com/victoriametrics/data-ingestion/telegraf/)
|
||||
- [Vector](https://docs.victoriametrics.com/victoriametrics/data-ingestion/vector/)
|
||||
- [vmagent](https://docs.victoriametrics.com/victoriametrics/data-ingestion/vmagent)
|
||||
- [Grafana Alloy](https://docs.victoriametrics.com/victoriametrics/data-ingestion/alloy/)
|
||||
- [Prometheus](https://docs.victoriametrics.com/victoriametrics/data-ingestion/prometheus/)
|
||||
|
||||
|
||||
## Supported Platforms
|
||||
|
||||
- [Proxmox Virtual Environment and Proxmox Backup Server](https://docs.victoriametrics.com/data-ingestion/proxmox/)
|
||||
- [Proxmox Virtual Environment and Proxmox Backup Server](https://docs.victoriametrics.com/victoriametrics/data-ingestion/proxmox/)
|
||||
|
||||
|
||||
@@ -7,12 +7,13 @@ menu:
|
||||
parent: "data-ingestion"
|
||||
weight: 5
|
||||
aliases:
|
||||
- /data-ingestion/telegraf/
|
||||
- /data-ingestion/telegraf.html
|
||||
- /data-ingestion/Telegraf.html
|
||||
- /data-ingestion/telegraf/index.html
|
||||
- /data-ingestion/telegraf/
|
||||
---
|
||||
This document covers various output configurations for Telegraf for shipping data [via HTTP](https://docs.victoriametrics.com/#how-to-send-data-from-influxdb-compatible-agents-such-as-telegraf)
|
||||
This document covers various output configurations for Telegraf for shipping data [via HTTP](https://docs.victoriametrics.com/victoriametrics#how-to-send-data-from-influxdb-compatible-agents-such-as-telegraf)
|
||||
to VictoriaMetrics. All the options examples below can be combined to fit your use case.
|
||||
|
||||
To avoid storing Passwords in configuration files you can store as a key value pair in `/etc/default/telegraf` on Linux as follows
|
||||
|
||||
@@ -7,6 +7,7 @@ menu:
|
||||
parent: "data-ingestion"
|
||||
weight: 4
|
||||
aliases:
|
||||
- /data-ingestion/vector/
|
||||
- /data-ingestion/Vector.html
|
||||
- /data-ingestion/vector.html
|
||||
- /data-ingestion/vector/index.html
|
||||
|
||||
@@ -7,6 +7,7 @@ menu:
|
||||
identifier: 'data-ingestion'
|
||||
weight: 26
|
||||
aliases:
|
||||
- /data-ingestion/
|
||||
- /data-ingestion.html
|
||||
- /data-ingestion.html
|
||||
- /data-ingestion/index.html
|
||||
|
||||
@@ -7,6 +7,7 @@ menu:
|
||||
parent: data-ingestion
|
||||
weight: 2
|
||||
aliases:
|
||||
- /data-ingestion/vmagent/
|
||||
- /data-ingestion/vmagent.html
|
||||
- /data-ingestion/vmagent/index.html
|
||||
- /data-ingestion/vmagent/
|
||||
@@ -15,10 +16,10 @@ aliases:
|
||||
|
||||
vmagent can receive data via the same protocols as VictoriaMetrics Single or Cluster versions,
|
||||
as well as scrape Prometheus endpoints. In other words,
|
||||
it supports both [Push](https://docs.victoriametrics.com/keyconcepts/#push-model) and [Pull](https://docs.victoriametrics.com/keyconcepts/#pull-model) models.
|
||||
it supports both [Push](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#push-model) and [Pull](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#pull-model) models.
|
||||
|
||||
This section of the documentation only covers forwarding data from vmagent to another destination.
|
||||
For extra information about vmagent as well as quickstart guide please refer to the [vmagent documentation](https://docs.victoriametrics.com/vmagent/).
|
||||
For extra information about vmagent as well as quickstart guide please refer to the [vmagent documentation](https://docs.victoriametrics.com/victoriametrics/vmagent/).
|
||||
|
||||
To configure vmagent to push metrics to VictoriaMetrics via Prometheus remote write protocol,
|
||||
configure the `-remoteWrite.url` cmd-line flag:
|
||||
@@ -28,17 +29,17 @@ configure the `-remoteWrite.url` cmd-line flag:
|
||||
```
|
||||
|
||||
For pushing data to VictoriaMetrics cluster the `-remoteWrite.url` should point to vminsert and include
|
||||
the [tenantID](https://docs.victoriametrics.com/cluster-victoriametrics/#url-format):
|
||||
the [tenantID](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#url-format):
|
||||
```sh
|
||||
/path/to/vmagent -remoteWrite.url=https://<vminsert-addr>/insert/<tenant_id>/prometheus/api/v1/write
|
||||
```
|
||||
|
||||
> Note: read more about [multitenancy](https://docs.victoriametrics.com/cluster-victoriametrics/#multitenancy)
|
||||
> or [multitenancy via labels](https://docs.victoriametrics.com/cluster-victoriametrics/#multitenancy-via-labels).
|
||||
> Note: read more about [multitenancy](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#multitenancy)
|
||||
> or [multitenancy via labels](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#multitenancy-via-labels).
|
||||
|
||||
Please note, `-remoteWrite.url` cmd-line flag can be specified multiple times with different values. In this case,
|
||||
vmagent will [replicate](https://docs.victoriametrics.com/vmagent/#replication-and-high-availability) data to each
|
||||
specified destination. In addition, it is possible to configure [metrics sharding](https://docs.victoriametrics.com/vmagent/#sharding-among-remote-storages)
|
||||
vmagent will [replicate](https://docs.victoriametrics.com/victoriametrics/vmagent/#replication-and-high-availability) data to each
|
||||
specified destination. In addition, it is possible to configure [metrics sharding](https://docs.victoriametrics.com/victoriametrics/vmagent/#sharding-among-remote-storages)
|
||||
across `-remoteWrite.url` destinations.
|
||||
|
||||
## Remote write with basic authentication
|
||||
@@ -82,5 +83,5 @@ to a file containing the self-signed CA certificate:
|
||||
|
||||
## References
|
||||
|
||||
- [vmagent docs](https://docs.victoriametrics.com/vmagent/)
|
||||
- [vmagent commandline flags](https://docs.victoriametrics.com/vmagent/#advanced-usage)
|
||||
- [vmagent docs](https://docs.victoriametrics.com/victoriametrics/vmagent/)
|
||||
- [vmagent commandline flags](https://docs.victoriametrics.com/victoriametrics/vmagent/#advanced-usage)
|
||||
|
||||
@@ -7,6 +7,7 @@ menu:
|
||||
parent: 'victoriametrics'
|
||||
weight: 80
|
||||
aliases:
|
||||
- /enterprise/
|
||||
- /enterprise.html
|
||||
- /enterprise/index.html
|
||||
- /enterprise/
|
||||
@@ -36,10 +37,10 @@ See [these docs](#running-victoriametrics-enterprise) for details on how to run
|
||||
|
||||
## VictoriaMetrics enterprise features
|
||||
|
||||
VictoriaMetrics Enterprise includes [all the features of the community edition](https://docs.victoriametrics.com/#prominent-features),
|
||||
VictoriaMetrics Enterprise includes [all the features of the community edition](https://docs.victoriametrics.com/victoriametrics#prominent-features),
|
||||
plus the following additional features:
|
||||
|
||||
- Stable releases with long-term support, which contains important bugfixes and security fixes. See [these docs](https://docs.victoriametrics.com/lts-releases/).
|
||||
- Stable releases with long-term support, which contains important bugfixes and security fixes. See [these docs](https://docs.victoriametrics.com/victoriametrics/lts-releases/).
|
||||
- First-class consulting and technical support provided by the core VictoriaMetrics dev team.
|
||||
- [Monitoring of monitoring](https://victoriametrics.com/products/mom/) - this feature allows forecasting
|
||||
and preventing possible issues in VictoriaMetrics setups.
|
||||
@@ -48,26 +49,26 @@ plus the following additional features:
|
||||
|
||||
On top of this, Enterprise package of VictoriaMetrics includes the following features:
|
||||
|
||||
- [Downsampling](https://docs.victoriametrics.com/#downsampling) - this feature allows reducing storage costs
|
||||
- [Downsampling](https://docs.victoriametrics.com/victoriametrics#downsampling) - this feature allows reducing storage costs
|
||||
and increasing performance for queries over historical data.
|
||||
- [Multiple retentions](https://docs.victoriametrics.com/#retention-filters) - this feature allows reducing storage costs
|
||||
- [Multiple retentions](https://docs.victoriametrics.com/victoriametrics#retention-filters) - this feature allows reducing storage costs
|
||||
by specifying different retentions for different datasets.
|
||||
- [Automatic discovery of vmstorage nodes](https://docs.victoriametrics.com/cluster-victoriametrics/#automatic-vmstorage-discovery) -
|
||||
- [Automatic discovery of vmstorage nodes](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#automatic-vmstorage-discovery) -
|
||||
this feature allows updating the list of `vmstorage` nodes at `vminsert` and `vmselect` without the need to restart these services.
|
||||
- [Anomaly Detection Service](https://docs.victoriametrics.com/anomaly-detection) - this feature allows automation and simplification of your alerting rules, covering [complex anomalies](https://victoriametrics.com/blog/victoriametrics-anomaly-detection-handbook-chapter-2/) found in metrics data.
|
||||
- [Backup automation](https://docs.victoriametrics.com/vmbackupmanager/).
|
||||
- [Backup automation](https://docs.victoriametrics.com/victoriametrics/vmbackupmanager/).
|
||||
- [Advanced per-tenant stats](https://docs.victoriametrics.com/pertenantstatistic/).
|
||||
- [Advanced query execution stats](https://docs.victoriametrics.com/query-stats/).
|
||||
- [Advanced auth and rate limiter](https://docs.victoriametrics.com/vmgateway/).
|
||||
- [Automatic issuing of TLS certificates](https://docs.victoriametrics.com/#automatic-issuing-of-tls-certificates).
|
||||
- [mTLS for all the VictoriaMetrics components](https://docs.victoriametrics.com/#mtls-protection).
|
||||
- [mTLS for communications between cluster components](https://docs.victoriametrics.com/cluster-victoriametrics/#mtls-protection).
|
||||
- [mTLS-based request routing](https://docs.victoriametrics.com/vmauth/#mtls-based-request-routing).
|
||||
- [Kafka integration](https://docs.victoriametrics.com/vmagent/#kafka-integration).
|
||||
- [Google PubSub integration](https://docs.victoriametrics.com/vmagent/#google-pubsub-integration).
|
||||
- [Multitenant support in vmalert](https://docs.victoriametrics.com/vmalert/#multitenancy).
|
||||
- [Ability to read alerting and recording rules from Object Storage](https://docs.victoriametrics.com/vmalert/#reading-rules-from-object-storage).
|
||||
- [Ability to filter incoming requests by IP at vmauth](https://docs.victoriametrics.com/vmauth/#ip-filters).
|
||||
- [Advanced query execution stats](https://docs.victoriametrics.com/victoriametrics/query-stats/).
|
||||
- [Advanced auth and rate limiter](https://docs.victoriametrics.com/victoriametrics/vmgateway/).
|
||||
- [Automatic issuing of TLS certificates](https://docs.victoriametrics.com/victoriametrics#automatic-issuing-of-tls-certificates).
|
||||
- [mTLS for all the VictoriaMetrics components](https://docs.victoriametrics.com/victoriametrics#mtls-protection).
|
||||
- [mTLS for communications between cluster components](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#mtls-protection).
|
||||
- [mTLS-based request routing](https://docs.victoriametrics.com/victoriametrics/vmauth/#mtls-based-request-routing).
|
||||
- [Kafka integration](https://docs.victoriametrics.com/victoriametrics/vmagent/#kafka-integration).
|
||||
- [Google PubSub integration](https://docs.victoriametrics.com/victoriametrics/vmagent/#google-pubsub-integration).
|
||||
- [Multitenant support in vmalert](https://docs.victoriametrics.com/victoriametrics/vmalert/#multitenancy).
|
||||
- [Ability to read alerting and recording rules from Object Storage](https://docs.victoriametrics.com/victoriametrics/vmalert/#reading-rules-from-object-storage).
|
||||
- [Ability to filter incoming requests by IP at vmauth](https://docs.victoriametrics.com/victoriametrics/vmauth/#ip-filters).
|
||||
|
||||
Contact us via [this page](https://victoriametrics.com/products/enterprise/) if you are interested in VictoriaMetrics Enterprise.
|
||||
|
||||
@@ -223,7 +224,7 @@ In order to use Enterprise components it is required to provide the license key
|
||||
|
||||
Enterprise license key can be obtained at [this page](https://victoriametrics.com/products/enterprise/trial/).
|
||||
|
||||
For example, the following custom resource for [VictoriaMetrics single-node](https://docs.victoriametrics.com/single-server-victoriametrics/)
|
||||
For example, the following custom resource for [VictoriaMetrics single-node](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/)
|
||||
is used to provide key in plain-text:
|
||||
|
||||
```yaml
|
||||
@@ -283,7 +284,7 @@ All the VictoriaMetrics Enterprise components expose the following metrics at th
|
||||
* `vm_license_expires_at` - license expiration date in unix timestamp format
|
||||
* `vm_license_expires_in_seconds` - the number of seconds left until the license expires
|
||||
|
||||
Example alerts for [vmalert](https://docs.victoriametrics.com/vmalert/) based on these metrics:
|
||||
Example alerts for [vmalert](https://docs.victoriametrics.com/victoriametrics/vmalert/) based on these metrics:
|
||||
|
||||
```yaml
|
||||
groups:
|
||||
|
||||
@@ -11,15 +11,15 @@ aliases:
|
||||
---
|
||||
## Goals
|
||||
|
||||
1. The main goal - **to help users and [clients](https://docs.victoriametrics.com/enterprise/) resolving issues with VictoriaMetrics components,
|
||||
1. The main goal - **to help users and [clients](https://docs.victoriametrics.com/victoriametrics/enterprise/) resolving issues with VictoriaMetrics components,
|
||||
so they could use these components in the most efficient way**.
|
||||
1. Fixing bugs in the essential functionality of VictoriaMetrics components. Small usability bugs are usually the most annoying,
|
||||
so they **must be fixed first**.
|
||||
1. Improving [docs](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/docs) for VictoriaMetrics components,
|
||||
so users could find answers to their questions via Google or [Perplexity](https://www.perplexity.ai/) without the need
|
||||
to ask these questions at our [support channels](https://docs.victoriametrics.com/#community-and-contributions).
|
||||
to ask these questions at our [support channels](https://docs.victoriametrics.com/victoriametrics#community-and-contributions).
|
||||
1. Simplifying usage of VictoriaMetrics components without breaking backwards compatibility, so users could regularly
|
||||
upgrade to [the latest available release](https://docs.victoriametrics.com/CHANGELOG) and remain happy.
|
||||
upgrade to [the latest available release](https://docs.victoriametrics.com/victoriametrics/changelog) and remain happy.
|
||||
1. Improving usability for the existing functionality of VictoriaMetrics components.
|
||||
1. Improving the readability and maintainability of the code base by removing unnecessary abstractions and simplifying the code whenever possible.
|
||||
1. Improving development velocity by optimizing and simplifying CI/CD tasks, so they take less time to execute and debug.
|
||||
@@ -52,7 +52,7 @@ aliases:
|
||||
with VictoriaMetrics components in the most efficient way. Nobody uses new shiny feature if it isn't documented properly.
|
||||
|
||||
- **Happy users are more important than the momentary revenue.** Happy users spread the word about VictoriaMetrics,
|
||||
so more people convert to VictoriaMetrics users. Happy users are eager to become happy [customers](https://docs.victoriametrics.com/enterprise/).
|
||||
so more people convert to VictoriaMetrics users. Happy users are eager to become happy [customers](https://docs.victoriametrics.com/victoriametrics/enterprise/).
|
||||
This increases long-term revenue.
|
||||
|
||||
- **Simple solution is better than smart solution.** Simple solution is easier to setup, operate, debug and troubleshoot than the smart solution.
|
||||
|
||||
@@ -7,9 +7,10 @@ menu:
|
||||
parent: victoriametrics
|
||||
weight: 34
|
||||
aliases:
|
||||
- /keyconcepts/
|
||||
- /keyConcepts.html
|
||||
- /keyсoncepts.html
|
||||
- /keyconcepts/index.html
|
||||
- /keyconcepts/
|
||||
---
|
||||
{{% content "keyConcepts.md" %}}
|
||||
{{% content "keyConcepts.md" %}}
|
||||
|
||||
@@ -44,10 +44,10 @@ requests_total{path="/", code="200"}
|
||||
```
|
||||
|
||||
Labels can be automatically attached to the [time series](#time-series)
|
||||
written via [vmagent](https://docs.victoriametrics.com/vmagent/#adding-labels-to-metrics)
|
||||
or [Prometheus](https://docs.victoriametrics.com/single-server-victoriametrics/#prometheus-setup).
|
||||
VictoriaMetrics supports enforcing of label filters for [query API](https://docs.victoriametrics.com/single-server-victoriametrics/#prometheus-querying-api-enhancements)
|
||||
to emulate data isolation. However, the real data isolation can be achieved via [multi-tenancy](https://docs.victoriametrics.com/cluster-victoriametrics/#multitenancy).
|
||||
written via [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/#adding-labels-to-metrics)
|
||||
or [Prometheus](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#prometheus-setup).
|
||||
VictoriaMetrics supports enforcing of label filters for [query API](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#prometheus-querying-api-enhancements)
|
||||
to emulate data isolation. However, the real data isolation can be achieved via [multi-tenancy](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#multitenancy).
|
||||
|
||||
#### Time series
|
||||
|
||||
@@ -56,14 +56,14 @@ A combination of a metric name and its labels defines a `time series`. For examp
|
||||
are two different time series because they have different values for `code` label.
|
||||
|
||||
The number of unique time series has an impact on database resource usage.
|
||||
See [what is an active time series](https://docs.victoriametrics.com/faq/#what-is-an-active-time-series) and
|
||||
[what is high churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate) docs for details.
|
||||
See [what is an active time series](https://docs.victoriametrics.com/victoriametrics/faq/#what-is-an-active-time-series) and
|
||||
[what is high churn rate](https://docs.victoriametrics.com/victoriametrics/faq/#what-is-high-churn-rate) docs for details.
|
||||
|
||||
#### Cardinality
|
||||
|
||||
The number of unique [time series](#time-series) is named `cardinality`. Too big number of unique time series is named `high cardinality`.
|
||||
High cardinality may result in increased resource usage at VictoriaMetrics.
|
||||
See [these docs](https://docs.victoriametrics.com/faq/#what-is-high-cardinality) for more details.
|
||||
See [these docs](https://docs.victoriametrics.com/victoriametrics/faq/#what-is-high-cardinality) for more details.
|
||||
|
||||
#### Raw samples
|
||||
|
||||
@@ -89,8 +89,8 @@ requests_total{path="/", code="200"} 123 4567890
|
||||
|
||||
#### Time series resolution
|
||||
|
||||
Resolution is the minimum interval between [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples)
|
||||
of the [time series](https://docs.victoriametrics.com/keyconcepts/#time-series). Consider the following example:
|
||||
Resolution is the minimum interval between [raw samples](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#raw-samples)
|
||||
of the [time series](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#time-series). Consider the following example:
|
||||
```
|
||||
----------------------------------------------------------------------
|
||||
| <time series> | <value> | <timestamp> |
|
||||
@@ -103,9 +103,9 @@ of the [time series](https://docs.victoriametrics.com/keyconcepts/#time-series).
|
||||
Here we have a time series `requests_total{path="/health", code="200"}` which has a value update each `30s`.
|
||||
This means, its resolution is also a `30s`.
|
||||
|
||||
> In terms of [pull model](https://docs.victoriametrics.com/keyconcepts/#pull-model), resolution is equal
|
||||
> In terms of [pull model](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#pull-model), resolution is equal
|
||||
> to `scrape_interval` and is controlled by the monitoring system (server).
|
||||
> For [push model](https://docs.victoriametrics.com/keyconcepts/#push-model), resolution is an interval between
|
||||
> For [push model](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#push-model), resolution is an interval between
|
||||
> samples timestamps and is controlled by a client (metrics collector).
|
||||
|
||||
Try to keep time series resolution consistent, since some [MetricsQL](#metricsql) functions may expect it to be so.
|
||||
@@ -134,9 +134,9 @@ was rapidly changing from 1:38 pm to 1:39 pm, then there were no changes until 1
|
||||
Counter is used for measuring the number of events, like the number of requests, errors, logs, messages, etc.
|
||||
The most common [MetricsQL](#metricsql) functions used with counters are:
|
||||
|
||||
* [rate](https://docs.victoriametrics.com/metricsql/#rate) - calculates the average per-second speed of metric change.
|
||||
* [rate](https://docs.victoriametrics.com/victoriametrics/metricsql/#rate) - calculates the average per-second speed of metric change.
|
||||
For example, `rate(requests_total)` shows how many requests are served per second on average;
|
||||
* [increase](https://docs.victoriametrics.com/metricsql/#increase) - calculates the growth of a metric on the given
|
||||
* [increase](https://docs.victoriametrics.com/victoriametrics/metricsql/#increase) - calculates the growth of a metric on the given
|
||||
time period specified in square brackets.
|
||||
For example, `increase(requests_total[1h])` shows the number of requests served over the last hour.
|
||||
|
||||
@@ -166,7 +166,7 @@ Gauge is used in the following scenarios:
|
||||
can store the timestamp of the last successful configuration reload.
|
||||
|
||||
The most common [MetricsQL](#metricsql) functions used with gauges are [aggregation functions](#aggregation-and-grouping-functions)
|
||||
and [rollup functions](https://docs.victoriametrics.com/metricsql/#rollup-functions).
|
||||
and [rollup functions](https://docs.victoriametrics.com/victoriametrics/metricsql/#rollup-functions).
|
||||
|
||||
#### Histogram
|
||||
|
||||
@@ -193,7 +193,7 @@ that there were 2 queries with the number of rows in the range `(408.4 - 464.2]`
|
||||
since the last VictoriaMetrics start.
|
||||
|
||||
The counters ending with `_bucket` suffix allow estimating arbitrary percentile
|
||||
for the observed measurement with the help of [histogram_quantile](https://docs.victoriametrics.com/metricsql/#histogram_quantile)
|
||||
for the observed measurement with the help of [histogram_quantile](https://docs.victoriametrics.com/victoriametrics/metricsql/#histogram_quantile)
|
||||
function. For example, the following query returns the estimated 99th percentile
|
||||
on the number of rows read per each query during the last hour (see `1h` in square brackets):
|
||||
|
||||
@@ -251,7 +251,7 @@ and calculating [quantiles](https://prometheus.io/docs/practices/histograms/#qua
|
||||
|
||||

|
||||
|
||||
Grafana doesn't understand buckets with `vmrange` labels, so the [prometheus_buckets](https://docs.victoriametrics.com/metricsql/#prometheus_buckets)
|
||||
Grafana doesn't understand buckets with `vmrange` labels, so the [prometheus_buckets](https://docs.victoriametrics.com/victoriametrics/metricsql/#prometheus_buckets)
|
||||
function must be used for converting buckets with `vmrange` labels to buckets with `le` labels before building heatmaps in Grafana.
|
||||
|
||||
Histograms are usually used for measuring the distribution of latency, sizes of elements (batch size, for example) etc. There are two
|
||||
@@ -344,13 +344,13 @@ avoid excessive resource usage and database slowdown.
|
||||
|
||||
### Multi-tenancy
|
||||
|
||||
[Cluster version](https://docs.victoriametrics.com/cluster-victoriametrics/) of VictoriaMetrics
|
||||
supports [multi-tenancy](https://docs.victoriametrics.com/cluster-victoriametrics/#multitenancy)
|
||||
[Cluster version](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/) of VictoriaMetrics
|
||||
supports [multi-tenancy](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#multitenancy)
|
||||
for data isolation.
|
||||
|
||||
Multi-tenancy can be emulated for [single-server](https://docs.victoriametrics.com/single-server-victoriametrics/)
|
||||
Multi-tenancy can be emulated for [single-server](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/)
|
||||
version of VictoriaMetrics by adding [labels](#labels) on [write path](#write-data)
|
||||
and enforcing [labels filtering](https://docs.victoriametrics.com/single-server-victoriametrics/#prometheus-querying-api-enhancements)
|
||||
and enforcing [labels filtering](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#prometheus-querying-api-enhancements)
|
||||
on [read path](#query-data).
|
||||
|
||||
|
||||
@@ -365,13 +365,13 @@ Client regularly sends the collected metrics to the server in the push model:
|
||||

|
||||
|
||||
The client (application) decides when and where to send its metrics. VictoriaMetrics supports many protocols
|
||||
for data ingestion (aka `push protocols`) - see [the full list here](https://docs.victoriametrics.com/#how-to-import-time-series-data).
|
||||
for data ingestion (aka `push protocols`) - see [the full list here](https://docs.victoriametrics.com/victoriametrics#how-to-import-time-series-data).
|
||||
All the protocols are fully compatible with VictoriaMetrics [data model](#data-model) and can be used in production.
|
||||
We recommend using the [github.com/VictoriaMetrics/metrics](https://github.com/VictoriaMetrics/metrics) package
|
||||
for pushing application metrics to VictoriaMetrics.
|
||||
It is also possible to use already existing clients compatible with the protocols listed above
|
||||
like [Telegraf](https://github.com/influxdata/telegraf)
|
||||
for [InfluxDB line protocol](https://docs.victoriametrics.com/single-server-victoriametrics/#how-to-send-data-from-influxdb-compatible-agents-such-as-telegraf).
|
||||
for [InfluxDB line protocol](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#how-to-send-data-from-influxdb-compatible-agents-such-as-telegraf).
|
||||
|
||||
Creating custom clients or instrumenting the application for metrics writing is as easy as sending a POST request:
|
||||
|
||||
@@ -379,14 +379,14 @@ Creating custom clients or instrumenting the application for metrics writing is
|
||||
curl -d '{"metric":{"__name__":"foo","job":"node_exporter"},"values":[0,1,2],"timestamps":[1549891472010,1549891487724,1549891503438]}' -X POST 'http://localhost:8428/api/v1/import'
|
||||
```
|
||||
|
||||
It is allowed to push/write metrics to [single-node VictoriaMetrics](https://docs.victoriametrics.com/single-server-victoriametrics/),
|
||||
to [cluster component vminsert](https://docs.victoriametrics.com/cluster-victoriametrics/#architecture-overview)
|
||||
and to [vmagent](https://docs.victoriametrics.com/vmagent/).
|
||||
It is allowed to push/write metrics to [single-node VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/),
|
||||
to [cluster component vminsert](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#architecture-overview)
|
||||
and to [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/).
|
||||
|
||||
The pros of push model:
|
||||
|
||||
* Simpler configuration at VictoriaMetrics side - there is no need to configure VictoriaMetrics with locations of the monitored applications.
|
||||
There is no need in complex [service discovery schemes](https://docs.victoriametrics.com/sd_configs/).
|
||||
There is no need in complex [service discovery schemes](https://docs.victoriametrics.com/victoriametrics/sd_configs/).
|
||||
* Simpler security setup - there is no need to set up access from VictoriaMetrics to each monitored application.
|
||||
|
||||
See [Foiled by the Firewall: A Tale of Transition From Prometheus to VictoriaMetrics](https://www.percona.com/blog/2020/12/01/foiled-by-the-firewall-a-tale-of-transition-from-prometheus-to-victoriametrics/)
|
||||
@@ -413,10 +413,10 @@ In pull model, the monitoring system needs to be aware of all the applications i
|
||||
scraped (pulled) from the known applications (aka `scrape targets`) via HTTP protocol on a regular basis (aka `scrape_interval`).
|
||||
|
||||
VictoriaMetrics supports discovering Prometheus-compatible targets and scraping metrics from them in the same way as Prometheus does -
|
||||
see [these docs](https://docs.victoriametrics.com/#how-to-scrape-prometheus-exporters-such-as-node-exporter).
|
||||
see [these docs](https://docs.victoriametrics.com/victoriametrics#how-to-scrape-prometheus-exporters-such-as-node-exporter).
|
||||
|
||||
Metrics scraping is supported by [single-node VictoriaMetrics](https://docs.victoriametrics.com/#how-to-scrape-prometheus-exporters-such-as-node-exporter)
|
||||
and by [vmagent](https://docs.victoriametrics.com/vmagent/).
|
||||
Metrics scraping is supported by [single-node VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics#how-to-scrape-prometheus-exporters-such-as-node-exporter)
|
||||
and by [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/).
|
||||
|
||||
The pros of the pull model:
|
||||
|
||||
@@ -429,7 +429,7 @@ The pros of the pull model:
|
||||
The cons of the pull model:
|
||||
|
||||
* Harder security setup - monitoring system needs to have access to applications it monitors.
|
||||
* Pull model needs non-trivial [service discovery schemes](https://docs.victoriametrics.com/sd_configs/).
|
||||
* Pull model needs non-trivial [service discovery schemes](https://docs.victoriametrics.com/victoriametrics/sd_configs/).
|
||||
|
||||
### Common approaches for data collection
|
||||
|
||||
@@ -440,7 +440,7 @@ The most common approach for data collection is using both models:
|
||||
|
||||

|
||||
|
||||
In this approach the additional component is used - [vmagent](https://docs.victoriametrics.com/vmagent/). Vmagent is
|
||||
In this approach the additional component is used - [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/). Vmagent is
|
||||
a lightweight agent whose main purpose is to collect, filter, relabel and deliver metrics to VictoriaMetrics.
|
||||
It supports all [push](#push-model) and [pull](#pull-model) protocols mentioned above.
|
||||
|
||||
@@ -455,17 +455,17 @@ VictoriaMetrics components allow building more advanced topologies. For example,
|
||||
|
||||

|
||||
|
||||
VictoriaMetrics in this example may be either [single-node VictoriaMetrics](https://docs.victoriametrics.com/single-server-victoriametrics/)
|
||||
or [VictoriaMetrics Cluster](https://docs.victoriametrics.com/cluster-victoriametrics/). Vmagent also allows
|
||||
[replicating the same data to multiple destinations](https://docs.victoriametrics.com/vmagent/#replication-and-high-availability).
|
||||
VictoriaMetrics in this example may be either [single-node VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/)
|
||||
or [VictoriaMetrics Cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/). Vmagent also allows
|
||||
[replicating the same data to multiple destinations](https://docs.victoriametrics.com/victoriametrics/vmagent/#replication-and-high-availability).
|
||||
|
||||
## Query data
|
||||
|
||||
VictoriaMetrics provides
|
||||
an [HTTP API](https://docs.victoriametrics.com/single-server-victoriametrics/#prometheus-querying-api-usage)
|
||||
an [HTTP API](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#prometheus-querying-api-usage)
|
||||
for serving read queries. The API is used in various integrations such as
|
||||
[Grafana](https://docs.victoriametrics.com/single-server-victoriametrics/#grafana-setup). The same API is also used by
|
||||
[VMUI](https://docs.victoriametrics.com/single-server-victoriametrics/#vmui) - a graphical User Interface for querying
|
||||
[Grafana](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#grafana-setup). The same API is also used by
|
||||
[VMUI](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#vmui) - a graphical User Interface for querying
|
||||
and visualizing metrics.
|
||||
|
||||
The API consists of two main handlers for serving [instant queries](#instant-query) and [range queries](#range-query).
|
||||
@@ -480,10 +480,10 @@ GET | POST /api/v1/query?query=...&time=...&step=...&timeout=...
|
||||
|
||||
Params:
|
||||
|
||||
* `query` - [MetricsQL](https://docs.victoriametrics.com/metricsql/) expression.
|
||||
* `time` - optional, [timestamp](https://docs.victoriametrics.com/single-server-victoriametrics/#timestamp-formats)
|
||||
* `query` - [MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/) expression.
|
||||
* `time` - optional, [timestamp](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#timestamp-formats)
|
||||
in millisecond precision to evaluate the `query` at. If omitted, `time` is set to `now()` (current timestamp).
|
||||
The `time` param can be specified in [multiple allowed formats](https://docs.victoriametrics.com/#timestamp-formats).
|
||||
The `time` param can be specified in [multiple allowed formats](https://docs.victoriametrics.com/victoriametrics#timestamp-formats).
|
||||
* `step` - optional [interval](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-durations)
|
||||
for searching for raw samples in the past when executing the `query` (used when a sample is missing at the specified `time`).
|
||||
For example, the request `/api/v1/query?query=up&step=1m` looks for the last written raw sample for the metric `up`
|
||||
@@ -493,7 +493,7 @@ Params:
|
||||
By default the timeout is set to the value of `-search.maxQueryDuration` command-line flag passed to single-node VictoriaMetrics
|
||||
or to `vmselect` component of VictoriaMetrics cluster.
|
||||
|
||||
The result of Instant query is a list of [time series](https://docs.victoriametrics.com/keyconcepts/#time-series)
|
||||
The result of Instant query is a list of [time series](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#time-series)
|
||||
matching the filter in `query` expression. Each returned series contains exactly one `(timestamp, value)` entry,
|
||||
where `timestamp` equals to the `time` query arg, while the `value` contains `query` result at the requested `time`.
|
||||
|
||||
@@ -563,7 +563,7 @@ Instant queries can return multiple time series, but always only one data sample
|
||||
the following scenarios:
|
||||
|
||||
* Getting the last recorded value;
|
||||
* For [rollup functions](https://docs.victoriametrics.com/metricsql/#rollup-functions) such as `count_over_time`;
|
||||
* For [rollup functions](https://docs.victoriametrics.com/victoriametrics/metricsql/#rollup-functions) such as `count_over_time`;
|
||||
* For alerts and recording rules evaluation;
|
||||
* Plotting Stat or Table panels in Grafana.
|
||||
|
||||
@@ -576,10 +576,10 @@ GET | POST /api/v1/query_range?query=...&start=...&end=...&step=...&timeout=...
|
||||
```
|
||||
|
||||
Params:
|
||||
* `query` - [MetricsQL](https://docs.victoriametrics.com/metricsql/) expression.
|
||||
* `start` - the starting [timestamp](https://docs.victoriametrics.com/single-server-victoriametrics/#timestamp-formats)
|
||||
* `query` - [MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/) expression.
|
||||
* `start` - the starting [timestamp](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#timestamp-formats)
|
||||
of the time range for `query` evaluation.
|
||||
* `end` - the ending [timestamp](https://docs.victoriametrics.com/single-server-victoriametrics/#timestamp-formats)
|
||||
* `end` - the ending [timestamp](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#timestamp-formats)
|
||||
of the time range for `query` evaluation.
|
||||
If the `end` isn't set, then the `end` is automatically set to the current time.
|
||||
* `step` - the [interval](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-durations)
|
||||
@@ -592,7 +592,7 @@ Params:
|
||||
By default the timeout is set to the value of `-search.maxQueryDuration` command-line flag passed to single-node VictoriaMetrics
|
||||
or to `vmselect` component in VictoriaMetrics cluster.
|
||||
|
||||
The result of Range query is a list of [time series](https://docs.victoriametrics.com/keyconcepts/#time-series)
|
||||
The result of Range query is a list of [time series](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#time-series)
|
||||
matching the filter in `query` expression. Each returned series contains `(timestamp, value)` results for the `query` executed
|
||||
at `start`, `start+step`, `start+2*step`, ..., `start+N*step` timestamps. In other words, Range query is an [Instant query](#instant-query)
|
||||
executed independently at `start`, `start+step`, ..., `start+N*step` timestamps with the only difference that an instant query
|
||||
@@ -729,7 +729,7 @@ useful in the following scenarios:
|
||||
* Correlate changes between multiple metrics on the time interval;
|
||||
* Observe trends and dynamics of the metric change.
|
||||
|
||||
If you need to export raw samples from VictoriaMetrics, then take a look at [export APIs](https://docs.victoriametrics.com/#how-to-export-time-series).
|
||||
If you need to export raw samples from VictoriaMetrics, then take a look at [export APIs](https://docs.victoriametrics.com/victoriametrics#how-to-export-time-series).
|
||||
|
||||
### Query latency
|
||||
|
||||
@@ -756,14 +756,14 @@ VictoriaMetrics buffers recently ingested samples in memory for up to a few seco
|
||||
This buffering improves data ingestion performance. The buffered samples are invisible in query results, even if `-search.latencyOffset` command-line flag is set to 0,
|
||||
or if `latency_offset` query arg is set to 0.
|
||||
You can send GET request to `/internal/force_flush` http handler at single-node VictoriaMetrics
|
||||
or to `vmstorage` at [cluster version of VictoriaMetrics](https://docs.victoriametrics.com/cluster-victoriametrics/)
|
||||
or to `vmstorage` at [cluster version of VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/)
|
||||
in order to forcibly flush the buffered samples to disk, so they become visible for querying. The `/internal/force_flush` handler
|
||||
is provided for debugging and testing purposes only. Do not call it in production, since this may significantly slow down data ingestion
|
||||
performance and increase resource usage.
|
||||
|
||||
### MetricsQL
|
||||
|
||||
VictoriaMetrics provide a special query language for executing read queries - [MetricsQL](https://docs.victoriametrics.com/metricsql/).
|
||||
VictoriaMetrics provide a special query language for executing read queries - [MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/).
|
||||
It is a [PromQL](https://prometheus.io/docs/prometheus/latest/querying/basics)-like query language with a powerful set of
|
||||
functions and features for working specifically with time series data. MetricsQL is backward-compatible with PromQL,
|
||||
so it shares most of the query concepts. The basic concepts for PromQL and MetricsQL are
|
||||
@@ -821,7 +821,7 @@ The query above returns series for two metrics: `requests_error_total` and `requ
|
||||
|
||||
#### Filtering by multiple "or" filters
|
||||
|
||||
[MetricsQL](https://docs.victoriametrics.com/metricsql/) supports selecting time series, which match at least one of multiple "or" filters.
|
||||
[MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/) supports selecting time series, which match at least one of multiple "or" filters.
|
||||
Such filters must be delimited by `or` inside curly braces. For example, the following query selects time series with
|
||||
`{job="app1",env="prod"}` or `{job="app2",env="dev"}` labels:
|
||||
|
||||
@@ -832,9 +832,9 @@ Such filters must be delimited by `or` inside curly braces. For example, the fol
|
||||
The number of `or` groups can be arbitrary. The number of `,`-delimited label filters per each `or` group can be arbitrary.
|
||||
Per-group filters are applied with `and` operation, e.g. they select series simultaneously matching all the filters in the group.
|
||||
|
||||
This functionality allows passing the selected series to [rollup functions](https://docs.victoriametrics.com/metricsql/#rollup-functions)
|
||||
such as [rate()](https://docs.victoriametrics.com/metricsql/#rate)
|
||||
without the need to use [subqueries](https://docs.victoriametrics.com/metricsql/#subqueries):
|
||||
This functionality allows passing the selected series to [rollup functions](https://docs.victoriametrics.com/victoriametrics/metricsql/#rollup-functions)
|
||||
such as [rate()](https://docs.victoriametrics.com/victoriametrics/metricsql/#rate)
|
||||
without the need to use [subqueries](https://docs.victoriametrics.com/victoriametrics/metricsql/#subqueries):
|
||||
|
||||
```metricsql
|
||||
rate({job="app1",env="prod" or job="app2",env="dev"}[5m])
|
||||
@@ -905,12 +905,12 @@ summary memory usage for each `job`:
|
||||
sum(process_resident_memory_bytes) by (job)
|
||||
```
|
||||
|
||||
See [docs for aggregate functions in MetricsQL](https://docs.victoriametrics.com/metricsql/#aggregate-functions).
|
||||
See [docs for aggregate functions in MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/#aggregate-functions).
|
||||
|
||||
#### Calculating rates
|
||||
|
||||
One of the most widely used functions for [counters](#counter)
|
||||
is [rate](https://docs.victoriametrics.com/metricsql/#rate). It calculates the average per-second increase rate individually
|
||||
is [rate](https://docs.victoriametrics.com/victoriametrics/metricsql/#rate). It calculates the average per-second increase rate individually
|
||||
per each matching time series. For example, the following query shows the average per-second data receive speed
|
||||
per each monitored `node_exporter` instance, which exposes the `node_network_receive_bytes_total` metric:
|
||||
|
||||
@@ -931,7 +931,7 @@ In this case VictoriaMetrics uses the specified lookbehind window - `5m` (5 minu
|
||||
Bigger lookbehind windows usually lead to smoother graphs.
|
||||
|
||||
`rate` strips metric name while leaving all the labels for the inner time series. If you need to keep the metric name,
|
||||
then add [keep_metric_names](https://docs.victoriametrics.com/metricsql/#keep_metric_names) modifier
|
||||
then add [keep_metric_names](https://docs.victoriametrics.com/victoriametrics/metricsql/#keep_metric_names) modifier
|
||||
after the `rate(..)`. For example, the following query leaves metric names after calculating the `rate()`:
|
||||
|
||||
```metricsql
|
||||
@@ -943,13 +943,13 @@ rate(node_network_receive_bytes_total) keep_metric_names
|
||||
### Visualizing time series
|
||||
|
||||
VictoriaMetrics has a built-in graphical User Interface for querying and visualizing metrics -
|
||||
[VMUI](https://docs.victoriametrics.com/single-server-victoriametrics/#vmui).
|
||||
[VMUI](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#vmui).
|
||||
Open `http://victoriametrics:8428/vmui` page, type the query and see the results:
|
||||
|
||||

|
||||
|
||||
VictoriaMetrics supports [Prometheus HTTP API](https://docs.victoriametrics.com/single-server-victoriametrics/#prometheus-querying-api-usage)
|
||||
which makes it possible to [query it with Grafana](https://docs.victoriametrics.com/single-server-victoriametrics/#grafana-setup)
|
||||
VictoriaMetrics supports [Prometheus HTTP API](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#prometheus-querying-api-usage)
|
||||
which makes it possible to [query it with Grafana](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#grafana-setup)
|
||||
in the same way as Grafana queries Prometheus.
|
||||
|
||||
## Modify data
|
||||
@@ -961,20 +961,20 @@ it is stored. Due to this limitation, VictoriaMetrics does not support direct da
|
||||
|
||||
### Deletion
|
||||
|
||||
See [How to delete time series](https://docs.victoriametrics.com/single-server-victoriametrics/#how-to-delete-time-series)
|
||||
See [How to delete time series](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#how-to-delete-time-series)
|
||||
.
|
||||
|
||||
### Relabeling
|
||||
|
||||
Relabeling is a powerful mechanism for modifying time series before they have been written to the database. Relabeling
|
||||
may be applied for both [push](#push-model) and [pull](#pull-model) models. See more
|
||||
details [here](https://docs.victoriametrics.com/single-server-victoriametrics/#relabeling).
|
||||
details [here](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#relabeling).
|
||||
|
||||
### Deduplication
|
||||
|
||||
VictoriaMetrics supports data deduplication. See [these docs](https://docs.victoriametrics.com/single-server-victoriametrics/#deduplication).
|
||||
VictoriaMetrics supports data deduplication. See [these docs](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#deduplication).
|
||||
|
||||
|
||||
### Downsampling
|
||||
|
||||
VictoriaMetrics supports data downsampling. See [these docs](https://docs.victoriametrics.com/single-server-victoriametrics/#downsampling).
|
||||
VictoriaMetrics supports data downsampling. See [these docs](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#downsampling).
|
||||
|
||||
@@ -6,15 +6,16 @@ menu:
|
||||
parent: 'victoriametrics'
|
||||
weight: 82
|
||||
aliases:
|
||||
- /query-stats
|
||||
- /query-stats.html
|
||||
- /query-stats/index.html
|
||||
- /query-stats/
|
||||
---
|
||||
|
||||
[Enterprise version of VictoriaMetrics](https://docs.victoriametrics.com/enterprise/) supports statistics logging {{% available_from "#" %}} for
|
||||
executed read queries for [/api/v1/query](https://docs.victoriametrics.com/keyconcepts/#instant-query)
|
||||
and [/api/v1/query_range](https://docs.victoriametrics.com/keyconcepts/#range-query) API. To enable statistics
|
||||
logging specify `-search.logSlowQueryStats=<duration>` command line flag on [vmselect](https://docs.victoriametrics.com/cluster-victoriametrics/)
|
||||
[Enterprise version of VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics/enterprise/) supports statistics logging {{% available_from "#" %}} for
|
||||
executed read queries for [/api/v1/query](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#instant-query)
|
||||
and [/api/v1/query_range](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#range-query) API. To enable statistics
|
||||
logging specify `-search.logSlowQueryStats=<duration>` command line flag on [vmselect](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/)
|
||||
or [Single-node VictoriaMetrics](https://docs.victoriametrics.com/).
|
||||
Where `<duration>` is a threshold for query duration after which it must be logged:
|
||||
* `-search.logSlowQueryStats=5s` will log stats for queries with execution duration exceeding `5s`;
|
||||
@@ -26,17 +27,17 @@ The example of query statistics log is the following:
|
||||
2025-03-25T11:23:29.520Z info VictoriaMetrics/app/vmselect/promql/query_stats.go:60 vm_slow_query_stats type=instant query="vm_promscrape_config_last_reload_successful != 1\nor\nvmagent_relabel_config_last_reload_successful != 1\n" query_hash=1585303298 start_ms=1742901750000 end_ms=1742901750000 step_ms=300000 range_ms=0 tenant="0" execution_duration_ms=0 series_fetched=2 samples_fetched=163 bytes=975 memory_estimated_bytes=2032
|
||||
```
|
||||
|
||||
* `type` is either [instant](https://docs.victoriametrics.com/keyconcepts/#instant-query)
|
||||
or [range](https://docs.victoriametrics.com/keyconcepts/#range-query) query;
|
||||
* `query` is the executed [MetricsQL](https://docs.victoriametrics.com/metricsql/) query;
|
||||
* `type` is either [instant](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#instant-query)
|
||||
or [range](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#range-query) query;
|
||||
* `query` is the executed [MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/) query;
|
||||
* `query_hash` is a hashed `query` and is used to simplify filtering logs by a specific query;
|
||||
* `start_ms`, `end_ms`, `step_ms` are query params described [here](https://docs.victoriametrics.com/keyconcepts/#range-query);
|
||||
* `start_ms`, `end_ms`, `step_ms` are query params described [here](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#range-query);
|
||||
* `range_ms` is a difference between `start_ms` and `end_ms`. If `range_ms==0` it means this query is instant;
|
||||
* `tenant` is a tenant ID. Is available only for cluster version of VictoriaMetrics;
|
||||
* `execution_duration_ms` is execution duration of the query. It doesn't include time spent on transferring query results to the requester over network;
|
||||
* `series_fetched` is the amount of unique [time series](https://docs.victoriametrics.com/keyconcepts/#time-series) fetched during query execution. The number could be bigger than
|
||||
* `series_fetched` is the amount of unique [time series](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#time-series) fetched during query execution. The number could be bigger than
|
||||
the actual number of returned series, as it accounts for series before filtering by bool conditions (like `cpu_usage > 0`);
|
||||
* `samples_fetched` is the amount of [data samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) fetched
|
||||
* `samples_fetched` is the amount of [data samples](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#raw-samples) fetched
|
||||
during query execution;
|
||||
* `bytes` is the amount of bytes transferred from storage to execute the query;
|
||||
* `memory_estimated_bytes` is the estimated amount of memory that is needed to evaluate query. See `-search.maxMemoryPerQuery` cmd-line flag.
|
||||
@@ -73,4 +74,4 @@ we can build a dashboard with various stats and query filtering options:
|
||||
|
||||

|
||||
|
||||
The example of Grafana datasource is available [here](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/dashboards/query-stats.json).
|
||||
The example of Grafana datasource is available [here](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/dashboards/query-stats.json).
|
||||
|
||||
@@ -6,17 +6,18 @@ menu:
|
||||
parent: 'victoriametrics'
|
||||
weight: 38
|
||||
aliases:
|
||||
- /relabeling/
|
||||
- /relabeling.html
|
||||
- /relabeling/index.html
|
||||
- /relabeling/
|
||||
---
|
||||
VictoriaMetrics and [vmagent](https://docs.victoriametrics.com/vmagent/) support
|
||||
[Prometheus-compatible relabeling](https://docs.victoriametrics.com/vmagent/#relabeling)
|
||||
with [additional enhancements](https://docs.victoriametrics.com/vmagent/#relabeling-enhancements).
|
||||
VictoriaMetrics and [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/) support
|
||||
[Prometheus-compatible relabeling](https://docs.victoriametrics.com/victoriametrics/vmagent/#relabeling)
|
||||
with [additional enhancements](https://docs.victoriametrics.com/victoriametrics/vmagent/#relabeling-enhancements).
|
||||
|
||||
The relabeling is mostly used for the following tasks:
|
||||
|
||||
* Dropping unneeded scrape targets during [service discovery](https://docs.victoriametrics.com/sd_configs/#prometheus-service-discovery).
|
||||
* Dropping unneeded scrape targets during [service discovery](https://docs.victoriametrics.com/victoriametrics/sd_configs/#prometheus-service-discovery).
|
||||
See [how to drop discovered targets](#how-to-drop-discovered-targets).
|
||||
* Adding or updating static labels at scrape targets. See [how to add labels to scrape targets](#how-to-add-labels-to-scrape-targets).
|
||||
* Copying target labels from another labels. See [how to copy labels in scrape targets](#how-to-copy-labels-in-scrape-targets).
|
||||
@@ -33,7 +34,7 @@ The relabeling is mostly used for the following tasks:
|
||||
* Removing some labels from metrics matching some [series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors).
|
||||
See [how to remove labels from metrics subset](#how-to-remove-labels-from-metrics-subset).
|
||||
|
||||
See also [relabeling docs at vmagent](https://docs.victoriametrics.com/vmagent/#relabeling).
|
||||
See also [relabeling docs at vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/#relabeling).
|
||||
|
||||
|
||||
## How to remove labels from metrics subset
|
||||
@@ -63,7 +64,7 @@ See also:
|
||||
|
||||
## How to rename scraped metrics
|
||||
|
||||
Metric name is a regular label with special name - `__name__` (see [these docs](https://docs.victoriametrics.com/keyconcepts/#labels)).
|
||||
Metric name is a regular label with special name - `__name__` (see [these docs](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#labels)).
|
||||
So renaming of metric name is performed in the same way as changing label value.
|
||||
|
||||
Let's look at a few examples.
|
||||
@@ -115,7 +116,7 @@ See also [useful tips for metric relabeling](#useful-tips-for-metric-relabeling)
|
||||
|
||||
## How to add labels to scraped metrics
|
||||
|
||||
The following config sets `foo="bar"` [label](https://docs.victoriametrics.com/keyconcepts/#labels) across all the scraped metrics:
|
||||
The following config sets `foo="bar"` [label](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#labels) across all the scraped metrics:
|
||||
|
||||
```yaml
|
||||
scrape_configs:
|
||||
@@ -127,7 +128,7 @@ scrape_configs:
|
||||
replacement: bar
|
||||
```
|
||||
|
||||
The following config sets `foo="bar"` label only for metrics matching `{job=~"my-app-.*",env!="dev"}` [series selector](https://docs.victoriametrics.com/keyconcepts/#filtering):
|
||||
The following config sets `foo="bar"` label only for metrics matching `{job=~"my-app-.*",env!="dev"}` [series selector](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#filtering):
|
||||
|
||||
```yaml
|
||||
scrape_configs:
|
||||
@@ -159,7 +160,7 @@ scrape_configs:
|
||||
```
|
||||
|
||||
The following config adds `foo_` prefix to `job` label values only for metrics
|
||||
matching `{job=~"my-app-.*",env!="dev"}` [series selector](https://docs.victoriametrics.com/keyconcepts/#filtering):
|
||||
matching `{job=~"my-app-.*",env!="dev"}` [series selector](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#filtering):
|
||||
|
||||
```yaml
|
||||
scrape_configs:
|
||||
@@ -179,8 +180,8 @@ See also [useful tips for metric relabeling](#useful-tips-for-metric-relabeling)
|
||||
## How to remove labels from scraped metrics
|
||||
|
||||
Sometimes it may be needed to remove labels from scraped metrics. For example, if some labels
|
||||
lead to [high cardinality](https://docs.victoriametrics.com/faq/#what-is-high-cardinality)
|
||||
or [high churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate) issues,
|
||||
lead to [high cardinality](https://docs.victoriametrics.com/victoriametrics/faq/#what-is-high-cardinality)
|
||||
or [high churn rate](https://docs.victoriametrics.com/victoriametrics/faq/#what-is-high-churn-rate) issues,
|
||||
then it may be a good idea to drop these labels during scrapes.
|
||||
This can be done with `action: labeldrop` or `action: labelkeep` relabeling rules at `metric_relabel_configs` section:
|
||||
|
||||
@@ -216,8 +217,8 @@ See also [useful tips for metric relabeling](#useful-tips-for-metric-relabeling)
|
||||
## How to drop metrics during scrape
|
||||
|
||||
Sometimes it is needed to drop some metrics during scrapes. For example, if some metrics result
|
||||
in [high cardinality](https://docs.victoriametrics.com/faq/#what-is-high-cardinality)
|
||||
or [high churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate) issues,
|
||||
in [high cardinality](https://docs.victoriametrics.com/victoriametrics/faq/#what-is-high-cardinality)
|
||||
or [high churn rate](https://docs.victoriametrics.com/victoriametrics/faq/#what-is-high-churn-rate) issues,
|
||||
then it may be a good idea to drop these metrics during scrapes. This can be done with the `action: drop` or `action: keep`
|
||||
relabeling rules at `metric_relabel_configs` section:
|
||||
|
||||
@@ -246,12 +247,12 @@ See also [useful tips for metric relabeling](#useful-tips-for-metric-relabeling)
|
||||
|
||||
## How to remove labels from a subset of targets
|
||||
|
||||
Sometimes it is needed to remove some labels from a subset of [discovered targets](https://docs.victoriametrics.com/sd_configs/),
|
||||
Sometimes it is needed to remove some labels from a subset of [discovered targets](https://docs.victoriametrics.com/victoriametrics/sd_configs/),
|
||||
while leaving these labels in the rest of discovered targets.
|
||||
In this case the `if` [selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors)
|
||||
can be added to `action: labeldrop` or `action: labelkeep` relabeling rule.
|
||||
|
||||
For example, the following config discovers pod targets in [Kubernetes](https://docs.victoriametrics.com/sd_configs/#kubernetes_sd_configs),
|
||||
For example, the following config discovers pod targets in [Kubernetes](https://docs.victoriametrics.com/victoriametrics/sd_configs/#kubernetes_sd_configs),
|
||||
[extracts pod-level labels](#how-to-remove-prefixes-from-target-label-names) into labels with `foo_` prefix and then drops all the labels
|
||||
with `foo_bar_` prefix in their names for targets matching `{__address__=~"pod123.+"}` selector:
|
||||
|
||||
@@ -274,13 +275,13 @@ See also [how to remove labels from targets](#how-to-remove-labels-from-targets)
|
||||
|
||||
## How to remove labels from targets
|
||||
|
||||
Sometimes it is needed to remove some labels from [discovered targets](https://docs.victoriametrics.com/sd_configs/).
|
||||
Sometimes it is needed to remove some labels from [discovered targets](https://docs.victoriametrics.com/victoriametrics/sd_configs/).
|
||||
In this case the `action: labeldrop` and `action: labelkeep` relabeling options can be used:
|
||||
|
||||
* `action: labeldrop` drops all the labels with names matching the `regex` option
|
||||
* `action: labelkeep` drops all the labels with names not matching the `regex` option
|
||||
|
||||
For example, the following config discovers pod targets in [Kubernetes](https://docs.victoriametrics.com/sd_configs/#kubernetes_sd_configs),
|
||||
For example, the following config discovers pod targets in [Kubernetes](https://docs.victoriametrics.com/victoriametrics/sd_configs/#kubernetes_sd_configs),
|
||||
[extracts pod-level labels](#how-to-remove-prefixes-from-target-label-names) into labels with `foo_` prefix and then drops all the labels
|
||||
with `foo_bar_` prefix in their names:
|
||||
|
||||
@@ -311,8 +312,8 @@ See also [useful tips for target relabeling](#useful-tips-for-target-relabeling)
|
||||
|
||||
## How to remove prefixes from target label names
|
||||
|
||||
Sometimes it is needed to remove `__meta_*` prefixes from meta-labels of the [discovered targets](https://docs.victoriametrics.com/sd_configs/).
|
||||
For example, [Kubernetes service discovery](https://docs.victoriametrics.com/sd_configs/#kubernetes_sd_configs) adds `__meta_kubernetes_pod_label_<labelname>`
|
||||
Sometimes it is needed to remove `__meta_*` prefixes from meta-labels of the [discovered targets](https://docs.victoriametrics.com/victoriametrics/sd_configs/).
|
||||
For example, [Kubernetes service discovery](https://docs.victoriametrics.com/victoriametrics/sd_configs/#kubernetes_sd_configs) adds `__meta_kubernetes_pod_label_<labelname>`
|
||||
labels per each pod-level label. In this case it may be needed to leave only the `<labelname>` part of such label names,
|
||||
while removing the `__meta_kubernetes_pod_label_` prefix. This can be done with `action: labelmap` relabeling option:
|
||||
|
||||
@@ -342,7 +343,7 @@ See also [useful tips for target relabeling](#useful-tips-for-target-relabeling)
|
||||
Relabeling allows extracting parts from label values and storing them into arbitrary labels.
|
||||
This is performed with `regex` and `replacement` options in relabeling rules.
|
||||
|
||||
For example, the following config discovers pod targets in [Kubernetes](https://docs.victoriametrics.com/sd_configs/#kubernetes_sd_configs),
|
||||
For example, the following config discovers pod targets in [Kubernetes](https://docs.victoriametrics.com/victoriametrics/sd_configs/#kubernetes_sd_configs),
|
||||
extracts `bar` part from `foo/bar` container name and stores it into the `xyz` label with `abc_` prefix:
|
||||
|
||||
```yaml
|
||||
@@ -373,16 +374,16 @@ See also [useful tips for target relabeling](#useful-tips-for-target-relabeling)
|
||||
|
||||
## How to modify instance and job
|
||||
|
||||
Single-node VictoriaMetrics and [vmagent](https://docs.victoriametrics.com/vmagent/) automatically add `instance` and `job` labels per each discovered target:
|
||||
Single-node VictoriaMetrics and [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/) automatically add `instance` and `job` labels per each discovered target:
|
||||
|
||||
* The `job` label is set to `job_name` value specified in the corresponding [scrape_config](https://docs.victoriametrics.com/sd_configs/#scrape_configs).
|
||||
* The `job` label is set to `job_name` value specified in the corresponding [scrape_config](https://docs.victoriametrics.com/victoriametrics/sd_configs/#scrape_configs).
|
||||
* The `instance` label is set to the `host:port` part of `__address__` label value after target-level relabeling.
|
||||
The `__address__` label value is automatically set to the most suitable value depending
|
||||
on the used [service discovery type](https://docs.victoriametrics.com/sd_configs/#supported-service-discovery-configs).
|
||||
on the used [service discovery type](https://docs.victoriametrics.com/victoriametrics/sd_configs/#supported-service-discovery-configs).
|
||||
The `__address__` label can be overridden during relabeling - see [these docs](#how-to-modify-scrape-urls-in-targets).
|
||||
|
||||
Both `instance` and `job` labels can be overridden during relabeling. For example, the following config discovers pod targets
|
||||
in [Kubernetes](https://docs.victoriametrics.com/sd_configs/#kubernetes_sd_configs) and overrides `job` label from `k8s` to `foo`:
|
||||
in [Kubernetes](https://docs.victoriametrics.com/victoriametrics/sd_configs/#kubernetes_sd_configs) and overrides `job` label from `k8s` to `foo`:
|
||||
|
||||
```yaml
|
||||
scrape_configs:
|
||||
@@ -403,10 +404,10 @@ URLs for scrape targets are composed of the following parts:
|
||||
|
||||
* Scheme (e.g. `http` or `https`). The scheme is available during target relabeling in a special label - `__scheme__`.
|
||||
By default, the scheme is set to `http`. It can be overridden either by specifying the `scheme` option
|
||||
at [scrape_config](https://docs.victoriametrics.com/sd_configs/#scrape_configs) level
|
||||
at [scrape_config](https://docs.victoriametrics.com/victoriametrics/sd_configs/#scrape_configs) level
|
||||
or by updating the `__scheme__` label during relabeling.
|
||||
* Host and port (e.g. `host12:3456`). This information is available during target relabeling in a special label - `__address__`.
|
||||
Every [supported service discovery type](https://docs.victoriametrics.com/sd_configs/#supported-service-discovery-configs)
|
||||
Every [supported service discovery type](https://docs.victoriametrics.com/victoriametrics/sd_configs/#supported-service-discovery-configs)
|
||||
sets the `__address__` label to the most suitable value. Sometimes this value needs to be modified. In this case
|
||||
just update the `__address__` label during relabeling to the needed value.
|
||||
The port part is optional. If it is missing, then it is automatically set either to `80` or `443` depending
|
||||
@@ -417,12 +418,12 @@ URLs for scrape targets are composed of the following parts:
|
||||
In this case the `__scheme__` and `__metrics_path__` labels are ignored.
|
||||
* URL path (e.g. `/metrics`). This information is available during target relabeling in a special label - `__metrics_path__`.
|
||||
By default, the `__metrics_path__` is set to `/metrics`. It can be overridden either by specifying the `metrics_path`
|
||||
option at [scrape_config](https://docs.victoriametrics.com/sd_configs/#scrape_configs)
|
||||
option at [scrape_config](https://docs.victoriametrics.com/victoriametrics/sd_configs/#scrape_configs)
|
||||
or by updating the `__metrics_path__` label during relabeling.
|
||||
* Query args (e.g. `?foo=bar&baz=xyz`). This information is available during target relabeling in special labels
|
||||
with `__param_` prefix. For example, `__param_foo` would have the `bar` value, while `__param_baz` would have the `xyz` value
|
||||
for `?foo=bar&baz=xyz` query string. The query args can be specified either via `params` section
|
||||
at [scrape_config](https://docs.victoriametrics.com/sd_configs/#scrape_configs)
|
||||
at [scrape_config](https://docs.victoriametrics.com/victoriametrics/sd_configs/#scrape_configs)
|
||||
or by updating/setting the corresponding `__param_*` labels during relabeling.
|
||||
|
||||
The resulting scrape url looks like the following:
|
||||
@@ -436,7 +437,7 @@ in [Prometheus text exposition format](https://github.com/prometheus/docs/blob/m
|
||||
at the resulting scrape url.
|
||||
|
||||
Given the scrape url construction rules above, the following config discovers pod targets
|
||||
in [Kubernetes](https://docs.victoriametrics.com/sd_configs/#kubernetes_sd_configs)
|
||||
in [Kubernetes](https://docs.victoriametrics.com/victoriametrics/sd_configs/#kubernetes_sd_configs)
|
||||
and constructs per-target scrape url as `https://<pod_name>/foo/bar?baz=<container_name>`:
|
||||
|
||||
```yaml
|
||||
@@ -462,7 +463,7 @@ See also [useful tips for target relabeling](#useful-tips-for-target-relabeling)
|
||||
Labels can be copied by specifying the source labels via `source_labels` relabeling option
|
||||
and specifying the target label via `target_label` relabeling option.
|
||||
For example, the following config copies `__meta_kubernetes_pod_name` label to `pod` label
|
||||
for all the discovered pods in [Kubernetes](https://docs.victoriametrics.com/sd_configs/#kubernetes_sd_configs):
|
||||
for all the discovered pods in [Kubernetes](https://docs.victoriametrics.com/victoriametrics/sd_configs/#kubernetes_sd_configs):
|
||||
|
||||
```yaml
|
||||
scrape_configs:
|
||||
@@ -478,7 +479,7 @@ Note that the `source_labels` option accepts a list of labels in square brackets
|
||||
in the `source_labels` list, then the specified label values are joined into a single string with `;` delimiter by default.
|
||||
The delimiter can be modified by specifying it via `separator` option.
|
||||
For example, the following config sets the `pod_name:container_port` value to the `host_port` label
|
||||
for all the discovered pod targets in [Kubernetes](https://docs.victoriametrics.com/sd_configs/#kubernetes_sd_configs):
|
||||
for all the discovered pod targets in [Kubernetes](https://docs.victoriametrics.com/victoriametrics/sd_configs/#kubernetes_sd_configs):
|
||||
|
||||
```yaml
|
||||
scrape_configs:
|
||||
@@ -500,7 +501,7 @@ Additional labels can be added to scrape targets by specifying the label name in
|
||||
and by specifying the label value in `replacement` relabeling option.
|
||||
The same approach can be used for updating already existing label values at target level.
|
||||
|
||||
For example, the following config adds `{foo="bar"}` label to all the discovered pods in [Kubernetes](https://docs.victoriametrics.com/sd_configs/#kubernetes_sd_configs):
|
||||
For example, the following config adds `{foo="bar"}` label to all the discovered pods in [Kubernetes](https://docs.victoriametrics.com/victoriametrics/sd_configs/#kubernetes_sd_configs):
|
||||
|
||||
```yaml
|
||||
scrape_configs:
|
||||
@@ -541,7 +542,7 @@ If a particular discovered target shouldn't be scraped, then `action: keep` or `
|
||||
must be used inside `relabel_configs` section.
|
||||
|
||||
The `action: keep` keeps only scrape targets with labels matching the `if` [selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors),
|
||||
while dropping the rest of targets. For example, the following config discovers pod targets in [Kubernetes](https://docs.victoriametrics.com/sd_configs/#kubernetes_sd_configs)
|
||||
while dropping the rest of targets. For example, the following config discovers pod targets in [Kubernetes](https://docs.victoriametrics.com/victoriametrics/sd_configs/#kubernetes_sd_configs)
|
||||
and scrapes only pods with names starting with `foo` prefix:
|
||||
|
||||
```yaml
|
||||
@@ -555,7 +556,7 @@ scrape_configs:
|
||||
```
|
||||
|
||||
The `action: drop` drops all the scrape targets with labels matching the `if` [selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors),
|
||||
while keeping the rest of targets. For example, the following config discovers pod targets in [Kubernetes](https://docs.victoriametrics.com/sd_configs/#kubernetes_sd_configs)
|
||||
while keeping the rest of targets. For example, the following config discovers pod targets in [Kubernetes](https://docs.victoriametrics.com/victoriametrics/sd_configs/#kubernetes_sd_configs)
|
||||
and scrapes only pods with names starting with prefixes other than `foo`:
|
||||
|
||||
```yaml
|
||||
@@ -574,10 +575,10 @@ See also [useful tips for target relabeling](#useful-tips-for-target-relabeling)
|
||||
## Useful tips for target relabeling
|
||||
|
||||
* Target relabeling can be debugged by clicking the `debug` link for the needed target on the `http://vmagent:8429/target`
|
||||
or on the `http://vmagent:8429/service-discovery` pages. See [these docs](https://docs.victoriametrics.com/vmagent/#relabel-debug).
|
||||
or on the `http://vmagent:8429/service-discovery` pages. See [these docs](https://docs.victoriametrics.com/victoriametrics/vmagent/#relabel-debug).
|
||||
* Every discovered target contains a set of meta-labels, which start with `__meta_` prefix.
|
||||
The specific sets of labels per each supported service discovery option are listed
|
||||
[here](https://docs.victoriametrics.com/sd_configs/#prometheus-service-discovery).
|
||||
[here](https://docs.victoriametrics.com/victoriametrics/sd_configs/#prometheus-service-discovery).
|
||||
* Every discovered target contains additional labels with `__` prefix other than `__meta_` labels.
|
||||
See [these docs](#how-to-modify-scrape-urls-in-targets) for more details.
|
||||
* All the labels, which start with `__` prefix, are automatically removed from targets after the relabeling.
|
||||
@@ -592,7 +593,7 @@ See also [useful tips for target relabeling](#useful-tips-for-target-relabeling)
|
||||
## Useful tips for metric relabeling
|
||||
|
||||
* Metric relabeling can be debugged at `http://vmagent:8429/metric-relabel-debug` page.
|
||||
See [these docs](https://docs.victoriametrics.com/vmagent/#relabel-debug).
|
||||
See [these docs](https://docs.victoriametrics.com/victoriametrics/vmagent/#relabel-debug).
|
||||
* All the labels, which start with `__` prefix, are automatically removed from metrics after the relabeling.
|
||||
So it is common practice to store temporary labels with names starting with `__` during metrics relabeling.
|
||||
* All the target-level labels are automatically added to all the metrics scraped from targets,
|
||||
|
||||
@@ -6,6 +6,7 @@ menu:
|
||||
parent: 'victoriametrics'
|
||||
weight: 37
|
||||
aliases:
|
||||
- /scrape_config_examples/
|
||||
- /scrape_config_examples.html
|
||||
- /scrape_config_examples/index.html
|
||||
- /scrape_config_examples/
|
||||
@@ -38,7 +39,7 @@ tar xzf victoria-metrics-linux-amd64-v1.115.0.tar.gz
|
||||
```
|
||||
|
||||
Then start VictoriaMetrics and instruct it to scrape targets defined in `scrape.yaml` and save scraped metrics
|
||||
to local storage according to [these docs](https://docs.victoriametrics.com/#how-to-scrape-prometheus-exporters-such-as-node-exporter):
|
||||
to local storage according to [these docs](https://docs.victoriametrics.com/victoriametrics#how-to-scrape-prometheus-exporters-such-as-node-exporter):
|
||||
|
||||
```
|
||||
./victoria-metrics-prod -promscrape.config=scrape.yaml
|
||||
@@ -48,7 +49,7 @@ Now open the `http://localhost:8428/targets` page in web browser in order to see
|
||||
The page must contain the information about the target at `http://localhost:9100/metrics` url.
|
||||
It is likely the target has `state: down` if you didn't start [`node-exporter`](https://github.com/prometheus/node_exporter) on `localhost`.
|
||||
|
||||
Let's add a new scrape config to `scrape.yaml` for scraping [VictoriaMetrics metrics](https://docs.victoriametrics.com/#monitoring):
|
||||
Let's add a new scrape config to `scrape.yaml` for scraping [VictoriaMetrics metrics](https://docs.victoriametrics.com/victoriametrics#monitoring):
|
||||
|
||||
```yaml
|
||||
scrape_configs:
|
||||
@@ -63,10 +64,10 @@ scrape_configs:
|
||||
```
|
||||
|
||||
Note that the last specified target contains the full url instead of host and port.
|
||||
This is an extension supported by VictoriaMetrics and [vmagent](https://docs.victoriametrics.com/vmagent/) - you can use both `host:port`
|
||||
This is an extension supported by VictoriaMetrics and [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/) - you can use both `host:port`
|
||||
and full urls in scrape target lists.
|
||||
|
||||
Send `SIGHUP` signal `victoria-metrics-prod` process, so it [reloads the updated `scrape.yaml`](https://docs.victoriametrics.com/vmagent/#configuration-update):
|
||||
Send `SIGHUP` signal `victoria-metrics-prod` process, so it [reloads the updated `scrape.yaml`](https://docs.victoriametrics.com/victoriametrics/vmagent/#configuration-update):
|
||||
|
||||
```
|
||||
kill -HUP `pidof victoria-metrics-prod`
|
||||
@@ -75,10 +76,10 @@ kill -HUP `pidof victoria-metrics-prod`
|
||||
Now the `http://localhost:8428/targets` page must contain two targets - `http://localhost:9100/metrics` and `http://localhost:8428/metrics`.
|
||||
The last one should have `state: up`, since this is VictoriaMetrics itself.
|
||||
|
||||
Let's query the scraped metrics. Open `http://localhost:8428/vmui/` aka [vmui](https://docs.victoriametrics.com/#vmui), enter `up` in the query input field
|
||||
Let's query the scraped metrics. Open `http://localhost:8428/vmui/` aka [vmui](https://docs.victoriametrics.com/victoriametrics#vmui), enter `up` in the query input field
|
||||
and press `enter`. You'll see a graph for `up` metrics. It must contain two lines for the targets defined in `scrape.yaml` file above.
|
||||
See [these docs](https://docs.victoriametrics.com/vmagent/#automatically-generated-metrics) about `up` metric. You can explore other scraped metrics
|
||||
in `vmui` via [Prometheus metrics explorer](https://docs.victoriametrics.com/#metrics-explorer).
|
||||
See [these docs](https://docs.victoriametrics.com/victoriametrics/vmagent/#automatically-generated-metrics) about `up` metric. You can explore other scraped metrics
|
||||
in `vmui` via [Prometheus metrics explorer](https://docs.victoriametrics.com/victoriametrics#metrics-explorer).
|
||||
|
||||
Let's look closely to the contents of the `scrape.yaml` file created above:
|
||||
|
||||
@@ -94,35 +95,35 @@ scrape_configs:
|
||||
- http://localhost:8428/metrics
|
||||
```
|
||||
|
||||
The [`scrape_configs`](https://docs.victoriametrics.com/sd_configs/#scrape_configs) section contains a list of scrape configs.
|
||||
The [`scrape_configs`](https://docs.victoriametrics.com/victoriametrics/sd_configs/#scrape_configs) section contains a list of scrape configs.
|
||||
Our `scrape.yaml` file contains two scrape configs - for `job_name: node-exporter` and for `job_name: victoriametrics`.
|
||||
[vmagent](https://docs.victoriametrics.com/vmagent/) and [single-node VictoriaMetrics](https://docs.victoriametrics.com/)
|
||||
[vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/) and [single-node VictoriaMetrics](https://docs.victoriametrics.com/)
|
||||
can efficiently process thousands of scrape configs in production.
|
||||
|
||||
Every scrape config in the list **must** contain `job_name` field - its' value is used as [`job`](https://prometheus.io/docs/concepts/jobs_instances/) label
|
||||
in all the metrics scraped from targets defined in this scrape config.
|
||||
Every scrape config must contain at least a single section from [this list](https://docs.victoriametrics.com/sd_configs/#supported-service-discovery-configs).
|
||||
Every scrape config may contain other options described [here](https://docs.victoriametrics.com/sd_configs/#scrape_configs).
|
||||
Every scrape config must contain at least a single section from [this list](https://docs.victoriametrics.com/victoriametrics/sd_configs/#supported-service-discovery-configs).
|
||||
Every scrape config may contain other options described [here](https://docs.victoriametrics.com/victoriametrics/sd_configs/#scrape_configs).
|
||||
|
||||
In our case only [`static_configs`](https://docs.victoriametrics.com/sd_configs/#static_configs) sections are used.
|
||||
These sections consist of a list of static configs according to [these docs](https://docs.victoriametrics.com/sd_configs/#static_configs).
|
||||
In our case only [`static_configs`](https://docs.victoriametrics.com/victoriametrics/sd_configs/#static_configs) sections are used.
|
||||
These sections consist of a list of static configs according to [these docs](https://docs.victoriametrics.com/victoriametrics/sd_configs/#static_configs).
|
||||
Every static config contains a list of `targets`, which need to be scraped. The target address is used as [`instance`](https://prometheus.io/docs/concepts/jobs_instances/)
|
||||
label in all the metrics scraped from the target.
|
||||
|
||||
[vmagent](https://docs.victoriametrics.com/vmagent/) and [single-node VictoriaMetrics](https://docs.victoriametrics.com/)
|
||||
[vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/) and [single-node VictoriaMetrics](https://docs.victoriametrics.com/)
|
||||
can efficiently process tens of thousands of targets in production. If you need scraping more targets,
|
||||
then see [these docs](https://docs.victoriametrics.com/vmagent/#scraping-big-number-of-targets).
|
||||
then see [these docs](https://docs.victoriametrics.com/victoriametrics/vmagent/#scraping-big-number-of-targets).
|
||||
|
||||
Targets are scraped at `http` or `https` urls, which are formed according to [these rules](https://docs.victoriametrics.com/relabeling/#how-to-modify-scrape-urls-in-targets).
|
||||
It is possible to modify scrape urls via [relabeling](https://docs.victoriametrics.com/relabeling/) if needed.
|
||||
Targets are scraped at `http` or `https` urls, which are formed according to [these rules](https://docs.victoriametrics.com/victoriametrics/relabeling/#how-to-modify-scrape-urls-in-targets).
|
||||
It is possible to modify scrape urls via [relabeling](https://docs.victoriametrics.com/victoriametrics/relabeling/) if needed.
|
||||
|
||||
|
||||
## File-based target discovery
|
||||
|
||||
It may be not so convenient updating `scrape.yaml` file with [`static_configs`](https://docs.victoriametrics.com/sd_configs/#static_configs)
|
||||
every time new scrape target is added, changed or removed. In this case [`file_sd_configs`](https://docs.victoriametrics.com/sd_configs/#file_sd_configs)
|
||||
It may be not so convenient updating `scrape.yaml` file with [`static_configs`](https://docs.victoriametrics.com/victoriametrics/sd_configs/#static_configs)
|
||||
every time new scrape target is added, changed or removed. In this case [`file_sd_configs`](https://docs.victoriametrics.com/victoriametrics/sd_configs/#file_sd_configs)
|
||||
can come to rescue. It allows defining a list of scrape targets in `JSON` files, and automatically updating the list of scrape targets
|
||||
at [vmagent](https://docs.victoriametrics.com/vmagent/) or [single-node VictoriaMetrics](https://docs.victoriametrics.com/) side
|
||||
at [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/) or [single-node VictoriaMetrics](https://docs.victoriametrics.com/) side
|
||||
when the corresponding `JSON` files are updated.
|
||||
|
||||
Let's create `node_exporter_targets.json` file with the following contents:
|
||||
@@ -145,7 +146,7 @@ scrape_configs:
|
||||
- node_exporter_targets.json
|
||||
```
|
||||
|
||||
Then start [single-node VictoriaMetrics](https://docs.victoriametrics.com/) according to [these docs](https://docs.victoriametrics.com/#how-to-scrape-prometheus-exporters-such-as-node-exporter):
|
||||
Then start [single-node VictoriaMetrics](https://docs.victoriametrics.com/) according to [these docs](https://docs.victoriametrics.com/victoriametrics#how-to-scrape-prometheus-exporters-such-as-node-exporter):
|
||||
|
||||
```yaml
|
||||
# Download and unpack single-node VictoriaMetrics
|
||||
@@ -169,13 +170,13 @@ Now let's add more targets to `node_exporter_targets.json`:
|
||||
```
|
||||
|
||||
Note that the added targets contains full urls instead of host and port.
|
||||
This is an extension supported by VictoriaMetrics and [vmagent](https://docs.victoriametrics.com/vmagent/) - you can use both `host:port`
|
||||
This is an extension supported by VictoriaMetrics and [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/) - you can use both `host:port`
|
||||
and full urls in scrape target lists.
|
||||
|
||||
Save the updated `node_exporter_targets.json`, wait for 30 seconds and then refresh the `http://localhost:8428/targets` page.
|
||||
Now this page must contain all the targets defined in the updated `node_exporter_targets.json`.
|
||||
By default [vmagent](https://docs.victoriametrics.com/vmagent/) and [single-node VictoriaMetrics](https://docs.victoriametrics.com/)
|
||||
check for updates in `files` specified at [`file_sd_configs`](https://docs.victoriametrics.com/sd_configs/#file_sd_configs)
|
||||
By default [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/) and [single-node VictoriaMetrics](https://docs.victoriametrics.com/)
|
||||
check for updates in `files` specified at [`file_sd_configs`](https://docs.victoriametrics.com/victoriametrics/sd_configs/#file_sd_configs)
|
||||
every 30 seconds. This interval can be changed via `-promscrape.fileSDCheckInterval` command-line flag.
|
||||
For example, the following command starts VictoriaMetrics, which checks for updates in `file_sd_configs` every 5 seconds:
|
||||
|
||||
@@ -199,22 +200,22 @@ scrape_configs:
|
||||
```
|
||||
|
||||
It is possible to specify directories with `*` wildcards for distinct sets of targets at `file_sd_configs`.
|
||||
See [these docs](https://docs.victoriametrics.com/sd_configs/#file_sd_configs) for details.
|
||||
See [these docs](https://docs.victoriametrics.com/victoriametrics/sd_configs/#file_sd_configs) for details.
|
||||
|
||||
[vmagent](https://docs.victoriametrics.com/vmagent/) and [single-node VictoriaMetrics](https://docs.victoriametrics.com/)
|
||||
[vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/) and [single-node VictoriaMetrics](https://docs.victoriametrics.com/)
|
||||
can efficiently scrape tens of thousands of scrape targets. If you need scraping more targets,
|
||||
then see [these docs](https://docs.victoriametrics.com/vmagent/#scraping-big-number-of-targets).
|
||||
then see [these docs](https://docs.victoriametrics.com/victoriametrics/vmagent/#scraping-big-number-of-targets).
|
||||
|
||||
Targets are scraped at `http` or `https` urls, which are formed according to [these rules](https://docs.victoriametrics.com/relabeling/#how-to-modify-scrape-urls-in-targets).
|
||||
It is possible to modify scrape urls via [relabeling](https://docs.victoriametrics.com/relabeling/) if needed.
|
||||
Targets are scraped at `http` or `https` urls, which are formed according to [these rules](https://docs.victoriametrics.com/victoriametrics/relabeling/#how-to-modify-scrape-urls-in-targets).
|
||||
It is possible to modify scrape urls via [relabeling](https://docs.victoriametrics.com/victoriametrics/relabeling/) if needed.
|
||||
|
||||
|
||||
## HTTP-based target discovery
|
||||
|
||||
It may not so convenient maintaining a list of local files for [`file_sd_configs`](https://docs.victoriametrics.com/sd_configs/#file_sd_configs).
|
||||
In this case [`http_sd_configs`](https://docs.victoriametrics.com/sd_configs/#http_sd_configs) can help.
|
||||
It may not so convenient maintaining a list of local files for [`file_sd_configs`](https://docs.victoriametrics.com/victoriametrics/sd_configs/#file_sd_configs).
|
||||
In this case [`http_sd_configs`](https://docs.victoriametrics.com/victoriametrics/sd_configs/#http_sd_configs) can help.
|
||||
They allow specifying a list of `http` or `https` urls, which return targets, which need to be scraped.
|
||||
For example, the following [`-promscrape.config`](https://docs.victoriametrics.com/#how-to-scrape-prometheus-exporters-such-as-node-exporter)
|
||||
For example, the following [`-promscrape.config`](https://docs.victoriametrics.com/victoriametrics#how-to-scrape-prometheus-exporters-such-as-node-exporter)
|
||||
periodically fetches the list of targets from the specified url:
|
||||
|
||||
```yaml
|
||||
@@ -235,7 +236,7 @@ If you feel brave, let's look at a few typical cases for Kubernetes monitoring.
|
||||
|
||||
### Discovering and scraping `node-exporter` targets in Kubernetes
|
||||
|
||||
The following [`-promscrape.config`](https://docs.victoriametrics.com/#how-to-scrape-prometheus-exporters-such-as-node-exporter)
|
||||
The following [`-promscrape.config`](https://docs.victoriametrics.com/victoriametrics#how-to-scrape-prometheus-exporters-such-as-node-exporter)
|
||||
instructs discovering and scraping all the [`node-exporter`](https://github.com/prometheus/node_exporter) targets inside Kubernetes cluster:
|
||||
|
||||
```yaml
|
||||
@@ -260,18 +261,18 @@ scrape_configs:
|
||||
target_label: node
|
||||
```
|
||||
|
||||
See [`kubernetes_sd_configs` docs](https://docs.victoriametrics.com/sd_configs/#kubernetes_sd_configs) for more details.
|
||||
See [`kubernetes_sd_configs` docs](https://docs.victoriametrics.com/victoriametrics/sd_configs/#kubernetes_sd_configs) for more details.
|
||||
|
||||
See [relabeling docs](https://docs.victoriametrics.com/vmagent/#relabeling) for details on `relabel_configs`.
|
||||
See [relabeling docs](https://docs.victoriametrics.com/victoriametrics/vmagent/#relabeling) for details on `relabel_configs`.
|
||||
|
||||
### Discovering and scraping `kube-state-metrics` in Kubernetes
|
||||
|
||||
[kube-state-metrics](https://github.com/kubernetes/kube-state-metrics) is a special metrics exporter,
|
||||
which exposes `state` metrics for all the Kubernetes objects such as `container`, `pod`, `node`, etc.
|
||||
It already sets `namespace`, `container`, `pod` and `node` labels for every exposed metric,
|
||||
so these metrics shouldn't be set in [target relabeling](https://docs.victoriametrics.com/vmagent/#relabeling).
|
||||
so these metrics shouldn't be set in [target relabeling](https://docs.victoriametrics.com/victoriametrics/vmagent/#relabeling).
|
||||
|
||||
The following [`-promscrape.config`](https://docs.victoriametrics.com/#how-to-scrape-prometheus-exporters-such-as-node-exporter)
|
||||
The following [`-promscrape.config`](https://docs.victoriametrics.com/victoriametrics#how-to-scrape-prometheus-exporters-such-as-node-exporter)
|
||||
instructs discovering and scraping [kube-state-metrics](https://github.com/kubernetes/kube-state-metrics) target inside Kubernetes cluster:
|
||||
|
||||
```yaml
|
||||
@@ -298,14 +299,14 @@ scrape_configs:
|
||||
action: keep
|
||||
```
|
||||
|
||||
See [`kubernetes_sd_configs` docs](https://docs.victoriametrics.com/sd_configs/#kubernetes_sd_configs) for more details.
|
||||
See [`kubernetes_sd_configs` docs](https://docs.victoriametrics.com/victoriametrics/sd_configs/#kubernetes_sd_configs) for more details.
|
||||
|
||||
See [relabeling docs](https://docs.victoriametrics.com/vmagent/#relabeling) for details on `relabel_configs`.
|
||||
See [relabeling docs](https://docs.victoriametrics.com/victoriametrics/vmagent/#relabeling) for details on `relabel_configs`.
|
||||
|
||||
### Discovering and scraping metrics from `cadvisor`
|
||||
|
||||
[cadvisor](https://github.com/google/cadvisor) exposes resource usage metrics for every container in Kubernetes.
|
||||
The following [`-promscrape.config`](https://docs.victoriametrics.com/#how-to-scrape-prometheus-exporters-such-as-node-exporter)
|
||||
The following [`-promscrape.config`](https://docs.victoriametrics.com/victoriametrics#how-to-scrape-prometheus-exporters-such-as-node-exporter)
|
||||
can be used for collecting `cadvisor` metrics in Kubernetes:
|
||||
|
||||
```yaml
|
||||
@@ -336,15 +337,15 @@ scrape_configs:
|
||||
target_label: instance
|
||||
```
|
||||
|
||||
See [`kubernetes_sd_configs` docs](https://docs.victoriametrics.com/sd_configs/#kubernetes_sd_configs) for more details.
|
||||
See [`kubernetes_sd_configs` docs](https://docs.victoriametrics.com/victoriametrics/sd_configs/#kubernetes_sd_configs) for more details.
|
||||
|
||||
See [relabeling docs](https://docs.victoriametrics.com/vmagent/#relabeling) for details on `relabel_configs`.
|
||||
See [relabeling docs](https://docs.victoriametrics.com/victoriametrics/vmagent/#relabeling) for details on `relabel_configs`.
|
||||
|
||||
See [these docs](https://docs.victoriametrics.com/sd_configs/#http-api-client-options) for details on `bearer_token_file` and `tls_config` options.
|
||||
See [these docs](https://docs.victoriametrics.com/victoriametrics/sd_configs/#http-api-client-options) for details on `bearer_token_file` and `tls_config` options.
|
||||
|
||||
### Discovering and scraping metrics for a particular container in Kubernetes
|
||||
|
||||
The following [`-promscrape.config`](https://docs.victoriametrics.com/#how-to-scrape-prometheus-exporters-such-as-node-exporter)
|
||||
The following [`-promscrape.config`](https://docs.victoriametrics.com/victoriametrics#how-to-scrape-prometheus-exporters-such-as-node-exporter)
|
||||
instructs discovering and scraping metrics for all the containers with the name `my-super-app`.
|
||||
It is expected that these containers expose only a single TCP port, which serves its metrics at `/metrics` page
|
||||
according to [Prometheus text exposition format](https://github.com/prometheus/docs/blob/master/content/docs/instrumenting/exposition_formats.md#text-based-format):
|
||||
@@ -357,7 +358,7 @@ scrape_configs:
|
||||
relabel_configs:
|
||||
|
||||
# Leave only targets with the container name, which matches the `job_name` specified above
|
||||
# See https://docs.victoriametrics.com/relabeling/#how-to-modify-instance-and-job for details on `job` label.
|
||||
# See https://docs.victoriametrics.com/victoriametrics/relabeling/#how-to-modify-instance-and-job for details on `job` label.
|
||||
#
|
||||
- source_labels: [__meta_kubernetes_pod_container_name]
|
||||
target_label: job
|
||||
@@ -377,6 +378,6 @@ scrape_configs:
|
||||
target_label: container
|
||||
```
|
||||
|
||||
See [`kubernetes_sd_configs` docs](https://docs.victoriametrics.com/sd_configs/#kubernetes_sd_configs) for more details.
|
||||
See [`kubernetes_sd_configs` docs](https://docs.victoriametrics.com/victoriametrics/sd_configs/#kubernetes_sd_configs) for more details.
|
||||
|
||||
See [relabeling docs](https://docs.victoriametrics.com/vmagent/#relabeling) for details on `relabel_configs`.
|
||||
See [relabeling docs](https://docs.victoriametrics.com/victoriametrics/vmagent/#relabeling) for details on `relabel_configs`.
|
||||
|
||||
@@ -6,13 +6,14 @@ menu:
|
||||
parent: 'victoriametrics'
|
||||
weight: 36
|
||||
aliases:
|
||||
- /sd_configs/
|
||||
- /sd_configs.html
|
||||
- /sd_configs/index.html
|
||||
- /sd_configs/
|
||||
---
|
||||
# Supported service discovery configs
|
||||
|
||||
[vmagent](https://docs.victoriametrics.com/vmagent/) and [single-node VictoriaMetrics](https://docs.victoriametrics.com/#how-to-scrape-prometheus-exporters-such-as-node-exporter)
|
||||
[vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/) and [single-node VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics#how-to-scrape-prometheus-exporters-such-as-node-exporter)
|
||||
supports the following Prometheus-compatible service discovery options for Prometheus-compatible scrape targets in the file pointed by `-promscrape.config` command-line flag:
|
||||
|
||||
* `azure_sd_configs` is for scraping the targets registered in [Azure Cloud](https://azure.microsoft.com/en-us/). See [these docs](#azure_sd_configs).
|
||||
@@ -93,14 +94,14 @@ scrape_configs:
|
||||
# port: ...
|
||||
|
||||
# Additional HTTP API client options can be specified here.
|
||||
# See https://docs.victoriametrics.com/sd_configs/#http-api-client-options
|
||||
# See https://docs.victoriametrics.com/victoriametrics/sd_configs/#http-api-client-options
|
||||
```
|
||||
|
||||
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/relabeling/#how-to-modify-scrape-urls-in-targets) label set
|
||||
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/victoriametrics/relabeling/#how-to-modify-scrape-urls-in-targets) label set
|
||||
to `<private_ip>:<port>`, where `<private_ip>` is the machine's private IP and the `<port>` is the `port`
|
||||
option specified in the `azure_sd_configs`.
|
||||
|
||||
The following meta labels are available on discovered targets during [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling):
|
||||
The following meta labels are available on discovered targets during [relabeling](https://docs.victoriametrics.com/victoriametrics/vmagent/#relabeling):
|
||||
|
||||
* `__meta_azure_machine_id`: the machine ID
|
||||
* `__meta_azure_machine_location`: the location the machine runs in
|
||||
@@ -201,14 +202,14 @@ scrape_configs:
|
||||
# allow_stale: ...
|
||||
|
||||
# Additional HTTP API client options can be specified here.
|
||||
# See https://docs.victoriametrics.com/sd_configs/#http-api-client-options
|
||||
# See https://docs.victoriametrics.com/victoriametrics/sd_configs/#http-api-client-options
|
||||
```
|
||||
|
||||
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/relabeling/#how-to-modify-scrape-urls-in-targets) label set
|
||||
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/victoriametrics/relabeling/#how-to-modify-scrape-urls-in-targets) label set
|
||||
to `<service_or_node_addr>:<service_port>`, where `<service_or_node_addr>` is the service address. If the service address is empty,
|
||||
then the node address is used instead. The `<service_port>` is the service port.
|
||||
|
||||
The following meta labels are available on discovered targets during [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling):
|
||||
The following meta labels are available on discovered targets during [relabeling](https://docs.victoriametrics.com/victoriametrics/vmagent/#relabeling):
|
||||
|
||||
* `__meta_consul_address`: the address of the target
|
||||
* `__meta_consul_dc`: the datacenter name for the target
|
||||
@@ -293,14 +294,14 @@ scrape_configs:
|
||||
# filter: "..."
|
||||
|
||||
# Additional HTTP API client options can be specified here.
|
||||
# See https://docs.victoriametrics.com/sd_configs/#http-api-client-options
|
||||
# See https://docs.victoriametrics.com/victoriametrics/sd_configs/#http-api-client-options
|
||||
```
|
||||
|
||||
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/relabeling/#how-to-modify-scrape-urls-in-targets) label set
|
||||
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/victoriametrics/relabeling/#how-to-modify-scrape-urls-in-targets) label set
|
||||
to `<service_or_node_addr>:<service_port>`, where `<service_or_node_addr>` is the service address. If the service address is empty,
|
||||
then the node address is used instead. The `<service_port>` is the service port.
|
||||
|
||||
The following meta labels are available on discovered targets during [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling):
|
||||
The following meta labels are available on discovered targets during [relabeling](https://docs.victoriametrics.com/victoriametrics/vmagent/#relabeling):
|
||||
|
||||
* `__meta_consulagent_address`: the address of the target
|
||||
* `__meta_consulagent_dc`: the datacenter name for the target
|
||||
@@ -341,13 +342,13 @@ scrape_configs:
|
||||
# port: ...
|
||||
|
||||
# Additional HTTP API client options can be specified here.
|
||||
# See https://docs.victoriametrics.com/sd_configs/#http-api-client-options
|
||||
# See https://docs.victoriametrics.com/victoriametrics/sd_configs/#http-api-client-options
|
||||
```
|
||||
|
||||
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/relabeling/#how-to-modify-scrape-urls-in-targets) label set
|
||||
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/victoriametrics/relabeling/#how-to-modify-scrape-urls-in-targets) label set
|
||||
to `<public_ip>:<port>`, where `<public_ip>` is a public ipv4 address of the droplet, while `<port>` is the port specified in the `digitalocean_sd_configs`.
|
||||
|
||||
The following meta labels are available on discovered targets during [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling):
|
||||
The following meta labels are available on discovered targets during [relabeling](https://docs.victoriametrics.com/victoriametrics/vmagent/#relabeling):
|
||||
|
||||
* `__meta_digitalocean_droplet_id`: the id of the droplet
|
||||
* `__meta_digitalocean_droplet_name`: the name of the droplet
|
||||
@@ -393,11 +394,11 @@ scrape_configs:
|
||||
# port: ...
|
||||
```
|
||||
|
||||
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/relabeling/#how-to-modify-scrape-urls-in-targets) label set
|
||||
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/victoriametrics/relabeling/#how-to-modify-scrape-urls-in-targets) label set
|
||||
to the `<addr>:<port>`, where `<addr>` is the discovered DNS address, while `<port>` is either the discovered port for SRV records or the port
|
||||
specified in the `dns_sd_config`.
|
||||
|
||||
The following meta labels are available on discovered targets during [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling):
|
||||
The following meta labels are available on discovered targets during [relabeling](https://docs.victoriametrics.com/victoriametrics/vmagent/#relabeling):
|
||||
|
||||
* `__meta_dns_name`: the record name that produced the discovered target.
|
||||
* `__meta_dns_srv_record_target`: the target field of the SRV record
|
||||
@@ -445,15 +446,15 @@ scrape_configs:
|
||||
# values: ["...", "..."]
|
||||
|
||||
# Additional HTTP API client options can be specified here.
|
||||
# See https://docs.victoriametrics.com/sd_configs/#http-api-client-options
|
||||
# See https://docs.victoriametrics.com/victoriametrics/sd_configs/#http-api-client-options
|
||||
```
|
||||
|
||||
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/relabeling/#how-to-modify-scrape-urls-in-targets) label set
|
||||
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/victoriametrics/relabeling/#how-to-modify-scrape-urls-in-targets) label set
|
||||
to `<ip_address>:<port>`, where `<ip_address>` is the exposed ip address of the docker container, while the `<port>` is either the exposed port
|
||||
of the docker container or the port specified in the `docker_sd_configs` if the docker container has no exposed ports.
|
||||
If a container exposes multiple ip addresses, then multiple targets will be discovered - one per each exposed ip address.
|
||||
|
||||
The following meta labels are available on discovered targets during [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling):
|
||||
The following meta labels are available on discovered targets during [relabeling](https://docs.victoriametrics.com/victoriametrics/vmagent/#relabeling):
|
||||
|
||||
* `__meta_docker_container_id`: the id of the container
|
||||
* `__meta_docker_container_name`: the name of the container
|
||||
@@ -508,7 +509,7 @@ scrape_configs:
|
||||
# values: ["...", "..."]
|
||||
|
||||
# Additional HTTP API client options can be specified here.
|
||||
# See https://docs.victoriametrics.com/sd_configs/#http-api-client-options
|
||||
# See https://docs.victoriametrics.com/victoriametrics/sd_configs/#http-api-client-options
|
||||
```
|
||||
|
||||
One of the following roles can be configured to discover targets:
|
||||
@@ -517,12 +518,12 @@ One of the following roles can be configured to discover targets:
|
||||
|
||||
The `services` role discovers all Swarm services.
|
||||
|
||||
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/relabeling/#how-to-modify-scrape-urls-in-targets) label set
|
||||
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/victoriametrics/relabeling/#how-to-modify-scrape-urls-in-targets) label set
|
||||
to `<ip>:<port>`, where `<ip>` is the endpoint's virtual IP, while the `<port>` is the published port of the service.
|
||||
If the service has multiple published ports, then multiple targets are generated - one per each port.
|
||||
If the service has no published ports, then the `<port>` is set to the `port` value obtained from `dockerswarm_sd_configs`.
|
||||
|
||||
Available meta labels for `role: services` during [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling):
|
||||
Available meta labels for `role: services` during [relabeling](https://docs.victoriametrics.com/victoriametrics/vmagent/#relabeling):
|
||||
|
||||
* `__meta_dockerswarm_service_id`: the id of the service
|
||||
* `__meta_dockerswarm_service_name`: the name of the service
|
||||
@@ -544,12 +545,12 @@ One of the following roles can be configured to discover targets:
|
||||
|
||||
The `tasks` role discovers all Swarm tasks.
|
||||
|
||||
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/relabeling/#how-to-modify-scrape-urls-in-targets) label set
|
||||
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/victoriametrics/relabeling/#how-to-modify-scrape-urls-in-targets) label set
|
||||
to `<ip>:<port>`, where the `<ip>` is the node IP, while the `<port>` is the published port of the task.
|
||||
If the task has multiple published ports, then multiple targets are generated - one per each port.
|
||||
If the task has no published ports, then the `<port>` is set to the `port` value obtained from `dockerswarm_sd_configs`.
|
||||
|
||||
Available meta labels for `role: tasks` during [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling):
|
||||
Available meta labels for `role: tasks` during [relabeling](https://docs.victoriametrics.com/victoriametrics/vmagent/#relabeling):
|
||||
|
||||
* `__meta_dockerswarm_container_label_<labelname>`: each label of the container
|
||||
* `__meta_dockerswarm_task_id`: the id of the task
|
||||
@@ -585,10 +586,10 @@ One of the following roles can be configured to discover targets:
|
||||
|
||||
The `nodes` role is used to discover Swarm nodes.
|
||||
|
||||
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/relabeling/#how-to-modify-scrape-urls-in-targets) label set
|
||||
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/victoriametrics/relabeling/#how-to-modify-scrape-urls-in-targets) label set
|
||||
to `<ip>:<port>`, where `<ip>` is the node IP, while the `<port>` is the `port` value obtained from the `dockerswarm_sd_configs`.
|
||||
|
||||
Available meta labels for `role: nodes` during [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling):
|
||||
Available meta labels for `role: nodes` during [relabeling](https://docs.victoriametrics.com/victoriametrics/vmagent/#relabeling):
|
||||
|
||||
* `__meta_dockerswarm_node_address`: the address of the node
|
||||
* `__meta_dockerswarm_node_availability`: the availability of the node
|
||||
@@ -670,11 +671,11 @@ scrape_configs:
|
||||
# values: ["...", "..."]
|
||||
```
|
||||
|
||||
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/relabeling/#how-to-modify-scrape-urls-in-targets) label set
|
||||
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/victoriametrics/relabeling/#how-to-modify-scrape-urls-in-targets) label set
|
||||
to `<instance_ip>:<port>`, where `<instance_ip>` is the private IP of the instance, while the `<port>` is set to the `port` value
|
||||
obtain from `ec2_sd_configs`.
|
||||
|
||||
The following meta labels are available on discovered targets during [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling):
|
||||
The following meta labels are available on discovered targets during [relabeling](https://docs.victoriametrics.com/victoriametrics/vmagent/#relabeling):
|
||||
|
||||
* `__meta_ec2_ami`: the EC2 Amazon Machine Image
|
||||
* `__meta_ec2_architecture`: the architecture of the instance
|
||||
@@ -716,14 +717,14 @@ scrape_configs:
|
||||
- server: "..."
|
||||
|
||||
# Additional HTTP API client options can be specified here.
|
||||
# See https://docs.victoriametrics.com/sd_configs/#http-api-client-options
|
||||
# See https://docs.victoriametrics.com/victoriametrics/sd_configs/#http-api-client-options
|
||||
```
|
||||
|
||||
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/relabeling/#how-to-modify-scrape-urls-in-targets) label set
|
||||
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/victoriametrics/relabeling/#how-to-modify-scrape-urls-in-targets) label set
|
||||
to `<instance_host>:<instance_port>`, where `<instance_host>` is the discovered instance hostname, while the `<instance_port>`
|
||||
is the discovered instance port. If the instance has no port, then port 80 is used.
|
||||
|
||||
The following meta labels are available on discovered targets during [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling):
|
||||
The following meta labels are available on discovered targets during [relabeling](https://docs.victoriametrics.com/victoriametrics/vmagent/#relabeling):
|
||||
|
||||
* `__meta_eureka_app_name`: the name of the app
|
||||
* `__meta_eureka_app_instance_id`: the ID of the app instance
|
||||
@@ -768,7 +769,7 @@ scrape_configs:
|
||||
- "http://central-config-server/targets?type=foobar"
|
||||
```
|
||||
|
||||
See [these examples](https://docs.victoriametrics.com/scrape_config_examples/#file-based-target-discovery) on how to configure file-based target discovery.
|
||||
See [these examples](https://docs.victoriametrics.com/victoriametrics/scrape_config_examples/#file-based-target-discovery) on how to configure file-based target discovery.
|
||||
|
||||
The referred files and urls must contain a list of static configs in one of the following formats:
|
||||
|
||||
@@ -797,10 +798,10 @@ The referred files and urls must contain a list of static configs in one of the
|
||||
...
|
||||
```
|
||||
|
||||
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/relabeling/#how-to-modify-scrape-urls-in-targets) label set
|
||||
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/victoriametrics/relabeling/#how-to-modify-scrape-urls-in-targets) label set
|
||||
to one of the `target` value specified in the target files.
|
||||
|
||||
The following meta labels are available on discovered targets during [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling):
|
||||
The following meta labels are available on discovered targets during [relabeling](https://docs.victoriametrics.com/victoriametrics/vmagent/#relabeling):
|
||||
|
||||
* `__meta_filepath`: the filepath from which the target was extracted
|
||||
|
||||
@@ -853,11 +854,11 @@ Credentials are discovered by looking in the following places, preferring the fi
|
||||
1. a JSON file in the well-known path `$HOME/.config/gcloud/application_default_credentials.json`
|
||||
1. fetched from the GCE metadata server
|
||||
|
||||
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/relabeling/#how-to-modify-scrape-urls-in-targets) label set
|
||||
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/victoriametrics/relabeling/#how-to-modify-scrape-urls-in-targets) label set
|
||||
to `<iface_ip>:<port>`, where `<iface_ip>` is private IP of the discovered instance, while `<port>` is the `port` value
|
||||
specified in the `gce_sd_configs`.
|
||||
|
||||
The following meta labels are available on discovered targets during [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling):
|
||||
The following meta labels are available on discovered targets during [relabeling](https://docs.victoriametrics.com/victoriametrics/vmagent/#relabeling):
|
||||
|
||||
* `__meta_gce_instance_id`: the numeric id of the instance
|
||||
* `__meta_gce_instance_name`: the name of the instance
|
||||
@@ -912,13 +913,13 @@ scrape_configs:
|
||||
# port: ...
|
||||
|
||||
# Additional HTTP API client options can be specified here.
|
||||
# See https://docs.victoriametrics.com/sd_configs/#http-api-client-options
|
||||
# See https://docs.victoriametrics.com/victoriametrics/sd_configs/#http-api-client-options
|
||||
```
|
||||
|
||||
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/relabeling/#how-to-modify-scrape-urls-in-targets) label set
|
||||
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/victoriametrics/relabeling/#how-to-modify-scrape-urls-in-targets) label set
|
||||
to `<FQDN>:<port>`, where FQDN is discovered instance address and `<port>` is the port from the `hetzner_sd_configs` (default port is `80`).
|
||||
|
||||
The following meta labels are available on discovered targets during [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling):
|
||||
The following meta labels are available on discovered targets during [relabeling](https://docs.victoriametrics.com/victoriametrics/vmagent/#relabeling):
|
||||
|
||||
Common labels for both `hcloud` and `robot` roles:
|
||||
|
||||
@@ -970,10 +971,10 @@ scrape_configs:
|
||||
- url: "http://..."
|
||||
|
||||
# Additional HTTP API client options can be specified here.
|
||||
# See https://docs.victoriametrics.com/sd_configs/#http-api-client-options
|
||||
# See https://docs.victoriametrics.com/victoriametrics/sd_configs/#http-api-client-options
|
||||
```
|
||||
|
||||
See [these examples](https://docs.victoriametrics.com/scrape_config_examples/#http-based-target-discovery) on how to configure http-based target discovery.
|
||||
See [these examples](https://docs.victoriametrics.com/victoriametrics/scrape_config_examples/#http-based-target-discovery) on how to configure http-based target discovery.
|
||||
|
||||
The service at `url` must return JSON response in the following format:
|
||||
|
||||
@@ -993,10 +994,10 @@ The service at `url` must return JSON response in the following format:
|
||||
The `url` is queried periodically with the interval specified in `-promscrape.httpSDCheckInterval` command-line flag.
|
||||
Discovery errors are tracked in `promscrape_discovery_http_errors_total` metric.
|
||||
|
||||
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/relabeling/#how-to-modify-scrape-urls-in-targets) label set
|
||||
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/victoriametrics/relabeling/#how-to-modify-scrape-urls-in-targets) label set
|
||||
to one of the targets returned by the http service.
|
||||
|
||||
The following meta labels are available on discovered targets during [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling):
|
||||
The following meta labels are available on discovered targets during [relabeling](https://docs.victoriametrics.com/victoriametrics/vmagent/#relabeling):
|
||||
|
||||
* `__meta_url`: the URL from which the target was extracted
|
||||
|
||||
@@ -1061,10 +1062,10 @@ scrape_configs:
|
||||
# node: <boolean>
|
||||
|
||||
# Additional HTTP API client options can be specified here.
|
||||
# See https://docs.victoriametrics.com/sd_configs/#http-api-client-options
|
||||
# See https://docs.victoriametrics.com/victoriametrics/sd_configs/#http-api-client-options
|
||||
```
|
||||
|
||||
See [these examples](https://docs.victoriametrics.com/scrape_config_examples/#kubernetes-target-discovery) on how to discover and scrape Kubernetes targets.
|
||||
See [these examples](https://docs.victoriametrics.com/victoriametrics/scrape_config_examples/#kubernetes-target-discovery) on how to discover and scrape Kubernetes targets.
|
||||
|
||||
One of the following `role` types can be configured to discover targets:
|
||||
|
||||
@@ -1072,12 +1073,12 @@ One of the following `role` types can be configured to discover targets:
|
||||
|
||||
The `role: node` discovers one target per cluster node.
|
||||
|
||||
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/relabeling/#how-to-modify-scrape-urls-in-targets) label set
|
||||
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/victoriametrics/relabeling/#how-to-modify-scrape-urls-in-targets) label set
|
||||
to `<ip>:<port>`, where `<ip>` is to the first existing address of the Kubernetes node object in the address type order
|
||||
of `NodeInternalIP`, `NodeExternalIP`, `NodeLegacyHostIP` and `NodeHostName`,
|
||||
while `<port>` is the kubelet port on the given node.
|
||||
|
||||
Available meta labels for `role: node` during [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling):
|
||||
Available meta labels for `role: node` during [relabeling](https://docs.victoriametrics.com/victoriametrics/vmagent/#relabeling):
|
||||
|
||||
* `__meta_kubernetes_node_name`: The name of the node object.
|
||||
* `__meta_kubernetes_node_provider_id`: The cloud provider's name for the node object.
|
||||
@@ -1093,7 +1094,7 @@ One of the following `role` types can be configured to discover targets:
|
||||
|
||||
The `role: service` discovers Kubernetes services.
|
||||
|
||||
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/relabeling/#how-to-modify-scrape-urls-in-targets) label set
|
||||
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/victoriametrics/relabeling/#how-to-modify-scrape-urls-in-targets) label set
|
||||
to `<service_name>.<namespace>:<port>`, where `<service_name>` is the service name, `<namespace>` is the service namespace
|
||||
and `<port>` is the service port.
|
||||
If the service has multiple ports, then multiple targets are discovered for the service - one per each port.
|
||||
@@ -1101,7 +1102,7 @@ One of the following `role` types can be configured to discover targets:
|
||||
This is generally useful for blackbox monitoring of a service. The target address will be set to the Kubernetes DNS name
|
||||
of the service and respective service port.
|
||||
|
||||
Available meta labels for `role: service` during [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling):
|
||||
Available meta labels for `role: service` during [relabeling](https://docs.victoriametrics.com/victoriametrics/vmagent/#relabeling):
|
||||
|
||||
* `__meta_kubernetes_namespace`: The namespace of the service object.
|
||||
* `__meta_kubernetes_service_annotation_<annotationname>`: Each annotation from the service object.
|
||||
@@ -1120,12 +1121,12 @@ One of the following `role` types can be configured to discover targets:
|
||||
|
||||
The `role: pod` discovers all pods and exposes their containers as targets.
|
||||
|
||||
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/relabeling/#how-to-modify-scrape-urls-in-targets) label set
|
||||
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/victoriametrics/relabeling/#how-to-modify-scrape-urls-in-targets) label set
|
||||
to `<ip>:<port>`, where `<ip>` is pod IP, while `<port>` is the exposed container port.
|
||||
If the pod has multiple container ports, then multiple targets are generated for the pod - one per each exposed container port.
|
||||
If the pod has no exposed container ports, then the `__address__` for pod target is set to the pod IP.
|
||||
|
||||
Available meta labels for `role: pod` during [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling):
|
||||
Available meta labels for `role: pod` during [relabeling](https://docs.victoriametrics.com/victoriametrics/vmagent/#relabeling):
|
||||
|
||||
* `__meta_kubernetes_namespace`: The namespace of the pod object.
|
||||
* `__meta_kubernetes_pod_name`: The name of the pod object.
|
||||
@@ -1154,12 +1155,12 @@ One of the following `role` types can be configured to discover targets:
|
||||
|
||||
The `role: endpoints` discovers targets from listed endpoints of a service.
|
||||
|
||||
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/relabeling/#how-to-modify-scrape-urls-in-targets) label set
|
||||
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/victoriametrics/relabeling/#how-to-modify-scrape-urls-in-targets) label set
|
||||
to `<addr>:<port>`, where `<addr>` is the endpoint address, while `<port>` is the endpoint port.
|
||||
If the endpoint has multiple ports, then a single target per each port is generated.
|
||||
If the endpoint is backed by a pod, all additional container ports of the pod, not bound to an endpoint port, are discovered as targets as well.
|
||||
|
||||
Available meta labels for `role: endpoints` during [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling):
|
||||
Available meta labels for `role: endpoints` during [relabeling](https://docs.victoriametrics.com/victoriametrics/vmagent/#relabeling):
|
||||
|
||||
* `__meta_kubernetes_namespace`: The namespace of the endpoints object.
|
||||
* `__meta_kubernetes_endpoints_name`: The names of the endpoints object.
|
||||
@@ -1183,12 +1184,12 @@ One of the following `role` types can be configured to discover targets:
|
||||
|
||||
The `role: endpointslice` discovers targets from existing endpointslices.
|
||||
|
||||
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/relabeling/#how-to-modify-scrape-urls-in-targets) label set
|
||||
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/victoriametrics/relabeling/#how-to-modify-scrape-urls-in-targets) label set
|
||||
to `<addr>:<port>`, where `<addr>` is the endpoint address, while `<port>` is the endpoint port.
|
||||
If the endpoint has multiple ports, then a single target per each port is generated.
|
||||
If the endpoint is backed by a pod, all additional container ports of the pod, not bound to an endpoint port, are discovered as targets as well.
|
||||
|
||||
Available meta labels for `role: endpointslice` during [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling):
|
||||
Available meta labels for `role: endpointslice` during [relabeling](https://docs.victoriametrics.com/victoriametrics/vmagent/#relabeling):
|
||||
|
||||
* `__meta_kubernetes_namespace`: The namespace of the endpointslice object.
|
||||
* `__meta_kubernetes_endpointslice_name`: The name of endpointslice object.
|
||||
@@ -1212,13 +1213,13 @@ One of the following `role` types can be configured to discover targets:
|
||||
|
||||
The `role: ingress` discovers a target for each path of each ingress.
|
||||
|
||||
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/relabeling/#how-to-modify-scrape-urls-in-targets) label set
|
||||
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/victoriametrics/relabeling/#how-to-modify-scrape-urls-in-targets) label set
|
||||
to the host obtained from ingress spec.
|
||||
If the ingress has multiple specs with multiple hosts, then a target per each host is created.
|
||||
|
||||
This is generally useful for blackbox monitoring of an ingress.
|
||||
|
||||
Available meta labels for `role: ingress` during [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling):
|
||||
Available meta labels for `role: ingress` during [relabeling](https://docs.victoriametrics.com/victoriametrics/vmagent/#relabeling):
|
||||
|
||||
* `__meta_kubernetes_namespace`: The namespace of the ingress object.
|
||||
* `__meta_kubernetes_ingress_name`: The name of the ingress object.
|
||||
@@ -1258,16 +1259,16 @@ scrape_configs:
|
||||
# client_id: "..."
|
||||
|
||||
# Additional HTTP API client options can be specified here.
|
||||
# See https://docs.victoriametrics.com/sd_configs/#http-api-client-options
|
||||
# See https://docs.victoriametrics.com/victoriametrics/sd_configs/#http-api-client-options
|
||||
```
|
||||
|
||||
The `server` is queried periodically with the interval specified in `-promscrape.kumaSDCheckInterval` command-line flag.
|
||||
Discovery errors are tracked in `promscrape_discovery_kuma_errors_total` metric.
|
||||
|
||||
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/relabeling/#how-to-modify-scrape-urls-in-targets) label set
|
||||
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/victoriametrics/relabeling/#how-to-modify-scrape-urls-in-targets) label set
|
||||
to one of the targets returned by the http service.
|
||||
|
||||
The following meta labels are available on discovered targets during [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling):
|
||||
The following meta labels are available on discovered targets during [relabeling](https://docs.victoriametrics.com/victoriametrics/vmagent/#relabeling):
|
||||
|
||||
* `__meta_kuma_mesh`: the name of the mesh
|
||||
* `__meta_kuma_dataplane`: the name of the proxy
|
||||
@@ -1296,10 +1297,10 @@ scrape_configs:
|
||||
- "..."
|
||||
|
||||
# Additional HTTP API client options can be specified here.
|
||||
# See https://docs.victoriametrics.com/sd_configs/#http-api-client-options
|
||||
# See https://docs.victoriametrics.com/victoriametrics/sd_configs/#http-api-client-options
|
||||
```
|
||||
|
||||
The following meta labels are available on discovered targets during [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling):
|
||||
The following meta labels are available on discovered targets during [relabeling](https://docs.victoriametrics.com/victoriametrics/vmagent/#relabeling):
|
||||
|
||||
* `__meta_marathon_app`: the name of the app (with slashes replaced by dashes)
|
||||
* `__meta_marathon_image`: the name of the Docker image used (if available)
|
||||
@@ -1352,13 +1353,13 @@ scrape_configs:
|
||||
# allow_stale: ...
|
||||
|
||||
# Additional HTTP API client options can be specified here.
|
||||
# See https://docs.victoriametrics.com/sd_configs/#http-api-client-options
|
||||
# See https://docs.victoriametrics.com/victoriametrics/sd_configs/#http-api-client-options
|
||||
```
|
||||
|
||||
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/relabeling/#how-to-modify-scrape-urls-in-targets) label set
|
||||
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/victoriametrics/relabeling/#how-to-modify-scrape-urls-in-targets) label set
|
||||
to `<addr>:<port>`, where `<addr>` is the service address, while `<port>` is the service port.
|
||||
|
||||
The following meta labels are available on discovered targets during [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling):
|
||||
The following meta labels are available on discovered targets during [relabeling](https://docs.victoriametrics.com/victoriametrics/vmagent/#relabeling):
|
||||
|
||||
* `__meta_nomad_address`: the address of the target
|
||||
* `__meta_nomad_dc`: the datacenter name for the target
|
||||
@@ -1471,10 +1472,10 @@ One of the following `role` types can be configured to discover targets:
|
||||
|
||||
The `role: hypervisor` discovers one target per Nova hypervisor node.
|
||||
|
||||
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/relabeling/#how-to-modify-scrape-urls-in-targets) label set
|
||||
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/victoriametrics/relabeling/#how-to-modify-scrape-urls-in-targets) label set
|
||||
to `<host>:<port>`, where `<host>` is the discovered node IP, while `<port>` is the port specified in the `openstack_sd_configs`.
|
||||
|
||||
The following meta labels are available on discovered targets during [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling):
|
||||
The following meta labels are available on discovered targets during [relabeling](https://docs.victoriametrics.com/victoriametrics/vmagent/#relabeling):
|
||||
|
||||
* `__meta_openstack_hypervisor_host_ip`: the hypervisor node's IP address.
|
||||
* `__meta_openstack_hypervisor_hostname`: the hypervisor node's name.
|
||||
@@ -1487,10 +1488,10 @@ One of the following `role` types can be configured to discover targets:
|
||||
|
||||
The `role: instance` discovers one target per network interface of Nova instance.
|
||||
|
||||
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/relabeling/#how-to-modify-scrape-urls-in-targets) label set
|
||||
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/victoriametrics/relabeling/#how-to-modify-scrape-urls-in-targets) label set
|
||||
to `<host>:<port>`, where `<host>` is the private IP address of the discovered instance, while `<port>` is the port specified in the `openstack_sd_configs`.
|
||||
|
||||
The following meta labels are available on discovered targets during [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling):
|
||||
The following meta labels are available on discovered targets during [relabeling](https://docs.victoriametrics.com/victoriametrics/vmagent/#relabeling):
|
||||
|
||||
* `__meta_openstack_address_pool`: the pool of the private IP.
|
||||
* `__meta_openstack_instance_flavor`: the flavor of the OpenStack instance.
|
||||
@@ -1542,14 +1543,14 @@ scrape_configs:
|
||||
service: "..."
|
||||
|
||||
# Additional HTTP API client options can be specified here.
|
||||
# See https://docs.victoriametrics.com/sd_configs.html#http-api-client-options
|
||||
# See https://docs.victoriametrics.com/victoriametrics/sd_configs.html#http-api-client-options
|
||||
```
|
||||
|
||||
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/relabeling/#how-to-modify-scrape-urls-in-targets) label set to either `<ipv4>` address or `<ipv6>` address.
|
||||
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/victoriametrics/relabeling/#how-to-modify-scrape-urls-in-targets) label set to either `<ipv4>` address or `<ipv6>` address.
|
||||
|
||||
In addition, the `instance` label for the VPS/dedicated server will be set to the VPS/dedicated server name as retrieved from OVH Cloud API.
|
||||
|
||||
The following meta labels are available on discovered targets during [relabeling](https://docs.victoriametrics.com/vmagent.html#relabeling).
|
||||
The following meta labels are available on discovered targets during [relabeling](https://docs.victoriametrics.com/victoriametrics/vmagent.html#relabeling).
|
||||
|
||||
VPS:
|
||||
* `__meta_ovhcloud_vps_cluster`: the cluster of the server.
|
||||
@@ -1623,7 +1624,7 @@ scrape_configs:
|
||||
# port: <int> | default = 80
|
||||
|
||||
# Additional HTTP API client options can be specified here.
|
||||
# See https://docs.victoriametrics.com/sd_configs.html#http-api-client-options
|
||||
# See https://docs.victoriametrics.com/victoriametrics/sd_configs.html#http-api-client-options
|
||||
```
|
||||
|
||||
The resource address is the `certname` of the resource and can be changed during relabeling.
|
||||
@@ -1657,10 +1658,10 @@ scrape_configs:
|
||||
# The `http://host:port/metrics` endpoint is scraped per each configured target then.
|
||||
# The `http` scheme can be changed to `https` by setting it via `scheme` field at `scrape_config` level.
|
||||
# The `/metrics` path can be changed to arbitrary path via `metrics_path` field at `scrape_config` level.
|
||||
# See https://docs.victoriametrics.com/sd_configs/#scrape_configs .
|
||||
# See https://docs.victoriametrics.com/victoriametrics/sd_configs/#scrape_configs .
|
||||
#
|
||||
# Alternatively the scheme and path can be changed via `relabel_configs` section at `scrape_config` level.
|
||||
# See https://docs.victoriametrics.com/vmagent/#relabeling .
|
||||
# See https://docs.victoriametrics.com/victoriametrics/vmagent/#relabeling .
|
||||
#
|
||||
# It is also possible specifying full target urls here, e.g. "http://host:port/metrics/path?query_args"
|
||||
#
|
||||
@@ -1676,7 +1677,7 @@ scrape_configs:
|
||||
# <labelnameN>: "<labelvalueN>"
|
||||
```
|
||||
|
||||
See [these examples](https://docs.victoriametrics.com/scrape_config_examples/#static-configs) on how to configure scraping for static targets.
|
||||
See [these examples](https://docs.victoriametrics.com/victoriametrics/scrape_config_examples/#static-configs) on how to configure scraping for static targets.
|
||||
|
||||
## vultr_sd_configs
|
||||
Vultr SD configuration discovers scrape targets from [Vultr](https://www.vultr.com/) Instances.
|
||||
@@ -1721,15 +1722,15 @@ scrape_configs:
|
||||
# port: ...
|
||||
|
||||
# Additional HTTP API client options can be specified here.
|
||||
# See https://docs.victoriametrics.com/sd_configs.html#http-api-client-options
|
||||
# See https://docs.victoriametrics.com/victoriametrics/sd_configs.html#http-api-client-options
|
||||
|
||||
|
||||
```
|
||||
|
||||
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/relabeling/#how-to-modify-scrape-urls-in-targets) label set
|
||||
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/victoriametrics/relabeling/#how-to-modify-scrape-urls-in-targets) label set
|
||||
to `<FQDN>:<port>`, where FQDN is discovered instance address and `<port>` is the port from the `vultr_sd_configs` (default port is `80`).
|
||||
|
||||
The following meta labels are available on discovered targets during [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling):
|
||||
The following meta labels are available on discovered targets during [relabeling](https://docs.victoriametrics.com/victoriametrics/vmagent/#relabeling):
|
||||
|
||||
* `__meta_vultr_instance_allowed_bandwidth_gb`: monthly bandwidth quota in GB.
|
||||
* `__meta_vultr_instance_disk_gb`: the size of the disk in GB.
|
||||
@@ -1807,10 +1808,10 @@ scrape_configs:
|
||||
replacement: "$1:9100"
|
||||
```
|
||||
|
||||
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/relabeling/#how-to-modify-scrape-urls-in-targets) label set
|
||||
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/victoriametrics/relabeling/#how-to-modify-scrape-urls-in-targets) label set
|
||||
to the FQDN of the discovered instance.
|
||||
|
||||
The following meta labels are available on discovered targets during [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling):
|
||||
The following meta labels are available on discovered targets during [relabeling](https://docs.victoriametrics.com/victoriametrics/vmagent/#relabeling):
|
||||
|
||||
* `__meta_yandexcloud_instance_name`: the name of instance
|
||||
* `__meta_yandexcloud_instance_id`: the id of instance
|
||||
@@ -1852,7 +1853,7 @@ scrape_configs:
|
||||
# - "2m" - 2 minutes
|
||||
# The scrape_interval can be set on a per-target basis by specifying `__scrape_interval__`
|
||||
# label during target relabeling phase.
|
||||
# See https://docs.victoriametrics.com/vmagent/#relabeling
|
||||
# See https://docs.victoriametrics.com/victoriametrics/vmagent/#relabeling
|
||||
#
|
||||
# scrape_interval: <duration>
|
||||
|
||||
@@ -1867,7 +1868,7 @@ scrape_configs:
|
||||
# The `scrape_timeout` cannot exceed the `scrape_interval`.
|
||||
# The scrape_timeout can be set on a per-target basis by specifying `__scrape_timeout__`
|
||||
# label during target relabeling phase.
|
||||
# See https://docs.victoriametrics.com/vmagent/#relabeling
|
||||
# See https://docs.victoriametrics.com/victoriametrics/vmagent/#relabeling
|
||||
#
|
||||
# scrape_timeout: <duration>
|
||||
|
||||
@@ -1933,14 +1934,14 @@ scrape_configs:
|
||||
|
||||
# relabel_configs is an optional relabeling configurations
|
||||
# for the specified and discovered scrape targets.
|
||||
# See https://docs.victoriametrics.com/vmagent/#relabeling
|
||||
# See https://docs.victoriametrics.com/victoriametrics/vmagent/#relabeling
|
||||
#
|
||||
# relabel_configs:
|
||||
# - <relabel_config> ...
|
||||
|
||||
# metric_relabel_configs is an optional relabeling configs
|
||||
# for the collected metrics from active scrape targets.
|
||||
# See https://docs.victoriametrics.com/vmagent/#relabeling
|
||||
# See https://docs.victoriametrics.com/victoriametrics/vmagent/#relabeling
|
||||
#
|
||||
# metric_relabel_configs:
|
||||
# - <relabel_config> ...
|
||||
@@ -1952,30 +1953,30 @@ scrape_configs:
|
||||
# By default, the limit is disabled.
|
||||
# The sample_limit can be set on a per-target basis by specifying `__sample_limit__`
|
||||
# label during target relabeling phase. Available starting from v1.103.0.
|
||||
# See https://docs.victoriametrics.com/vmagent/#relabeling
|
||||
# See https://docs.victoriametrics.com/victoriametrics/vmagent/#relabeling
|
||||
#
|
||||
# sample_limit: <int>
|
||||
|
||||
# disable_compression allows disabling HTTP compression for responses received from scrape targets.
|
||||
# By default, scrape targets are queried with `Accept-Encoding: gzip` http request header,
|
||||
# so targets could send compressed responses in order to save network bandwidth.
|
||||
# See https://docs.victoriametrics.com/vmagent/#scrape_config-enhancements
|
||||
# See https://docs.victoriametrics.com/victoriametrics/vmagent/#scrape_config-enhancements
|
||||
#
|
||||
# disable_compression: <boolean>
|
||||
|
||||
# disable_keepalive allows disabling HTTP keep-alive when scraping targets.
|
||||
# By default, HTTP keep-alive is enabled, so TCP connections to scrape targets
|
||||
# could be reused.
|
||||
# See https://docs.victoriametrics.com/vmagent/#scrape_config-enhancements
|
||||
# See https://docs.victoriametrics.com/victoriametrics/vmagent/#scrape_config-enhancements
|
||||
#
|
||||
# disable_keepalive: <boolean>
|
||||
|
||||
# stream_parse allows enabling stream parsing mode when scraping targets.
|
||||
# By default, stream parsing mode is disabled for targets which return up to a few thousands samples.
|
||||
# See https://docs.victoriametrics.com/vmagent/#stream-parsing-mode .
|
||||
# See https://docs.victoriametrics.com/victoriametrics/vmagent/#stream-parsing-mode .
|
||||
# The stream_parse can be set on a per-target basis by specifying `__stream_parse__`
|
||||
# label during target relabeling phase.
|
||||
# See https://docs.victoriametrics.com/vmagent/#relabeling
|
||||
# See https://docs.victoriametrics.com/victoriametrics/vmagent/#relabeling
|
||||
#
|
||||
# stream_parse: <boolean>
|
||||
|
||||
@@ -1983,7 +1984,7 @@ scrape_configs:
|
||||
# Example values:
|
||||
# - "5m" - align scrapes to every 5 minutes.
|
||||
# - "1h" - align scrapes to every hour.
|
||||
# See https://docs.victoriametrics.com/vmagent/#scrape_config-enhancements
|
||||
# See https://docs.victoriametrics.com/victoriametrics/vmagent/#scrape_config-enhancements
|
||||
#
|
||||
# scrape_align_interval: <duration>
|
||||
|
||||
@@ -1991,28 +1992,28 @@ scrape_configs:
|
||||
# Example values:
|
||||
# - "5m" - align scrapes to every 5 minutes.
|
||||
# - "1h" - align scrapes to every hour.
|
||||
# See https://docs.victoriametrics.com/vmagent/#scrape_config-enhancements
|
||||
# See https://docs.victoriametrics.com/victoriametrics/vmagent/#scrape_config-enhancements
|
||||
#
|
||||
# scrape_offset: <duration>
|
||||
|
||||
# series_limit is an optional limit on the number of unique time series
|
||||
# a single target can expose during all the scrapes on the time window of 24h.
|
||||
# By default, there is no limit on the number of exposed series.
|
||||
# See https://docs.victoriametrics.com/vmagent/#cardinality-limiter .
|
||||
# See https://docs.victoriametrics.com/victoriametrics/vmagent/#cardinality-limiter .
|
||||
# The series_limit can be set on a per-target basis by specifying `__series_limit__`
|
||||
# label during target relabeling phase.
|
||||
# See https://docs.victoriametrics.com/vmagent/#relabeling
|
||||
# See https://docs.victoriametrics.com/victoriametrics/vmagent/#relabeling
|
||||
#
|
||||
# series_limit: ...
|
||||
|
||||
# no_stale_markers allows disabling staleness tracking.
|
||||
# By default, staleness tracking is enabled for all the discovered scrape targets.
|
||||
# See https://docs.victoriametrics.com/vmagent/#prometheus-staleness-markers
|
||||
# See https://docs.victoriametrics.com/victoriametrics/vmagent/#prometheus-staleness-markers
|
||||
#
|
||||
# no_stale_markers: <boolean>
|
||||
|
||||
# Additional HTTP client options for target scraping can be specified here.
|
||||
# See https://docs.victoriametrics.com/sd_configs/#http-api-client-options
|
||||
# See https://docs.victoriametrics.com/victoriametrics/sd_configs/#http-api-client-options
|
||||
```
|
||||
|
||||
## HTTP API client options
|
||||
|
||||
@@ -6,18 +6,19 @@ menu:
|
||||
parent: 'victoriametrics'
|
||||
weight: 39
|
||||
aliases:
|
||||
- /stream-aggregation/
|
||||
- /stream-aggregation.html
|
||||
- /stream-aggregation/index.html
|
||||
- /stream-aggregation/
|
||||
---
|
||||
[vmagent](https://docs.victoriametrics.com/vmagent/) and [single-node VictoriaMetrics](https://docs.victoriametrics.com/single-server-victoriametrics/)
|
||||
can aggregate incoming [samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) in streaming mode by time and by labels before data is written to remote storage
|
||||
[vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/) and [single-node VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/)
|
||||
can aggregate incoming [samples](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#raw-samples) in streaming mode by time and by labels before data is written to remote storage
|
||||
(or local storage for single-node VictoriaMetrics).
|
||||
The aggregation is applied to all the metrics received via any [supported data ingestion protocol](https://docs.victoriametrics.com/#how-to-import-time-series-data)
|
||||
and/or scraped from [Prometheus-compatible targets](https://docs.victoriametrics.com/#how-to-scrape-prometheus-exporters-such-as-node-exporter)
|
||||
after applying all the configured [relabeling stages](https://docs.victoriametrics.com/vmagent/#relabeling).
|
||||
The aggregation is applied to all the metrics received via any [supported data ingestion protocol](https://docs.victoriametrics.com/victoriametrics#how-to-import-time-series-data)
|
||||
and/or scraped from [Prometheus-compatible targets](https://docs.victoriametrics.com/victoriametrics#how-to-scrape-prometheus-exporters-such-as-node-exporter)
|
||||
after applying all the configured [relabeling stages](https://docs.victoriametrics.com/victoriametrics/vmagent/#relabeling).
|
||||
|
||||
**By default, stream aggregation ignores timestamps associated with the input [samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples).
|
||||
**By default, stream aggregation ignores timestamps associated with the input [samples](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#raw-samples).
|
||||
It expects that the ingested samples have timestamps close to the current time. See [how to ignore old samples](#ignoring-old-samples).**
|
||||
|
||||
# Use cases
|
||||
@@ -39,17 +40,17 @@ Stream aggregation can be used as [statsd](https://github.com/statsd/statsd) alt
|
||||
* [Histograms over input metrics](#histograms-over-input-metrics)
|
||||
* [Aggregating histograms](#aggregating-histograms)
|
||||
|
||||
Currently, streaming aggregation is available only for [supported data ingestion protocols](https://docs.victoriametrics.com/#how-to-import-time-series-data)
|
||||
Currently, streaming aggregation is available only for [supported data ingestion protocols](https://docs.victoriametrics.com/victoriametrics#how-to-import-time-series-data)
|
||||
and not available for [Statsd metrics format](https://github.com/statsd/statsd/blob/master/docs/metric_types.md).
|
||||
|
||||
## Recording rules alternative
|
||||
|
||||
Sometimes [alerting queries](https://docs.victoriametrics.com/vmalert/#alerting-rules) may require non-trivial amounts of CPU, RAM,
|
||||
Sometimes [alerting queries](https://docs.victoriametrics.com/victoriametrics/vmalert/#alerting-rules) may require non-trivial amounts of CPU, RAM,
|
||||
disk IO and network bandwidth at metrics storage side. For example, if `http_request_duration_seconds` histogram is generated by thousands
|
||||
of application instances, then the alerting query `histogram_quantile(0.99, sum(increase(http_request_duration_seconds_bucket[5m])) without (instance)) > 0.5`
|
||||
can become slow, since it needs to scan too big number of unique [time series](https://docs.victoriametrics.com/keyconcepts/#time-series)
|
||||
can become slow, since it needs to scan too big number of unique [time series](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#time-series)
|
||||
with `http_request_duration_seconds_bucket` name. This alerting query can be accelerated by pre-calculating
|
||||
the `sum(increase(http_request_duration_seconds_bucket[5m])) without (instance)` via [recording rule](https://docs.victoriametrics.com/vmalert/#recording-rules).
|
||||
the `sum(increase(http_request_duration_seconds_bucket[5m])) without (instance)` via [recording rule](https://docs.victoriametrics.com/victoriametrics/vmalert/#recording-rules).
|
||||
But this recording rule may take too much time to execute too. In this case the slow recording rule can be substituted
|
||||
with the following [stream aggregation config](#stream-aggregation-config):
|
||||
|
||||
@@ -61,7 +62,7 @@ with the following [stream aggregation config](#stream-aggregation-config):
|
||||
```
|
||||
|
||||
This stream aggregation generates `http_request_duration_seconds_bucket:5m_without_instance_total` output series according to [output metric naming](#output-metric-names).
|
||||
Then these series can be used in [alerting rules](https://docs.victoriametrics.com/vmalert/#alerting-rules):
|
||||
Then these series can be used in [alerting rules](https://docs.victoriametrics.com/victoriametrics/vmalert/#alerting-rules):
|
||||
|
||||
```metricsql
|
||||
histogram_quantile(0.99, last_over_time(http_request_duration_seconds_bucket:5m_without_instance_total[5m])) > 0.5
|
||||
@@ -76,7 +77,7 @@ Field `interval` is recommended to be set to a value at least several times high
|
||||
|
||||
## Reducing the number of stored samples
|
||||
|
||||
If per-[series](https://docs.victoriametrics.com/keyconcepts/#time-series) samples are ingested at high frequency,
|
||||
If per-[series](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#time-series) samples are ingested at high frequency,
|
||||
then this may result in high disk space usage, since too much data must be stored to disk. This also may result
|
||||
in slow queries, since too much data must be processed during queries.
|
||||
|
||||
@@ -87,13 +88,13 @@ to one sample per 5 minutes per each input time series (this operation is also k
|
||||
|
||||
```yaml
|
||||
# Aggregate metrics ending with _total with `total` output.
|
||||
# See https://docs.victoriametrics.com/stream-aggregation/#aggregation-outputs
|
||||
# See https://docs.victoriametrics.com/victoriametrics/stream-aggregation/#aggregation-outputs
|
||||
- match: '{__name__=~".+_total"}'
|
||||
interval: 5m
|
||||
outputs: [total]
|
||||
|
||||
# Downsample other metrics with `count_samples`, `sum_samples`, `min` and `max` outputs
|
||||
# See https://docs.victoriametrics.com/stream-aggregation/#aggregation-outputs
|
||||
# See https://docs.victoriametrics.com/victoriametrics/stream-aggregation/#aggregation-outputs
|
||||
- match: '{__name__!~".+_total"}'
|
||||
interval: 5m
|
||||
outputs: [count_samples, sum_samples, min, max]
|
||||
@@ -117,7 +118,7 @@ See also [aggregating histograms](#aggregating-histograms) and [aggregating by l
|
||||
|
||||
## Reducing the number of stored series
|
||||
|
||||
Sometimes applications may generate too many [time series](https://docs.victoriametrics.com/keyconcepts/#time-series).
|
||||
Sometimes applications may generate too many [time series](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#time-series).
|
||||
For example, the `http_requests_total` metric may have `path` or `user` label with too big number of unique values.
|
||||
In this case the following stream aggregation can be used for reducing the number metrics stored in VictoriaMetrics:
|
||||
|
||||
@@ -225,7 +226,7 @@ See also [histograms over input metrics](#histograms-over-input-metrics) and [ag
|
||||
## Histograms over input metrics
|
||||
|
||||
If the monitored application generates measurement metrics per each request, then it may be useful to calculate
|
||||
a [histogram](https://docs.victoriametrics.com/keyconcepts/#histogram) over these metrics.
|
||||
a [histogram](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#histogram) over these metrics.
|
||||
|
||||
For example, if the monitored application generates `request_duration_seconds N` and `response_size_bytes M` metrics
|
||||
per each incoming request, then the following [stream aggregation config](#stream-aggregation-config)
|
||||
@@ -254,7 +255,7 @@ response_size_bytes:60s_histogram_bucket{vmrange="start2...end2"} count2
|
||||
response_size_bytes:60s_histogram_bucket{vmrange="startN...endN"} countN
|
||||
```
|
||||
|
||||
The resulting histogram buckets can be queried with [MetricsQL](https://docs.victoriametrics.com/metricsql/) in the following ways:
|
||||
The resulting histogram buckets can be queried with [MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/) in the following ways:
|
||||
|
||||
1. An estimated 50th and 99th [percentiles](https://en.wikipedia.org/wiki/Percentile) of the request duration over the last hour:
|
||||
|
||||
@@ -262,7 +263,7 @@ The resulting histogram buckets can be queried with [MetricsQL](https://docs.vic
|
||||
histogram_quantiles("quantile", 0.50, 0.99, sum(increase(request_duration_seconds:60s_histogram_bucket[1h])) by (vmrange))
|
||||
```
|
||||
|
||||
This query uses [histogram_quantiles](https://docs.victoriametrics.com/metricsql/#histogram_quantiles) function.
|
||||
This query uses [histogram_quantiles](https://docs.victoriametrics.com/victoriametrics/metricsql/#histogram_quantiles) function.
|
||||
|
||||
1. An estimated [standard deviation](https://en.wikipedia.org/wiki/Standard_deviation) of the request duration over the last hour:
|
||||
|
||||
@@ -270,7 +271,7 @@ The resulting histogram buckets can be queried with [MetricsQL](https://docs.vic
|
||||
histogram_stddev(sum(increase(request_duration_seconds:60s_histogram_bucket[1h])) by (vmrange))
|
||||
```
|
||||
|
||||
This query uses [histogram_stddev](https://docs.victoriametrics.com/metricsql/#histogram_stddev) function.
|
||||
This query uses [histogram_stddev](https://docs.victoriametrics.com/victoriametrics/metricsql/#histogram_stddev) function.
|
||||
|
||||
1. An estimated share of requests with the duration smaller than `0.5s` over the last hour:
|
||||
|
||||
@@ -278,17 +279,17 @@ The resulting histogram buckets can be queried with [MetricsQL](https://docs.vic
|
||||
histogram_share(0.5, sum(increase(request_duration_seconds:60s_histogram_bucket[1h])) by (vmrange))
|
||||
```
|
||||
|
||||
This query uses [histogram_share](https://docs.victoriametrics.com/metricsql/#histogram_share) function.
|
||||
This query uses [histogram_share](https://docs.victoriametrics.com/victoriametrics/metricsql/#histogram_share) function.
|
||||
|
||||
See [the list of aggregate output](#aggregation-outputs), which can be specified at `output` field.
|
||||
See also [quantiles over input metrics](#quantiles-over-input-metrics) and [aggregating by labels](#aggregating-by-labels).
|
||||
|
||||
## Aggregating histograms
|
||||
|
||||
[Histogram](https://docs.victoriametrics.com/keyconcepts/#histogram) is a set of [counter](https://docs.victoriametrics.com/keyconcepts/#counter)
|
||||
[Histogram](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#histogram) is a set of [counter](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#counter)
|
||||
metrics with different `vmrange` or `le` labels. Since typical usage of histograms is to calculate quantiles over the
|
||||
buckets change via [histogram_quantile](https://docs.victoriametrics.com/metricsql/#histogram_quantile) function the
|
||||
appropriate aggregation output for this is [total](https://docs.victoriametrics.com/stream-aggregation/#rate_sum):
|
||||
buckets change via [histogram_quantile](https://docs.victoriametrics.com/victoriametrics/metricsql/#histogram_quantile) function the
|
||||
appropriate aggregation output for this is [total](https://docs.victoriametrics.com/victoriametrics/stream-aggregation/#rate_sum):
|
||||
|
||||
```yaml
|
||||
- match: 'http_request_duration_seconds_bucket'
|
||||
@@ -309,7 +310,7 @@ http_request_duration_seconds_bucket:5m_without_instance_rate_sum{le="3"} val
|
||||
http_request_duration_seconds_bucket:5m_without_instance_rate_sum{le="+Inf"} value6
|
||||
```
|
||||
|
||||
The resulting metrics can be passed to [histogram_quantile](https://docs.victoriametrics.com/metricsql/#histogram_quantile)
|
||||
The resulting metrics can be passed to [histogram_quantile](https://docs.victoriametrics.com/victoriametrics/metricsql/#histogram_quantile)
|
||||
function:
|
||||
|
||||
```metricsql
|
||||
@@ -329,9 +330,9 @@ See also [histograms over input metrics](#histograms-over-input-metrics) and [qu
|
||||
|
||||
Stream aggregation can be configured via the following command-line flags:
|
||||
|
||||
- `-streamAggr.config` at [single-node VictoriaMetrics](https://docs.victoriametrics.com/single-server-victoriametrics/)
|
||||
and at [vmagent](https://docs.victoriametrics.com/vmagent/).
|
||||
- `-remoteWrite.streamAggr.config` at [vmagent](https://docs.victoriametrics.com/vmagent/) only. This flag can be specified individually
|
||||
- `-streamAggr.config` at [single-node VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/)
|
||||
and at [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/).
|
||||
- `-remoteWrite.streamAggr.config` at [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/) only. This flag can be specified individually
|
||||
per each `-remoteWrite.url`, so the aggregation happens independently per each remote storage destination.
|
||||
This allows writing different aggregates to different remote storage systems.
|
||||
|
||||
@@ -345,12 +346,12 @@ By default, the following data is written to the storage when stream aggregation
|
||||
|
||||
This behaviour can be changed via the following command-line flags:
|
||||
|
||||
- `-streamAggr.keepInput` at [single-node VictoriaMetrics](https://docs.victoriametrics.com/single-server-victoriametrics/)
|
||||
and [vmagent](https://docs.victoriametrics.com/vmagent/). At [vmagent](https://docs.victoriametrics.com/vmagent/)
|
||||
- `-streamAggr.keepInput` at [single-node VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/)
|
||||
and [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/). At [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/)
|
||||
`-remoteWrite.streamAggr.keepInput` flag can be specified individually per each `-remoteWrite.url`.
|
||||
If one of these flags is set, then all the input samples are written to the storage alongside the aggregated samples.
|
||||
- `-streamAggr.dropInput` at [single-node VictoriaMetrics](https://docs.victoriametrics.com/single-server-victoriametrics/)
|
||||
and [vmagent](https://docs.victoriametrics.com/vmagent/). At [vmagent](https://docs.victoriametrics.com/vmagent/)
|
||||
- `-streamAggr.dropInput` at [single-node VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/)
|
||||
and [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/). At [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/)
|
||||
`-remoteWrite.streamAggr.dropInput` flag can be specified individually per each `-remoteWrite.url`.
|
||||
If one of these flags are set, then all the input samples are dropped, while only the aggregated samples are written to the storage.
|
||||
|
||||
@@ -385,10 +386,10 @@ Below are aggregation functions that can be put in the `outputs` list at [stream
|
||||
|
||||
### avg
|
||||
|
||||
`avg` returns the average over input [sample values](https://docs.victoriametrics.com/keyconcepts/#raw-samples).
|
||||
`avg` makes sense only for aggregating [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
|
||||
`avg` returns the average over input [sample values](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#raw-samples).
|
||||
`avg` makes sense only for aggregating [gauges](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#gauge).
|
||||
|
||||
The results of `avg` is equal to the following [MetricsQL](https://docs.victoriametrics.com/metricsql/) query:
|
||||
The results of `avg` is equal to the following [MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/) query:
|
||||
|
||||
```metricsql
|
||||
sum(sum_over_time(some_metric[interval])) / sum(count_over_time(some_metric[interval]))
|
||||
@@ -408,9 +409,9 @@ See also:
|
||||
|
||||
### count_samples
|
||||
|
||||
`count_samples` counts the number of input [samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) over the given `interval`.
|
||||
`count_samples` counts the number of input [samples](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#raw-samples) over the given `interval`.
|
||||
|
||||
The results of `count_samples` is equal to the following [MetricsQL](https://docs.victoriametrics.com/metricsql/) query:
|
||||
The results of `count_samples` is equal to the following [MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/) query:
|
||||
|
||||
```metricsql
|
||||
sum(count_over_time(some_metric[interval]))
|
||||
@@ -423,9 +424,9 @@ See also:
|
||||
|
||||
### count_series
|
||||
|
||||
`count_series` counts the number of unique [time series](https://docs.victoriametrics.com/keyconcepts/#time-series) over the given `interval`.
|
||||
`count_series` counts the number of unique [time series](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#time-series) over the given `interval`.
|
||||
|
||||
The results of `count_series` is equal to the following [MetricsQL](https://docs.victoriametrics.com/metricsql/) query:
|
||||
The results of `count_series` is equal to the following [MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/) query:
|
||||
|
||||
```metricsql
|
||||
count(last_over_time(some_metric[interval]))
|
||||
@@ -439,11 +440,11 @@ See also:
|
||||
### histogram_bucket
|
||||
|
||||
`histogram_bucket` returns [VictoriaMetrics histogram buckets](https://valyala.medium.com/improving-histogram-usability-for-prometheus-and-grafana-bc7e5df0e350)
|
||||
for the input [sample values](https://docs.victoriametrics.com/keyconcepts/#raw-samples) over the given `interval`.
|
||||
`histogram_bucket` makes sense only for aggregating [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
|
||||
for the input [sample values](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#raw-samples) over the given `interval`.
|
||||
`histogram_bucket` makes sense only for aggregating [gauges](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#gauge).
|
||||
See how to aggregate regular histograms [here](#aggregating-histograms).
|
||||
|
||||
The results of `histogram_bucket` is equal to the following [MetricsQL](https://docs.victoriametrics.com/metricsql/) query:
|
||||
The results of `histogram_bucket` is equal to the following [MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/) query:
|
||||
|
||||
Aggregating irregular and sporadic metrics (received from [Lambdas](https://aws.amazon.com/lambda/)
|
||||
or [Cloud Functions](https://cloud.google.com/functions)) can be controlled via [staleness_interval](#staleness) option.
|
||||
@@ -461,16 +462,16 @@ See also:
|
||||
|
||||
### increase
|
||||
|
||||
`increase` returns the increase of input [time series](https://docs.victoriametrics.com/keyconcepts/#time-series) over the given 'interval'.
|
||||
`increase` makes sense only for aggregating [counters](https://docs.victoriametrics.com/keyconcepts/#counter).
|
||||
`increase` returns the increase of input [time series](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#time-series) over the given 'interval'.
|
||||
`increase` makes sense only for aggregating [counters](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#counter).
|
||||
|
||||
The results of `increase` is equal to the following [MetricsQL](https://docs.victoriametrics.com/metricsql/) query:
|
||||
The results of `increase` is equal to the following [MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/) query:
|
||||
|
||||
```metricsql
|
||||
sum(increase_pure(some_counter[interval]))
|
||||
```
|
||||
|
||||
`increase` assumes that all the counters start from 0. For example, if the first seen sample for new [time series](https://docs.victoriametrics.com/keyconcepts/#time-series)
|
||||
`increase` assumes that all the counters start from 0. For example, if the first seen sample for new [time series](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#time-series)
|
||||
is `10`, then `increase` assumes that the time series has been increased by `10`. If you need ignoring the first sample for new time series,
|
||||
then take a look at [increase_prometheus](#increase_prometheus).
|
||||
|
||||
@@ -490,16 +491,16 @@ See also:
|
||||
|
||||
### increase_prometheus
|
||||
|
||||
`increase_prometheus` returns the increase of input [time series](https://docs.victoriametrics.com/keyconcepts/#time-series) over the given `interval`.
|
||||
`increase_prometheus` makes sense only for aggregating [counters](https://docs.victoriametrics.com/keyconcepts/#counter).
|
||||
`increase_prometheus` returns the increase of input [time series](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#time-series) over the given `interval`.
|
||||
`increase_prometheus` makes sense only for aggregating [counters](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#counter).
|
||||
|
||||
The results of `increase_prometheus` is equal to the following [MetricsQL](https://docs.victoriametrics.com/metricsql/) query:
|
||||
The results of `increase_prometheus` is equal to the following [MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/) query:
|
||||
|
||||
```metricsql
|
||||
sum(increase_prometheus(some_counter[interval]))
|
||||
```
|
||||
|
||||
`increase_prometheus` skips the first seen sample value per each [time series](https://docs.victoriametrics.com/keyconcepts/#time-series).
|
||||
`increase_prometheus` skips the first seen sample value per each [time series](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#time-series).
|
||||
If you need taking into account the first sample per time series, then take a look at [increase](#increase).
|
||||
|
||||
Aggregating irregular and sporadic metrics (received from [Lambdas](https://aws.amazon.com/lambda/)
|
||||
@@ -515,9 +516,9 @@ See also:
|
||||
|
||||
### last
|
||||
|
||||
`last` returns the last input [sample value](https://docs.victoriametrics.com/keyconcepts/#raw-samples) over the given `interval`.
|
||||
`last` returns the last input [sample value](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#raw-samples) over the given `interval`.
|
||||
|
||||
The results of `last` is roughly equal to the following [MetricsQL](https://docs.victoriametrics.com/metricsql/) query:
|
||||
The results of `last` is roughly equal to the following [MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/) query:
|
||||
|
||||
```metricsql
|
||||
last_over_time(some_metric[interval])
|
||||
@@ -532,9 +533,9 @@ See also:
|
||||
|
||||
### max
|
||||
|
||||
`max` returns the maximum input [sample value](https://docs.victoriametrics.com/keyconcepts/#raw-samples) over the given `interval`.
|
||||
`max` returns the maximum input [sample value](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#raw-samples) over the given `interval`.
|
||||
|
||||
The results of `max` is equal to the following [MetricsQL](https://docs.victoriametrics.com/metricsql/) query:
|
||||
The results of `max` is equal to the following [MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/) query:
|
||||
|
||||
```metricsql
|
||||
max(max_over_time(some_metric[interval]))
|
||||
@@ -553,9 +554,9 @@ See also:
|
||||
|
||||
### min
|
||||
|
||||
`min` returns the minimum input [sample value](https://docs.victoriametrics.com/keyconcepts/#raw-samples) over the given `interval`.
|
||||
`min` returns the minimum input [sample value](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#raw-samples) over the given `interval`.
|
||||
|
||||
The results of `min` is equal to the following [MetricsQL](https://docs.victoriametrics.com/metricsql/) query:
|
||||
The results of `min` is equal to the following [MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/) query:
|
||||
|
||||
```metricsql
|
||||
min(min_over_time(some_metric[interval]))
|
||||
@@ -574,10 +575,10 @@ See also:
|
||||
|
||||
### rate_avg
|
||||
|
||||
`rate_avg` returns the average of average per-second increase rates across input [time series](https://docs.victoriametrics.com/keyconcepts/#time-series) over the given `interval`.
|
||||
`rate_avg` makes sense only for aggregating [counters](https://docs.victoriametrics.com/keyconcepts/#counter).
|
||||
`rate_avg` returns the average of average per-second increase rates across input [time series](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#time-series) over the given `interval`.
|
||||
`rate_avg` makes sense only for aggregating [counters](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#counter).
|
||||
|
||||
The results of `rate_avg` are equal to the following [MetricsQL](https://docs.victoriametrics.com/metricsql/) query:
|
||||
The results of `rate_avg` are equal to the following [MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/) query:
|
||||
|
||||
```metricsql
|
||||
avg(rate(some_counter[interval]))
|
||||
@@ -591,10 +592,10 @@ See also:
|
||||
|
||||
### rate_sum
|
||||
|
||||
`rate_sum` returns the sum of average per-second increase rates across input [time series](https://docs.victoriametrics.com/keyconcepts/#time-series) over the given `interval`.
|
||||
`rate_sum` makes sense only for aggregating [counters](https://docs.victoriametrics.com/keyconcepts/#counter).
|
||||
`rate_sum` returns the sum of average per-second increase rates across input [time series](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#time-series) over the given `interval`.
|
||||
`rate_sum` makes sense only for aggregating [counters](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#counter).
|
||||
|
||||
The results of `rate_sum` are equal to the following [MetricsQL](https://docs.victoriametrics.com/metricsql/) query:
|
||||
The results of `rate_sum` are equal to the following [MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/) query:
|
||||
|
||||
```metricsql
|
||||
sum(rate(some_counter[interval]))
|
||||
@@ -608,11 +609,11 @@ See also:
|
||||
|
||||
### stddev
|
||||
|
||||
`stddev` returns [standard deviation](https://en.wikipedia.org/wiki/Standard_deviation) for the input [sample values](https://docs.victoriametrics.com/keyconcepts/#raw-samples)
|
||||
`stddev` returns [standard deviation](https://en.wikipedia.org/wiki/Standard_deviation) for the input [sample values](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#raw-samples)
|
||||
over the given `interval`.
|
||||
`stddev` makes sense only for aggregating [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
|
||||
`stddev` makes sense only for aggregating [gauges](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#gauge).
|
||||
|
||||
The results of `stddev` is roughly equal to the following [MetricsQL](https://docs.victoriametrics.com/metricsql/) query:
|
||||
The results of `stddev` is roughly equal to the following [MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/) query:
|
||||
|
||||
```metricsql
|
||||
histogram_stddev(sum(histogram_over_time(some_metric[interval])) by (vmrange))
|
||||
@@ -626,11 +627,11 @@ See also:
|
||||
|
||||
### stdvar
|
||||
|
||||
`stdvar` returns [standard variance](https://en.wikipedia.org/wiki/Variance) for the input [sample values](https://docs.victoriametrics.com/keyconcepts/#raw-samples)
|
||||
`stdvar` returns [standard variance](https://en.wikipedia.org/wiki/Variance) for the input [sample values](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#raw-samples)
|
||||
over the given `interval`.
|
||||
`stdvar` makes sense only for aggregating [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
|
||||
`stdvar` makes sense only for aggregating [gauges](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#gauge).
|
||||
|
||||
The results of `stdvar` is roughly equal to the following [MetricsQL](https://docs.victoriametrics.com/metricsql/) query:
|
||||
The results of `stdvar` is roughly equal to the following [MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/) query:
|
||||
|
||||
```metricsql
|
||||
histogram_stdvar(sum(histogram_over_time(some_metric[interval])) by (vmrange))
|
||||
@@ -648,10 +649,10 @@ See also:
|
||||
|
||||
### sum_samples
|
||||
|
||||
`sum_samples` sums input [sample values](https://docs.victoriametrics.com/keyconcepts/#raw-samples) over the given `interval`.
|
||||
`sum_samples` makes sense only for aggregating [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
|
||||
`sum_samples` sums input [sample values](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#raw-samples) over the given `interval`.
|
||||
`sum_samples` makes sense only for aggregating [gauges](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#gauge).
|
||||
|
||||
The results of `sum_samples` is equal to the following [MetricsQL](https://docs.victoriametrics.com/metricsql/) query:
|
||||
The results of `sum_samples` is equal to the following [MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/) query:
|
||||
|
||||
```metricsql
|
||||
sum(sum_over_time(some_metric[interval]))
|
||||
@@ -668,16 +669,16 @@ See also:
|
||||
|
||||
### total
|
||||
|
||||
`total` generates output [counter](https://docs.victoriametrics.com/keyconcepts/#counter) by summing the input counters over the given `interval`.
|
||||
`total` makes sense only for aggregating [counters](https://docs.victoriametrics.com/keyconcepts/#counter).
|
||||
`total` generates output [counter](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#counter) by summing the input counters over the given `interval`.
|
||||
`total` makes sense only for aggregating [counters](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#counter).
|
||||
|
||||
The results of `total` is roughly equal to the following [MetricsQL](https://docs.victoriametrics.com/metricsql/) query:
|
||||
The results of `total` is roughly equal to the following [MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/) query:
|
||||
|
||||
```metricsql
|
||||
sum(running_sum(increase_pure(some_counter)))
|
||||
```
|
||||
|
||||
`total` assumes that all the counters start from 0. For example, if the first seen sample for new [time series](https://docs.victoriametrics.com/keyconcepts/#time-series)
|
||||
`total` assumes that all the counters start from 0. For example, if the first seen sample for new [time series](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#time-series)
|
||||
is `10`, then `total` assumes that the time series has been increased by `10`. If you need ignoring the first sample for new time series,
|
||||
then take a look at [total_prometheus](#total_prometheus).
|
||||
|
||||
@@ -685,7 +686,7 @@ For example, see below time series produced by config with aggregation interval
|
||||
|
||||

|
||||
|
||||
`total` is not affected by [counter resets](https://docs.victoriametrics.com/keyconcepts/#counter) -
|
||||
`total` is not affected by [counter resets](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#counter) -
|
||||
it continues to increase monotonically with respect to the previous value.
|
||||
The counters are most often reset when the application is restarted.
|
||||
|
||||
@@ -711,19 +712,19 @@ See also:
|
||||
|
||||
### total_prometheus
|
||||
|
||||
`total_prometheus` generates output [counter](https://docs.victoriametrics.com/keyconcepts/#counter) by summing the input counters over the given `interval`.
|
||||
`total_prometheus` makes sense only for aggregating [counters](https://docs.victoriametrics.com/keyconcepts/#counter).
|
||||
`total_prometheus` generates output [counter](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#counter) by summing the input counters over the given `interval`.
|
||||
`total_prometheus` makes sense only for aggregating [counters](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#counter).
|
||||
|
||||
The results of `total_prometheus` is roughly equal to the following [MetricsQL](https://docs.victoriametrics.com/metricsql/) query:
|
||||
The results of `total_prometheus` is roughly equal to the following [MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/) query:
|
||||
|
||||
```metricsql
|
||||
sum(running_sum(increase_prometheus(some_counter)))
|
||||
```
|
||||
|
||||
`total_prometheus` skips the first seen sample value per each [time series](https://docs.victoriametrics.com/keyconcepts/#time-series).
|
||||
`total_prometheus` skips the first seen sample value per each [time series](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#time-series).
|
||||
If you need taking into account the first sample per time series, then take a look at [total](#total).
|
||||
|
||||
`total_prometheus` is not affected by [counter resets](https://docs.victoriametrics.com/keyconcepts/#counter) -
|
||||
`total_prometheus` is not affected by [counter resets](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#counter) -
|
||||
it continues to increase monotonically with respect to the previous value.
|
||||
The counters are most often reset when the application is restarted.
|
||||
|
||||
@@ -741,9 +742,9 @@ See also:
|
||||
### unique_samples
|
||||
|
||||
`unique_samples` counts the number of unique sample values over the given `interval`.
|
||||
`unique_samples` makes sense only for aggregating [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
|
||||
`unique_samples` makes sense only for aggregating [gauges](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#gauge).
|
||||
|
||||
The results of `unique_samples` is equal to the following [MetricsQL](https://docs.victoriametrics.com/metricsql/) query:
|
||||
The results of `unique_samples` is equal to the following [MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/) query:
|
||||
|
||||
```metricsql
|
||||
count(count_values_over_time(some_metric[interval]))
|
||||
@@ -757,11 +758,11 @@ See also:
|
||||
### quantiles
|
||||
|
||||
`quantiles(phi1, ..., phiN)` returns [percentiles](https://en.wikipedia.org/wiki/Percentile) for the given `phi*`
|
||||
over the input [sample values](https://docs.victoriametrics.com/keyconcepts/#raw-samples) on the given `interval`.
|
||||
over the input [sample values](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#raw-samples) on the given `interval`.
|
||||
`phi` must be in the range `[0..1]`, where `0` means `0th` percentile, while `1` means `100th` percentile.
|
||||
`quantiles(...)` makes sense only for aggregating [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
|
||||
`quantiles(...)` makes sense only for aggregating [gauges](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#gauge).
|
||||
|
||||
The results of `quantiles(phi1, ..., phiN)` is equal to the following [MetricsQL](https://docs.victoriametrics.com/metricsql/) query:
|
||||
The results of `quantiles(phi1, ..., phiN)` is equal to the following [MetricsQL](https://docs.victoriametrics.com/victoriametrics/metricsql/) query:
|
||||
|
||||
```metricsql
|
||||
histogram_quantiles("quantile", phi1, ..., phiN, sum(histogram_over_time(some_metric[interval])) by (vmrange))
|
||||
@@ -777,8 +778,8 @@ See also:
|
||||
## Stream aggregation config
|
||||
|
||||
Below is the format for stream aggregation config file, which may be referred via `-streamAggr.config` command-line flag at
|
||||
[single-node VictoriaMetrics](https://docs.victoriametrics.com/single-server-victoriametrics/) and [vmagent](https://docs.victoriametrics.com/vmagent/).
|
||||
At [vmagent](https://docs.victoriametrics.com/vmagent/) `-remoteWrite.streamAggr.config` command-line flag can be
|
||||
[single-node VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/) and [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/).
|
||||
At [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/) `-remoteWrite.streamAggr.config` command-line flag can be
|
||||
specified individually per each `-remoteWrite.url`:
|
||||
|
||||
```yaml
|
||||
@@ -787,12 +788,12 @@ specified individually per each `-remoteWrite.url`:
|
||||
#
|
||||
# If it is set, then it is used as `name` label in the exposed metrics
|
||||
# for the given aggregation config at /metrics page.
|
||||
# See https://docs.victoriametrics.com/vmagent/#monitoring and https://docs.victoriametrics.com/#monitoring
|
||||
# See https://docs.victoriametrics.com/victoriametrics/vmagent/#monitoring and https://docs.victoriametrics.com/victoriametrics#monitoring
|
||||
- name: 'foobar'
|
||||
|
||||
# match is an optional filter for incoming samples to aggregate.
|
||||
# It can contain arbitrary Prometheus series selector
|
||||
# according to https://docs.victoriametrics.com/keyconcepts/#filtering .
|
||||
# according to https://docs.victoriametrics.com/victoriametrics/keyconcepts/#filtering .
|
||||
# If match isn't set, then all the incoming samples are aggregated.
|
||||
#
|
||||
# match also can contain a list of series selectors. Then the incoming samples are aggregated
|
||||
@@ -806,8 +807,8 @@ specified individually per each `-remoteWrite.url`:
|
||||
interval: 1m
|
||||
|
||||
# dedup_interval is an optional interval for de-duplication of input samples before the aggregation.
|
||||
# Samples are de-duplicated on a per-series basis. See https://docs.victoriametrics.com/keyconcepts/#time-series
|
||||
# and https://docs.victoriametrics.com/#deduplication
|
||||
# Samples are de-duplicated on a per-series basis. See https://docs.victoriametrics.com/victoriametrics/keyconcepts/#time-series
|
||||
# and https://docs.victoriametrics.com/victoriametrics#deduplication
|
||||
# The deduplication is performed after input_relabel_configs relabeling is applied.
|
||||
# By default, the deduplication is disabled unless -remoteWrite.streamAggr.dedupInterval or -streamAggr.dedupInterval
|
||||
# command-line flags are set.
|
||||
@@ -815,7 +816,7 @@ specified individually per each `-remoteWrite.url`:
|
||||
# dedup_interval: 30s
|
||||
|
||||
# enable_windows is a boolean option to enable fixed aggregation windows.
|
||||
# See https://docs.victoriametrics.com/stream-aggregation#aggregation-windows
|
||||
# See https://docs.victoriametrics.com/victoriametrics/stream-aggregation#aggregation-windows
|
||||
#
|
||||
# enable_windows: true
|
||||
|
||||
@@ -828,7 +829,7 @@ specified individually per each `-remoteWrite.url`:
|
||||
# - rate_sum
|
||||
# - total
|
||||
# - total_prometheus
|
||||
# See https://docs.victoriametrics.com/stream-aggregation/#staleness for more details.
|
||||
# See https://docs.victoriametrics.com/victoriametrics/stream-aggregation/#staleness for more details.
|
||||
#
|
||||
# staleness_interval: 2m
|
||||
|
||||
@@ -847,17 +848,17 @@ specified individually per each `-remoteWrite.url`:
|
||||
# flush_on_shutdown: false
|
||||
|
||||
# without is an optional list of labels, which must be removed from the output aggregation.
|
||||
# See https://docs.victoriametrics.com/stream-aggregation/#aggregating-by-labels
|
||||
# See https://docs.victoriametrics.com/victoriametrics/stream-aggregation/#aggregating-by-labels
|
||||
#
|
||||
# without: [instance]
|
||||
|
||||
# by is an optional list of labels, which must be preserved in the output aggregation.
|
||||
# See https://docs.victoriametrics.com/stream-aggregation/#aggregating-by-labels
|
||||
# See https://docs.victoriametrics.com/victoriametrics/stream-aggregation/#aggregating-by-labels
|
||||
#
|
||||
# by: [job, vmrange]
|
||||
|
||||
# outputs is the list of unique aggregations to perform on the input data.
|
||||
# See https://docs.victoriametrics.com/stream-aggregation/#aggregation-outputs
|
||||
# See https://docs.victoriametrics.com/victoriametrics/stream-aggregation/#aggregation-outputs
|
||||
#
|
||||
outputs: [total]
|
||||
|
||||
@@ -865,25 +866,25 @@ specified individually per each `-remoteWrite.url`:
|
||||
# This option can't be enabled together with `-streamAggr.keepInput` or `-remoteWrite.streamAggr.keepInput`.
|
||||
# This option can be set only if outputs list contains a single output.
|
||||
# By default, a special suffix is added to original metric names in the aggregated samples.
|
||||
# See https://docs.victoriametrics.com/stream-aggregation/#output-metric-names
|
||||
# See https://docs.victoriametrics.com/victoriametrics/stream-aggregation/#output-metric-names
|
||||
#
|
||||
# keep_metric_names: false
|
||||
|
||||
# ignore_old_samples instructs ignoring input samples with old timestamps outside the current aggregation interval.
|
||||
# See https://docs.victoriametrics.com/stream-aggregation/#ignoring-old-samples
|
||||
# See https://docs.victoriametrics.com/victoriametrics/stream-aggregation/#ignoring-old-samples
|
||||
# See also -remoteWrite.streamAggr.ignoreOldSamples and -streamAggr.ignoreOldSamples command-line flag.
|
||||
#
|
||||
# ignore_old_samples: false
|
||||
|
||||
# ignore_first_intervals instructs ignoring the first N aggregation intervals after process start.
|
||||
# See https://docs.victoriametrics.com/stream-aggregation/#ignore-aggregation-intervals-on-start
|
||||
# See https://docs.victoriametrics.com/victoriametrics/stream-aggregation/#ignore-aggregation-intervals-on-start
|
||||
# See also -remoteWrite.streamAggr.ignoreFirstIntervals and -streamAggr.ignoreFirstIntervals command-line flags.
|
||||
#
|
||||
# ignore_first_intervals: N
|
||||
|
||||
# drop_input_labels instructs dropping the given labels from input samples.
|
||||
# The labels' dropping is performed before input_relabel_configs are applied.
|
||||
# This also means that the labels are dropped before de-duplication ( https://docs.victoriametrics.com/stream-aggregation/#deduplication )
|
||||
# This also means that the labels are dropped before de-duplication ( https://docs.victoriametrics.com/victoriametrics/stream-aggregation/#deduplication )
|
||||
# and stream aggregation.
|
||||
#
|
||||
# drop_input_labels: [replica, availability_zone]
|
||||
@@ -891,7 +892,7 @@ specified individually per each `-remoteWrite.url`:
|
||||
# input_relabel_configs is an optional relabeling rules,
|
||||
# which are applied to the incoming samples after they pass the match filter
|
||||
# and before being aggregated.
|
||||
# See https://docs.victoriametrics.com/stream-aggregation/#relabeling
|
||||
# See https://docs.victoriametrics.com/victoriametrics/stream-aggregation/#relabeling
|
||||
#
|
||||
input_relabel_configs:
|
||||
- target_label: vmaggr
|
||||
@@ -910,7 +911,7 @@ per each specified config entry.
|
||||
|
||||
### Configuration update
|
||||
|
||||
[vmagent](https://docs.victoriametrics.com/vmagent/) and [single-node VictoriaMetrics](https://docs.victoriametrics.com/single-server-victoriametrics/)
|
||||
[vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/) and [single-node VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/)
|
||||
support the following approaches for hot reloading stream aggregation configs from `-remoteWrite.streamAggr.config` and `-streamAggr.config`:
|
||||
|
||||
* By sending `SIGHUP` signal to `vmagent` or `victoria-metrics` process:
|
||||
@@ -923,18 +924,18 @@ support the following approaches for hot reloading stream aggregation configs fr
|
||||
|
||||
# Routing
|
||||
|
||||
[Single-node VictoriaMetrics](https://docs.victoriametrics.com/single-server-victoriametrics/) supports relabeling,
|
||||
[Single-node VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/) supports relabeling,
|
||||
deduplication and stream aggregation for all the received data, scraped or pushed.
|
||||
The processed data is then stored in local storage and **can't be forwarded further**.
|
||||
|
||||
[vmagent](https://docs.victoriametrics.com/vmagent/) supports relabeling, deduplication and stream aggregation for all
|
||||
[vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/) supports relabeling, deduplication and stream aggregation for all
|
||||
the received data, scraped or pushed. Then, the collected data will be forwarded to specified `-remoteWrite.url` destinations.
|
||||
The data processing order is the following:
|
||||
|
||||
1. all the received data is relabeled according to the specified [`-remoteWrite.relabelConfig`](https://docs.victoriametrics.com/vmagent/#relabeling) (if it is set)
|
||||
1. all the received data is deduplicated according to specified [`-streamAggr.dedupInterval`](https://docs.victoriametrics.com/stream-aggregation/#deduplication)
|
||||
1. all the received data is relabeled according to the specified [`-remoteWrite.relabelConfig`](https://docs.victoriametrics.com/victoriametrics/vmagent/#relabeling) (if it is set)
|
||||
1. all the received data is deduplicated according to specified [`-streamAggr.dedupInterval`](https://docs.victoriametrics.com/victoriametrics/stream-aggregation/#deduplication)
|
||||
(if it is set to duration bigger than 0)
|
||||
1. all the received data is aggregated according to specified [`-streamAggr.config`](https://docs.victoriametrics.com/stream-aggregation/#configuration) (if it is set)
|
||||
1. all the received data is aggregated according to specified [`-streamAggr.config`](https://docs.victoriametrics.com/victoriametrics/stream-aggregation/#configuration) (if it is set)
|
||||
1. the resulting data is then replicated to each `-remoteWrite.url`
|
||||
1. data sent to each `-remoteWrite.url` can be additionally relabeled according to the corresponding `-remoteWrite.urlRelabelConfig` (set individually per URL)
|
||||
1. data sent to each `-remoteWrite.url` can be additionally deduplicated according to the corresponding `-remoteWrite.streamAggr.dedupInterval` (set individually per URL)
|
||||
@@ -943,44 +944,44 @@ The data processing order is the following:
|
||||
|
||||
Typical scenarios for data routing with `vmagent`:
|
||||
|
||||
1. **Aggregate incoming data and replicate to N destinations**. Specify [`-streamAggr.config`](https://docs.victoriametrics.com/stream-aggregation/#configuration) command-line flag
|
||||
1. **Aggregate incoming data and replicate to N destinations**. Specify [`-streamAggr.config`](https://docs.victoriametrics.com/victoriametrics/stream-aggregation/#configuration) command-line flag
|
||||
to aggregate the incoming data before replicating it to all the configured `-remoteWrite.url` destinations.
|
||||
2. **Individually aggregate incoming data for each destination**. Specify [`-remoteWrite.streamAggr.config`](https://docs.victoriametrics.com/stream-aggregation/#configuration)
|
||||
command-line flag for each `-remoteWrite.url` destination. [Relabeling](https://docs.victoriametrics.com/vmagent/#relabeling) via `-remoteWrite.urlRelabelConfig`
|
||||
2. **Individually aggregate incoming data for each destination**. Specify [`-remoteWrite.streamAggr.config`](https://docs.victoriametrics.com/victoriametrics/stream-aggregation/#configuration)
|
||||
command-line flag for each `-remoteWrite.url` destination. [Relabeling](https://docs.victoriametrics.com/victoriametrics/vmagent/#relabeling) via `-remoteWrite.urlRelabelConfig`
|
||||
can be used for routing only the selected metrics to each `-remoteWrite.url` destination.
|
||||
|
||||
# Deduplication
|
||||
|
||||
[vmagent](https://docs.victoriametrics.com/vmagent/) supports online [de-duplication](https://docs.victoriametrics.com/#deduplication) of samples
|
||||
[vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/) supports online [de-duplication](https://docs.victoriametrics.com/victoriametrics#deduplication) of samples
|
||||
before sending them to the configured `-remoteWrite.url`. The de-duplication can be enabled via the following options:
|
||||
|
||||
- By specifying the desired de-duplication interval via `-streamAggr.dedupInterval` command-line flag for all received data
|
||||
or via `-remoteWrite.streamAggr.dedupInterval` command-line flag for the particular `-remoteWrite.url` destination.
|
||||
For example, `./vmagent -remoteWrite.url=http://remote-storage/api/v1/write -remoteWrite.streamAggr.dedupInterval=30s` instructs `vmagent` to leave
|
||||
only the last sample per each seen [time series](https://docs.victoriametrics.com/keyconcepts/#time-series) per every 30 seconds.
|
||||
The de-deduplication is performed after applying [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling) and
|
||||
only the last sample per each seen [time series](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#time-series) per every 30 seconds.
|
||||
The de-deduplication is performed after applying [relabeling](https://docs.victoriametrics.com/victoriametrics/vmagent/#relabeling) and
|
||||
before performing the aggregation.
|
||||
|
||||
- By specifying `dedup_interval` option individually per each [stream aggregation config](#stream-aggregation-config)
|
||||
in `-remoteWrite.streamAggr.config` or `-streamAggr.config` configs.
|
||||
|
||||
[Single-node VictoriaMetrics](https://docs.victoriametrics.com/single-server-victoriametrics/) supports two types of de-duplication:
|
||||
- After storing the duplicate samples to local storage. See [`-dedup.minScrapeInterval`](https://docs.victoriametrics.com/#deduplication) command-line option.
|
||||
[Single-node VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/) supports two types of de-duplication:
|
||||
- After storing the duplicate samples to local storage. See [`-dedup.minScrapeInterval`](https://docs.victoriametrics.com/victoriametrics#deduplication) command-line option.
|
||||
- Before storing the duplicate samples to local storage. This type of de-duplication can be enabled via the following options:
|
||||
- By specifying the desired de-duplication interval via `-streamAggr.dedupInterval` command-line flag.
|
||||
For example, `./victoria-metrics -streamAggr.dedupInterval=30s` instructs VictoriaMetrics to leave only the last sample per each
|
||||
seen [time series](https://docs.victoriametrics.com/keyconcepts/#time-series) per every 30 seconds.
|
||||
The de-duplication is performed after applying `-relabelConfig` [relabeling](https://docs.victoriametrics.com/#relabeling).
|
||||
seen [time series](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#time-series) per every 30 seconds.
|
||||
The de-duplication is performed after applying `-relabelConfig` [relabeling](https://docs.victoriametrics.com/victoriametrics#relabeling).
|
||||
|
||||
- By specifying `dedup_interval` option individually per each [stream aggregation config](#stream-aggregation-config) at `-streamAggr.config`.
|
||||
|
||||
It is possible to drop the given labels before applying the de-duplication. See [these docs](#dropping-unneeded-labels).
|
||||
|
||||
The online de-duplication uses the same logic as [`-dedup.minScrapeInterval` command-line flag](https://docs.victoriametrics.com/#deduplication) at VictoriaMetrics.
|
||||
The online de-duplication uses the same logic as [`-dedup.minScrapeInterval` command-line flag](https://docs.victoriametrics.com/victoriametrics#deduplication) at VictoriaMetrics.
|
||||
|
||||
# Relabeling
|
||||
|
||||
It is possible to apply [arbitrary relabeling](https://docs.victoriametrics.com/vmagent/#relabeling) to input and output metrics
|
||||
It is possible to apply [arbitrary relabeling](https://docs.victoriametrics.com/victoriametrics/vmagent/#relabeling) to input and output metrics
|
||||
during stream aggregation via `input_relabel_configs` and `output_relabel_configs` options in [stream aggregation config](#stream-aggregation-config).
|
||||
|
||||
Relabeling rules inside `input_relabel_configs` are applied to samples matching the `match` filters before optional [deduplication](#deduplication).
|
||||
@@ -1015,7 +1016,7 @@ By default, all the input samples are taken into account during stream aggregati
|
||||
outside the current [aggregation interval](#stream-aggregation-config) must be ignored, then the following options can be used:
|
||||
|
||||
- To pass `-streamAggr.ignoreOldSamples` command-line flag to [single-node VictoriaMetrics](https://docs.victoriametrics.com/)
|
||||
or to [vmagent](https://docs.victoriametrics.com/vmagent/). At [vmagent](https://docs.victoriametrics.com/vmagent/)
|
||||
or to [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/). At [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/)
|
||||
`-remoteWrite.streamAggr.ignoreOldSamples` flag can be specified individually per each `-remoteWrite.url`.
|
||||
This enables ignoring old samples for all the [aggregation configs](#stream-aggregation-config).
|
||||
|
||||
@@ -1024,16 +1025,16 @@ outside the current [aggregation interval](#stream-aggregation-config) must be i
|
||||
|
||||
## Ignore aggregation intervals on start
|
||||
|
||||
Streaming aggregation results may be incorrect for some time after the restart of [vmagent](https://docs.victoriametrics.com/vmagent/)
|
||||
or [single-node VictoriaMetrics](https://docs.victoriametrics.com/) until all the buffered [samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples)
|
||||
are sent from remote sources to the `vmagent` or single-node VictoriaMetrics via [supported data ingestion protocols](https://docs.victoriametrics.com/vmagent/#how-to-push-data-to-vmagent).
|
||||
Streaming aggregation results may be incorrect for some time after the restart of [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/)
|
||||
or [single-node VictoriaMetrics](https://docs.victoriametrics.com/) until all the buffered [samples](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#raw-samples)
|
||||
are sent from remote sources to the `vmagent` or single-node VictoriaMetrics via [supported data ingestion protocols](https://docs.victoriametrics.com/victoriametrics/vmagent/#how-to-push-data-to-vmagent).
|
||||
In this case it may be a good idea to drop the aggregated data during the first `N` [aggregation intervals](#stream-aggregation-config)
|
||||
just after the restart of `vmagent` or single-node VictoriaMetrics. This can be done via the following options:
|
||||
|
||||
- The `-streamAggr.ignoreFirstIntervals=N` command-line flag at `vmagent` and single-node VictoriaMetrics. This flag instructs skipping the first `N`
|
||||
[aggregation intervals](#stream-aggregation-config) just after the restart across all the [configured stream aggregation configs](#configuration).
|
||||
|
||||
The `-remoteWrite.streamAggr.ignoreFirstIntervals` command-line flag can be specified individually per each `-remoteWrite.url` at [vmagent](https://docs.victoriametrics.com/vmagent/).
|
||||
The `-remoteWrite.streamAggr.ignoreFirstIntervals` command-line flag can be specified individually per each `-remoteWrite.url` at [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/).
|
||||
|
||||
- The `ignore_first_intervals: N` option at the particular [aggregation config](#stream-aggregation-config).
|
||||
|
||||
@@ -1164,7 +1165,7 @@ incoming samples with timestamps close to the aggregation interval's end will ge
|
||||
with `interval: 1m` a data sample with timestamp 1739473078 (18:57:59) can fall into aggregation round `18:58:00` or `18:59:00`.
|
||||
It depends on network lag, load, clock synchronization, etc. In most scenarios it doesn't impact aggregation or
|
||||
deduplication results, which are consistent within margin of error. But for metrics represented as a collection of series,
|
||||
like [histograms](https://docs.victoriametrics.com/keyconcepts/#histogram), such inaccuracy leads to invalid aggregation results.
|
||||
like [histograms](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#histogram), such inaccuracy leads to invalid aggregation results.
|
||||
|
||||
For this case, streaming aggregation and deduplication support mode with aggregation windows for current and previous state.
|
||||
With this mode, flush doesn't happen immediately but is shifted by a calculated samples lag that improves correctness for delayed data. {{% available_from "v1.112.0" %}}
|
||||
@@ -1173,12 +1174,12 @@ Enabling of this mode has increased resource usage: memory usage is expected to
|
||||
instead of one. However, this significantly improves accuracy of calculations. Aggregation windows can be enabled via
|
||||
the following settings:
|
||||
|
||||
- `-streamAggr.enableWindows` at [single-node VictoriaMetrics](https://docs.victoriametrics.com/single-server-victoriametrics/)
|
||||
and [vmagent](https://docs.victoriametrics.com/vmagent/). At [vmagent](https://docs.victoriametrics.com/vmagent/)
|
||||
- `-streamAggr.enableWindows` at [single-node VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/)
|
||||
and [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/). At [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/)
|
||||
`-remoteWrite.streamAggr.enableWindows` flag can be specified individually per each `-remoteWrite.url`.
|
||||
If one of these flags is set, then all aggregators will be using fixed windows. In conjunction with `-remoteWrite.streamAggr.dedupInterval` or
|
||||
`-streamAggr.dedupInterval` fixed aggregation windows are enabled on deduplicator as well.
|
||||
- `enable_windows` option in [aggregation config](https://docs.victoriametrics.com/stream-aggregation/#stream-aggregation-config).
|
||||
- `enable_windows` option in [aggregation config](https://docs.victoriametrics.com/victoriametrics/stream-aggregation/#stream-aggregation-config).
|
||||
It allows enabling aggregation windows for a specific aggregator.
|
||||
|
||||
## Staleness
|
||||
@@ -1214,26 +1215,26 @@ These issues can be fixed in the following ways:
|
||||
The following solutions can help reducing memory usage and CPU usage during streaming aggregation:
|
||||
|
||||
- To use more specific `match` filters at [streaming aggregation config](#stream-aggregation-config), so only the really needed
|
||||
[raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) are aggregated.
|
||||
[raw samples](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#raw-samples) are aggregated.
|
||||
- To increase aggregation interval by specifying bigger duration for the `interval` option at [streaming aggregation config](#stream-aggregation-config).
|
||||
- To generate lower number of output time series by using less specific [`by` list](#aggregating-by-labels) or more specific [`without` list](#aggregating-by-labels).
|
||||
- To drop unneeded long labels in input samples via [input_relabel_configs](#relabeling).
|
||||
|
||||
## Cluster mode
|
||||
|
||||
If you use [vmagent in cluster mode](https://docs.victoriametrics.com/vmagent/#scraping-big-number-of-targets) for streaming aggregation
|
||||
If you use [vmagent in cluster mode](https://docs.victoriametrics.com/victoriametrics/vmagent/#scraping-big-number-of-targets) for streaming aggregation
|
||||
then be careful when using [`by` or `without` options](#aggregating-by-labels) or when modifying sample labels
|
||||
via [relabeling](#relabeling), since incorrect usage may result in duplicates and data collision.
|
||||
|
||||
For example, if more than one `vmagent` instance calculates [increase](#increase) for `http_requests_total` metric
|
||||
with `by: [path]` option, then all the `vmagent` instances will aggregate samples to the same set of time series with different `path` labels.
|
||||
The proper fix would be [adding a unique label](https://docs.victoriametrics.com/vmagent/#adding-labels-to-metrics) for all the output samples
|
||||
produced by each `vmagent`, so they are aggregated into distinct sets of [time series](https://docs.victoriametrics.com/keyconcepts/#time-series).
|
||||
The proper fix would be [adding a unique label](https://docs.victoriametrics.com/victoriametrics/vmagent/#adding-labels-to-metrics) for all the output samples
|
||||
produced by each `vmagent`, so they are aggregated into distinct sets of [time series](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#time-series).
|
||||
These time series then can be aggregated later as needed during querying.
|
||||
|
||||
If `vmagent` instances run in Docker or Kubernetes, then you can refer `POD_NAME` or `HOSTNAME` environment variables
|
||||
as a unique label value per each `vmagent` via `-remoteWrite.label=vmagent=%{HOSTNAME}` command-line flag.
|
||||
See [these docs](https://docs.victoriametrics.com/#environment-variables) on how to refer environment variables in VictoriaMetrics components.
|
||||
See [these docs](https://docs.victoriametrics.com/victoriametrics#environment-variables) on how to refer environment variables in VictoriaMetrics components.
|
||||
|
||||
## Common mistakes
|
||||
|
||||
@@ -1245,7 +1246,7 @@ and produce incomplete aggregations.
|
||||
|
||||
To keep aggregation results consistent, make sure that vmagent receives all the required data for aggregation. In case if you need to
|
||||
split the load across multiple vmagents, try sharding the traffic among them via metric names or labels.
|
||||
For example, see how vmagent could consistently [shard data across remote write destinations](https://docs.victoriametrics.com/vmagent/#sharding-among-remote-storages)
|
||||
For example, see how vmagent could consistently [shard data across remote write destinations](https://docs.victoriametrics.com/victoriametrics/vmagent/#sharding-among-remote-storages)
|
||||
via `-remoteWrite.shardByURL.labels` or `-remoteWrite.shardByURL.ignoreLabels` cmd-line flags.
|
||||
|
||||
### Create aggregator per each recording rule
|
||||
|
||||
@@ -8,6 +8,7 @@ menu:
|
||||
identifier: vm-api-examples
|
||||
|
||||
aliases:
|
||||
- /url-examples/
|
||||
- /url-examples.html
|
||||
- /url-examples/index.html
|
||||
- /url-examples/
|
||||
@@ -74,8 +75,8 @@ The expected output should return [HTTP Status 204](https://datatracker.ietf.org
|
||||
|
||||
Additional information:
|
||||
|
||||
* [How to delete time series](https://docs.victoriametrics.com/#how-to-delete-time-series)
|
||||
* [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#url-format)
|
||||
* [How to delete time series](https://docs.victoriametrics.com/victoriametrics#how-to-delete-time-series)
|
||||
* [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#url-format)
|
||||
|
||||
### /api/v1/export
|
||||
|
||||
@@ -97,10 +98,10 @@ curl http://<vmselect>:8481/select/0/prometheus/api/v1/export -d 'match[]=vm_htt
|
||||
|
||||
Additional information:
|
||||
|
||||
* [How to export time series](https://docs.victoriametrics.com/#how-to-export-time-series)
|
||||
* [How to import time series](https://docs.victoriametrics.com/#how-to-import-time-series-data)
|
||||
* [How to export data in JSON line format](https://docs.victoriametrics.com/#how-to-export-data-in-json-line-format)
|
||||
* [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#url-format)
|
||||
* [How to export time series](https://docs.victoriametrics.com/victoriametrics#how-to-export-time-series)
|
||||
* [How to import time series](https://docs.victoriametrics.com/victoriametrics#how-to-import-time-series-data)
|
||||
* [How to export data in JSON line format](https://docs.victoriametrics.com/victoriametrics#how-to-export-data-in-json-line-format)
|
||||
* [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#url-format)
|
||||
|
||||
### /api/v1/export/csv
|
||||
|
||||
@@ -122,9 +123,9 @@ curl http://<vmselect>:8481/select/0/prometheus/api/v1/export/csv -d 'format=__n
|
||||
|
||||
Additional information:
|
||||
|
||||
* [How to export time series](https://docs.victoriametrics.com/#how-to-export-time-series)
|
||||
* [How to import time series](https://docs.victoriametrics.com/#how-to-import-time-series-data)
|
||||
* [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#url-format)
|
||||
* [How to export time series](https://docs.victoriametrics.com/victoriametrics#how-to-export-time-series)
|
||||
* [How to import time series](https://docs.victoriametrics.com/victoriametrics#how-to-import-time-series-data)
|
||||
* [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#url-format)
|
||||
|
||||
### /api/v1/export/native
|
||||
|
||||
@@ -146,9 +147,9 @@ curl http://<vmselect>:8481/select/0/prometheus/api/v1/export/native -d 'match[]
|
||||
|
||||
More information:
|
||||
|
||||
* [How to export time series](https://docs.victoriametrics.com/#how-to-export-time-series)
|
||||
* [How to import time series](https://docs.victoriametrics.com/#how-to-import-time-series-data)
|
||||
* [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#url-format)
|
||||
* [How to export time series](https://docs.victoriametrics.com/victoriametrics#how-to-export-time-series)
|
||||
* [How to import time series](https://docs.victoriametrics.com/victoriametrics#how-to-import-time-series-data)
|
||||
* [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#url-format)
|
||||
|
||||
### /api/v1/import
|
||||
|
||||
@@ -170,9 +171,9 @@ curl -H 'Content-Type: application/json' --data-binary "@filename.json" -X POST
|
||||
|
||||
More information:
|
||||
|
||||
* [How to import time series](https://docs.victoriametrics.com/#how-to-import-time-series-data)
|
||||
* [How to export time series](https://docs.victoriametrics.com/#how-to-export-time-series)
|
||||
* [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#url-format)
|
||||
* [How to import time series](https://docs.victoriametrics.com/victoriametrics#how-to-import-time-series-data)
|
||||
* [How to export time series](https://docs.victoriametrics.com/victoriametrics#how-to-export-time-series)
|
||||
* [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#url-format)
|
||||
|
||||
### /api/v1/import/csv
|
||||
|
||||
@@ -198,9 +199,9 @@ curl -X POST 'http://<vminsert>:8480/insert/0/prometheus/api/v1/import/csv?forma
|
||||
|
||||
Additional information:
|
||||
|
||||
* [How to import time series](https://docs.victoriametrics.com/#how-to-import-time-series-data)
|
||||
* [How to export time series](https://docs.victoriametrics.com/#how-to-export-time-series)
|
||||
* [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#url-format)
|
||||
* [How to import time series](https://docs.victoriametrics.com/victoriametrics#how-to-import-time-series-data)
|
||||
* [How to export time series](https://docs.victoriametrics.com/victoriametrics#how-to-export-time-series)
|
||||
* [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#url-format)
|
||||
|
||||
### /api/v1/import/native
|
||||
|
||||
@@ -220,9 +221,9 @@ curl -X POST http://<vminsert>:8480/insert/0/prometheus/api/v1/import/native -T
|
||||
|
||||
Additional information:
|
||||
|
||||
* [How to import time series](https://docs.victoriametrics.com/#how-to-import-time-series-data)
|
||||
* [How to export time series](https://docs.victoriametrics.com/#how-to-export-time-series)
|
||||
* [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#url-format)
|
||||
* [How to import time series](https://docs.victoriametrics.com/victoriametrics#how-to-import-time-series-data)
|
||||
* [How to export time series](https://docs.victoriametrics.com/victoriametrics#how-to-export-time-series)
|
||||
* [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#url-format)
|
||||
|
||||
### /api/v1/import/prometheus
|
||||
|
||||
@@ -243,9 +244,9 @@ curl -d 'metric_name{foo="bar"} 123' -X POST http://<vminsert>:8480/insert/0/pro
|
||||
|
||||
Additional information:
|
||||
|
||||
* [How to import time series](https://docs.victoriametrics.com/#how-to-import-time-series-data)
|
||||
* [How to export time series](https://docs.victoriametrics.com/#how-to-export-time-series)
|
||||
* [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#url-format)
|
||||
* [How to import time series](https://docs.victoriametrics.com/victoriametrics#how-to-import-time-series-data)
|
||||
* [How to export time series](https://docs.victoriametrics.com/victoriametrics#how-to-export-time-series)
|
||||
* [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#url-format)
|
||||
|
||||
### /api/v1/labels
|
||||
|
||||
@@ -265,13 +266,13 @@ curl http://<vmselect>:8481/select/0/prometheus/api/v1/labels
|
||||
```
|
||||
|
||||
By default, VictoriaMetrics returns labels seen during the last day starting at 00:00 UTC because of performance reasons.
|
||||
An arbitrary time range can be set via [`start` and `end` query args](https://docs.victoriametrics.com/#timestamp-formats).
|
||||
An arbitrary time range can be set via [`start` and `end` query args](https://docs.victoriametrics.com/victoriametrics#timestamp-formats).
|
||||
The specified `start..end` time range is rounded to UTC day granularity because of performance reasons.
|
||||
|
||||
Additional information:
|
||||
* [Getting label names](https://prometheus.io/docs/prometheus/latest/querying/api/#getting-label-names)
|
||||
* [Prometheus querying API usage](https://docs.victoriametrics.com/#prometheus-querying-api-usage)
|
||||
* [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#url-format)
|
||||
* [Prometheus querying API usage](https://docs.victoriametrics.com/victoriametrics#prometheus-querying-api-usage)
|
||||
* [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#url-format)
|
||||
|
||||
### /api/v1/label/.../values
|
||||
|
||||
@@ -295,8 +296,8 @@ The specified `start..end` time range is rounded to UTC day granularity because
|
||||
|
||||
Additional information:
|
||||
* [Querying label values](https://prometheus.io/docs/prometheus/latest/querying/api/#querying-label-values)
|
||||
* [Prometheus querying API usage](https://docs.victoriametrics.com/#prometheus-querying-api-usage)
|
||||
* [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#url-format)
|
||||
* [Prometheus querying API usage](https://docs.victoriametrics.com/victoriametrics#prometheus-querying-api-usage)
|
||||
* [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#url-format)
|
||||
|
||||
### /api/v1/query
|
||||
|
||||
@@ -317,10 +318,10 @@ curl http://<vmselect>:8481/select/0/prometheus/api/v1/query -d 'query=vm_http_r
|
||||
|
||||
|
||||
Additional information:
|
||||
* [Instant queries](https://docs.victoriametrics.com/keyconcepts/#instant-query)
|
||||
* [Prometheus querying API usage](https://docs.victoriametrics.com/#prometheus-querying-api-usage)
|
||||
* [Query language](https://docs.victoriametrics.com/keyconcepts/#metricsql)
|
||||
* [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#url-format)
|
||||
* [Instant queries](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#instant-query)
|
||||
* [Prometheus querying API usage](https://docs.victoriametrics.com/victoriametrics#prometheus-querying-api-usage)
|
||||
* [Query language](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#metricsql)
|
||||
* [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#url-format)
|
||||
|
||||
### /api/v1/query_range
|
||||
|
||||
@@ -341,10 +342,10 @@ curl http://<vmselect>:8481/select/0/prometheus/api/v1/query_range -d 'query=sum
|
||||
|
||||
|
||||
Additional information:
|
||||
* [Range queries](https://docs.victoriametrics.com/keyconcepts/#range-query)
|
||||
* [Prometheus querying API usage](https://docs.victoriametrics.com/#prometheus-querying-api-usage)
|
||||
* [Query language](https://docs.victoriametrics.com/keyconcepts/#metricsql)
|
||||
* [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#url-format)
|
||||
* [Range queries](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#range-query)
|
||||
* [Prometheus querying API usage](https://docs.victoriametrics.com/victoriametrics#prometheus-querying-api-usage)
|
||||
* [Query language](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#metricsql)
|
||||
* [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#url-format)
|
||||
|
||||
### /api/v1/series
|
||||
|
||||
@@ -369,8 +370,8 @@ The specified `start..end` time range is rounded to UTC day granularity because
|
||||
|
||||
Additional information:
|
||||
* [Finding series by label matchers](https://prometheus.io/docs/prometheus/latest/querying/api/#finding-series-by-label-matchers)
|
||||
* [Prometheus querying API usage](https://docs.victoriametrics.com/#prometheus-querying-api-usage)
|
||||
* [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#url-format)
|
||||
* [Prometheus querying API usage](https://docs.victoriametrics.com/victoriametrics#prometheus-querying-api-usage)
|
||||
* [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#url-format)
|
||||
VictoriaMetrics accepts `limit` query arg for `/api/v1/series` handlers for limiting the number of returned entries. For example, the query to `/api/v1/series?limit=5` returns a sample of up to 5 series, while ignoring the rest. If the provided `limit` value exceeds the corresponding `-search.maxSeries` command-line flag values, then limits specified in the command-line flags are used.
|
||||
|
||||
### /api/v1/status/tsdb
|
||||
@@ -393,8 +394,8 @@ curl http://<vmselect>:8481/select/0/prometheus/api/v1/status/tsdb
|
||||
|
||||
Additional information:
|
||||
* [TSDB Stats](https://prometheus.io/docs/prometheus/latest/querying/api/#tsdb-stats)
|
||||
* [Prometheus querying API usage](https://docs.victoriametrics.com/#prometheus-querying-api-usage)
|
||||
* [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#url-format)
|
||||
* [Prometheus querying API usage](https://docs.victoriametrics.com/victoriametrics#prometheus-querying-api-usage)
|
||||
* [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#url-format)
|
||||
|
||||
### /datadog
|
||||
|
||||
@@ -470,8 +471,8 @@ echo '
|
||||
|
||||
Additional information:
|
||||
|
||||
* [How to send data from DataDog agent](https://docs.victoriametrics.com/#how-to-send-data-from-datadog-agent)
|
||||
* [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#url-format)
|
||||
* [How to send data from DataDog agent](https://docs.victoriametrics.com/victoriametrics#how-to-send-data-from-datadog-agent)
|
||||
* [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#url-format)
|
||||
|
||||
|
||||
### /datadog/api/v2/series
|
||||
@@ -538,8 +539,8 @@ echo '
|
||||
|
||||
Additional information:
|
||||
|
||||
* [How to send data from DataDog agent](https://docs.victoriametrics.com/#how-to-send-data-from-datadog-agent)
|
||||
* [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#url-format)
|
||||
* [How to send data from DataDog agent](https://docs.victoriametrics.com/victoriametrics#how-to-send-data-from-datadog-agent)
|
||||
* [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#url-format)
|
||||
|
||||
### /federate
|
||||
|
||||
@@ -561,9 +562,9 @@ curl http://<vmselect>:8481/select/0/prometheus/federate -d 'match[]=vm_http_req
|
||||
|
||||
Additional information:
|
||||
|
||||
* [Federation](https://docs.victoriametrics.com/#federation)
|
||||
* [Federation](https://docs.victoriametrics.com/victoriametrics#federation)
|
||||
* [Prometheus-compatible federation data](https://prometheus.io/docs/prometheus/latest/federation/#configuring-federation)
|
||||
* [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#url-format)
|
||||
* [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#url-format)
|
||||
|
||||
### /graphite/metrics/find
|
||||
|
||||
@@ -586,9 +587,9 @@ curl http://<vmselect>:8481/select/0/graphite/metrics/find -d 'query=vm_http_req
|
||||
Additional information:
|
||||
|
||||
* [Metrics find API in Graphite](https://graphite-api.readthedocs.io/en/latest/api.html#metrics-find)
|
||||
* [Graphite API in VictoriaMetrics](https://docs.victoriametrics.com/#graphite-api-usage)
|
||||
* [How to send Graphite data to VictoriaMetrics](https://docs.victoriametrics.com/single-server-victoriametrics/#how-to-send-data-from-graphite-compatible-agents-such-as-statsd)
|
||||
* [URL Format](https://docs.victoriametrics.com/cluster-victoriametrics/#url-format)
|
||||
* [Graphite API in VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics#graphite-api-usage)
|
||||
* [How to send Graphite data to VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#how-to-send-data-from-graphite-compatible-agents-such-as-statsd)
|
||||
* [URL Format](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#url-format)
|
||||
|
||||
### /influx/write
|
||||
|
||||
@@ -610,12 +611,12 @@ curl -d 'measurement,tag1=value1,tag2=value2 field1=123,field2=1.23' -X POST htt
|
||||
|
||||
Additional information:
|
||||
|
||||
* [How to send Influx data to VictoriaMetrics](https://docs.victoriametrics.com/#how-to-send-data-from-influxdb-compatible-agents-such-as-telegraf)
|
||||
* [URL Format](https://docs.victoriametrics.com/cluster-victoriametrics/#url-format)
|
||||
* [How to send Influx data to VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics#how-to-send-data-from-influxdb-compatible-agents-such-as-telegraf)
|
||||
* [URL Format](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#url-format)
|
||||
|
||||
### /internal/resetRollupResultCache
|
||||
|
||||
**Resets the response cache for previously served queries. It is recommended to invoke after [backfilling](https://docs.victoriametrics.com/#backfilling) procedure.**
|
||||
**Resets the response cache for previously served queries. It is recommended to invoke after [backfilling](https://docs.victoriametrics.com/victoriametrics#backfilling) procedure.**
|
||||
|
||||
Single-node VictoriaMetrics:
|
||||
|
||||
@@ -674,7 +675,7 @@ curl -H 'Content-Type: application/json' -d '[{"metric":"foo","value":45.34},{"m
|
||||
Additional information:
|
||||
|
||||
* [OpenTSDB http put API](http://opentsdb.net/docs/build/html/api_http/put.html)
|
||||
* [How to send data OpenTSDB data to VictoriaMetrics](https://docs.victoriametrics.com/single-server-victoriametrics/#how-to-send-data-from-opentsdb-compatible-agents)
|
||||
* [How to send data OpenTSDB data to VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#how-to-send-data-from-opentsdb-compatible-agents)
|
||||
|
||||
#### How to send Graphite data to VictoriaMetrics
|
||||
|
||||
@@ -696,5 +697,5 @@ echo "foo.bar.baz;tag1=value1;tag2=value2 123 `date +%s`" | nc -N http://<vminse
|
||||
|
||||
Additional information:
|
||||
|
||||
* [How to send Graphite data to VictoriaMetrics](https://docs.victoriametrics.com/single-server-victoriametrics/#how-to-send-data-from-graphite-compatible-agents-such-as-statsd)
|
||||
* [Multitenancy in cluster version of VictoriaMetrics](https://docs.victoriametrics.com/cluster-victoriametrics/#multitenancy)
|
||||
* [How to send Graphite data to VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#how-to-send-data-from-graphite-compatible-agents-such-as-statsd)
|
||||
* [Multitenancy in cluster version of VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#multitenancy)
|
||||
|
||||
@@ -1,6 +1,5 @@
|
||||
---
|
||||
weight: 40
|
||||
editLink: https://github.com/VictoriaMetrics/victoriametrics-datasource/blob/main/README.md
|
||||
title: Grafana datasource
|
||||
menu:
|
||||
docs:
|
||||
@@ -8,6 +7,7 @@ menu:
|
||||
parent: victoriametrics
|
||||
weight: 40
|
||||
aliases:
|
||||
- /victoriametrics-datasource/
|
||||
- /victoriametrics-datasource.html
|
||||
# old doc name before it was renamed
|
||||
- /grafana-datasource/
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user