Compare commits

...

30 Commits

Author SHA1 Message Date
Artem Fetishev
b70e2ddd66 fix time ranges, use 10k series
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
2026-08-21 16:14:42 +02:00
Artem Fetishev
a2020f2d38 add 15 and 30 day trs
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
2026-08-21 13:42:55 +02:00
Artem Fetishev
a80619a7d0 more benchmark fixes
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
2026-08-21 11:02:43 +02:00
Artem Fetishev
de7695b011 subtract 1ms from tr.MaxTimestamp so that the search op does not search the next interval
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
2026-08-21 11:02:43 +02:00
Artem Fetishev
032d03ddb8 fix cases with series repeated on each sub-interval
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
2026-08-21 11:02:42 +02:00
Artem Fetishev
ac5e435675 lib/storage: another variable time range search benchmark
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
2026-08-21 11:02:39 +02:00
Max Kotliar
420f18b219 docs: follow-up fix on 364b3b6823 2026-08-21 09:36:52 +03:00
Vandit Singh
364b3b6823 lib/flagutil: show how dynamic flag defaults are calculated in -help output
This commit adds new methods into  `lib/flagutil` package. Which allows to add a hint for dynamic flags. It adds a new -help formatting formula - ``(default <number> = <text>)`.

For example:

```
-maxConcurrentInserts int
     The maximum number of concurrent insert requests. ... (default 16 = 2*cgroup.AvailableCPUs())
```

 This commit also updates `docs/Makefile` with a simpler sed regex.

Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/9680
2026-08-20 21:38:16 +02:00
Zasda Yusuf Mikail
400f01a114 lib/appmetrics: expose unclean shutdown metric
This commit adds new metric - `vm_app_prev_shutdown_unclean`. Which indicate wether previous
shutdown of application was not clean ( OOM, kill -9, etc).

 In order to achieve it, application creates new file - `.vm_app_running` at the root of persistent directory.
And removes it during graceful shutdown. If file exists at the start of application, it indicates that application was not
stopped gracefully.

This commit adds the `vm_app_prev_shutdown_unclean` metric, which indicates
whether the application was shut down uncleanly (e.g. OOM, kill -9, etc.).
 
 Now application creates a `.vm_app_running` file at the
root of the storage directory when it starts and removes it during a
graceful shutdown. If the file exists when the application starts, it
indicates that the previous shutdown was not graceful.

Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8443
2026-08-20 15:28:06 +02:00
angelo
00abfbf043 docs/integrations: update outdated Grafana UI screenshot (#11432)
Modifies the [Grafana integration
docs](https://docs.victoriametrics.com/victoriametrics/integrations/grafana/index.html#prometheus-datasource)
to update the image and text for setting up the Prometheus datasource,
reflecting changes in the Grafana UI.

Before, the docs suggested to navigate to the "Type and version" section
which no longer exists in the latest Grafana build:

<img width="400" alt="image"
src="https://github.com/user-attachments/assets/1833790d-25be-4675-bb49-419b5aa96462"
/>

This is the UI in the latest Grafana build which has the Prometheus type
and Prometheus version field in a "Performance" section instead:

<img width="400" alt="image"
src="https://github.com/user-attachments/assets/506220de-9962-4ce0-99cb-488d47937e8f"
/>

I was trying to integrate Grafana and VictoriaMetrics and I was somewhat
tripped up by this mixup in the docs so I'm making a PR to fix it :)

### Checklist

- [x] My change adheres to the [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2026-08-20 10:29:16 +02:00
hagen1778
2f2c6bcdec docs/vmalert: update example of debug mode
In updated example we also display the new added `series_fetched`
field as part of debug message. This field can be useful
for identifying whether series where filtered before returning.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2026-08-20 10:27:17 +02:00
hagen1778
636a50bb9c readme: fix brokeb release badge
For some reason, `sort=semver` was breaking the badge rendering.
It works without it, but it also worked before with it. If badge
will continue breaking in future - let's remove it.

Also removed link from the badge url since we already have it in
wrapping markdown.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2026-08-20 10:17:48 +02:00
Evgeny
a441d7e94e app/vmalert-tool: reuse connections to -remoteWrite.url
DebugClient sends every series in a separate request, but it left
MaxIdleConnsPerHost at the two connections of http.DefaultTransport.
Under concurrent rule evaluation most requests could not find an idle
connection and had to dial a new one, leaving many sockets in TIME_WAIT
state.

Set MaxIdleConnsPerHost from the new -remoteWrite.maxIdleConnections
flag, mirroring -datasource.maxIdleConnections, and apply the already
existing -remoteWrite.idleConnTimeout to the transport.
In my tests 640 concurrent pushes now open 17 connections instead of
227.

Related PR https://github.com/VictoriaMetrics/VictoriaMetrics/pull/11387/
2026-08-19 13:38:04 +02:00
Hui Wang
ac1d77e3de docs/changelog: add missing update note on v1.129.0 (#11430)
The breaking change was introduced with https://github.com/VictoriaMetrics/VictoriaMetrics/issues/9779.
2026-08-19 12:05:23 +03:00
Max Kotliar
d149d4c91d app/vmselect: fix panic in sort_by_label_numeric() for label values with 309+ digit numbers (#11423)
`sort_by_label_numeric()` and `sort_by_label_numeric_desc()` call
`mustParseNum()`, which panics when `strconv.ParseFloat` returns
`ErrRange` for numbers with 309 or more digits.

Fix it by treating `ErrRange` as `Inf`, which is semantically correct
for sorting purposes.

Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/security/advisories/GHSA-9g98-8jgr-x2vv
PR https://github.com/VictoriaMetrics/VictoriaMetrics/pull/11423
2026-08-18 18:55:21 +03:00
Max Kotliar
4946a403eb lib/protoparser: fix infinite loop on incomplete varint in Firehose ingestion endpoint (#11424)
When `binary.Uvarint` returns `(0, 0)` for an incomplete varint (e.g. a
single `0x80` byte), the parser loop made no progress and spun forever.

Fix it by treating `varIntLength <= 0` as an error.

Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/security/advisories/GHSA-89v2-864p-v3xc
PR https://github.com/VictoriaMetrics/VictoriaMetrics/pull/11424

---------

Signed-off-by: Nikolay <nik@victoriametrics.com>
Co-authored-by: Nikolay <nik@victoriametrics.com>
2026-08-18 18:00:46 +03:00
Yury Moladau
a491b930e4 app/vmui: show selected time zone offset in header controls (#11332)
### Describe Your Changes

Make the selected timezone offset more visible and provide quicker access to timezone settings.

This makes the timezone immediately visible at the top when sharing metrics with others, which is especially useful during screen sharing. It also helps avoid ambiguity when sharing screenshots, where the timezone context may otherwise be unclear.

### Changes

- display the selected UTC offset in the header
- make the timezone shown in the date picker clickable
- update the mobile settings menu layout and styling

### Screenshots

<img width="623" height="54" alt="image"
src="https://github.com/user-attachments/assets/13500485-f038-4777-a99e-2ea8516fe024"
/>

<hr/>

| Before | After |
|---|---|
| <img width="404" height="430" alt="image"
src="https://github.com/user-attachments/assets/69bd9bbf-74f1-4cef-b423-094a2bbc77e8"
/> | <img width="404" height="430" alt="image"
src="https://github.com/user-attachments/assets/fde0e71f-4b7c-4a82-bca3-e43d4501daca"
/> |

---------

Signed-off-by: Yury Molodov <yurymolodov@gmail.com>
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Max Kotliar <mkotlyar@victoriametrics.com>
2026-08-18 17:48:52 +03:00
Pablo (Tomas) Fernandez
9caf74fbb2 docs: Update guide "VictoriaMetrics Multi-Regional Setup: Dedicated Monitoring" (#11334)
Updates the [VictoriaMetrics Multi-Regional Setup: Dedicated
Monitoring](https://docs.victoriametrics.com/guides/multi-regional-setup-dedicated-regions/)
guide.

* New diagrams using the official template
* Expanded sections with examples and config snippets

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2026-08-18 14:51:50 +02:00
Roman Khavronenko
5e5ea9283e docs: explain new HA option with -vmselectAddr (#11410)
With single-node support of `-vmselectAddr` users can build an HA
topology using vmselect, that wasn't available before.
The new option is more preferable for data completeness but is more
resource-costly.

Adding docs how this can be achieved.

Related to https://github.com/VictoriaMetrics/VictoriaMetrics/pull/11334

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Signed-off-by: Roman Khavronenko <hagen1778@gmail.com>
Signed-off-by: Pablo (Tomas) Fernandez <46322567+TomFern@users.noreply.github.com>
Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>
Co-authored-by: Pablo Fernandez <46322567+TomFern@users.noreply.github.com>
2026-08-18 14:51:20 +02:00
Cuong Le
7ba71f7f65 app/vmalert: fix a data race on the compress buffer after failed remote write requests
When the Go http client fails to send a request, the request body buffer
shouldn't be reused: the http transport may still be reading the body in
a separate goroutine even after the request returns (see
https://pkg.go.dev/net/http#RoundTripper). Reusing the buffer while that
goroutine is still running creates a data race.

This PR returns the compress buffer to the pool only when no send
attempt has failed. Otherwise the buffer is left for GC to collect.

A similar problem in VictoriaLogs was detected during the review of
https://github.com/VictoriaMetrics/VictoriaLogs/pull/1616.
2026-08-18 10:05:47 +02:00
Max Kotliar
f5b7c8795a dashboards: refine version annotation (#11397)
- Add the version annotation to the missing components (vmbackupmanager,
vmauth).
- Fix the incorrect placeholder: used `{{version}}`, while query
returned `{{short_version}}`.
- Update the query to include both `short_version` and `version`,
preferring short_version when available. Same as in
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/11047.

Query used:
```
sum by(version) (
    label_replace(vm_app_version{job=~"$job", instance=~"$instance", short_version!=""}, "version", "$1", "short_version", "(.*)")
    OR
    vm_app_version{job=~"$job", instance=~"$instance", short_version=""}
) 
unless 
(
    sum by(version) (
        label_replace(vm_app_version{job=~"$job", instance=~"$instance", short_version!=""}, "version", "$1", "short_version", "(.*)")
        OR
        vm_app_version{job=~"$job", instance=~"$instance", short_version=""}
    ) offset $__interval
)
```

Previously, custom build versions were detected, but the version itself
wasn’t displayed:
<img width="799" height="389" alt="Screenshot 2026-08-12 at 21 21 12"
src="https://github.com/user-attachments/assets/bbfb0342-3f7a-4b2f-a8b8-4bbd04a9e1ab"
/>
2026-08-17 16:29:35 +03:00
f41gh7
28e138c0ae docs: update flags with actual v1.150.0 binaries
Signed-off-by: f41gh7 <nik@victoriametrics.com>
2026-08-17 11:56:34 +02:00
f41gh7
c28aa8b7fc docs: bump version to v1.150.0
Signed-off-by: f41gh7 <nik@victoriametrics.com>
2026-08-17 11:54:15 +02:00
f41gh7
c4566e7706 deplyoment/docker: bump version to v1.150.0
Signed-off-by: f41gh7 <nik@victoriametrics.com>
2026-08-17 11:53:48 +02:00
f41gh7
16c8dd18ed docs: forward port LTS v1.136.16 changelog to upstream
Signed-off-by: f41gh7 <nik@victoriametrics.com>
2026-08-17 11:52:24 +02:00
f41gh7
4ff25e5a28 docs: forward port LTS v1.148.2 changelog to upstream
Signed-off-by: f41gh7 <nik@victoriametrics.com>
2026-08-17 11:51:48 +02:00
Artem Fetishev
14d6461aee lib/storage: fix search benchmarks (#11399)
Change benchmark time ranges:

- Several day timeranges: 1d, 2d, 4d, 8d, 16d, 32d, 64d to see how x2 time range increase affects the retrieval of a fixed number of metrics (100k).
- Several month time ranges: 1m, 2m, 4m. Same as above, but involves global index.

Related to #11388.

---------

Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
2026-08-17 07:50:48 +02:00
MarkHe1222
4edd8aa257 docs: fix grammar issues in README.md (#11409)
Signed-off-by: MarkHe1222 <20268851+MarkHe1222@users.noreply.github.com>
Co-authored-by: MarkHe1222 <20268851+MarkHe1222@users.noreply.github.com>
2026-08-15 22:40:22 +02:00
Pablo (Tomas) Fernandez
49b9c78460 docs: Add meta descriptions for SEO optimization (#11402)
This PR add meta descriptions to every page in the docs in this
repository. Right now, we have one general meta description for every
page. This PR adds page-specific descriptions.

Meta descriptions are not likely to improve rankings on search engines
but can improve CTRs. They can also be used to keep LLMs.txt updated as
new pages are added (by adding an automation later).

The descriptions were taken from LLMs.txt, so they were already reviewed
in https://github.com/VictoriaMetrics/vmdocs/pull/251

I only tweaked those that needed trimming to fit into the 160 SEO
character limit. We also revised them with our SEO specialist
(Jonathan). So I think it's a good staring point.

I'm going to open a PR for every repository that contributes to the docs
and add descriptions so eventually every page in the docs has a
dedicated meta descriptions.
2026-08-15 22:39:22 +02:00
Phuong Le
d712941d8c docs: adds VictoriaMetrics blog posts to documentation (#11249) 2026-08-15 22:38:48 +02:00
188 changed files with 44584 additions and 266 deletions

View File

@@ -1,6 +1,6 @@
# VictoriaMetrics
[![Latest Release](https://img.shields.io/github/v/release/VictoriaMetrics/VictoriaMetrics?sort=semver&label=&filter=!*-victorialogs&logo=github&labelColor=gray&color=gray&link=https%3A%2F%2Fgithub.com%2FVictoriaMetrics%2FVictoriaMetrics%2Freleases%2Flatest)](https://github.com/VictoriaMetrics/VictoriaMetrics/releases)
[![Latest Release](https://img.shields.io/github/v/release/VictoriaMetrics/VictoriaMetrics?logo=github&labelColor=gray&color=gray&label=Release)](https://github.com/VictoriaMetrics/VictoriaMetrics/releases)
[![Docker Pulls](https://img.shields.io/docker/pulls/victoriametrics/victoria-metrics?label=&logo=docker&logoColor=white&labelColor=2496ED&color=2496ED&link=https%3A%2F%2Fhub.docker.com%2Fr%2Fvictoriametrics%2Fvictoria-metrics)](https://hub.docker.com/u/victoriametrics)
[![Build Status](https://github.com/VictoriaMetrics/VictoriaMetrics/actions/workflows/build.yml/badge.svg?branch=master&link=https%3A%2F%2Fgithub.com%2FVictoriaMetrics%2FVictoriaMetrics%2Factions)](https://github.com/VictoriaMetrics/VictoriaMetrics/actions/workflows/build.yml)
[![License](https://img.shields.io/github/license/VictoriaMetrics/VictoriaMetrics?labelColor=green&label=&link=https%3A%2F%2Fgithub.com%2FVictoriaMetrics%2FVictoriaMetrics%2Fblob%2Fmaster%2FLICENSE)](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/LICENSE)
@@ -41,8 +41,8 @@ VictoriaMetrics is optimized for timeseries data, even when old time series are
* **Ideal for big data**: Works well with large amounts of time series data from APM, Kubernetes, IoT sensors, connected cars, industrial telemetry, financial data and various [Enterprise workloads](https://docs.victoriametrics.com/victoriametrics/enterprise/).
* **Query language**: Supports both PromQL and the more performant MetricsQL.
* **Easy to setup**: No dependencies, single [small binary](https://medium.com/@valyala/stripping-dependency-bloat-in-victoriametrics-docker-image-983fb5912b0d), configuration through command-line flags, but the default is also fine-tuned; backup and restore with [instant snapshots](https://medium.com/@valyala/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282).
* **Global query view**: Multiple Prometheus instances or any other data sources may ingest data into VictoriaMetrics and queried via a single query.
* **Various Protocols**: Support metric scraping, ingestion and backfilling in various protocol.
* **Global query view**: Multiple Prometheus instances or any other data sources may ingest data into VictoriaMetrics and be queried via a single query.
* **Various Protocols**: Support metric scraping, ingestion and backfilling in various protocols.
* [Prometheus exporters](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#how-to-scrape-prometheus-exporters-such-as-node-exporter), [Prometheus remote write API](https://docs.victoriametrics.com/victoriametrics/integrations/prometheus/), [Prometheus exposition format](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#how-to-import-data-in-prometheus-exposition-format).
* [InfluxDB line protocol](https://docs.victoriametrics.com/victoriametrics/integrations/influxdb/) over HTTP, TCP and UDP.
* [Graphite plaintext protocol](https://docs.victoriametrics.com/victoriametrics/integrations/graphite/#ingesting) with [tags](https://graphite.readthedocs.io/en/latest/tags.html#carbon).
@@ -78,7 +78,7 @@ We strictly apply security measures in everything we do. VictoriaMetrics has ach
Some good benchmarks VictoriaMetrics achieved:
* **Minimal memory footprint**: handling millions of unique timeseries with [10x less RAM](https://medium.com/@valyala/insert-benchmarks-with-inch-influxdb-vs-victoriametrics-e31a41ae2893) than InfluxDB, up to [7x less RAM](https://valyala.medium.com/prometheus-vs-victoriametrics-benchmark-on-node-exporter-metrics-4ca29c75590f) than Prometheus, Thanos or Cortex.
* **Highly scalable and performance** for [data ingestion](https://medium.com/@valyala/high-cardinality-tsdb-benchmarks-victoriametrics-vs-timescaledb-vs-influxdb-13e6ee64dd6b) and [querying](https://medium.com/@valyala/when-size-matters-benchmarking-victoriametrics-vs-timescale-and-influxdb-6035811952d4), [20x outperforms](https://medium.com/@valyala/insert-benchmarks-with-inch-influxdb-vs-victoriametrics-e31a41ae2893) InfluxDB and TimescaleDB.
* **Highly scalable and performant** for [data ingestion](https://medium.com/@valyala/high-cardinality-tsdb-benchmarks-victoriametrics-vs-timescaledb-vs-influxdb-13e6ee64dd6b) and [querying](https://medium.com/@valyala/when-size-matters-benchmarking-victoriametrics-vs-timescale-and-influxdb-6035811952d4), [20x outperforms](https://medium.com/@valyala/insert-benchmarks-with-inch-influxdb-vs-victoriametrics-e31a41ae2893) InfluxDB and TimescaleDB.
* **High data compression**: [70x more data points](https://medium.com/@valyala/when-size-matters-benchmarking-victoriametrics-vs-timescale-and-influxdb-6035811952d4) may be stored into limited storage than TimescaleDB, [7x less storage](https://valyala.medium.com/prometheus-vs-victoriametrics-benchmark-on-node-exporter-metrics-4ca29c75590f) space is required than Prometheus, Thanos or Cortex.
* **Reducing storage costs**: [10x more effective](https://docs.victoriametrics.com/victoriametrics/casestudies/#grammarly) than Graphite according to the Grammarly case study.
* **A single-node VictoriaMetrics** can replace medium-sized clusters built with competing solutions such as Thanos, M3DB, Cortex, InfluxDB or TimescaleDB. See [VictoriaMetrics vs Thanos](https://medium.com/@valyala/comparing-thanos-to-victoriametrics-cluster-b193bea1683), [Measuring vertical scalability](https://medium.com/@valyala/measuring-vertical-scalability-for-time-series-databases-in-google-cloud-92550d78d8ae), [Remote write storage wars - PromCon 2019](https://promcon.io/2019-munich/talks/remote-write-storage-wars/).
@@ -86,7 +86,7 @@ Some good benchmarks VictoriaMetrics achieved:
## Community and contributions
Feel free asking any questions regarding VictoriaMetrics:
Feel free to ask any questions regarding VictoriaMetrics:
* [Slack Inviter](https://slack.victoriametrics.com/) and [Slack channel](https://victoriametrics.slack.com/)
* [X (Twitter)](https://x.com/VictoriaMetrics/)

View File

@@ -13,6 +13,7 @@ import (
"github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect"
"github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect/promql"
"github.com/VictoriaMetrics/VictoriaMetrics/app/vmstorage"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/appmetrics"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/buildinfo"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/cgroup"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/envflag"
@@ -34,9 +35,10 @@ var (
"This can be changed with -promscrape.config.strictParse=false command-line flag")
maxIngestionRate = flag.Int("maxIngestionRate", 0, "The maximum number of samples vmsingle can receive per second. Data ingestion is paused when the limit is exceeded. "+
"By default there are no limits on samples ingestion rate.")
vmselectMaxConcurrentRequests = flag.Int("search.maxConcurrentRequests", getDefaultMaxConcurrentRequests(), "The maximum number of concurrent search requests. "+
"It shouldn't be high, since a single request can saturate all the CPU cores, while many concurrently executed requests may require high amounts of memory. "+
"See also -search.maxQueueDuration and -search.maxMemoryPerQuery")
vmselectMaxConcurrentRequests = flagutil.NewIntWithDynamicDefault("search.maxConcurrentRequests", getDefaultMaxConcurrentRequests(), "vmselect.getDefaultMaxConcurrentRequests()",
"The maximum number of concurrent search requests. "+
"It shouldn't be high, since a single request can saturate all the CPU cores, while many concurrently executed requests may require high amounts of memory. "+
"See also -search.maxQueueDuration and -search.maxMemoryPerQuery")
vmselectMaxQueueDuration = flag.Duration("search.maxQueueDuration", 10*time.Second, "The maximum time the request waits for execution when -search.maxConcurrentRequests "+
"limit is reached; see also -search.maxQueryDuration")
)
@@ -90,7 +92,9 @@ func main() {
}
logger.Infof("starting VictoriaMetrics at %q...", listenAddrs)
startTime := time.Now()
vmstorage.Init(*vmselectMaxConcurrentRequests, *vmselectMaxQueueDuration, promql.ResetRollupResultCacheIfNeeded)
appmetrics.MustCreateUncleanShutdownMarker(vmstorage.DataPath())
vmselect.Init(*vmselectMaxConcurrentRequests, *vmselectMaxQueueDuration)
vminsertcommon.StartIngestionRateLimiter(*maxIngestionRate)
vminsert.Init()
@@ -120,6 +124,7 @@ func main() {
vmstorage.Stop()
vmselect.Stop()
appmetrics.MustRemoveUncleanShutdownMarker(vmstorage.DataPath())
logger.Infof("the VictoriaMetrics has been stopped in %.3f seconds", time.Since(startTime).Seconds())
}

View File

@@ -15,6 +15,7 @@ import (
"github.com/VictoriaMetrics/metrics"
"github.com/cespare/xxhash/v2"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/appmetrics"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/auth"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/bloomfilter"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/cgroup"
@@ -62,9 +63,10 @@ var (
"See also -remoteWrite.maxDiskUsagePerURL and -remoteWrite.disableOnDiskQueue")
keepDanglingQueues = flag.Bool("remoteWrite.keepDanglingQueues", false, "Keep persistent queues contents at -remoteWrite.tmpDataPath in case there are no matching -remoteWrite.url. "+
"Useful when -remoteWrite.url is changed temporarily and persistent queue files will be needed later on.")
queues = flagutil.NewArrayInt("remoteWrite.queues", cgroup.AvailableCPUs()*2, "The number of concurrent queues to each -remoteWrite.url. Set more queues if default number of queues "+
"isn't enough for sending high volume of collected data to remote storage. "+
"Default value depends on the number of available CPU cores. It should work fine in most cases since it minimizes resource usage")
queues = flagutil.NewArrayIntWithDynamicDefault("remoteWrite.queues", cgroup.AvailableCPUs()*2, "2*cgroup.AvailableCPUs()",
"The number of concurrent queues to each -remoteWrite.url. Set more queues if default number of queues "+
"isn't enough for sending high volume of collected data to remote storage. "+
"Default value depends on the number of available CPU cores. It should work fine in most cases since it minimizes resource usage")
inmemoryQueues = flagutil.NewArrayInt("remoteWrite.inmemoryQueues", 0, "The number of additional workers per each -remoteWrite.url, which send only recently ingested data from the in-memory queue, "+
"while the file-based queue at -remoteWrite.tmpDataPath is drained by workers configured via -remoteWrite.queues. "+
"This reduces delivery lag for fresh samples when the file-based queue contains a backlog accumulated during remote storage outages.")
@@ -233,6 +235,7 @@ func Init() {
initStreamAggrConfigGlobal()
initRemoteWriteCtxs(*remoteWriteURLs)
appmetrics.MustCreateUncleanShutdownMarker(*tmpDataPath)
disableOnDiskQueues := []bool(*disableOnDiskQueue)
disableOnDiskQueueAny = slices.Contains(disableOnDiskQueues, true)
@@ -391,6 +394,8 @@ func Stop() {
if sl := dailySeriesLimiter; sl != nil {
sl.MustStop()
}
appmetrics.MustRemoveUncleanShutdownMarker(*tmpDataPath)
}
// PushDropSamplesOnFailure pushes wr to the configured remote storage systems set via -remoteWrite.url

View File

@@ -284,7 +284,15 @@ func (c *Client) flush(ctx context.Context, wr *prompb.WriteRequest) {
bb := writeRequestBufPool.Get()
bb.B = wr.MarshalProtobuf(bb.B[:0])
zb := compressBufPool.Get()
defer compressBufPool.Put(zb)
// A failed send may leave the http transport still reading zb.B in a separate goroutine
// even after send returns, so zb is returned to the pool only if no send attempt has failed.
// See https://pkg.go.dev/net/http#RoundTripper
sendFailed := false
defer func() {
if !sendFailed {
compressBufPool.Put(zb)
}
}()
if c.isVMRemoteWrite.Load() {
zb.B = zstd.CompressLevel(zb.B[:0], bb.B, 0)
} else {
@@ -303,10 +311,13 @@ func (c *Client) flush(ctx context.Context, wr *prompb.WriteRequest) {
L:
for {
err := c.send(ctx, zb.B)
if err != nil && (errors.Is(err, io.EOF) || netutil.IsTrivialNetworkError(err)) {
// Something in the middle between client and destination might be closing
// the connection. So we do a one more attempt in hope request will succeed.
err = c.send(ctx, zb.B)
if err != nil {
sendFailed = true
if errors.Is(err, io.EOF) || netutil.IsTrivialNetworkError(err) {
// Something in the middle between client and destination might be closing
// the connection. So we do a one more attempt in hope request will succeed.
err = c.send(ctx, zb.B)
}
}
if err == nil {
sentRows.Add(len(wr.Timeseries))

View File

@@ -36,6 +36,13 @@ func NewDebugClient() (*DebugClient, error) {
if err != nil {
return nil, fmt.Errorf("failed to create transport for -remoteWrite.url=%q: %w", *addr, err)
}
tr.IdleConnTimeout = *idleConnectionTimeout
// DebugClient sends every series in a separate request, so it needs more idle
// connections than the two http.DefaultTransport keeps per host.
tr.MaxIdleConnsPerHost = *maxIdleConnections
if tr.MaxIdleConns != 0 && tr.MaxIdleConns < tr.MaxIdleConnsPerHost {
tr.MaxIdleConns = tr.MaxIdleConnsPerHost
}
c := &DebugClient{
c: &http.Client{
Timeout: *sendTimeout,

View File

@@ -0,0 +1,45 @@
package remotewrite
import (
"net/http"
"testing"
)
// TestDebugClient_IdleConns makes sure DebugClient keeps enough idle connections
// to -remoteWrite.url. Every series is pushed in a separate request, so with the
// two idle connections per host of http.DefaultTransport most of the concurrent
// requests would open a new connection and leave a socket in TIME_WAIT state.
func TestDebugClient_IdleConns(t *testing.T) {
f := func(maxIdle int) {
t.Helper()
oldAddr, oldMaxIdle := *addr, *maxIdleConnections
*addr, *maxIdleConnections = "http://localhost:8428", maxIdle
defer func() {
*addr, *maxIdleConnections = oldAddr, oldMaxIdle
}()
client, err := NewDebugClient()
if err != nil {
t.Fatalf("failed to create debug client: %s", err)
}
tr, ok := client.c.Transport.(*http.Transport)
if !ok {
t.Fatalf("unexpected transport type %T", client.c.Transport)
}
if tr.MaxIdleConnsPerHost != maxIdle {
t.Fatalf("unexpected MaxIdleConnsPerHost; got %d; want %d", tr.MaxIdleConnsPerHost, maxIdle)
}
if tr.MaxIdleConns != 0 && tr.MaxIdleConns < maxIdle {
t.Fatalf("MaxIdleConns=%d is lower than MaxIdleConnsPerHost=%d", tr.MaxIdleConns, maxIdle)
}
if tr.IdleConnTimeout != *idleConnectionTimeout {
t.Fatalf("unexpected IdleConnTimeout; got %s; want %s", tr.IdleConnTimeout, *idleConnectionTimeout)
}
}
f(100)
// the number of idle connections must be raised together with the total limit
f(1000)
}

View File

@@ -34,10 +34,12 @@ var (
bearerTokenFile = flag.String("remoteWrite.bearerTokenFile", "", "Optional path to bearer token file to use for -remoteWrite.url.")
idleConnectionTimeout = flag.Duration("remoteWrite.idleConnTimeout", 50*time.Second, `Defines a duration for idle (keep-alive connections) to exist. Consider settings this value less to the value of "-http.idleConnTimeout". It must prevent possible "write: broken pipe" and "read: connection reset by peer" errors.`)
maxIdleConnections = flag.Int("remoteWrite.maxIdleConnections", 100, `Defines the number of idle (keep-alive connections) to -remoteWrite.url for the vmalert-tool debug writer, which sends every series in a separate request. Too low a value may result in a high number of sockets in TIME_WAIT state.`)
maxQueueSize = flag.Int("remoteWrite.maxQueueSize", defaultMaxQueueSize, "Defines the max number of pending datapoints to remote write endpoint")
maxBatchSize = flag.Int("remoteWrite.maxBatchSize", defaultMaxBatchSize, "Defines max number of timeseries to be flushed at once")
concurrency = flag.Int("remoteWrite.concurrency", defaultConcurrency, "Defines number of writers for concurrent writing into remote write endpoint. Default value depends on the number of available CPU cores.")
maxQueueSize = flag.Int("remoteWrite.maxQueueSize", defaultMaxQueueSize, "Defines the max number of pending datapoints to remote write endpoint")
maxBatchSize = flag.Int("remoteWrite.maxBatchSize", defaultMaxBatchSize, "Defines max number of timeseries to be flushed at once")
concurrency = flagutil.NewIntWithDynamicDefault("remoteWrite.concurrency", defaultConcurrency, "2*cgroup.AvailableCPUs()",
"Defines number of writers for concurrent writing into remote write endpoint. Default value depends on the number of available CPU cores.")
flushInterval = flag.Duration("remoteWrite.flushInterval", defaultFlushInterval, "Defines interval of flushes to remote write endpoint")
tlsInsecureSkipVerify = flag.Bool("remoteWrite.tlsInsecureSkipVerify", false, "Whether to skip tls verification when connecting to -remoteWrite.url")

View File

@@ -20,6 +20,7 @@ import (
"github.com/VictoriaMetrics/VictoriaMetrics/lib/bytesutil"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/cgroup"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/fasttime"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/flagutil"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/querytracer"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/storage"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/storage/metricnamestats"
@@ -30,11 +31,12 @@ var (
maxSamplesPerSeries = flag.Int("search.maxSamplesPerSeries", 30e6, "The maximum number of raw samples a single query can scan per each time series. This option allows limiting memory usage")
maxSamplesPerQuery = flag.Int("search.maxSamplesPerQuery", 1e9, "The maximum number of raw samples a single query can process across all time series. "+
"This protects from heavy queries, which select unexpectedly high number of raw samples. See also -search.maxSamplesPerSeries")
maxWorkersPerQuery = flag.Int("search.maxWorkersPerQuery", defaultMaxWorkersPerQuery, "The maximum number of CPU cores a single query can use. "+
"The default value should work good for most cases. "+
"The flag can be set to lower values for improving performance of big number of concurrently executed queries. "+
"The flag can be set to bigger values for improving performance of heavy queries, which scan big number of time series (>10K) and/or big number of samples (>100M). "+
"There is no sense in setting this flag to values bigger than the number of CPU cores available on the system")
maxWorkersPerQuery = flagutil.NewIntWithDynamicDefault("search.maxWorkersPerQuery", defaultMaxWorkersPerQuery, "netstorage.defaultMaxWorkersPerQuery()",
"The maximum number of CPU cores a single query can use. "+
"The default value should work good for most cases. "+
"The flag can be set to lower values for improving performance of big number of concurrently executed queries. "+
"The flag can be set to bigger values for improving performance of heavy queries, which scan big number of time series (>10K) and/or big number of samples (>100M). "+
"There is no sense in setting this flag to values bigger than the number of CPU cores available on the system")
)
// Result is a single timeseries result.

View File

@@ -2,6 +2,7 @@ package promql
import (
"bytes"
"errors"
"fmt"
"math"
"math/rand"
@@ -2566,6 +2567,11 @@ func isDecimalChar(ch byte) bool {
func mustParseNum(s string) float64 {
f, err := strconv.ParseFloat(s, 64)
if err != nil {
if errors.Is(err, strconv.ErrRange) {
// The number is too large to fit into float64; ParseFloat returns ±Inf in this case.
// Use ±Inf for sorting purposes — it is semantically correct.
return f
}
logger.Panicf("BUG: unexpected error when parsing the number %q: %s", s, err)
}
return f

View File

@@ -385,4 +385,12 @@ func TestNumericLess(t *testing.T) {
f("12.9", "12.56", false)
f("12.56", "12.9", true)
f("12.9", "12.9", false)
// 309-digit numbers - must not panic (regression test for GHSA-9g98-8jgr-x2vv)
big := strings.Repeat("9", 309)
f(big, "1", false)
f("1", big, true)
f(big, big, false)
f("-"+big, big, true)
f(big, "-"+big, false)
}

View File

@@ -1,4 +1,4 @@
import { FC, useRef } from "preact/compat";
import { forwardRef, useImperativeHandle, useRef } from "preact/compat";
import ServerConfigurator from "./ServerConfigurator/ServerConfigurator";
import { ArrowDownIcon, SettingsIcon } from "../../Main/Icons";
import Button from "../../Main/Button/Button";
@@ -21,7 +21,11 @@ export interface ChildComponentHandle {
handleApply: () => void;
}
const GlobalSettings: FC = () => {
export interface GlobalSettingsHandle {
open: () => void;
}
const GlobalSettings = forwardRef<GlobalSettingsHandle>((_, ref) => {
const { isMobile } = useDeviceDetect();
const appModeEnable = getAppModeEnable();
@@ -74,6 +78,10 @@ const GlobalSettings: FC = () => {
},
].filter(control => control.show);
useImperativeHandle(ref, () => ({
open: handleOpen,
}));
return <>
{isMobile ? (
<div
@@ -139,6 +147,6 @@ const GlobalSettings: FC = () => {
</Modal>
)}
</>;
};
});
export default GlobalSettings;

View File

@@ -0,0 +1,52 @@
import { FC } from "preact/compat";
import Button from "../../../Main/Button/Button";
import { useTimeState } from "../../../../state/time/TimeStateContext";
import useDeviceDetect from "../../../../hooks/useDeviceDetect";
import { getUTCByTimezone } from "../../../../utils/time";
import { useMemo } from "react";
import { ArrowDownIcon, PlanetIcon } from "../../../Main/Icons";
type Props = {
onOpenSettings?: () => void;
}
const TimeZonePreview: FC<Props> = ({ onOpenSettings }) => {
const { isMobile } = useDeviceDetect();
const { timezone } = useTimeState();
const utcOffset = useMemo(() => getUTCByTimezone(timezone), [timezone]);
const handleOpenSettings = () => {
onOpenSettings && onOpenSettings();
};
if (isMobile) {
return (
<button
className="vm-mobile-option"
onClick={handleOpenSettings}
>
<span className="vm-mobile-option__icon"><PlanetIcon/></span>
<div className="vm-mobile-option-text">
<span className="vm-mobile-option-text__label">Time zone</span>
<span className="vm-mobile-option-text__value">{utcOffset}</span>
</div>
<span className="vm-mobile-option__arrow"><ArrowDownIcon/></span>
</button>
);
}
return (
<Button
className="vm-header-button"
onClick={handleOpenSettings}
startIcon={<PlanetIcon/>}
>
{utcOffset}
</Button>
);
};
export default TimeZonePreview;

View File

@@ -113,6 +113,8 @@ const StepConfigurator: FC = () => {
setError("");
}, [defaultStep, prevDefaultStep, value, graphDispatch]);
const textValue = isAutoStep ? `auto (${customStep})` : customStep;
return (
<div
className="vm-step-control"
@@ -126,7 +128,7 @@ const StepConfigurator: FC = () => {
<span className="vm-mobile-option__icon"><TimelineIcon/></span>
<div className="vm-mobile-option-text">
<span className="vm-mobile-option-text__label">Step</span>
<span className="vm-mobile-option-text__value">{customStep}</span>
<span className="vm-mobile-option-text__value">{textValue}</span>
</div>
<span className="vm-mobile-option__arrow"><ArrowDownIcon/></span>
</div>
@@ -138,7 +140,7 @@ const StepConfigurator: FC = () => {
startIcon={<TimelineIcon/>}
onClick={toggleOpenOptions}
>
Step: {isAutoStep ? `auto (${customStep})` : customStep}
Step: {textValue}
</Button>
)}
<Popper

View File

@@ -19,7 +19,11 @@ import useBoolean from "../../../../hooks/useBoolean";
import useWindowSize from "../../../../hooks/useWindowSize";
import usePrevious from "../../../../hooks/usePrevious";
export const TimeSelector: FC = () => {
type Props = {
onOpenSettings?: () => void;
}
export const TimeSelector: FC<Props> = ({ onOpenSettings }) => {
const { isMobile } = useDeviceDetect();
const { isDarkTheme } = useAppState();
const wrapperRef = useRef<HTMLDivElement>(null);
@@ -53,7 +57,7 @@ export const TimeSelector: FC = () => {
setFrom(formatDateForNativeInput(dateFromSeconds(start)));
}, [timezone, start]);
const setDuration = ({ duration, until, id }: {duration: string, until: Date, id: string}) => {
const setDuration = ({ duration, until, id }: { duration: string, until: Date, id: string }) => {
dispatch({ type: "SET_RELATIVE_TIME", payload: { duration, until, id } });
handleCloseOptions();
};
@@ -75,16 +79,23 @@ export const TimeSelector: FC = () => {
const setTimeAndClosePicker = () => {
if (from && until) {
dispatch({ type: "SET_PERIOD", payload: {
from: dayjs.tz(from).toDate(),
to: dayjs.tz(until).toDate()
} });
dispatch({
type: "SET_PERIOD", payload: {
from: dayjs.tz(from).toDate(),
to: dayjs.tz(until).toDate()
}
});
}
handleCloseOptions();
};
const onSwitchToNow = () => dispatch({ type: "RUN_QUERY_TO_NOW" });
const handleOpenSettings = () => {
onOpenSettings && onOpenSettings();
handleCloseOptions();
};
const onCancelClick = () => {
setUntil(formatDateForNativeInput(dateFromSeconds(end)));
setFrom(formatDateForNativeInput(dateFromSeconds(start)));
@@ -140,6 +151,7 @@ export const TimeSelector: FC = () => {
</Tooltip>
)}
</div>
<Popper
open={openOptions}
buttonRef={buttonRef}
@@ -179,13 +191,17 @@ export const TimeSelector: FC = () => {
onEnter={setTimeAndClosePicker}
/>
</div>
<div className="vm-time-selector-left-timezone">
<div className="vm-time-selector-left-timezone__title">{activeTimezone.region}</div>
<div className="vm-time-selector-left-timezone__utc">{activeTimezone.utc}</div>
</div>
<button
type="button"
className="vm-time-selector-left-timezone"
onClick={handleOpenSettings}
>
<span className="vm-time-selector-left-timezone__title">{activeTimezone.region}</span>
<span className="vm-time-selector-left-timezone__utc">{activeTimezone.utc}</span>
</button>
<Button
variant="text"
startIcon={<AlarmIcon />}
startIcon={<AlarmIcon/>}
onClick={onSwitchToNow}
>
switch to now

View File

@@ -40,8 +40,13 @@
gap: $padding-small;
font-size: $font-size-small;
margin-bottom: $padding-small;
color: $color-text;
cursor: pointer;
&__title {}
&:hover {
color: $color-primary;
text-decoration: underline;
}
&__utc {
display: inline-flex;

View File

@@ -634,6 +634,17 @@ export const DebugIcon = () => (
</svg>
);
export const PlanetIcon = () => (
<svg
viewBox="0 0 24 24"
fill="currentColor"
>
<path
d="M12 2C6.48 2 2 6.48 2 12s4.48 10 10 10 10-4.48 10-10S17.52 2 12 2M4 12c0-.61.08-1.21.21-1.78L8.99 15v1c0 1.1.9 2 2 2v1.93C7.06 19.43 4 16.07 4 12m13.89 5.4c-.26-.81-1-1.4-1.9-1.4h-1v-3c0-.55-.45-1-1-1h-6v-2h2c.55 0 1-.45 1-1V7h2c1.1 0 2-.9 2-2v-.41C17.92 5.77 20 8.65 20 12c0 2.08-.81 3.98-2.11 5.4"
></path>
</svg>
);
export const SystemIcon = () => (
<svg
viewBox="0 0 24 24"

View File

@@ -11,6 +11,7 @@
&_mobile {
display: grid;
grid-template-columns: 1fr;
gap: 0;
padding: 0;
flex-grow: initial;

View File

@@ -6,9 +6,11 @@ import StepConfigurator from "../../components/Configurators/StepConfigurator/St
import { TimeSelector } from "../../components/Configurators/TimeRangeSettings/TimeSelector/TimeSelector";
import CardinalityDatePicker from "../../components/Configurators/CardinalityDatePicker/CardinalityDatePicker";
import { ExecutionControls } from "../../components/Configurators/TimeRangeSettings/ExecutionControls/ExecutionControls";
import GlobalSettings from "../../components/Configurators/GlobalSettings/GlobalSettings";
import GlobalSettings, { GlobalSettingsHandle } from "../../components/Configurators/GlobalSettings/GlobalSettings";
import ShortcutKeys from "../../components/Main/ShortcutKeys/ShortcutKeys";
import { ControlsProps } from "../Header/HeaderControls/HeaderControls";
import { useRef } from "react";
import TimeZonePreview from "../../components/Configurators/GlobalSettings/TimeZonePreview/TimeZonePreview";
const ControlsMainLayout: FC<ControlsProps> = ({
displaySidebar,
@@ -17,6 +19,7 @@ const ControlsMainLayout: FC<ControlsProps> = ({
accountIds,
closeModal,
}) => {
const settingsRef = useRef<GlobalSettingsHandle>(null);
return (
<div
@@ -27,14 +30,15 @@ const ControlsMainLayout: FC<ControlsProps> = ({
>
{headerSetup?.tenant && <TenantsConfiguration accountIds={accountIds || []}/>}
{headerSetup?.stepControl && <StepConfigurator/>}
{headerSetup?.timeSelector && <TimeSelector/>}
{headerSetup?.timeSelector && <TimeSelector onOpenSettings={() => settingsRef.current?.open()}/>}
{headerSetup?.cardinalityDatePicker && <CardinalityDatePicker/>}
<TimeZonePreview onOpenSettings={() => settingsRef.current?.open()}/>
{headerSetup?.executionControls && <ExecutionControls
tooltip={headerSetup?.executionControls?.tooltip}
useAutorefresh={headerSetup?.executionControls?.useAutorefresh}
closeModal={closeModal}
/>}
<GlobalSettings/>
<GlobalSettings ref={settingsRef}/>
{!displaySidebar && <ShortcutKeys/>}
</div>
);

View File

@@ -1,11 +1,12 @@
@use "src/styles/variables" as *;
.vm-mobile-option {
display: flex;
display: grid;
grid-template-columns: auto 1fr auto;
align-items: center;
justify-content: flex-start;
gap: $padding-small;
padding: calc($padding-medium/2) 0;
gap: $padding-global;
padding: $padding-global $padding-small;
width: 100%;
user-select: none;
@@ -17,14 +18,33 @@
}
&__icon {
width: 22px;
height: 22px;
position: relative;
display: flex;
width: 40px;
height: 40px;
color: $color-primary;
&:after {
content: "";
position: absolute;
top: 0;
left: 0;
width: 100%;
height: 100%;
opacity: 0.1;
background-color: currentColor;
border-radius: $border-radius-medium;
}
svg {
width: 21px;
height: auto;
}
}
&__arrow {
width: 14px;
height: 14px;
width: 20px;
height: 20px;
transform: rotate(-90deg);
color: $color-primary;
}
@@ -32,11 +52,13 @@
&-text {
display: grid;
align-items: center;
gap: 2px;
height: 100%;
gap: calc($padding-small / 2);
flex-grow: 1;
text-align: left;
&__label {
font-weight: bold;
font-weight: 600;
}
&__value {

View File

@@ -59,6 +59,19 @@
},
"type": "dashboard"
},
{
"datasource": {
"type": "prometheus",
"uid": "$ds"
},
"enable": true,
"expr": "sum by(version) (\n label_replace(vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version!=\"\"}, \"version\", \"$1\", \"short_version\", \"(.*)\")\n OR\n vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version=\"\"}\n) \nunless \n(\n sum by(version) (\n label_replace(vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version!=\"\"}, \"version\", \"$1\", \"short_version\", \"(.*)\")\n OR\n vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version=\"\"}\n ) offset $__interval\n)",
"hide": true,
"iconColor": "dark-blue",
"name": "version",
"textFormat": "{{version}}",
"titleFormat": "Version change"
},
{
"datasource": {
"type": "prometheus",

View File

@@ -37,7 +37,7 @@
"uid": "$ds"
},
"enable": true,
"expr": "sum(vm_app_version{job=~\"$job\", instance=~\"$instance\"}) by(version) unless (sum(vm_app_version{job=~\"$job\", instance=~\"$instance\"} offset $__interval) by(version))",
"expr": "sum by(version) (\n label_replace(vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version!=\"\"}, \"version\", \"$1\", \"short_version\", \"(.*)\")\n OR\n vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version=\"\"}\n) \nunless \n(\n sum by(version) (\n label_replace(vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version!=\"\"}, \"version\", \"$1\", \"short_version\", \"(.*)\")\n OR\n vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version=\"\"}\n ) offset $__interval\n)",
"hide": true,
"iconColor": "dark-blue",
"name": "version change",

View File

@@ -37,7 +37,7 @@
"uid": "$ds"
},
"enable": true,
"expr": "sum(vm_app_version{job=~\"$job\", instance=~\"$instance\"}) by(version) unless (sum(vm_app_version{job=~\"$job\", instance=~\"$instance\"} offset $__interval) by(version))",
"expr": "sum by(version) (\n label_replace(vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version!=\"\"}, \"version\", \"$1\", \"short_version\", \"(.*)\")\n OR\n vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version=\"\"}\n) \nunless \n(\n sum by(version) (\n label_replace(vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version!=\"\"}, \"version\", \"$1\", \"short_version\", \"(.*)\")\n OR\n vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version=\"\"}\n ) offset $__interval\n)",
"hide": true,
"iconColor": "dark-blue",
"name": "version",

View File

@@ -60,6 +60,19 @@
},
"type": "dashboard"
},
{
"datasource": {
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"enable": true,
"expr": "sum by(version) (\n label_replace(vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version!=\"\"}, \"version\", \"$1\", \"short_version\", \"(.*)\")\n OR\n vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version=\"\"}\n) \nunless \n(\n sum by(version) (\n label_replace(vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version!=\"\"}, \"version\", \"$1\", \"short_version\", \"(.*)\")\n OR\n vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version=\"\"}\n ) offset $__interval\n)",
"hide": true,
"iconColor": "dark-blue",
"name": "version",
"textFormat": "{{version}}",
"titleFormat": "Version change"
},
{
"datasource": {
"type": "victoriametrics-metrics-datasource",

View File

@@ -38,7 +38,7 @@
"uid": "$ds"
},
"enable": true,
"expr": "sum(vm_app_version{job=~\"$job\", instance=~\"$instance\"}) by(version) unless (sum(vm_app_version{job=~\"$job\", instance=~\"$instance\"} offset $__interval) by(version))",
"expr": "sum by(version) (\n label_replace(vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version!=\"\"}, \"version\", \"$1\", \"short_version\", \"(.*)\")\n OR\n vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version=\"\"}\n) \nunless \n(\n sum by(version) (\n label_replace(vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version!=\"\"}, \"version\", \"$1\", \"short_version\", \"(.*)\")\n OR\n vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version=\"\"}\n ) offset $__interval\n)",
"hide": true,
"iconColor": "dark-blue",
"name": "version change",

View File

@@ -38,7 +38,7 @@
"uid": "$ds"
},
"enable": true,
"expr": "sum(vm_app_version{job=~\"$job\", instance=~\"$instance\"}) by(version) unless (sum(vm_app_version{job=~\"$job\", instance=~\"$instance\"} offset $__interval) by(version))",
"expr": "sum by(version) (\n label_replace(vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version!=\"\"}, \"version\", \"$1\", \"short_version\", \"(.*)\")\n OR\n vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version=\"\"}\n) \nunless \n(\n sum by(version) (\n label_replace(vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version!=\"\"}, \"version\", \"$1\", \"short_version\", \"(.*)\")\n OR\n vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version=\"\"}\n ) offset $__interval\n)",
"hide": true,
"iconColor": "dark-blue",
"name": "version",

View File

@@ -26,11 +26,11 @@
"uid": "$ds"
},
"enable": true,
"expr": "sum(vm_app_version{job=~\"$job\", instance=~\"$instance\"}) by(short_version) unless (sum(vm_app_version{job=~\"$job\", instance=~\"$instance\"} offset $__interval) by(short_version))",
"expr": "sum by(version) (\n label_replace(vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version!=\"\"}, \"version\", \"$1\", \"short_version\", \"(.*)\")\n OR\n vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version=\"\"}\n) \nunless \n(\n sum by(version) (\n label_replace(vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version!=\"\"}, \"version\", \"$1\", \"short_version\", \"(.*)\")\n OR\n vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version=\"\"}\n ) offset $__interval\n)",
"hide": true,
"iconColor": "dark-blue",
"name": "version",
"textFormat": "{{short_version}}",
"textFormat": "{{version}}",
"titleFormat": "Version change"
},
{

View File

@@ -26,11 +26,11 @@
"uid": "$ds"
},
"enable": true,
"expr": "sum(vm_app_version{job=~\"$job\", instance=~\"$instance\"}) by(short_version) unless (sum(vm_app_version{job=~\"$job\", instance=~\"$instance\"} offset $__interval) by(short_version))",
"expr": "sum by(version) (\n label_replace(vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version!=\"\"}, \"version\", \"$1\", \"short_version\", \"(.*)\")\n OR\n vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version=\"\"}\n) \nunless \n(\n sum by(version) (\n label_replace(vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version!=\"\"}, \"version\", \"$1\", \"short_version\", \"(.*)\")\n OR\n vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version=\"\"}\n ) offset $__interval\n)",
"hide": true,
"iconColor": "dark-blue",
"name": "version",
"textFormat": "{{short_version}}",
"textFormat": "{{version}}",
"titleFormat": "Version change"
},
{

View File

@@ -54,6 +54,19 @@
},
"type": "dashboard"
},
{
"datasource": {
"type": "victoriametrics-metrics-datasource",
"uid": "$ds"
},
"enable": true,
"expr": "sum by(version) (\n label_replace(vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version!=\"\"}, \"version\", \"$1\", \"short_version\", \"(.*)\")\n OR\n vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version=\"\"}\n) \nunless \n(\n sum by(version) (\n label_replace(vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version!=\"\"}, \"version\", \"$1\", \"short_version\", \"(.*)\")\n OR\n vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version=\"\"}\n ) offset $__interval\n)",
"hide": true,
"iconColor": "dark-blue",
"name": "version",
"textFormat": "{{version}}",
"titleFormat": "Version change"
},
{
"datasource": {
"type": "victoriametrics-metrics-datasource",

View File

@@ -25,11 +25,11 @@
"uid": "$ds"
},
"enable": true,
"expr": "sum(vm_app_version{job=~\"$job\", instance=~\"$instance\"}) by(short_version) unless (sum(vm_app_version{job=~\"$job\", instance=~\"$instance\"} offset $__interval) by(short_version))",
"expr": "sum by(version) (\n label_replace(vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version!=\"\"}, \"version\", \"$1\", \"short_version\", \"(.*)\")\n OR\n vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version=\"\"}\n) \nunless \n(\n sum by(version) (\n label_replace(vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version!=\"\"}, \"version\", \"$1\", \"short_version\", \"(.*)\")\n OR\n vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version=\"\"}\n ) offset $__interval\n)",
"hide": true,
"iconColor": "dark-blue",
"name": "version",
"textFormat": "{{short_version}}",
"textFormat": "{{version}}",
"titleFormat": "Version change"
},
{

View File

@@ -25,11 +25,11 @@
"uid": "$ds"
},
"enable": true,
"expr": "sum(vm_app_version{job=~\"$job\", instance=~\"$instance\"}) by(short_version) unless (sum(vm_app_version{job=~\"$job\", instance=~\"$instance\"} offset $__interval) by(short_version))",
"expr": "sum by(version) (\n label_replace(vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version!=\"\"}, \"version\", \"$1\", \"short_version\", \"(.*)\")\n OR\n vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version=\"\"}\n) \nunless \n(\n sum by(version) (\n label_replace(vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version!=\"\"}, \"version\", \"$1\", \"short_version\", \"(.*)\")\n OR\n vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version=\"\"}\n ) offset $__interval\n)",
"hide": true,
"iconColor": "dark-blue",
"name": "version",
"textFormat": "{{short_version}}",
"textFormat": "{{version}}",
"titleFormat": "Version change"
},
{

View File

@@ -53,6 +53,19 @@
},
"type": "dashboard"
},
{
"datasource": {
"type": "prometheus",
"uid": "$ds"
},
"enable": true,
"expr": "sum by(version) (\n label_replace(vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version!=\"\"}, \"version\", \"$1\", \"short_version\", \"(.*)\")\n OR\n vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version=\"\"}\n) \nunless \n(\n sum by(version) (\n label_replace(vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version!=\"\"}, \"version\", \"$1\", \"short_version\", \"(.*)\")\n OR\n vm_app_version{job=~\"$job\", instance=~\"$instance\", short_version=\"\"}\n ) offset $__interval\n)",
"hide": true,
"iconColor": "dark-blue",
"name": "version",
"textFormat": "{{version}}",
"titleFormat": "Version change"
},
{
"datasource": {
"type": "prometheus",

View File

@@ -3,7 +3,7 @@ services:
# It scrapes targets defined in --promscrape.config
# And forward them to --remoteWrite.url
vmagent:
image: victoriametrics/vmagent:v1.149.0
image: victoriametrics/vmagent:v1.150.0
depends_on:
- "vmauth"
ports:
@@ -42,14 +42,14 @@ services:
# vmstorage shards. Each shard receives 1/N of all metrics sent to vminserts,
# where N is number of vmstorages (2 in this case).
vmstorage-1:
image: victoriametrics/vmstorage:v1.149.0-cluster
image: victoriametrics/vmstorage:v1.150.0-cluster
volumes:
- strgdata-1:/storage
command:
- "--storageDataPath=/storage"
restart: always
vmstorage-2:
image: victoriametrics/vmstorage:v1.149.0-cluster
image: victoriametrics/vmstorage:v1.150.0-cluster
volumes:
- strgdata-2:/storage
command:
@@ -59,7 +59,7 @@ services:
# vminsert is ingestion frontend. It receives metrics pushed by vmagent,
# pre-process them and distributes across configured vmstorage shards.
vminsert-1:
image: victoriametrics/vminsert:v1.149.0-cluster
image: victoriametrics/vminsert:v1.150.0-cluster
depends_on:
- "vmstorage-1"
- "vmstorage-2"
@@ -68,7 +68,7 @@ services:
- "--storageNode=vmstorage-2:8400"
restart: always
vminsert-2:
image: victoriametrics/vminsert:v1.149.0-cluster
image: victoriametrics/vminsert:v1.150.0-cluster
depends_on:
- "vmstorage-1"
- "vmstorage-2"
@@ -80,7 +80,7 @@ services:
# vmselect is a query fronted. It serves read queries in MetricsQL or PromQL.
# vmselect collects results from configured `--storageNode` shards.
vmselect-1:
image: victoriametrics/vmselect:v1.149.0-cluster
image: victoriametrics/vmselect:v1.150.0-cluster
depends_on:
- "vmstorage-1"
- "vmstorage-2"
@@ -90,7 +90,7 @@ services:
- "--vmalert.proxyURL=http://vmalert:8880"
restart: always
vmselect-2:
image: victoriametrics/vmselect:v1.149.0-cluster
image: victoriametrics/vmselect:v1.150.0-cluster
depends_on:
- "vmstorage-1"
- "vmstorage-2"
@@ -105,7 +105,7 @@ services:
# read requests from Grafana, vmui, vmalert among vmselects.
# It can be used as an authentication proxy.
vmauth:
image: victoriametrics/vmauth:v1.149.0
image: victoriametrics/vmauth:v1.150.0
depends_on:
- "vmselect-1"
- "vmselect-2"
@@ -119,7 +119,7 @@ services:
# vmalert executes alerting and recording rules
vmalert:
image: victoriametrics/vmalert:v1.149.0
image: victoriametrics/vmalert:v1.150.0
depends_on:
- "vmauth"
ports:

View File

@@ -3,7 +3,7 @@ services:
# It scrapes targets defined in --promscrape.config
# And forward them to --remoteWrite.url
vmagent:
image: victoriametrics/vmagent:v1.149.0
image: victoriametrics/vmagent:v1.150.0
depends_on:
- "victoriametrics"
ports:
@@ -18,7 +18,7 @@ services:
# VictoriaMetrics instance, a single process responsible for
# storing metrics and serve read requests.
victoriametrics:
image: victoriametrics/victoria-metrics:v1.149.0
image: victoriametrics/victoria-metrics:v1.150.0
ports:
- 8428:8428
- 8089:8089
@@ -59,7 +59,7 @@ services:
# vmalert executes alerting and recording rules
vmalert:
image: victoriametrics/vmalert:v1.149.0
image: victoriametrics/vmalert:v1.150.0
depends_on:
- "victoriametrics"
- "alertmanager"

View File

@@ -16,6 +16,19 @@ groups:
Job {{ $labels.job }} (instance {{ $labels.instance }}) has restarted more than twice in the last 15 minutes.
It might be crashlooping.
- alert: UncleanShutdown
expr: vm_app_prev_shutdown_unclean == 1 and time() - vm_app_start_timestamp < 600
labels:
severity: warning
annotations:
summary: "{{ $labels.job }} on instance {{ $labels.instance }} started after an unclean shutdown"
description: |
The previous process run didn't shut down cleanly. Check the logs for OOM, SIGKILL,
a host failure, or another unexpected termination. In Kubernetes, a pod may be forcefully
killed with SIGKILL if the shutdown takes longer than terminationGracePeriodSeconds.
This alert stops firing 10 minutes after startup.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8443 for more details.
- alert: ServiceDown
expr: up{job=~".*(victoriametrics|vmselect|vminsert|vmstorage|vmagent|vmalert|vmsingle|vmalertmanager|vmauth).*"} == 0
for: 2m

View File

@@ -1,6 +1,6 @@
services:
vmagent:
image: victoriametrics/vmagent:v1.149.0
image: victoriametrics/vmagent:v1.150.0
depends_on:
- "victoriametrics"
ports:
@@ -14,7 +14,7 @@ services:
restart: always
victoriametrics:
image: victoriametrics/victoria-metrics:v1.149.0
image: victoriametrics/victoria-metrics:v1.150.0
ports:
- 8428:8428
volumes:
@@ -40,7 +40,7 @@ services:
restart: always
vmalert:
image: victoriametrics/vmalert:v1.149.0
image: victoriametrics/vmalert:v1.150.0
depends_on:
- "victoriametrics"
ports:

View File

@@ -90,12 +90,9 @@ endif
sed -i 's/\t/ /g' docs/victoriametrics/victoria_metrics_common_flags.md
sed -i 's/\t/ /g' docs/victoriametrics/victoria_metrics_enterprise_flags.md
# adjust flags with dynamic default values
# remove after https://github.com/VictoriaMetrics/VictoriaMetrics/issues/9680 implemented
sed -i '/The maximum number of concurrent insert requests/ s/(default [0-9]\+)/(default 2*cgroup.AvailableCPUs())/' docs/victoriametrics/victoria_metrics_common_flags.md
sed -i '/The maximum number of concurrent search requests\./ s/(default [0-9]\+)/(default vmselect.getDefaultMaxConcurrentRequests())/' docs/victoriametrics/victoria_metrics_common_flags.md
sed -i '/The maximum number of CPU cores a single query can use\./ s/(default [0-9]\+)/(default netstorage.defaultMaxWorkersPerQuery())/' docs/victoriametrics/victoria_metrics_common_flags.md
sed -i '/The maximum number of concurrent goroutines to work with files;/ s/(default [0-9]\+)/(default fsutil.getDefaultConcurrency())/' docs/victoriametrics/victoria_metrics_common_flags.md
# hide the machine-specific value of dynamic defaults, keeping the formula.
# the flagutil.New*WithDynamicDefault constructors print them as "(default <value> = <formula>)".
sed -i 's/(default [0-9]\+ = \(.*\))$$/(default \1)/' docs/victoriametrics/victoria_metrics_common_flags.md
docs-update-vmauth-flags:
ifndef TAG
@@ -119,9 +116,9 @@ endif
sed -i 's/\t/ /g' docs/victoriametrics/vmauth_common_flags.md
sed -i 's/\t/ /g' docs/victoriametrics/vmauth_enterprise_flags.md
# adjust flags with dynamic default values
# remove after https://github.com/VictoriaMetrics/VictoriaMetrics/issues/9680 implemented
sed -i '/The maximum number of concurrent goroutines to work with files;/ s/(default [0-9]\+)/(default fsutil.getDefaultConcurrency())/' docs/victoriametrics/vmauth_common_flags.md
# hide the machine-specific value of dynamic defaults, keeping the formula.
# the flagutil.New*WithDynamicDefault constructors print them as "(default <value> = <formula>)".
sed -i 's/(default [0-9]\+ = \(.*\))$$/(default \1)/' docs/victoriametrics/vmauth_common_flags.md
docs-update-vmagent-flags:
ifndef TAG
@@ -145,11 +142,9 @@ endif
sed -i 's/\t/ /g' docs/victoriametrics/vmagent_common_flags.md
sed -i 's/\t/ /g' docs/victoriametrics/vmagent_enterprise_flags.md
# adjust flags with dynamic default values
# remove after https://github.com/VictoriaMetrics/VictoriaMetrics/issues/9680 implemented
sed -i '/The maximum number of concurrent insert requests/ s/(default [0-9]\+)/(default 2*cgroup.AvailableCPUs())/' docs/victoriametrics/vmagent_common_flags.md
sed -i '/The number of concurrent queues to each -remoteWrite.url./ s/(default [0-9]\+)/(default 2*cgroup.AvailableCPUs())/' docs/victoriametrics/vmagent_common_flags.md
sed -i '/The maximum number of concurrent goroutines to work with files;/ s/(default [0-9]\+)/(default fsutil.getDefaultConcurrency())/' docs/victoriametrics/vmagent_common_flags.md
# hide the machine-specific value of dynamic defaults, keeping the formula.
# the flagutil.New*WithDynamicDefault constructors print them as "(default <value> = <formula>)".
sed -i 's/(default [0-9]\+ = \(.*\))$$/(default \1)/' docs/victoriametrics/vmagent_common_flags.md
docs-update-vmalert-flags:
ifndef TAG
@@ -173,10 +168,9 @@ endif
sed -i 's/\t/ /g' docs/victoriametrics/vmalert_common_flags.md
sed -i 's/\t/ /g' docs/victoriametrics/vmalert_enterprise_flags.md
# adjust flags with dynamic default values
# remove after https://github.com/VictoriaMetrics/VictoriaMetrics/issues/9680 implemented
sed -i '/Defines number of writers for concurrent writing into remote write endpoint./ s/(default [0-9]\+)/(default 2*cgroup.AvailableCPUs())/' docs/victoriametrics/vmalert_common_flags.md
sed -i '/The maximum number of concurrent goroutines to work with files;/ s/(default [0-9]\+)/(default fsutil.getDefaultConcurrency())/' docs/victoriametrics/vmalert_common_flags.md
# hide the machine-specific value of dynamic defaults, keeping the formula.
# the flagutil.New*WithDynamicDefault constructors print them as "(default <value> = <formula>)".
sed -i 's/(default [0-9]\+ = \(.*\))$$/(default \1)/' docs/victoriametrics/vmalert_common_flags.md
docs-update-vmselect-flags:
ifndef TAG

View File

@@ -1,5 +1,6 @@
---
title: AI tools
description: "MCP servers, skills, and AI assistant integrations for querying metrics, logs, and traces with natural language."
weight: 61
menu:
docs:

View File

@@ -1,6 +1,7 @@
---
weight: 7
title: CHANGELOG
description: "Release history for vmanomaly."
menu:
docs:
identifier: "vmanomaly-changelog"

View File

@@ -1,6 +1,7 @@
---
weight: 6
title: FAQ
description: "Frequently asked questions about vmanomaly."
menu:
docs:
identifier: "vmanomaly-faq"

View File

@@ -1,6 +1,7 @@
---
weight: 5
title: Migration
description: "Migration guide to the latest vmanomaly version."
menu:
docs:
identifier: "vmanomaly-migration"

View File

@@ -1,6 +1,7 @@
---
weight: 2
title: Presets
description: "Preconfigured anomaly detection configurations for widely-recognized metrics (e.g., node_exporter)"
menu:
docs:
parent: "anomaly-detection"

View File

@@ -1,6 +1,7 @@
---
weight: 1
title: Quick Start
description: "Get started with vmanomaly. Install, configure, and run anomaly detection."
menu:
docs:
parent: "anomaly-detection"

View File

@@ -1,6 +1,7 @@
---
weight: 3
title: Scaling vmanomaly
description: "High availability and horizontal scaling for vmanomaly."
menu:
docs:
identifier: "vmanomaly-scaling"

View File

@@ -1,6 +1,7 @@
---
weight: 4
title: Self-monitoring
description: "Track vmanomaly health and operational performance."
menu:
docs:
identifier: "vmanomaly-self-monitoring"

View File

@@ -1,6 +1,7 @@
---
weight: 2
title: UI
description: "Built-in vmui-like UI for exploring anomaly detection results."
menu:
docs:
parent: "anomaly-detection"

View File

@@ -1,5 +1,6 @@
---
title: Anomaly Detection
description: "Use vmanomaly to detect anomalies in metrics and logs. Configure models, run inference, monitor the service, and connect results to alerts and dashboards."
weight: 50
menu:
docs:

View File

@@ -1,5 +1,6 @@
---
title: Components
description: "Architecture overview. Models, reader, writer, scheduler, monitoring, settings, server."
weight: 3
menu:
docs:

View File

@@ -1,5 +1,6 @@
---
title: Models
description: "Model types and configuration. Built-in and custom anomaly detection models."
weight: 1
menu:
docs:

View File

@@ -1,5 +1,6 @@
---
title: Monitoring
description: "Self-monitoring via push and pull models."
weight: 5
menu:
docs:

View File

@@ -1,5 +1,6 @@
---
title: Reader
description: "Data reader configuration. MetricsQL queries from VictoriaMetrics or LogsQL from VictoriaLogs/VictoriaTraces."
weight: 2
menu:
docs:

View File

@@ -1,5 +1,6 @@
---
title: Scheduler
description: "Scheduling configuration. Inference frequency and training time range."
weight: 3
menu:
docs:

View File

@@ -1,5 +1,6 @@
---
title: Server
description: "HTTP server. REST API, /metrics endpoint, and web UI."
weight: 7
menu:
docs:

View File

@@ -1,5 +1,6 @@
---
title: Settings
description: "Global settings for the anomaly detection service."
weight: 6
menu:
docs:

View File

@@ -1,5 +1,6 @@
---
title: Writer
description: "Data writer. Write anomaly scores back to VictoriaMetrics."
weight: 4
menu:
docs:

View File

@@ -1,5 +1,6 @@
---
title: Guides
description: "Step-by-step guides for deploying, configuring, integrating, and operating vmanomaly for anomaly detection."
weight: 3
menu:
docs:

View File

@@ -10,9 +10,9 @@ sitemap:
- To use *vmanomaly*, part of the enterprise package, a license key is required. Obtain your key [here](https://victoriametrics.com/products/enterprise/trial/) for this tutorial or for enterprise use.
- In the tutorial, we'll be using the following VictoriaMetrics components:
- [VictoriaMetrics Single-Node](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/) (v1.149.0)
- [vmalert](https://docs.victoriametrics.com/victoriametrics/vmalert/) (v1.149.0)
- [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/) (v1.149.0)
- [VictoriaMetrics Single-Node](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/) (v1.150.0)
- [vmalert](https://docs.victoriametrics.com/victoriametrics/vmalert/) (v1.150.0)
- [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/) (v1.150.0)
- [Grafana](https://grafana.com/) (v12.2.0)
- [Docker](https://docs.docker.com/get-docker/) and [Docker Compose](https://docs.docker.com/compose/)
- [Node exporter](https://github.com/prometheus/node_exporter#node-exporter) (v1.9.1) and [Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/) (v0.28.1)
@@ -328,7 +328,7 @@ Let's wrap it all up together into the `docker-compose.yml` file.
services:
vmagent:
container_name: vmagent
image: victoriametrics/vmagent:v1.149.0
image: victoriametrics/vmagent:v1.150.0
depends_on:
- "victoriametrics"
ports:
@@ -345,7 +345,7 @@ services:
victoriametrics:
container_name: victoriametrics
image: victoriametrics/victoria-metrics:v1.149.0
image: victoriametrics/victoria-metrics:v1.150.0
ports:
- 8428:8428
volumes:
@@ -378,7 +378,7 @@ services:
vmalert:
container_name: vmalert
image: victoriametrics/vmalert:v1.149.0
image: victoriametrics/vmalert:v1.150.0
depends_on:
- "victoriametrics"
ports:

View File

@@ -1,6 +1,7 @@
---
weight: 1
title: Anomaly Detection and Alerting Setup
description: "Tutorial integrating vmanomaly with vmalert, Alertmanager, and Grafana."
menu:
docs:
parent: "anomaly-detection-guides"

View File

@@ -1,6 +1,7 @@
---
weight: 0
title: Guides
description: "Practical guides for deploying and operating VictoriaMetrics."
disableToc: true
menu:

View File

@@ -1,6 +1,7 @@
---
weight: 12
title: Collecting OpenShift logs with Victoria Logs
description: "Collect and store OpenShift cluster logs in VictoriaLogs."
menu:
docs:
parent: "guides"

View File

@@ -1,6 +1,7 @@
---
weight: 5
title: Connecting VictoriaMetrics components to cloud storage
description: "Configure VictoriaMetrics components to use object storage for data, backups, and other storage workflows."
menu:
docs:
parent: guides

View File

@@ -1,6 +1,7 @@
---
weight: 5
title: How to use OpenTelemetry with VictoriaMetrics and VictoriaLogs
description: "Use OpenTelemetry with VictoriaMetrics and VictoriaLogs on Kubernetes."
menu:
docs:
parent: "guides"

View File

@@ -1,6 +1,7 @@
---
weight: 4
title: Getting started with VM Operator
description: "Deploy the VictoriaMetrics stack on Kubernetes with the Kubernetes Operator."
menu:
docs:
parent: "guides"

View File

@@ -248,23 +248,23 @@ vmagent will write data into VictoriaMetrics single-node and cluster (with tenan
# compose.yaml
services:
vmsingle:
image: victoriametrics/victoria-metrics:v1.149.0
image: victoriametrics/victoria-metrics:v1.150.0
vmstorage:
image: victoriametrics/vmstorage:v1.149.0-cluster
image: victoriametrics/vmstorage:v1.150.0-cluster
vminsert:
image: victoriametrics/vminsert:v1.149.0-cluster
image: victoriametrics/vminsert:v1.150.0-cluster
command:
- -storageNode=vmstorage:8400
vmselect:
image: victoriametrics/vmselect:v1.149.0-cluster
image: victoriametrics/vmselect:v1.150.0-cluster
command:
- -storageNode=vmstorage:8401
vmagent:
image: victoriametrics/vmagent:v1.149.0
image: victoriametrics/vmagent:v1.150.0
volumes:
- ./scrape.yaml:/etc/vmagent/config.yaml
command:
@@ -316,7 +316,7 @@ Now add the vmauth service to `compose.yaml`:
# compose.yaml
services:
vmauth:
image: docker.io/victoriametrics/vmauth:v1.149.0
image: docker.io/victoriametrics/vmauth:v1.150.0
ports:
- 8427:8427
volumes:

View File

@@ -1,6 +1,7 @@
---
weight: 5
title: Setup vmauth - Multi-Tenant Access with Grafana & OIDC
description: "Multi-tenant access for metrics, logs, and traces with Grafana and OIDC."
menu:
docs:
parent: guides

View File

@@ -1,6 +1,7 @@
---
weight: 16
title: Setup vmgateway - Multi-Tenant Access with Grafana & OIDC
description: "Configure vmgateway with Grafana and OpenID Connect for authenticated, multi-tenant access to VictoriaMetrics data."
menu: false
tags:
- metrics

View File

@@ -1,6 +1,7 @@
---
weight: 7
title: How to delete or replace metrics in VictoriaMetrics
description: "Guide to deleting or replacing time series data."
menu:
docs:
parent: "guides"

View File

@@ -1,6 +1,7 @@
---
weight: 10
title: Multi Retention Setup within VictoriaMetrics Cluster
description: "Configure multiple retention periods in VM Cluster."
menu:
docs:
parent: "guides"

View File

@@ -1,6 +1,7 @@
---
weight: 9
title: HA monitoring setup in Kubernetes via VictoriaMetrics Cluster
description: "High-availability Kubernetes monitoring with replication."
menu:
docs:
parent: "guides"

View File

@@ -1,6 +1,7 @@
---
weight: 3
title: Kubernetes monitoring with VictoriaMetrics Cluster
description: "Monitor Kubernetes with VM Cluster."
menu:
docs:
parent: "guides"

View File

@@ -1,6 +1,7 @@
---
weight: 2
title: Kubernetes monitoring via VictoriaMetrics Single
description: "Monitor Kubernetes with single-node VictoriaMetrics and Helm."
menu:
docs:
parent: "guides"

View File

@@ -1,6 +1,7 @@
---
weight: 13
title: Headlamp Kubernetes UI and VictoriaMetrics
description: "Point Headlamp's Prometheus integration at VictoriaMetrics."
menu:
docs:
parent: "guides"

View File

@@ -1,5 +1,6 @@
---
title: Migrate from InfluxDB to VictoriaMetrics
description: "Differences and approaches for migrating from InfluxDB."
weight: 8
menu:
docs:

View File

@@ -6,80 +6,237 @@ build:
sitemap:
disable: true
---
### Scenario
Let's cover the case. You have multiple regions with workloads and want to collect metrics.
## Overview {#scenario}
The monitoring setup is in the dedicated regions as shown below:
This guide shows how to run VictoriaMetrics across many regions in high-availability mode. Each workload runs a local vmagent and sends metrics to dedicated monitoring deployments, so metric data is duplicated and available even if one monitoring region is down.
![Multi-regional setup with VictoriaMetrics: Dedicated regions for monitoring](setup.webp)
Use this architecture when you need region-level resilience and want monitoring to keep working even if one region becomes unavailable.
Every workload region (Earth, Mars, Venus) has a vmagent that sends data to multiple regions with a monitoring setup.
The monitoring setup (Ground Control 1,2) contains VictoriaMetrics Time Series Database(TSDB) cluster or single.
This setup gives you:
Using this schema, you can achieve:
* High availability of metric data across regions.
* A single global query endpoint.
* Simpler disaster recovery.
* Global Querying View
* Querying all metrics from one monitoring installation
* High Availability
* You can lose one region, but your experience will be the same.
* Of course, that means you duplicate your traffic twice.
The trade-off is that you store and send the same data twice, so storage and compute requirements are increased.
## Architecture
The example architecture separates workloads into three regions, called Earth, Mars, and Venus. These represent the systems you want to monitor (e.g., your applications or your infrastructure). For monitoring, there are two separate regions, Ground Control 1 and 2, each running its own VictoriaMetrics deployment. The workload regions (the planets) run a local vmagent that forwards the same metrics to the two dedicated Ground Control regions.
![Multi-regional setup with VictoriaMetrics: Dedicated regions for monitoring](setup-1.webp)
{width="700"}
The role of the Ground Controls can be filled by VictoriaMetrics in [single-node](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/) or [cluster mode](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/).
## High Availability
The architecture provides high availability by storing two full copies of the data: one in Ground Control 1 and the other in Ground Control 2. Since both store the same data, losing one region doesn't result in a monitoring outage. You can still run queries, view dashboards, and receive alerts.
vmagent keeps a separate persistent queue for each `-remoteWrite.url` destination. If one Ground Control region is unavailable, vmagent continues sending data to the other region. The samples for the unavailable region stay in the file-based queue, and vmagent delivers them after the region recovers. The queue size is limited by disk space available to the vmagent or group of vmagents. This helps restore consistency across both regions.
This setup provides two logical copies of the data in separate monitoring regions. That lets you fail over to the healthy region if one region becomes unavailable, or spread read load across both regions if needed.
### How to write the data to Ground Control regions
* You need to pass two `-remoteWrite.url` command-line options to `vmagent`:
Run one or more vmagent nodes in each workload region and configure them to send metrics to both Ground Control regions. This gives each workload region a local write path and keeps delivery going if one monitoring region is unavailable.
For example, a vmagent that sends data to two single-node VictoriaMetrics instances looks like this:
```sh
/path/to/vmagent-prod \
-remoteWrite.url=<ground-control-1-remote-write> \
-remoteWrite.url=<ground-control-2-remote-write>
-remoteWrite.url=https://ground-control-1:8428/api/v1/write \
-remoteWrite.url=https://ground-control-2:8428/api/v1/write
```
* If you scrape data from Prometheus-compatible targets, then please specify `-promscrape.config` parameter as well.
For a VictoriaMetrics cluster, use the following URLs for [`accountID=0`](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#multitenancy)
Here is a Quickstart guide for [vmagent](https://docs.victoriametrics.com/victoriametrics/vmagent/#quick-start)
```sh
/path/to/vmagent-prod \
-remoteWrite.url=https://ground-control-1-vminsert:8480/insert/0/prometheus/api/v1/write \
-remoteWrite.url=https://ground-control-2-vminsert:8480/insert/0/prometheus/api/v1/write
```
For more details, see [data ingestion with vmagent](https://docs.victoriametrics.com/victoriametrics/data-ingestion/vmagent/).
vmagent [alerting rules and dashboards](https://docs.victoriametrics.com/vmagent/index.html#monitoring) help to monitor
the health state of each configured destination and its queue size.
### How to read the data from Ground Control regions
You can use one of the following options:
You can read data from Ground Control regions in a few different ways. The best option depends on your needs and operational complexity:
1. Multi-level [vmselect setup](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#multi-level-cluster-setup) in cluster setup, top-level vmselect(s) reads data from cluster-level vmselects
* Returns data in one of the clusters is unavailable
* Merges data from both sources. You need to turn on [deduplication](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#deduplication) to remove duplicates
1. Regional endpoints - use one regional endpoint as default and switch to another if there is an issue.
1. Load balancer - that sends queries to a particular region. The benefit and disadvantage of this setup is that it's simple.
1. Promxy - proxy that reads data from multiple Prometheus-like sources. It allows reading data more intelligently to cover the region's unavailability out of the box. It doesn't support MetricsQL yet (please check this issue).
1. Global vmselect in cluster setup - you can set up an additional subset of vmselects that knows about all storages in all regions.
* The [deduplication](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#deduplication) in 1ms on the vmselect side must be turned on. This setup allows you to query data using MetricsQL.
* The downside is that vmselect waits for a response from all storages in all regions.
* Choose region via load balancer: put a load balancer in front of both Ground Control regions. Route traffic to a preferred region, with automatic failover to the other region in case of failure.
* Merge results from multiple regions via vmselect: run a dedicated vmselect that would be configured to read from both regions and merge the results.
You can read more about choosing the right architecture in the [VictoriaMetrics topologies guide](https://docs.victoriametrics.com/guides/vm-architectures/).
### High Availability
#### Load balancer
The data is duplicated twice, and every region contains a full copy of the data. That means one region can be offline.
Use a load balancer when you want one stable query endpoint in front of your Ground Control regions. In this setup, dashboards and tools send queries to a single URL, and vmauth routes each request to one available region.
You don't need to set up a replication factor using the VictoriaMetrics cluster.
The following diagram shows [vmauth](https://docs.victoriametrics.com/victoriametrics/vmauth/) performing the role of [load balancer for HA setups](https://docs.victoriametrics.com/vmauth/index.html#high-availability).
### Alerting
![Diagram shows vmauth between Grafana and Ground Control regions](load-balancer-vmauth.webp)
{width="700"}
You can set up vmalert in each Ground control region that evaluates recording and alerting rules. As every region contains a full copy of the data, you don't need to synchronize recording rules from one region to another.
This approach is faster than [merging results with vmselect](#vmselect), because each query goes to only one region. It can also reduce query latency by roughly half compared with a topology that reads and merges data from both regions.
For alert deduplication, please use [cluster mode in Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/#high-availability).
The main downside is that vmauth does not know whether a recovered region has already finished replaying delayed data from the vmagent queue. If you send queries to that region too early, recent data may still be incomplete. In that case, it is better to wait until the region catches up before routing traffic there.
We also recommend adopting the list of [alerting rules](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker#alerts)
for VictoriaMetrics components.
For VictoriaMetrics single node, you can vmauth it with the following configuration:
### Monitoring
```yaml
unauthorized_user:
url_prefix:
- "http://ground-control-1:8428"
- "http://ground-control-2:8428"
load_balancing_policy: first_available
```
An additional VictoriaMetrics single can be set up in every region, scraping metrics from the main TSDB.
On the VictoriaMetrics cluster, the URLs must point to the Ground Control vmselect nodes. For example:
You also may evaluate the option to send these metrics to the neighbour region to achieve HA.
```yaml
unauthorized_user:
url_prefix:
- "http://ground-control-1-vmselect:8481"
- "http://ground-control-2-vmselect:8481"
load_balancing_policy: first_available
```
Additional context
* VictoriaMetrics Single - [https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#monitoring](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#monitoring)
* VictoriaMetrics Cluster - [https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#monitoring](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#monitoring)
The examples above show how to load balance requests without authentication. You can optionally configure authentication in several ways; for more details, read the [vmauth authorization section](https://docs.victoriametrics.com/victoriametrics/vmauth/#authorization).
To start vmauth with your configuration, use the `-auth.config` flag. For example:
### What more can we do?
```sh
/path/to/vmauth-prod -auth.config=/path/to/auth.yaml
```
You can test that queries work with curl:
```sh
# single node
curl http://vmauth-node:8427/api/v1/query?query=up
# cluster
curl http://vmauth-node:8427/select/0/prometheus/api/v1/query?query=up
```
For an example of this topology in Kubernetes, see the [`VMDistributed` resource](https://docs.victoriametrics.com/helm/victoriametrics-k8s-stack/#vmdistributed-enabled).
#### vmselect
> This option requires that Ground Control regions are deployed in one of these modes:
> - As a [VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/).
> - Or as VictoriaMetrics [single-node with multitenant support enabled](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#multi-tenancy). In other words, VictoriaMetrics should be started with the optional `-vmselectAddr=:8401` command line flag to enable the vmselect RPC server.
In this setup, each Ground Control region has its own local vmselect. A top-level vmselect queries these instead of connecting directly to vmstorage nodes.
![Diagram shows top-level vmselect connecting to the regional vmselect nodes in each Ground Control cluster](top-level-vmselect.webp)
{width="700"}
This option is useful when direct access to vmstorage nodes is not practical or desirable. For example, when running on Kubernetes, the vmstorage services don't provide an HTTP query endpoint by default.
To enable this setup, each Ground Control regional vmselect must listen for requests from the top layer by setting the `-clusternativeListenAddr` flag. The top-level vmselect must then use `-storageNode` to point to the regional vmselect nodes and must set a [deduplication](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#deduplication) interval to handle duplicated data.
For example, here's how we can run the local cluster vmselect nodes and a top-level vmselect node:
```sh
# Ground Control 1 cluster vmselect
/path/to/vmselect-prod \
-storageNode=ground-control-1-vmstorage-1:8401,ground-control-1-vmstorage-2:8401 \
-clusternativeListenAddr=:8401
# Ground Control 2 cluster vmselect
/path/to/vmselect-prod \
-storageNode=ground-control-2-vmstorage-1:8401,ground-control-2-vmstorage-2:8401 \
-clusternativeListenAddr=:8401
# Top-level vmselect
/path/to/vmselect-prod \
-storageNode=ground-control-1-vmselect:8401,ground-control-2-vmselect:8401 \
-dedup.minScrapeInterval=1ms \
-replicationFactor=2
```
This option provides a single query endpoint for both Ground Control regions. If one region becomes unavailable, the global vmselect can still query the healthy region, so dashboards and queries can continue to work.
The main trade-off is performance. In a two-level vmselect topology, queries pass through two query layers, so they usually take longer than using regional endpoints directly, or through a load balancer. The benefit is that the topology is easy to understand; it keeps working if one region is lost, and it can merge data from both regions while one region is still catching up after recovery.
## Alerting
Run a vmalert node in each Ground Control region and point it to the local VictoriaMetrics endpoint. Since each region stores the same data, you can deploy the same alerting and recording rules in every region without needing cross-region rule synchronization. Send alerts to an [Alertmanager cluster](https://prometheus.io/docs/alerting/latest/alertmanager/#high-availability) to deduplicate firing alerts.
![Diagram showing vmalert nodes running in each Ground Control region. An Alertmanager cluster connects to each vmalert and deduplicates notifications](vmalert-alertmanager.webp)
{width="700"}
A simple vmalert example for a single-node VictoriaMetrics looks like this:
```sh
/path/to/vmalert \
-rule=/path/to/rules.yaml \
-datasource.url=http://ground-control-1:8428 \
-notifier.url=http://alertmanager-1:9093 \
-notifier.url=http://alertmanager-2:9093
```
In VictoriaMetrics cluster mode, point `-datasource.url` to the regional vmselect endpoint. For example:
```sh
/path/to/vmalert \
-rule=/path/to/rules.yaml \
-datasource.url=http://ground-control-1-vmselect:8481/select/0/prometheus \
-notifier.url=http://alertmanager-1:9093,http://alertmanager-2:9093
```
If you want vmalert to preserve alert state and recording rule results across restarts, configure `-remoteWrite.url` and `-remoteRead.url` to point to VictoriaMetrics as well. For example, for a VictoriaMetrics cluster:
```sh
/path/to/vmalert \
-rule=/path/to/rules.yaml \
-datasource.url=http://ground-control-1-vmselect:8481/select/0/prometheus \
-remoteRead.url=http://ground-control-1-vmselect:8481/select/0/prometheus \
-remoteWrite.url=http://ground-control-1-vminsert:8480/insert/0/prometheus \
-notifier.url=http://alertmanager-1:9093,http://alertmanager-2:9093
```
We recommend using the list of [VictoriaMetrics alerting rules](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker#alerts).
## Monitoring
You can monitor Ground Control instances themselves using a separate monitoring path. In this setup, each region runs its own monitoring instance that scrapes metrics from the Ground Control components.
![Diagram of the original setup with monitoring of monitoring added. Each region has a dedicated VictoriaMetrics instance dedicated to monitoring the main TSDB](setup-mom-1.webp)
{width="700"}
You can optionally duplicate the monitored metrics to the neighboring region for extra resilience. That way, if a whole Ground Control region goes down, you still have access to the telemetry of the downed VictoriaMetrics instance, which can help you troubleshoot and restore service more easily.
Refer to the following pages on how to monitor your VictoriaMetrics deployments:
* [How to monitor VictoriaMetrics single node](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#monitoring)
* [How to monitor a VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#monitoring)
## What more can we do?
You can deploy extra vmagent instances in Ground Control regions and use them as regional ingestion proxies. This places the write endpoint closer to storage and adds another disk-backed buffer, which improves resilience when storage is temporarily unavailable.
![Diagram of the original setup where a vmagent node runs in front of each Ground Control region](setup-vmagent-1.webp)
{width="700"}
This pattern is useful when you want more reliable delivery, local relabeling, or a cleaner separation between cross-region traffic and local storage ingestion.
For a Ground Control running VictoriaMetrics single node, you can run vmagent as follows:
```sh
# vmagent next to Ground Control 1
/path/to/vmagent-prod \
-remoteWrite.url=http://ground-control-1:8428/api/v1/write
```
If running in cluster mode, use this instead:
```sh
# vmagent next to Ground Control 1 for cluster mode
/path/to/vmagent-prod \
-remoteWrite.url=http://ground-control-1-vminsert:8480/insert/0/prometheus/api/v1/write
```
Setup vmagents in Ground Control regions. That allows it to accept data close to storage and add more reliability if storage is temporarily offline.

View File

@@ -1,6 +1,7 @@
---
weight: 11
title: 'VictoriaMetrics Multi-Regional Setup: Dedicated Monitoring'
description: "Collect metrics across regions with dedicated monitoring."
menu:
docs:
parent: guides

Binary file not shown.

After

Width:  |  Height:  |  Size: 57 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 56 KiB

File diff suppressed because one or more lines are too long

Binary file not shown.

After

Width:  |  Height:  |  Size: 91 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 111 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 92 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 63 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 72 KiB

View File

@@ -1,6 +1,7 @@
---
weight: 9
title: Understand Your Setup Size
description: "Capacity planning. Active time series, ingestion rate, churn rate, QPS."
menu:
docs:
parent: "guides"

View File

@@ -1,6 +1,7 @@
---
weight: 14
title: VictoriaMetrics topologies
description: "Choose the right deployment topology for risk tolerance and performance needs."
menu:
docs:
parent: "guides"

View File

@@ -155,15 +155,15 @@ These services will store and query the metrics scraped by vmagent.
# compose.yaml
services:
vmstorage:
image: victoriametrics/vmstorage:v1.149.0-cluster
image: victoriametrics/vmstorage:v1.150.0-cluster
vminsert:
image: victoriametrics/vminsert:v1.149.0-cluster
image: victoriametrics/vminsert:v1.150.0-cluster
command:
- -storageNode=vmstorage:8400
vmselect:
image: victoriametrics/vmselect:v1.149.0-cluster
image: victoriametrics/vmselect:v1.150.0-cluster
command:
- -storageNode=vmstorage:8401
ports:
@@ -196,7 +196,7 @@ Add the vmauth service to `compose.yaml`:
# compose.yaml
services:
vmauth:
image: victoriametrics/vmauth:v1.149.0-enterprise
image: victoriametrics/vmauth:v1.150.0-enterprise
ports:
- 8427:8427
volumes:
@@ -251,7 +251,7 @@ Add the vmagent service to `compose.yaml` with OAuth2 configuration:
# compose.yaml
services:
vmagent:
image: victoriametrics/vmagent:v1.149.0
image: victoriametrics/vmagent:v1.150.0
volumes:
- ./scrape.yaml:/etc/vmagent/config.yaml
command:

View File

@@ -1,6 +1,7 @@
---
weight: 5
title: Setup vmagent - Multi-Tenant remote write & OIDC
description: "Multi-tenant remote write with OIDC and JWT tokens."
menu:
docs:
parent: guides

View File

@@ -107,7 +107,7 @@ The final piece is the Docker Compose file. This ties all the services together
# compose.yml
services:
victoriametrics:
image: victoriametrics/victoria-metrics:v1.149.0
image: victoriametrics/victoria-metrics:v1.150.0
command:
- "--storageDataPath=/victoria-metrics-data"
- "--selfScrapeInterval=10s"
@@ -128,7 +128,7 @@ services:
- ./alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro
vmalert:
image: victoriametrics/vmalert:v1.149.0
image: victoriametrics/vmalert:v1.150.0
depends_on:
- victoriametrics
- alertmanager

View File

@@ -1,6 +1,7 @@
---
weight: 16
title: Datasource-Managed Alerts with vmalert and Grafana
description: "Scalable alerting topology with vmalert and Grafana."
menu:
docs:
parent: "guides"

View File

@@ -1,5 +1,6 @@
---
title: OpenTelemetry
description: "OTLP ingestion for metrics, logs, and traces in VictoriaMetrics, VictoriaLogs, and VictoriaTraces, with signal correlation and configuration guides."
weight: 60
menu:
docs:

View File

@@ -1,5 +1,6 @@
---
title: Playgrounds
description: "Public demo environments for VictoriaMetrics, VictoriaLogs, and VictoriaTraces."
weight: 63
menu:
docs:

View File

@@ -1,6 +1,7 @@
---
weight: 29
title: Articles
description: "Third-party articles, slides, and videos about VictoriaMetrics."
menu:
docs:
parent: 'victoriametrics'
@@ -156,6 +157,13 @@ See [our blog](https://victoriametrics.com/blog) for the latest articles written
* [Why irate from Prometheus doesn't capture spikes](https://valyala.medium.com/why-irate-from-prometheus-doesnt-capture-spikes-45f9896d7832)
* [VictoriaMetrics: PromQL compliance](https://medium.com/@romanhavronenko/victoriametrics-promql-compliance-d4318203f51e)
* [How do open source solutions for logs work: Elasticsearch, Loki and VictoriaLogs](https://itnext.io/how-do-open-source-solutions-for-logs-work-elasticsearch-loki-and-victorialogs-9f7097ecbc2f)
* [How vmagent Collects and Ships Metrics Fast with Aggregation, Deduplication, and More](https://victoriametrics.com/blog/vmagent-how-it-works/)
* [When Metrics Meet vminsert: A Data-Delivery Story](https://victoriametrics.com/blog/vminsert-how-it-works/)
* [How vmstorage Handles Data Ingestion From vminsert](https://victoriametrics.com/blog/vmstorage-how-it-handles-data-ingestion/)
* [How vmstorage Processes Data: Retention, Merging, Deduplication...](https://victoriametrics.com/blog/vmstorage-retention-merging-deduplication/)
* [How vmstorage's IndexDB Works](https://victoriametrics.com/blog/vmstorage-how-indexdb-works/)
* [How vmstorage Handles Query Requests From vmselect](https://victoriametrics.com/blog/vmstorage-how-it-handles-query-requests/)
* [Inside vmselect: The Query Processing Engine of VictoriaMetrics](https://victoriametrics.com/blog/vmselect-how-it-works/)
### Tutorials, guides and how-to articles
@@ -173,6 +181,12 @@ See [our guides](https://docs.victoriametrics.com/guides/) for the up-to-date gu
* [Prometheus storage: tech terms for humans](https://valyala.medium.com/prometheus-storage-technical-terms-for-humans-4ab4de6c3d48)
* [Cardinality explorer](https://victoriametrics.com/blog/cardinality-explorer/)
* [Rules backfilling via vmalert](https://victoriametrics.com/blog/rules-replay/)
* [vmagent: Key Features Explained in Under 15 Minutes](https://victoriametrics.com/blog/vmagent-key-features-explained/)
* [Prometheus Metrics Explained: Counters, Gauges, Histograms & Summaries](https://victoriametrics.com/blog/prometheus-monitoring-metrics-counters-gauges-histogram-summaries/)
* [Prometheus Monitoring: Instant Queries and Range Queries Explained](https://victoriametrics.com/blog/prometheus-monitoring-instant-range-query/)
* [Prometheus Monitoring: Functions, Subqueries, Operators, and Modifiers](https://victoriametrics.com/blog/prometheus-monitoring-function-operator-modifier/)
* [Prometheus Alerting 101: Rules, Recording Rules, and Alertmanager](https://victoriametrics.com/blog/alerting-recording-rules-alertmanager/)
* [Alerting Best Practices](https://victoriametrics.com/blog/alerting-best-practices/)
### Other articles

View File

@@ -1,6 +1,7 @@
---
weight: 22
title: Best practices
description: "Production best practices for installation, configuration, hardware sizing, and maintenance."
menu:
docs:
identifier: vm-best-practices

View File

@@ -1,6 +1,7 @@
---
weight: 400
title: Contributing
description: "Guidelines for contributing to VictoriaMetrics."
menu:
docs:
identifier: vm-contributing

View File

@@ -1,6 +1,7 @@
---
weight: 25
title: Case studies and talks
description: "Production case studies from Grammarly, Roblox, Wix, Spotify, adidas, and more."
menu:
docs:
parent: "victoriametrics"

View File

@@ -6,6 +6,7 @@ menu:
parent: 'victoriametrics'
weight: 2
title: Cluster version
description: "Deploy and configure cluster mode with vminsert, vmselect, and vmstorage components, including replication and multi-tenancy."
tags:
- metrics
aliases:
@@ -59,6 +60,11 @@ It increases cluster availability, and simplifies cluster maintenance as well as
![Cluster Scheme](Cluster-VictoriaMetrics-components.webp)
> Further reading, deep dives into how each service works internally:
> - `vmstorage`: [How vmstorage Handles Data Ingestion From vminsert](https://victoriametrics.com/blog/vmstorage-how-it-handles-data-ingestion/), [How vmstorage's IndexDB Works](https://victoriametrics.com/blog/vmstorage-how-indexdb-works/), [How vmstorage Handles Query Requests From vmselect](https://victoriametrics.com/blog/vmstorage-how-it-handles-query-requests/).
> - `vminsert`: [When Metrics Meet vminsert: A Data-Delivery Story](https://victoriametrics.com/blog/vminsert-how-it-works/).
> - `vmselect`: [Inside vmselect: The Query Processing Engine of VictoriaMetrics](https://victoriametrics.com/blog/vmselect-how-it-works/).
## vmui
VictoriaMetrics cluster version provides UI for query troubleshooting and exploration. The UI is available at
@@ -828,8 +834,8 @@ See also [minimum downtime strategy](#minimum-downtime-strategy).
## Slowness-based re-routing
By default{{% available_from "v1.149.0" %}}, `vminsert` automatically re-routes writes away from the slowest `vmstorage` node
to preserve maximum ingestion throughput. This prevents a single slow `vmstorage` node
By default{{% available_from "v1.149.0" %}}, `vminsert` automatically [re-route writes](https://victoriametrics.com/blog/vminsert-how-it-works/#31-rerouting)
away from the slowest `vmstorage` node to preserve maximum ingestion throughput. This prevents a single slow `vmstorage` node
from throttling the entire cluster.
Re-routing occurs only when all of the following conditions hold:
@@ -877,7 +883,7 @@ See also [resource usage limits docs](#resource-usage-limits).
## Rebalancing
Every `vminsert` node evenly spreads (shards) incoming data among `vmstorage` nodes specified in the `-storageNode` command-line flag.
Every `vminsert` node [evenly spreads (shards) incoming data](https://victoriametrics.com/blog/vminsert-how-it-works/#3-sharding-and-buffering) among `vmstorage` nodes specified in the `-storageNode` command-line flag.
This guarantees even distribution of the ingested data among `vmstorage` nodes. When new `vmstorage` nodes are added to the `-storageNode`
command-line flag at `vminsert`, then only newly ingested data is distributed evenly among old and new `vmstorage` nodes, while
historical data remains on the old `vmstorage` nodes. This speeds up data ingestion and querying for the majority of production workloads,
@@ -1025,7 +1031,7 @@ By default, VictoriaMetrics offloads replication to the underlying storage point
which guarantees data durability. VictoriaMetrics supports application-level replication if replicated durable persistent disks cannot be used for some reason.
The replication can be enabled by passing `-replicationFactor=N` command-line flag to `vminsert`. This instructs `vminsert` to store `N` copies for every ingested sample
on `N` distinct `vmstorage` nodes. This guarantees that all the stored data remains available for querying if up to `N-1` `vmstorage` nodes are unavailable.
on `N` distinct `vmstorage` nodes. This guarantees that all the stored data remains available for querying if up to `N-1` `vmstorage` nodes are unavailable. See [how `vminsert` replicates each sample to `N` `vmstorage` nodes](https://victoriametrics.com/blog/vminsert-how-it-works/#4-replication-and-sending-data-to-vmstorage) for details.
Passing `-replicationFactor=N` command-line flag to `vmselect` instructs it to not mark responses as `partial` if less than `-replicationFactor` vmstorage nodes are unavailable during the query.
See [cluster availability docs](#cluster-availability) for details.
@@ -1060,7 +1066,7 @@ deduplication can't be guaranteed when samples and sample duplicates for the sam
- when `vmstorage` node has no enough capacity for processing incoming data stream. Then `vminsert` re-routes new samples to other `vmstorage` nodes.
It is recommended to set **the same** `-dedup.minScrapeInterval` command-line flag value to both `vmselect` and `vmstorage` nodes
to ensure query results consistency, even if storage layer didn't complete deduplication yet.
to ensure query results consistency, even if [storage layer didn't complete deduplication](https://victoriametrics.com/blog/vmstorage-retention-merging-deduplication/#deduplication) yet.
## Metrics Metadata

View File

@@ -1,6 +1,7 @@
---
weight: 24
title: FAQ
description: "Frequently asked questions comparing VM with Prometheus, InfluxDB, TimescaleDB, M3DB, Thanos, and Cortex."
menu:
docs:
parent: 'victoriametrics'

View File

@@ -1,6 +1,7 @@
---
weight: 300
title: Long-term support releases
description: "Long-term support release lines for enterprise customers."
menu:
docs:
parent: 'victoriametrics'
@@ -27,5 +28,5 @@ to [the latest available releases](https://docs.victoriametrics.com/victoriametr
## Currently supported LTS release lines
- v1.148.x - the latest one is [v1.148.1 LTS release](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.148.1)
- v1.136.x - the latest one is [v1.136.15 LTS release](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.136.15)
- v1.148.x - the latest one is [v1.148.2 LTS release](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.148.2)
- v1.136.x - the latest one is [v1.136.16 LTS release](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.136.16)

View File

@@ -1,6 +1,7 @@
---
weight: 23
title: MetricsQL
description: "Query language reference. Enhanced PromQL with additional functions, histogram helpers, and subquery support."
menu:
docs:
parent: 'victoriametrics'

View File

@@ -1,6 +1,7 @@
---
weight: 81
title: Cluster Per Tenant Statistic
description: "Enterprise cluster per-tenant usage tracking."
menu:
docs:
identifier: vm-cluster-per-tenant-statistic

View File

@@ -1,6 +1,7 @@
---
weight: 1
title: Quick start
description: "Get up and running with VictoriaMetrics. Download, run, scrape, and query."
menu:
docs:
identifier: vm-quick-start
@@ -53,8 +54,8 @@ and unpack it. It contains a single `victoria-metrics-prod` binary.
For example, on Linux with `amd64` architecture:
```sh
wget https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.149.0/victoria-metrics-linux-amd64-v1.149.0.tar.gz
tar xzf victoria-metrics-linux-amd64-v1.149.0.tar.gz
wget https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.150.0/victoria-metrics-linux-amd64-v1.150.0.tar.gz
tar xzf victoria-metrics-linux-amd64-v1.150.0.tar.gz
```
The binary is self-contained and requires no installation - it is ready to run as is.
@@ -229,9 +230,9 @@ Download the newest available [VictoriaMetrics release](https://docs.victoriamet
from [DockerHub](https://hub.docker.com/r/victoriametrics/victoria-metrics) or [Quay](https://quay.io/repository/victoriametrics/victoria-metrics?tab=tags):
```sh
docker pull victoriametrics/victoria-metrics:v1.149.0
docker pull victoriametrics/victoria-metrics:v1.150.0
docker run -it --rm -v `pwd`/victoria-metrics-data:/victoria-metrics-data -p 8428:8428 \
victoriametrics/victoria-metrics:v1.149.0 --selfScrapeInterval=5s -storageDataPath=victoria-metrics-data
victoriametrics/victoria-metrics:v1.150.0 --selfScrapeInterval=5s -storageDataPath=victoria-metrics-data
```
_For Enterprise images, see [this link](https://docs.victoriametrics.com/victoriametrics/enterprise/#docker-images)._

Some files were not shown because too many files have changed in this diff Show More