This PR rewrites the `TestSearchTSIDWithTimeRange`
`TestSearchLabelValues` tests:
- `TestSearchTSIDWithTimeRange` is split into several tests:
`TestIndexDBSearchTSIDs`, `TestIndexDBSearchLabelNames`,
`TestIndexDBSearchLabelValues`, `TestIndexDBGetTSDBStatus`,
`TestIndexDBDeleteSeries`.
- Previous `TestSearchLabelValues` was merged into
`TestIndexDBSearchLabelValues`
- All tests support disabling per-day index and can be easily extended
to support global index disabling ( related to #11196).
---------
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
This commit adds http client load-balancing with DNS discovery. For both A and SRV records.
It helps to route HTTP requests evenly for across discovered backends.
Discovered IP addresses are cached locally.
The main benefit of this feature is to remove intermediate vmauth as a load-balancer between
vmagent (vmalert) and remote targets. Which simplifies components management and
reduces operational overhead.
By default, it's disable and could be used with `dns+` or `srv+` hostname suffix at url.
For example, `-remoteWrite.url=http://dns+victoria-metrics:8428/api/v1/write`.
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2388
This commit adds `name` label to the `vm_persistentqueue_*` metrics. It provides more context for queues with human read-able format. Because `path` label contains only on-disk path, which is not really useful.
For vmagent case it will contain remote.writeURL position at command-line args.
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7944
Previously queue metainfo was flushed with fsync. However it may
corrupt file if OS filesystem not properly flush data.
This commit adds atomic file write with temporary file.
It must prevent possible file corruption issue.
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/11192
Target metrics relabel debug UI now preloads relabeling rules configured in `-promscrape.config`, `-remoteWrite.relabelConfig` and `-remoteWrite.urlRelabelConfig` flags.
Metrics relabel debug UI now preloads relabeling rules configured in `-remoteWrite.relabelConfig` and `-remoteWrite.urlRelabelConfig` flags.
The configs are ordered in the same order they are executed by vmagent. They also contain hint comments that highlight from which flag they come.
The select at the top lets you change which remote write configuration to use. The first one is preloaded by default.
Other UI improvements:
- renamed "Labels" -> "A Time Series", "Reset" -> "Reset All",
"Relabel Configs" -> "Configs"
- pre-populate "A Time Series" field with an `up` metric placeholder
- display YAML comments (lines starting with #) in gray
- added links to "Relabeling Cookbook" and "Relabeling Stages" docs
- added JSONPath hints for -promscrape.config relabel config fields
- use POST when reloading config content larger than 1KB
- improved error messages when multiple time series are provided
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/9918
PR https://github.com/VictoriaMetrics/VictoriaMetrics/pull/10607
---------
Signed-off-by: Zhu Jiekun <jiekun@victoriametrics.com>
Co-authored-by: Max Kotliar <mkotlyar@victoriametrics.com>
Remove empty db checks before and after test.
These checks are redundant and make it difficult to port them to the cluster
branch.
Follow-up for 52de485bc9.
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
Concurrent goroutines that ingest, flush, and then read their own data
may not see it. See `TestStorageAddFlushSearchDataConcurrently` in this
PR.
This is not the case for reading index. See
`TestStorageAddFlushSearchMetricNamesConcurrently`. This is because
index table handles concurrent flushes correctly while data table don't.
The solution is to use the same mechanism in data table
(`flushPendingItemsWG`).
The issue was originally discovered when rewriting the series deletion
test in https://github.com/VictoriaMetrics/VictoriaMetrics/pull/11271.
---------
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>
Currently, `rawRowsShard` and `rawRowsShards` lives in
`lib/partition.go` file. But these types and the `partition` type are
only loosely coupled. While the `rawRows` logic is scattered across two
files `lib/raw_row.go` and `lib/partition.go`.
This PR moves `rawRowsShard` and `rawRowsShards` and related constants
to `raw_row.go`. This allows to keep related funtionality together and
view raw row types as functionality that is independent from the
`partition` type.
Additionally, the `rawRowsShards` and the `partition` types were
decoupled by using the `func([][]rawRow)` callback.
---------
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
Previously, reader could advance writer offset at metadata without actual data write.
It may produce corrupted persistent queue, if vmagent crashed.
This commit changes flush behavior to perform f-sync flush, only if there is some recently written data.
Related to https://github.com/VictoriaMetrics/VictoriaMetrics/issues/11192
Previously, slow http_sd target may slow discovery pipeline. And for
example, if configuration contains 100 targets which respond in 1
second. It will take 100 seconds to refresh targets.
This commit adds background discovery for `http_sd` scrape config.
A new goroutine starts for each http_sd url. Which reduces discovery
time.
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8838
This commit adds new metrics:
* vm_data_size_bytes{type="storage/metaindex"}
* vm_data_size_bytes{type="indexdb/metaindex"}
Which are useful to track memory used to store meta index.
Related to https://github.com/VictoriaMetrics/VictoriaMetrics/issues/10203
Allows rejecting ingested samples with historical timestamps older than
now-maxBackfillAge, independently of -retentionPeriod. This is useful
for limiting ingestion of historical samples, for example when older
data has been moved between storage tiers (nvme/hdd, hot/cold).
Mirrors the -maxBackfillAge flag already available in VictoriaLogs.
Defaults to 0, in which case it is clamped to -retentionPeriod,
preserving previous ingestion behavior.
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/11199
---------
Signed-off-by: Ashwin Ramani <ashwinramanipsg@gmail.com>
Co-authored-by: Artem Fetishev <rtm@victoriametrics.com>
vmbackup fails immediately when S3 returns a 429. The SDK retries
TooManyRequestsException but not the short form TooManyRequests, and 429
isn't in the retryable status codes either (only 500/502/503/504 are).
Added both so a rate limit doesn't kill the whole backup.
Note: Followed the same pattern as ExpiredToken and IncompleteBody in
the same retryer config.
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/11218
Using nsPrefixTagToMetricIDs will stop working once global index will be
disabled by default. See:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/10641.
Instead, use nsPrefixMetricIDToTSID index entry type which must always
exist readless whether global or per-day index is disabled or not. Using
nsPrefixMetricIDToTSID may also be faster because the prefix to match is
shorter.
---------
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
Previously, creation of compressedLabels could require extra memory due
to re-allocation of tmpBuf and clone of 3 extra fields. It could result
into extra CPU usage for garbage-collection.
This commit adds sync.Pool for labels escape with JSON marshal and
allocates dedicated buffer for job, address and ID strings.
Optimisations was made based on the following profiles from reported
issue:
1) CPU:
```
Showing top 10 nodes out of 172
flat flat% sum% cum cum%
12.17s 17.19% 17.19% 12.25s 17.30% runtime.cgocall
5.87s 8.29% 25.48% 5.87s 8.29% runtime.memmove
3.45s 4.87% 30.35% 6.66s 9.41% runtime.tryDeferToSpanScan
```
2) memory go tool pprof -alloc_objects heap_profile.txt
```
Showing top 10 nodes out of 94
flat flat% sum% cum cum%
3673568660 26.09% 26.09% 4147984949 29.46%
github.com/valyala/quicktemplate.AppendJSONString
1657933055 11.77% 37.86% 1657933055 11.77% internal/stringslite.Clone
(inline)
1555166274 11.04% 48.91% 1555166274 11.04%
github.com/valyala/gozstd.compress
1254756359 8.91% 57.82% 9433313305 66.99%
github.com/VictoriaMetrics/VictoriaMetrics/lib/promscrape.newCompressedLabels
1067036870 7.58% 65.39% 1067036870 7.58%
github.com/VictoriaMetrics/VictoriaMetrics/lib/promscrape.appendExtraLabels
```
results of benchstat:
```
benchstat before after
goos: darwin
goarch: arm64
pkg: github.com/VictoriaMetrics/VictoriaMetrics/lib/promscrape
cpu: Apple M1 Pro
│ 134/before │ after │
│ sec/op │ sec/op vs base │
NewCompressedLabels-10 981.3n ± 2% 908.6n ± 2% -7.40% (p=0.000 n=10)
│ 134/before │ after │
│ B/op │ B/op vs base │
NewCompressedLabels-10 891.5 ± 0% 772.0 ± 0% -13.40% (p=0.000 n=10)
│ 134/before │ after │
│ allocs/op │ allocs/op vs base │
NewCompressedLabels-10 10.000 ± 0% 3.000 ± 0% -70.00% (p=0.000 n=10)
```
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/10919
The PR adds the support of vmselect RPC to vmsingle.
- The RPC server was copied from cluster branch as is and is disabled by
default. To enable, users must provide the listen address explicitly via
`-vmselectAddr` flag.
- Users must also specify which accountID and projectID the vmsingle
data corresponds to. To do this, `-accountID` and `-projectID` flags. By
default, these flags are 0. So the default tenantID is `"0:0"`.
Copying RPC server from cluster branch also required copying its
dependencies:
- Code located in `lib/handshake` (copied as is)
- Some parts of `SearchQuery` type from `lib/storage/search.go`.
The entire API surface is tested with an app test.
Implements https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4328
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
This commit adds a size limit for metric metadata fields.
Limit is hard-capped by `56kb` ( max uint16), since historically metric
metadata marshalled with those limit at cluster version. And it seems to
be a sane limit, it's unlikely that any valid metadata will exceed it.
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/11128
## Description
We have a simple vmagent setup using `remotewrite.shardByURL` to
horizontally scale stream aggregations, as suggested
[here](https://docs.victoriametrics.com/victoriametrics/vmagent/#sharding-among-remote-storages)
with minimal additional configuration and noticed there was some
optimization opportunities.
## Investigation
We took a
[pprof](https://github.com/user-attachments/files/28968177/cpu.pprof.zip)
and found that our CPU time was in a few primary places.
1. payload protobuf marshaling
2. zstd compresssion
3. Map access in sharding
4. Hashing in sharding
5. Memory copies for timeseries
## Changes
Zstd compression is pretty intractable(besides using
`remotewrite.vmProtoCompressLevel`, which we already do), but there were
some relatively easy gains for the other paths.
1. inline sov() when l < 128 on marshal. Skips calling a bunch of byte
math in a common case
2. Use slices instead of maps for shard keys, in the common case where
there are less than ~10 keys a slice iteration will be faster
3. use `xxhash.Digest.WriteString` rather than buffering values, avoids
some extra allocations and GC work
4. In the simple case where `insertRows` doesn't modify label names or
values construct the remotewrite request from the passed values instead
of copying them. Avoids a bunch of memory copy and GC work.
## Results
Modest but effective, we saw a ~10% reduction in total cpu cores used by
vmagent with these changes.
<img width="2358" height="709" alt="Screenshot 2026-06-15 at 2 24 43 PM"
src="https://github.com/user-attachments/assets/d2931242-c7ed-447b-9cfd-73b29cd0b893"
/>
Related PR https://github.com/VictoriaMetrics/VictoriaMetrics/pull/11113
This commit adds MDX(Monitoring Data eXchange) feature into vmagent.
It allows to detect VictoriaMetrics related components and forward metrics only for those instances.
In default state, mdx detects timeseries based on `vm_app_version` metric name and tracks instance based on "job:instance" key. It holds tracked instances in-memory for 1 hour, which must be enough for the most scrape intervals.
fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/10600
This commit relaxes JWT token format. Previously "vm_access" claim was required.
Which may not be a case for some users. For example, if there is no need at request templating.
The new `jwt` section field `default_vm_access_claim` was added for this purpose.
It's used as fall-back for templating data.
See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/11054
---------
Signed-off-by: Nikolay <nik@victoriametrics.com>
Co-authored-by: f41gh7 <nik@victoriametrics.com>
`http.Server.Shutdown()` waits for in-flight requests but never cancels
them, so long-lived ones (e.g. VictoriaLogs live tailing) make graceful
shutdown timeout, and the resulting `logger.Fatalf` -> `os.Exit` skips
the storage flush and loses data. So adding a cancelable `BaseContext`
that is canceled once `-http.maxGracefulShutdownDuration` elapses.
Updates https://github.com/VictoriaMetrics/VictoriaLogs/issues/1502
Also notes that VictoriaMetrics query execution is deadline-driven and
ignores ctx, so it's a no-op there.
The cross-link between `/metric-relabel-debug` and `/target-relabel-debug` pages has always been rendered, regardless of whether a target `id` query param was present or not. The target ID is responsible for preloading either metrics or the target relabeling config.
The commit hides the cross-link when no target ID is present. Without the ID, pages are essentially the same, so there is no need to link them. It should reduce user confusion.
This commit adds a new flag remoteWrite.inmemoryQueues, which starts a dedicated set of workers for processing only in-memory part of vmagent persistentqueue. It should help to mitigate an
issue when stale data at file-based queue prevents from ingestion recent data.
There is a downside - in case of remote storage is not reachable,
it's possible to get only a part of data ingested and other part queued
into file-based queue. But it should be acceptable, because there is no
strong guarantees for the data ingestion order.
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8833
When tenant filter is a long regexp, its content can be replaced with
`...`, causing tenants to be matched incorrectly.
`applyFiltersToTenants` converts tag filters using `tagFiltersToString`
c497c8c2e9/app/vmselect/netstorage/tenant_filters.go (L106-L112)
Which uses human-readable representation of `TagFilter`
c497c8c2e9/lib/storage/search.go (L390-L397)
This way I can see results from unexpected tenants. See the test, which
fails with
```
--- FAIL: TestApplyFiltersToTenants (0.00s)
tenant_filters_test.go:18: unexpected tenants result; got [{100 0} {116 0} {1239 0}]; want [{100 0} {108 0} {116 0}]
```
---------
Related PR https://github.com/VictoriaMetrics/VictoriaMetrics/pull/11096
Reduce the default staleness_interval from `2*rule_interval` to
`1*rule_interval`, so the lookbehind range in stream aggregation is more
consistent with metricsQL query. Also add a stale sample check during
sample push in case flush hasn't cleaned it in time.
Fixes fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/11102
A specifically crafted backup source (compromised S3/GCS/Azure bucket) could use `..`
components in object names to write files outside the `-storageDataPath`
directory.
Fix by validating all source parts against the destination directory
before restore begins (restore.go), and adding a defense-in-depth panic
guard in `NewDirectWriteCloser` (fslocal.go).
PR https://github.com/VictoriaMetrics/VictoriaMetrics-enterprise/pull/1051
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
Previously, targets were sharded among `vmagent` instances by all target labels after relabeling. The commit adds `-promscrape.cluster.shardByLabels` optional flag to shard targets by specified labels.
For example, with `-promscrape.cluster.shardByLabels=service`, the targets with the same `service` label value will be scraped by the same `vmagent` instance,
which is useful when performing stream aggregation (drop pod label) that requires all metrics with the same `service` label value to be processed on the same `vmagent` instance.
If none of the specified labels are present in the target labels, then all target labels will be used for sharding.
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/11044
PR https://github.com/VictoriaMetrics/VictoriaMetrics/pull/11114
Add a configurable `flushCallbackInterval` to `mergeset.MustOpenTable`
instead of the hardcoded 10s flushCallback ticker. This lets
VictoriaLogs lower it to 1s so filter-cache invalidation keeps up with
live tailing,
See https://github.com/VictoriaMetrics/VictoriaLogs/issues/1477.
In vminsert and vmagent, on shutdown, the `globalStopCh` channel is
closed. This is intended to cause the `runScraper` loop to exit.
However, nothing gave the `globalStopCh` priority over the input
channels, `sighupCh` or `tickerCh`. This causes unpredictable behavior.
For example, if `configCheckInterval` is set to 5s but `loadConfig`
takes 6s, then there will be a new message on `tickerCh `for every
iteration of the `runScraper` loop. Go will choose one channel to pull
from at random, meaning that shutdown can take arbitrarily long.
To address this, check for `globalStopCh` *first* at the start of each
loop, ensuring that it will be reached at most one loop iteration after
the shutdown starts.
Related PR https://github.com/VictoriaMetrics/VictoriaMetrics/pull/11107/
