## Description
We have a simple vmagent setup using `remotewrite.shardByURL` to
horizontally scale stream aggregations, as suggested
[here](https://docs.victoriametrics.com/victoriametrics/vmagent/#sharding-among-remote-storages)
with minimal additional configuration and noticed there was some
optimization opportunities.
## Investigation
We took a
[pprof](https://github.com/user-attachments/files/28968177/cpu.pprof.zip)
and found that our CPU time was in a few primary places.
1. payload protobuf marshaling
2. zstd compresssion
3. Map access in sharding
4. Hashing in sharding
5. Memory copies for timeseries
## Changes
Zstd compression is pretty intractable(besides using
`remotewrite.vmProtoCompressLevel`, which we already do), but there were
some relatively easy gains for the other paths.
1. inline sov() when l < 128 on marshal. Skips calling a bunch of byte
math in a common case
2. Use slices instead of maps for shard keys, in the common case where
there are less than ~10 keys a slice iteration will be faster
3. use `xxhash.Digest.WriteString` rather than buffering values, avoids
some extra allocations and GC work
4. In the simple case where `insertRows` doesn't modify label names or
values construct the remotewrite request from the passed values instead
of copying them. Avoids a bunch of memory copy and GC work.
## Results
Modest but effective, we saw a ~10% reduction in total cpu cores used by
vmagent with these changes.
<img width="2358" height="709" alt="Screenshot 2026-06-15 at 2 24 43 PM"
src="https://github.com/user-attachments/assets/d2931242-c7ed-447b-9cfd-73b29cd0b893"
/>
Related PR https://github.com/VictoriaMetrics/VictoriaMetrics/pull/11113
This commit adds MDX(Monitoring Data eXchange) feature into vmagent.
It allows to detect VictoriaMetrics related components and forward metrics only for those instances.
In default state, mdx detects timeseries based on `vm_app_version` metric name and tracks instance based on "job:instance" key. It holds tracked instances in-memory for 1 hour, which must be enough for the most scrape intervals.
fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/10600
This commit relaxes JWT token format. Previously "vm_access" claim was required.
Which may not be a case for some users. For example, if there is no need at request templating.
The new `jwt` section field `default_vm_access_claim` was added for this purpose.
It's used as fall-back for templating data.
See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/11054
---------
Signed-off-by: Nikolay <nik@victoriametrics.com>
Co-authored-by: f41gh7 <nik@victoriametrics.com>
`http.Server.Shutdown()` waits for in-flight requests but never cancels
them, so long-lived ones (e.g. VictoriaLogs live tailing) make graceful
shutdown timeout, and the resulting `logger.Fatalf` -> `os.Exit` skips
the storage flush and loses data. So adding a cancelable `BaseContext`
that is canceled once `-http.maxGracefulShutdownDuration` elapses.
Updates https://github.com/VictoriaMetrics/VictoriaLogs/issues/1502
Also notes that VictoriaMetrics query execution is deadline-driven and
ignores ctx, so it's a no-op there.
The cross-link between `/metric-relabel-debug` and `/target-relabel-debug` pages has always been rendered, regardless of whether a target `id` query param was present or not. The target ID is responsible for preloading either metrics or the target relabeling config.
The commit hides the cross-link when no target ID is present. Without the ID, pages are essentially the same, so there is no need to link them. It should reduce user confusion.
This commit adds a new flag remoteWrite.inmemoryQueues, which starts a dedicated set of workers for processing only in-memory part of vmagent persistentqueue. It should help to mitigate an
issue when stale data at file-based queue prevents from ingestion recent data.
There is a downside - in case of remote storage is not reachable,
it's possible to get only a part of data ingested and other part queued
into file-based queue. But it should be acceptable, because there is no
strong guarantees for the data ingestion order.
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8833
When tenant filter is a long regexp, its content can be replaced with
`...`, causing tenants to be matched incorrectly.
`applyFiltersToTenants` converts tag filters using `tagFiltersToString`
c497c8c2e9/app/vmselect/netstorage/tenant_filters.go (L106-L112)
Which uses human-readable representation of `TagFilter`
c497c8c2e9/lib/storage/search.go (L390-L397)
This way I can see results from unexpected tenants. See the test, which
fails with
```
--- FAIL: TestApplyFiltersToTenants (0.00s)
tenant_filters_test.go:18: unexpected tenants result; got [{100 0} {116 0} {1239 0}]; want [{100 0} {108 0} {116 0}]
```
---------
Related PR https://github.com/VictoriaMetrics/VictoriaMetrics/pull/11096
Reduce the default staleness_interval from `2*rule_interval` to
`1*rule_interval`, so the lookbehind range in stream aggregation is more
consistent with metricsQL query. Also add a stale sample check during
sample push in case flush hasn't cleaned it in time.
Fixes fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/11102
A specifically crafted backup source (compromised S3/GCS/Azure bucket) could use `..`
components in object names to write files outside the `-storageDataPath`
directory.
Fix by validating all source parts against the destination directory
before restore begins (restore.go), and adding a defense-in-depth panic
guard in `NewDirectWriteCloser` (fslocal.go).
PR https://github.com/VictoriaMetrics/VictoriaMetrics-enterprise/pull/1051
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
Previously, targets were sharded among `vmagent` instances by all target labels after relabeling. The commit adds `-promscrape.cluster.shardByLabels` optional flag to shard targets by specified labels.
For example, with `-promscrape.cluster.shardByLabels=service`, the targets with the same `service` label value will be scraped by the same `vmagent` instance,
which is useful when performing stream aggregation (drop pod label) that requires all metrics with the same `service` label value to be processed on the same `vmagent` instance.
If none of the specified labels are present in the target labels, then all target labels will be used for sharding.
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/11044
PR https://github.com/VictoriaMetrics/VictoriaMetrics/pull/11114
Add a configurable `flushCallbackInterval` to `mergeset.MustOpenTable`
instead of the hardcoded 10s flushCallback ticker. This lets
VictoriaLogs lower it to 1s so filter-cache invalidation keeps up with
live tailing,
See https://github.com/VictoriaMetrics/VictoriaLogs/issues/1477.
In vminsert and vmagent, on shutdown, the `globalStopCh` channel is
closed. This is intended to cause the `runScraper` loop to exit.
However, nothing gave the `globalStopCh` priority over the input
channels, `sighupCh` or `tickerCh`. This causes unpredictable behavior.
For example, if `configCheckInterval` is set to 5s but `loadConfig`
takes 6s, then there will be a new message on `tickerCh `for every
iteration of the `runScraper` loop. Go will choose one channel to pull
from at random, meaning that shutdown can take arbitrarily long.
To address this, check for `globalStopCh` *first* at the start of each
loop, ensuring that it will be reached at most one loop iteration after
the shutdown starts.
Related PR https://github.com/VictoriaMetrics/VictoriaMetrics/pull/11107/
The log message will display when deletion API was called, how many
series it deleted and what params were used. This should help
identifying events of metrics deletion.
Example:
```
2026-06-12T13:02:28.006Z info VictoriaMetrics/app/vmselect/prometheus/prometheus.go:529 /api/v1/admin/tsdb/delete_series has been called for "[{__name__=\"vm_http_request_errors_total\"}]". Deleted 0 series.
```
move increase and increase_prometheus outputs to a separate struct
to simplify the code.
Before, increase, increase_prometheus, total_prometheus
were implemented within one aggregator making it harder to use or update it.
The change should help surface why the test fails sometimes. My expectations
that CreateSnapshot returns another error than expect but since we never
check it we move to the next assert which than fail
```
2026-04-09T14:04:32.674Z info
/home/runner/work/VictoriaMetrics-enterprise/VictoriaMetrics-enterprise/lib/snapshot/snapshot.go:35
Creating snapshot
--- FAIL: TestHTTPClient_CreateSnapshot (0.00s)
client_test.go:53: unexpected error
2026-04-09T14:04:32.675Z info
/home/runner/work/VictoriaMetrics-enterprise/VictoriaMetrics-enterprise/lib/snapshot/snapshot.go:81
Deleting snapshot foo
```
PR https://github.com/VictoriaMetrics/VictoriaMetrics-enterprise/pull/1021
CI example: https://github.com/VictoriaMetrics/VictoriaMetrics-enterprise/actions/runs/24194217809/job/70619495033
Previously, bucket spans were incorrectly re-used due to missing `clear`
call.
It could produce unexpected results with span bucket value from
previously parsed histogram.
In addition, if timeseries had a multiple data values for the same
histogram, metric name could add unexpected `bucket` prefix multiple
times, producing `_bucket_bucket` metric names.
This commit properly clears reused memory.
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/11041
This commit adds four flags to allow managing label promotion for
resource attributes and OTel scope metadata, while staying compatible
with
[Prometheus](https://prometheus.io/docs/prometheus/latest/configuration/configuration/):
- `-opentelemetry.promoteScopeMetadata` - promote OTel scope metadata
(i.e. name, version, schema URL, and attributes) to metric labels.
- `opentelemetry.promoteAllResourceAttributes` - promote all resource
attributes to labels, except for the ones configured with
`-opentelemetry.ignoreResourceAttributes`.
- `opentelemetry.promoteResourceAttributes` - promote specific list of
resource attributes to labels. It cannot be configured simultaneously
with `opentelemetry.promoteAllResourceAttributes`.
- `opentelemetry.ignoreResourceAttributes` - which resource attributes
to ignore, can only be set when
`opentelemetry.promoteAllResourceAttributes` is true.
`-opentelemetry.promoteScopeMetadata` and
`opentelemetry.promoteAllResourceAttributes` are enabled by default in
order to preserve the current behavior.
fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/10931.
Previously, errors in app/vmalert-tool and lib packages used the %w verb
in logger.Errorf calls, which is intended for wrapping errors via fmt.Errorf.
Using %w with the logger package does not wrap the error — instead, it prints
a malformed %!w(...) placeholder rather than the actual error message.
This commit replaces all affected occurrences of %w with %s to correctly
format and display errors.
Related PR: https://github.com/VictoriaMetrics/VictoriaMetrics/pull/10962
This commit adds a warning message, if `-memory.allowedBytes` has value less than 1MB.
It should help to debug possible issues, if there is a problem with app start-up due to low memory limit.
For example, fastcache could panic at `-memory.allowedBytes=`
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/10935
Add '=' separator between label name and value when computing the hash
to prevent false collisions, like {a="bc"} and {ab="c"} hashing to the
same value.
getLabelsHashForShard is added to avoid sharding disruptions in vmagent
(-remoteWrite.shardByURL=true mode). The function preserves previous
behavior, without '=' between name and value.
PR https://github.com/VictoriaMetrics/VictoriaMetrics/pull/10937
### Describe Your Changes
Fix stale `quantiles(...)` stream aggregation output for series without
samples in the current aggregation interval.
Previously, `quantilesAggrConfig` reused the `quantiles` buffer across
aggregation values. If `quantilesAggrValue.flush` was called for a
series without samples after another series had already calculated
quantiles, the stale quantile
values could be emitted for the empty series.
This could produce unrealistic `*_quantiles` output values and make the
same aggregated value appear across unrelated labelsets.
The PR skips `quantiles(...)` output when there is no histogram for the
current interval and adds a regression test for this case.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/#pull-request-checklist).
- [x] My change adheres to [VictoriaMetrics development
goals](https://docs.victoriametrics.com/victoriametrics/goals/).
---------
Co-authored-by: hagen1778 <roman@victoriametrics.com>
synctest runs inner closure in a new goroutine, which makes `t.Helper` instruction
useless on `t.Fatalf` checks. So when test fails we observe the log line where `t.Fatalf`
was called, instead of where `f()` was called.
Moving checks out of synctest closure makes `t.Helper` useful again.
--
In the synctest we were waiting for ingest a new batch of samples for aggregation interval.
Because of this, the new batch had 50% chance to be ingested in the previous or current
aggregation interval, depending on whether go run time initiated flush() call or no.
This change waits for additional 1ms for flush to happen. Locally, it stopped producing
flaky tests.
---------
Signed-off-by: hagen1778 <roman@victoriametrics.com>
This commit introduces a new metric to expose fs type for the provided path.
For example:
```
vm_fs_info{path="/vmstorage-data", fs_type="xfs"}
```
Path must be registered with new method `fs.RegisterPathFsMetrics`.
fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/10482
This commit adds possibility to omit tenantID in the URL path. In this case,
tenantID will be fetched from HTTP headers `AccountID` and `ProjectID`.
If headers are missing too, then default `0:0` tenantID is used.
This functionality can be enabled only if -enableMultitenantHandlers
cmd-line flag was set to vminsert, vmselect or vmagent.
Motivation: this change makes VM configuration for multienancy
consistent with VL configuration - see
https://docs.victoriametrics.com/victorialogs/#multitenancy. And keeps
backward compatibility in the same time.
fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4241
On 2025-12-16, Hetzner Cloud deprecated the `datacenter` field in their
Servers API and introduced a top-level `location` field carrying the
same data. The `datacenter` field will be removed after 2026-07-01.
Without this change, `__meta_hetzner_hcloud_datacenter_location`, and
`__meta_hetzner_hcloud_datacenter_location_network_zone` would silently
become empty for the `hcloud` role after that date.
This mirrors the change made in Prometheus v3.11.0
([prometheus/prometheus#17850](https://github.com/prometheus/prometheus/pull/17850)).
## Changes
**`hcloud` role:**
- Add `HCloudLocation` struct and `Location` field on `HCloudServer`,
mapped to the new top-level `location` API field
- Emit two new canonical labels: `__meta_hetzner_hcloud_location` and
`__meta_hetzner_hcloud_location_network_zone`
- Keep the deprecated `__meta_hetzner_hcloud_datacenter_location` and
`__meta_hetzner_hcloud_datacenter_location_network_zone` labels, now
sourced from the new `location` field so they continue to work past
2026-07-01
- `__meta_hetzner_datacenter` (the datacenter name, e.g. `fsn1-dc14`) is
unaffected for this role — the datacenter name is a distinct concept
from location and is kept as-is (this will stop working starting
2026-07-01)
**`robot` role:**
- Add `__meta_hetzner_robot_datacenter` as the canonical replacement for
`__meta_hetzner_datacenter`; the old label is kept for backward
compatibility
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/10909
Previously, if `-tls` flag was provided, victoria metrics components
produced the following log error entry at health checks:
http: TLS handshake error from 10.244.0.1:46556: EOF
Such health checks are common for many orchestration systems, such as
consul
or kubernetes. And default http server already suppresses such EOF
health checks.
This commit adds suppression to the tls server as well.
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/10538
Previously, metricMetadata was not properly reset during parsing of
metrics. It could result into `Unit` suffix to be added from previously
parsed metric into next metric without Unit field.
For example, metric `http_request` with `Unit` `seconds` will be
converted into `http_request_seconds` and `Unit` field hold `seconds`.
Next parsed metric `cpu_usage_ratio` has no `Unit` and it will get
previous `seconds` `Unit` -> `cpu_usage_ratio_seconds`.
This commit adds metricMetadata reset call before parsing of next
metric.
Bug was introduced at 293d80910c
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/10889
Add the support of storage and retrieval of samples with future
timestamps as requested in https://github.com/VictoriaMetrics/VictoriaMetrics/issues/827
What to expect:
- By default, the max future timestamp is still limited to `now+2d`. To
change it, set the `-futureRetention` flag in `vmstorage`. The max flag
value is currently limited to `100y`. It can be extended if we see a
demand for this, but it can't be more than `~ 290y` due to how the time
duration is implemented in Go. The flag value can't be less than `2d`.
- downsampling and retention filters (available in enterprise edition)
are currently not supported for future timestamps
- If `vmstorage` restarts with a smaller value of `-futureRetention`
flag, any future partitions that are outside the new future retention
will be automatically deleted.
- Data ingestion, data retrieval, backup/restore, timeseries (soft)
deletion, and other operations work with future timestamps the same way
as with the historical timestamps.
- In the cluster version, the affected binaries are `vmstorage` and
`vmselect`. This means that `vmselect` version must match `vmstorage`
version if you want to query future timestamps. `vminsert` was not
affected, so its version can be a lower one.
- If you downgrade the `vmstorage`, the data with future timestamps will
remain on disk and memory (per-partition caches) but won't be available
for querying.
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
Signed-off-by: Artem Fetishev <149964189+rtm0@users.noreply.github.com>
Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>
Update `timeutil.ParseTimeAt` to check the time limits for all date/time formats, not just year.
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
