Without a delay, an attacker can attempt thousands of authentication requests per second at minimal cost. The only indication of such an attack is the `vmauth_http_request_errors_total{reason="invalid_auth_token"}` metric, which may go unnoticed if it isn't being monitored.
Introduce a random delay of 2–3 seconds before returning 401 Unauthorized responses. This significantly increases the cost of sustaining a brute-force attack while having a negligible impact on legitimate clients, which are not expected to generate unauthorized
requests frequently.
OWASP Top10 also recommends adding delays on failed auth: https://owasp.org/Top10/2025/A07_2025-Authentication_Failures.
The added delay could itself be abused as part of a DoS attack by tying up request-processing resources. However, such an attack is easier to detect because it generates sustained unauthorized traffic from identifiable source IPs, allowing operators to block or rate-limit the offending clients using specialized tools.
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/11180
---------
Signed-off-by: Nikolay <nik@victoriametrics.com>
Co-authored-by: f41gh7 <nik@victoriametrics.com>
## Description
We have a simple vmagent setup using `remotewrite.shardByURL` to
horizontally scale stream aggregations, as suggested
[here](https://docs.victoriametrics.com/victoriametrics/vmagent/#sharding-among-remote-storages)
with minimal additional configuration and noticed there was some
optimization opportunities.
## Investigation
We took a
[pprof](https://github.com/user-attachments/files/28968177/cpu.pprof.zip)
and found that our CPU time was in a few primary places.
1. payload protobuf marshaling
2. zstd compresssion
3. Map access in sharding
4. Hashing in sharding
5. Memory copies for timeseries
## Changes
Zstd compression is pretty intractable(besides using
`remotewrite.vmProtoCompressLevel`, which we already do), but there were
some relatively easy gains for the other paths.
1. inline sov() when l < 128 on marshal. Skips calling a bunch of byte
math in a common case
2. Use slices instead of maps for shard keys, in the common case where
there are less than ~10 keys a slice iteration will be faster
3. use `xxhash.Digest.WriteString` rather than buffering values, avoids
some extra allocations and GC work
4. In the simple case where `insertRows` doesn't modify label names or
values construct the remotewrite request from the passed values instead
of copying them. Avoids a bunch of memory copy and GC work.
## Results
Modest but effective, we saw a ~10% reduction in total cpu cores used by
vmagent with these changes.
<img width="2358" height="709" alt="Screenshot 2026-06-15 at 2 24 43 PM"
src="https://github.com/user-attachments/assets/d2931242-c7ed-447b-9cfd-73b29cd0b893"
/>
Related PR https://github.com/VictoriaMetrics/VictoriaMetrics/pull/11113
This commit adds MDX(Monitoring Data eXchange) feature into vmagent.
It allows to detect VictoriaMetrics related components and forward metrics only for those instances.
In default state, mdx detects timeseries based on `vm_app_version` metric name and tracks instance based on "job:instance" key. It holds tracked instances in-memory for 1 hour, which must be enough for the most scrape intervals.
fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/10600
This commit improves performance for sub query requests by properly cache binary operation results.
For context:
When using binary operator in query, vmselect will executes the left and right expressions in parallel and independently. When the left and right expressions contain the same sub-expression, e.g.
```
count( count(vm_requests_total) by (action,addr,cluster,endpoint) ) by (action,addr,cluster)
/
count( count(vm_requests_total) by (action,addr,cluster,endpoint) )
```
```count( count(vm_requests_total) by (action,addr,cluster,endpoint) is a sub expression in both left and right side.```
The sub expression will be executed twice (once in each of the left and right expressions). This introduces additional RPC and resource overhead.
Currently, this feature is hidden behind query request param - `optimize_repeated_binary_op_subexprs`.
Later it could be transited into default behavior.
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/10575
This commit relaxes JWT token format. Previously "vm_access" claim was required.
Which may not be a case for some users. For example, if there is no need at request templating.
The new `jwt` section field `default_vm_access_claim` was added for this purpose.
It's used as fall-back for templating data.
See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/11054
---------
Signed-off-by: Nikolay <nik@victoriametrics.com>
Co-authored-by: f41gh7 <nik@victoriametrics.com>
`http.Server.Shutdown()` waits for in-flight requests but never cancels
them, so long-lived ones (e.g. VictoriaLogs live tailing) make graceful
shutdown timeout, and the resulting `logger.Fatalf` -> `os.Exit` skips
the storage flush and loses data. So adding a cancelable `BaseContext`
that is canceled once `-http.maxGracefulShutdownDuration` elapses.
Updates https://github.com/VictoriaMetrics/VictoriaLogs/issues/1502
Also notes that VictoriaMetrics query execution is deadline-driven and
ignores ctx, so it's a no-op there.
I tried to adopt publish-docs CI job. But it's not possible to add a
content to docs/victoriametrics. CI job from vmestimator overwrite the
whole content.
That's why I'll be copy pasting doc updates from vmestimator repo to VM
manually.
- move SBOM page down as general security recommendations must be
mentioned first
- separate httplib security-related flags to a separate section, so it
can be cross-referenced by products that use this lib
- mention that /metrics might require to be protected
---------
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Signed-off-by: Pablo (Tomas) Fernandez <46322567+TomFern@users.noreply.github.com>
Co-authored-by: Pablo Fernandez <46322567+TomFern@users.noreply.github.com>
Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>
Rewrite the guidance for `-remoteWrite.shardByURL.labels` and
`-remoteWrite.shardByURL.ignoreLabels` from the perspective of the
aggregation config, rather than the other way around, since users
typically define aggregation rules first and then configure sharding.
And the previous statements are wrong.
For example, I have 3 raw series
```
{env=1, pod=1, cluster=1}
{env=1, pod=2, cluster=1}
{env=2, pod=1, cluster=2}
```
If I have `by: [env]`, then `-remoteWrite.shardByURL.labels` cannot be
set to `env,pod`, because that could route `{env=1, pod=1, cluster=1}`
and `{env=1, pod=2, cluster=1}` to different aggregators, causing each
aggregator to produce its own partial result for `env=1`.
In this case, `-remoteWrite.shardByURL.labels` can only use labels that
are a subset of `by` as `env`.
If I have `without: [env, pod]`, then
`-remoteWrite.shardByURL.ignoreLabels` cannot be set to just `env`,
because that would still allow `{env=1, pod=1, cluster=1}` and `{env=1,
pod=2, cluster=1}` to be routed to different aggregators, causing each
aggregator to produce its own partial result for `cluster=1`.
In this case, `-remoteWrite.shardByURL.ignoreLabels` must include all
labels listed in `without`as `env,pod`.
This PR adds a short `AI policy` section to the contributing guide.
We've received quite a few raw AI-generated PRs on VictoriaLogs
recently, so we'd like to clarify our view on AI usage. In short:
1. You can use AI.
2. You are responsible for what you submit, AI or not.
3. Raw AI slop will be closed without review.
* mention actual api path for opentlemetry in HowToPush docs for
consistency with other protocols;
* update vmagent=>vminsert remoteWrite.url path to prometheus-rw, as it
is the only one that is supported in vmagent/vminsert communication;
* add links to diagrams, so users could easier get to the corresponding
docs;
* add some visual notes to increase awarness of missconfigs
---------
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Pablo Fernandez <46322567+TomFern@users.noreply.github.com>
We have a `check-links` target in the vmdocs repository. This is a good
start but it only detects broken links after changes in docs are merged
into master in this repository.
It would be nice to have a way to check the documentation in this
repository (before creating a PR for instance). This PR introduces a
`docs-check-links` target to the docs/Makefile. This lets us see if we
have any new broken links in the current branch/PR before merging into
main.
Pros:
- We can preview changes in docs before merging to master
- No new tools/big changes required, only add a new target to the
Makefile
- Once we fix all links, we could add a GH Actions check
Cons:
- You need access to the vmdocs repo for this target to work
- Running the check in GH Actions could be potentially complex because
it needs to clone vmdocs, build the container, etc
If this seems like a good idea, I could add the same check to the other
repos (VL, VT, etc).
PR https://github.com/VictoriaMetrics/VictoriaMetrics/pull/11121
This commit adds a new flag remoteWrite.inmemoryQueues, which starts a dedicated set of workers for processing only in-memory part of vmagent persistentqueue. It should help to mitigate an
issue when stale data at file-based queue prevents from ingestion recent data.
There is a downside - in case of remote storage is not reachable,
it's possible to get only a part of data ingested and other part queued
into file-based queue. But it should be acceptable, because there is no
strong guarantees for the data ingestion order.
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8833
Before, empty tenant response was cached for 5min. Making all subsequent
read queries to return empty response, even if data for the tenants was
properly ingested.
With this change empty response won't be cached.
This change is important for integration tests, where reads and writes
are performed one after another.
Related PR https://github.com/VictoriaMetrics/VictoriaMetrics/pull/10982/
When tenant filter is a long regexp, its content can be replaced with
`...`, causing tenants to be matched incorrectly.
`applyFiltersToTenants` converts tag filters using `tagFiltersToString`
c497c8c2e9/app/vmselect/netstorage/tenant_filters.go (L106-L112)
Which uses human-readable representation of `TagFilter`
c497c8c2e9/lib/storage/search.go (L390-L397)
This way I can see results from unexpected tenants. See the test, which
fails with
```
--- FAIL: TestApplyFiltersToTenants (0.00s)
tenant_filters_test.go:18: unexpected tenants result; got [{100 0} {116 0} {1239 0}]; want [{100 0} {108 0} {116 0}]
```
---------
Related PR https://github.com/VictoriaMetrics/VictoriaMetrics/pull/11096
Reduce the default staleness_interval from `2*rule_interval` to
`1*rule_interval`, so the lookbehind range in stream aggregation is more
consistent with metricsQL query. Also add a stale sample check during
sample push in case flush hasn't cleaned it in time.
Fixes fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/11102
