### Describe Your Changes
Some messages were written to `stdout` using `fmt.Printf` and
`fmt.Println`, while the other messages like import statistics were
written to `stderr` through the `log` package.
This led to ordering problems where the `Import finished!` +
`VictoriaMetrics importer stats` messages, which expected to be the last
messages, appeared before `Continue import process with filter`
messages, creating confusing output for users.
```
2025/08/20 13:07:26 Import finished!
2025/08/20 13:07:26 VictoriaMetrics importer stats:
time spent while importing: 20h49m10.8497184s;
total bytes: 277.1 GB;
bytes/s: 3.7 MB;
requests: 7978614;
requests retries: 0;
2025/08/20 13:07:26 Total time: 20h49m10.851006088s
Continue import process with filter
filter: match[]={__name__!=""}
start: 2025-08-08T00:00:00Z
end: 2025-08-15T00:00:00Z:
Continue import process with filter
filter: match[]={__name__!=""}
start: 2025-08-15T00:00:00Z
end: 2025-08-19T16:18:15Z:
```
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/#pull-request-checklist).
- [x] My change adheres to [VictoriaMetrics development
goals](https://docs.victoriametrics.com/victoriametrics/goals/).
### Describe Your Changes
It seems db39f045e1 accidentally reverted
#9419 changes.
```patch
--- a/app/vmagent/remotewrite/client.go
+++ b/app/vmagent/remotewrite/client.go
@@ -448,7 +448,8 @@ again:
}
metrics.GetOrCreateCounter(fmt.Sprintf(`vmagent_remotewrite_requests_total{url=%q, status_code="%d"}`, c.sanitizedURL, statusCode)).Inc()
- if statusCode == 409 {
+ switch statusCode {
+ case 409:
logBlockRejected(block, c.sanitizedURL, resp)
// Just drop block on 409 status code like Prometheus does.
@@ -461,7 +462,13 @@ again:
// - Remote Write v2 specification explicitly specifies a `415 Unsupported Media Type` for unsupported encodings.
// - Real-world implementations of v1 use both 400 and 415 status codes.
// See more in research: https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8462#issuecomment-2786918054
- } else if statusCode == 415 || statusCode == 400 {
+ case 415, 400:
+ if c.canDowngradeVMProto.Swap(false) {
+ logger.Infof("received unsupported media type or bad request from remote storage at %q. Downgrading protocol from VictoriaMetrics to Prometheus remote write for all future requests. "+
+ "See https://docs.victoriametrics.com/victoriametrics/vmagent/#victoriametrics-remote-write-protocol", c.sanitizedURL)
+ c.useVMProto.Store(false)
+ }
+
if encoding.IsZstd(block) {
logger.Infof("received unsupported media type or bad request from remote storage at %q. Re-packing the block to Prometheus remote write and retrying."+
"See https://docs.victoriametrics.com/victoriametrics/vmagent/#victoriametrics-remote-write-protocol", c.sanitizedURL)
```
cc @makasim
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/#pull-request-checklist).
- [x] My change adheres to [VictoriaMetrics development
goals](https://docs.victoriametrics.com/victoriametrics/goals/).
Previously, if pushBlockPubSub function returned error, vmagent stopped
remote write worker thread assigned for it. Expected behavior for this
scenario is to retry error inside pushBlockPubSub function. It must
return only on vmagent shutdown.
This commit properly handles this error and prevents from ingestion
stop.
Make it clear that `-vmalert.proxyURL` needs to be applied to
VM single or vmselect.
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit f85fd161e4)
* add missing features
* re-organize text without breaking links to improve clarity
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit a2f033ce6c)
Add build tag `disable_grpc_modules` for vmbackup, vmrestore and
vmbackupmanager. Binary size increases only for 3MB with it. It's
acceptable trade-off for security and feature updates.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8008
* check if built binary is present for `make tsbs-build`. Before, if
build fails, the command stopped working.
* make ENV variables configurable from command line, so `TSBS_STEP=15s
make tsbs-generate-data` would respect the configured step.
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Another batch of documentation improvements
Fix Spelling in:
- Comments in code
- Displayed strings
One change was in a json file used for the anomaly dashboard in docker,
else no other code was changed.
Some Markdown changes, related to standards:
- URLs
- List numbering
- Empty spaces at the end of a line
Previously, vmagent treated differently the following configuration:
1) ./bin/vmagent --remoteWrite.url=url-0 --remoteWrite.url=url-1 --remoteWrite.disableOndiskQueue
and
2)./bin/vmagent --remoteWrite.url=url-0 --remoteWrite.url=url-1 --remoteWrite.disableOndiskQueue=true,true
In first case, it could produce duplicates and blocks ingestion requests if one of remote write targets were not accessible.
In second case, it implicitly added --remoteWrite.dropSamplesOnOverload as true and silently dropped samples for inaccessible target.
This commit treat this configuration as the same and silently drop samples on both cases to mitigate possible duplicates.
It's expected, that vmagent provides delivery guarantees, only if it has a single remote write target, when flag remoteWrite.disableOndiskQueue=true is set.
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/9565
While working on #9431 there has been introduced 2 bugs related to
indexDB.searchMetricName():
1. During the search the index records are unconditionally placed in
sparse index
2. If search touches index records in both prev and curr indexDBs, there
will be possible cases that metricIDs can be unintentionally removed
using `wasMetricIDMissingBefore()` logic
Additionally, the PR moves the searchMetricName from indexDB and Search
to Storage which simplifies the code and makes it spossible to reuse the
function as-is in enterprise code.
Follow up for #9431.
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
`-eula` was deprecated and made no-op in v1.123.0, so examples with
`-eula` will no longer work.
Replace those with proper license configuration.
While at it, remove license flags from vmbackupmanager CLI commands as
it is not required when using CLI.
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
### Describe Your Changes
Do not check vmauth_config_last_reload_success_timestamp_seconds since
it may contain the timestamp < time.Now() due to how lib/fasttime works.
Instead, compare the number of config reloads.
follow up on
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/9369 and
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/9572
Also, split the config update and reload into two separate functions.
master:
```
$gotest -race ./apptest/tests/ -run=TestSingleVMAuthRouterWithInternalAddr -count=40
ok github.com/VictoriaMetrics/VictoriaMetrics/apptest/tests 90.176s
```
pr:
```
$gotest -race ./apptest/tests/ -run=TestSingleVMAuthRouterWithInternalAddr -count=40
ok github.com/VictoriaMetrics/VictoriaMetrics/apptest/tests 46.130s
```
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/#pull-request-checklist).
- [ ] My change adheres to [VictoriaMetrics development
goals](https://docs.victoriametrics.com/victoriametrics/goals/).
(cherry picked from commit c3c802a61c)
Removing extDB from indexDB makes prev, curr, and next indexDBs independent.
I.e. the search is performed independently in prev and curr, the results are
then merged.
Additionally, since no search is now performed in extDB:
- all indexDB search methods now return the original maps used for populating
the result, without invermediate conversion to slices.
- `NoExtDB` suffix has been removed from method names
This has been extracted from #8134.
Signed-off-by: Andrei Baidarov <baidarov@nebius.com>
Co-authored-by: Artem Fetishev <rtm@victoriametrics.com>
Before, we showed summarized 99th percentile for query complexity across
all available instance. This doesn't make much sense, as it doesn't
answer on the following questions:
1. What complexity limits to set per vmselect
2. What are the most expensive queries
The change is to use `max` instead of `sum`, to show only outliers, the
heaviest served queries. The update should help answering on questions
above.
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/#pull-request-checklist).
- [ ] My change adheres to [VictoriaMetrics development
goals](https://docs.victoriametrics.com/victoriametrics/goals/).
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit f99e49c15d)
fixes case, when `histogram_quantile` result contains gaps, that occur
in same time range, where NaNs are present in a first bucket of a
histogram
---------
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 1ba994970b)
Fix flaky integration test `TestSingleVMAuthRouterWithAuth`.
The flakiness is caused by the
`vmauth_config_last_reload_success_timestamp_seconds` metric, which
reports time with second-level precision.
Update the test to account for this when verifying that the config
reloads correctly.
Previously, if limit was reached for cardinality limiter, vmstorage
started to perform index lookups for any series exceed limit. Since
storage must skip index creation for such series, it's not possible to
cache it. It resulted into opposite effect of cardinality limiter -
instead of reducing resource usage, it increased it instead.
This commit changes cardinality limit calculation from metricID to the
hash from raw metricName. It could slightly increase CPU usage if
cardinality limiter is configured, since hash must be calculated for
each metricName row. But it mitigates excessive CPU and memory usage on
limit hit
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/9554
### Describe Your Changes
The current one-second timeout for individual read or write operations
during the handshake phase has proven to be insufficient in some
scenarios
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/9345. For
example, short-lived CPU spikes lasting a few seconds can cause
handshake failures due to the low timeout threshold.
While a small timeout may work well in environments with fast and
reliable networking, such as within a single datacenter, it becomes
problematic in more complex setups—particularly in a [multi-level
cluster
setup](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#multi-level-cluster-setup)
where the top-level vmselect may reside in a different availability zone
and work on a less reliable network.
Another issue with the per-operation timeout approach is that it allows
the total time for a handshake to accumulate significantly in the
worst-case scenario. If each operation experiences a delay just under
the timeout threshold, the entire handshake process could take up to 6s.
Which accounts for 60% of `-search.maxQueueDuration` and leaves only 4s
for the actual query.
Introducing a single timeout for the entire handshake process would
provide more predictable behavior and improve usability from a
configuration standpoint. The timeout for the whole handshake op is also
easier to understand from the operator's point of view. Increasing the
timeout value and providing a configuration option for it would make the
system more resilient to transient conditions like CPU contention and
better suited for use cases involving cross-AZ communication.
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/9345
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/#pull-request-checklist).
- [x] My change adheres to [VictoriaMetrics development
goals](https://docs.victoriametrics.com/victoriametrics/goals/).
The commit changes CI behavior:
- Run build in parallel for different os\arch
- Run unit\integration\lint in parallel
- Remove the custom Go cache step in favor of the logic provided in
`actions/setup-go`. The custom cache was used to build key based on
go.sum and makefiles. This logic is preserved.
- Introduce cache for golangci-lint.
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/#pull-request-checklist).
- [ ] My change adheres to [VictoriaMetrics development
goals](https://docs.victoriametrics.com/victoriametrics/goals/).
vmselect is experiencing memory exhaustion and OOM kills
when processing complex Graphite queries with nested functions and large
numbers of label selectors (30k+ values).
The root cause was unbounded growth of the pathExpression field.
This commit adds configurable truncation for Graphite pathExpression fields to
prevent memory exhaustion while preserving query functionality:
New flag: -search.maxGraphitePathExpressionLen=1024 (default 1024
characters)
Safe truncation: Long expressions are truncated with "..." suffix
Zero disables: Set to 0 to disable truncation entirely
Related PR https://github.com/VictoriaMetrics/VictoriaMetrics/pull/9534/
Previously, tcp listener perform synchronous proxy protocol header
read during connection accept. It could significantly reduce vmauth
performance and lead to timeout at serving http requests.
This commit changes this logic and performs proxy protocol header
parsing during first Read request from connection or RemoteAddr method
call. It significantly improves performance and reduce possible
bottleneck at connections accept method.
Related PR https://github.com/VictoriaMetrics/VictoriaMetrics/pull/9546/
- Rename WriteRequestUnmarshaller to WriteRequestUnmarshaler
- Add a description to WriteRequestUnmarshaler struct
Review comments
b98e592752 (r163365472)
Follow up on
b98e592752
* remove duplicated content between single and cluster versions
* mention recommendation to group component types by jobs in scrape
config
* link the example of scrape configs
* update wording
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 516a454f0a)
This metric can be used for building alerts and graphs for free disk space usage percentage by using the following MetricsQL query:
100 * (vm_free_disk_space_bytes / vm_total_disk_space_bytes)
### Describe Your Changes
By default `zstd.Reader` creates multiple goroutines to process a single
connection:
- It doesn't match cgo behavior, which works synchronously, and creates
a lot more concurrent goroutines (0.5k -> 5k on my workload)
- It results in non-zero `vm_tcpdialer_errors_total{type="read"}` errors
on vmselect because an underlying connection is closed while a goroutine
is still reading from it. The goroutine created by
`zstd.NewReader`/`zstd.Reset`
abb348e4db/lib/handshake/buffered_conn.go (L113-L120)
- vmselect (and vmagent) doesn't benefit from async mode since it has
multiple readers in-use at the same time, which usually exceeds the
number of cpu cores
Partly related to #9218
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/#pull-request-checklist).
- [x] My change adheres to [VictoriaMetrics development
goals](https://docs.victoriametrics.com/victoriametrics/goals/).
This commit introduces new storage API: `/internal/log_new_series`.
It helps to dynamically debug newly created series. Changing `-logNewSeries` value requires storage restart,
which may introduce downtime and is not recommended for production deployments.
In addition, this commit adds flags: `-logNewSeriesAuthKey`, which protects newly added API.
Fixes: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8879
This change will expose any IPv6 addresses assigned to an instance under
the meta labels:
* `__meta_gce_public_ipv6` -native IPv6 address, globally routed
* `__meta_gce_internal_ipv6` - unique local address (ULA).
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/pull/9370
Previously, the tenant selector was hidden when only one tenant
was returned, making it impossible to run queries in multi-tenant mode.
Now, the selector is always shown as long as at least one tenant exists.
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/9396
The prompb and prompbmarshal share exactly the same models and provide
marshal and unmarshale capabilities for them. This creates duplication
(changes in one model has to be made in another, case with metadata) and
confusion where for example you compare same looking models but golang
says they are not the same (because of the type).
This commit merge prompbmarshal logic into prompb so the rest of the
code is aligned on prompb models.
Moves samplesPool and labelsPool to WriteRequestUnmarshaller.
Make WriteRequest struct clean from unmarshal logic.
The benchmark shows no significant changes:
$benchstat prompbmarshal.bench prompb2.bench
goos: darwin
goarch: arm64
pkg: github.com/VictoriaMetrics/VictoriaMetrics/lib/prompb
cpu: Apple M1 Pro
│ prompbmarshal.bench │
prompb2.bench │
│ sec/op │ sec/op vs
base │
WriteRequestUnmarshalProtobuf-10 189.2µ ± 5% 190.8µ ± 8%
~ (p=0.579 n=10)
WriteRequestMarshalProtobuf-10 145.3µ ± 7% 143.6µ ± 2%
~ (p=0.143 n=10)
geomean 165.8µ 165.5µ
-0.14%
│ prompbmarshal.bench │
prompb2.bench │
│ B/s │ B/s
vs base │
WriteRequestUnmarshalProtobuf-10 50.42Mi ± 5% 49.99Mi ± 8%
~ (p=0.593 n=10)
WriteRequestMarshalProtobuf-10 65.64Mi ± 7% 66.39Mi ± 2%
~ (p=0.143 n=10)
geomean 57.53Mi 57.61Mi
+0.14%
│ prompbmarshal.bench │
prompb2.bench │
│ B/op │ B/op
vs base │
WriteRequestUnmarshalProtobuf-10 27.70Ki ± 4% 26.90Ki ± 7%
~ (p=0.190 n=10)
WriteRequestMarshalProtobuf-10 3.267Ki ± 12% 3.273Ki ± 12%
~ (p=0.971 n=10)
geomean 9.514Ki 9.383Ki
-1.38%
│ prompbmarshal.bench │
prompb2.bench │
│ allocs/op │ allocs/op vs
base │
WriteRequestUnmarshalProtobuf-10 0.000 ± 0% 0.000 ± 0%
~ (p=1.000 n=10) ¹
WriteRequestMarshalProtobuf-10 0.000 ± 0% 0.000 ± 0%
~ (p=1.000 n=10) ¹
geomean ²
+0.00% ²
¹ all samples are equal
² summaries must be >0 to compute geomean
This commit introduces new flag: `-storage.idbPrefillStart` with
default value of `1h`. It allows to adjust start time of the prefill
process for the data written into the next indexDB.
By default, VictoriaMetrics starts prefill indexDB at 3 A.M UTC, while
indexDB rotates at 4 A.M UTC. It could be useful to change start time
from 3 A.M. to 1 A.M or 00:00 A.M. It should smooth overall resource
usage.
However, changing value to the number bigger than 4 hours for the
default
installations doesn't make much sense ( like 11 P.M. for the day before
rotation). Since, VictoriaMetrics maintenances daily-indexes and it have
to repopulate it twice.
But it could be useful in conjunction with `-retentionTimezoneOffset`,
which could delay index rotation for the current day and give more time
for the prefill process.
As an example, `-retentionTimezoneOffset=4h` adds an additional 4 hours
to the rotation time and `-storage.idbPrefillStart` could be changed
accordingly.
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/9393
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/#pull-request-checklist).
- [ ] My change adheres to [VictoriaMetrics development
goals](https://docs.victoriametrics.com/victoriametrics/goals/).
---------
Signed-off-by: f41gh7 <nik@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
Previously, vmctl tried to create a tmp directory in a directory with
Prometheus snapshot. This is not always possible as snapshot can be
mounted in read-only mode.
Use a system default temporary location and allow user to customize the
tmp path in order to avoid issues with custom tmp locations.
Closes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/9505
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
The `groupWatchersCleaner` iterates through all watchers, attempting to
lock them sequentially while holding the global `groupWatchersLock`.
Therefore, a large number of group watchers can cause the
`groupWatchersLock.Lock()` to block for a noticeable period.
Proposing to use `TryLock` as an optimistic case because if the watcher
lock is held, it's more likely that the watcher is still in-use so no
further cleanup is required.
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/9354#issuecomment-3089108889
### Describe Your Changes
Currently, all errors that occur during the handshake and dial phases
(except for timeouts) are retried in
[execOnConnWithPossibleRetry](0c4062b727/app/vmselect/netstorage/netstorage.go (L2431)).
However, such errors typically result from network issues or CPU
exhaustion on the storage side. In both cases, retrying is unlikely to
succeed and may instead contribute to additional, unnecessary load on
the system.
This PR disables retries for all errors encountered during the handshake
and dial process. The goal is to avoid redundant retry attempts in
scenarios where they are unlikely to help and may worsen the underlying
problem.
Related to
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/9345
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/#pull-request-checklist).
- [ ] My change adheres to [VictoriaMetrics development
goals](https://docs.victoriametrics.com/victoriametrics/goals/).
The lib/fs.MustCloseParallel() accepts a slice of MustWriter items, which must implement only
a single method - MustWrite(). The previous lib/filestream.MustCloseWritersParallel() was
accepting CloseWriter items, which must implement Write() and Path() methods additionally
to MustClose() method. This was adding artificial restrictions on the applicability
of the MustCloseWritersParallel() method. Remove these restrictions.
This should reduce the time needed for the deletion of VictoriaLogs parts
with big number of files, which are created when wide events are ingested into VictoriaLogs
(e.g. logs with big number of log fields).
This may help improving scalability of VictoriaLogs at systems with big number of CPU cores,
which store data into high-latency storage such as Ceph or NFS.
See https://github.com/VictoriaMetrics/VictoriaLogs/issues/517
The import aliases may complicate maintenance of the code in the long term
if they aren't used consistently, e.g. if one file imports the apptest under the default name
while the other file imports the apptest under the "at" name.
The aliases also complicate grepping the code by apptest.* or prompbmarshal.* .
- Drop the code needed for asynchronous removal of the directory on NFS shares.
This code was needed when VictoriaMetrics could keep open files after their deletion
or renaming. This is no longer the case after the commit 43b24164ef .
Now files are deleted only after all the readers close them.
This updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/61
- Unify MustRemoveAll() and MustRemoveDirAtomic() into MustRemoveDir() and MustRemovePath()
functions:
- The MustRemoveDir() deletes the given directory with all its contents, in an "atomic" way:
it creates a special `.delete-this-dir` file in the directory, then removes all its contents
except of this file, and later removes the `.delete-this-dir` file together with the directory
itself. This makes possible easily determining whether the given directory needs to be deleted
after unclean shutdown - if it contains the `.delete-this-dir` file or if it is empty, it must be deleted.
Add IsPartiallyRemovedDir() function, which can be used for detecting whether the given directory must be removed
at starup.
Previously the MustRemoveDirAtomic() was using a "trick" for atomic directory removal: it was "atomically" renaming
the directory to a temporary directory with '.must-remove.' marker in the directory name, and after that it
was removing the renamed directory. On startup all the directories with the `.must-remove.` marker were deleted
if they are left after unclean shutdown. This "trick" doesn't work for NFS and object storage such as S3,
since these storage systems do not support atomic renaming of directories with multiple entries inside.
The new MustRemoveDir() function doesn't use this "trick", so it can be safely used in NFS and S3-like storage systems.
This is based on the pull request from @func25 - https://github.com/VictoriaMetrics/VictoriaMetrics/pull/9486/files .
- The MustRemovePath() deletes the given file or an empty directory.
- Delete the existing parts and partitions at startup if they were partially deleted.
- Consistently use fs.MustRemoveDir() and fs.MustRemovePath() instead of os.RemoveAll() across the codebase.
This reduces the amounts of bolierplate code related to error handling.
- Consistently use fs.MustWriteSync() instead of os.WriteFile() across the codebase.
- Consistently use fs.IsPathExist() for checking whether the given path exists on the filesystem.
- Consistently use fs.MustWriteAtomic() for atomic store of the serialized state into file.
- Consistently panic with 'BUG:' prefix on unexpected errors.
- Read and write the state file contents in one go. This simplifies the code for loading and storing the state.
This shouldn't increase memory usage too much, since the parsed state is already stored in RAM.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/9074
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/9102
The following renamings were requested in #8134 to be done in a separate
PR:
- Rename putTSIDToCache(s) to storeTSIDToCache(s). This is because
get/put prefixes are normally reserved for pools and ref counters. This
unclear though whether name `getTSIDFromCache` is okay in this context.
- Rename `addPartitionNolock` to `addPartitionLocked`, because the
`Locked` suffix is used everywhere else
- Rename deleteMetricIDs to saveDeletedMetricIDs, because no deletion is
actually happening. The deleted metric ids are still present in the
index. One needs to filter the deleted metric ids out after the
retrieval.
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
Previously, it was not possible to compile netBSD binary due to missing OS constrains at lib/fs and lib/filestream packages.
This commit fixes it by:
* apply proper constrains at lib/filestream
* Introduce statfs_t and statfs() to abstract unix internals: NetBSD
needs to use unix.Statvfs_t and unix.Statvfs() unlike other Unix-es.
* apply proper constrain for vmctl terminal package
Related PR https://github.com/VictoriaMetrics/VictoriaMetrics/pull/9473
while there, add a bunch of vlogs articles we've missed to add.
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 282d6fa998)
The newly created data part could become missing after unclean shutdown (such as hardware power off),
since the contents of the parent directory wasn't synced to disk before storing the newly created data part in the parts.json file.
Fix this by syncing the parent directory contents before storing the newly created part in the parts.json file.
This commit is based on https://github.com/VictoriaMetrics/VictoriaLogs/pull/507
This commit adds tmp inmemory and data blocks buffers for
index search requests. It allows to reduce memory allocations on block
cache misses. Since block cache puts block into cache only on after
configured number of cache misses.
Related PR https://github.com/VictoriaMetrics/VictoriaMetrics/pull/9324
This commit respects online CPU count for process_cpu_cores_available metric. Since CPUQuota may exceed it.
Detailed description:
There might be a case when `getCPUQuota()` returns a value bigger than
available logic CPU cores. In this case the lower value should be used.
These changes also reflect upcoming go1.25 changes
https://tip.golang.org/doc/go1.25#container-aware-gomaxprocs
> If the CPU bandwidth limit is lower than the number of logical CPUs
available, GOMAXPROCS will default to the lower limit
In practice that happens with [CPU
Manager](https://kubernetes.io/blog/2022/12/27/cpumanager-ga/) enabled.
It requires setting `/sys/devices/system/cpu/online` which shows the
total number of cores on a node. The container doesn't have any cgroup
limits, but it's pinned to a limited subset of cores.
```
root@vmselect:/# cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us
-1
root@vmselect:/# cat /sys/fs/cgroup/cpu/cpu.cfs_period_us
100000
root@vmselect:/# cat /sys/devices/system/cpu/online
0-255
```
go1.25 and go.uber.org/automaxprocs don't look into
`/sys/devices/system/cpu/online`, VictoriaMetrics does
The chunkedbuffer is released twice, leading to concurrent use of the
buffer after it's acquired from a pool by two different goroutines.
Issue was introduced at v1.115.0 at the following commit 5b87aff830
### Describe Your Changes
Previously, if I run `make publish-vmstorage` it composed the tag name
like:
```
docker.io/victoriametrics/vmstorage:heads-tcpdialer-increase-idle-timeout-0-g1b20c2dbd3-dirty-84b03c0eEXTRA_DOCKER_TAG_SUFFIX
```
It would work okay if I explisitly provide empty
`EXTRA_DOCKER_TAG_SUFFIX=` env var.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/#pull-request-checklist).
- [ ] My change adheres to [VictoriaMetrics development
goals](https://docs.victoriametrics.com/victoriametrics/goals/).
This reverts commit 63e1bf5d97. The reason
for revert is that the change makes increase/rate behavior less
predictable for users. See
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/9420.
Another negative impact of this change is that it negatively affects
recording rules that calculate increases or rates over time series on
big intervals. For example,
`increase(http_errors_total{instance="foo"}[30d])` would stop producing
results if `http_errors_total{instance="foo"}` was marked as stale. But
this is logically incorrect, as `http_errors_total{instance="foo"}` over
last 30d should still return results.
-------------
This change doesn't revert
63e1bf5d97
completely. It keeps changes made to `removeCounterResets` in order to
preserve original `staleNaN` values. Before, `staleNaN` were compared
with float values and producing `NaN` instead. Which could confuse us in
future. So keeping the fix and tests. It shouldn't have any negative
effect.
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/#pull-request-checklist).
- [ ] My change adheres to [VictoriaMetrics development
goals](https://docs.victoriametrics.com/victoriametrics/goals/).
---------
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 194df32a30)
Previously, snapshots were never deleted automatically. This lead to
disk space waste in case snapshots were left behind and never deleted,
which happened in case of backups failure. This required manual
investigation and snapshots cleanup.
This change enables removal of snapshots older than 3 days by default.
This should give enough time to upload backup in time and also make sure
old snapshots are not wasting disk space.
Closes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/9344
---------
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
lib/netutil/tcpdialer.go:35:4: QF1008: could remove embedded field
"connMetrics" from selector (staticcheck)
d.connMetrics.init(ms, "vm_tcpdialer", name, addr)
### Describe Your Changes
Related issue: #9388
- removed all the code related to VictoriaLogs
- updated dependencies to the latest versions
- removed unnecessary `React` import from components
- removed deprecated dependencies such as `lodash.get`,
`@babel/plugin-proposal-nullish-coalescing-operator`,
`@babel/plugin-proposal-private-property-in-object`
- removed unused packages
- fixed proxy for local playground development (after refresh the page
app cannot properly parse the `/#/` in the path)
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/#pull-request-checklist).
- [ ] My change adheres to [VictoriaMetrics development
goals](https://docs.victoriametrics.com/victoriametrics/goals/).
Add unit tests that verify filesystem hierarchy of the indexdb after
opening the storage and after rotation. The test have been originally
added in #8134 but are still applicable to the current version and will
also reduce the diff.
Co-authored-by: Artem Fetishev <rtm@victoriametrics.com>
- Move `prefetchMetricNames`, `SearchMetricNames`, and
`SearchLabelValues` from `Storage` to `indexDB`
- Rename `searchLabelValuesWithFiltersOnTimeRange` to
`searchLabelValuesOnTimeRange`
- Rename `searchLabelValuesWithFiltersOnDate` to
`searchLabelValuesOnDate`
Extracted from #8134.
Co-authored-by: Artem Fetishev <rtm@victoriametrics.com>
### Describe Your Changes
- Split the release process into two steps.
- Add some links
- Described in more details what should be done to check branches in
sync.
- Added some small checks, like running tests, and testing final release
on sandbox.
- Changed the order of some actions (like build final image before
publish release).]
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/#pull-request-checklist).
- [ ] My change adheres to [VictoriaMetrics development
goals](https://docs.victoriametrics.com/victoriametrics/goals/).
### Describe Your Changes
The example for using the `activeAt` template in `vmalert.md` was wrong,
with the `from` field being before the `to` field. This switches them
round to be correct.
### Checklist
The following checks are **mandatory**:
- [X] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/#pull-request-checklist).
When you try to run `make docker-cluster-up` and try to check metrics in
the Explore page of the Grafana, you will see the basic auth request. If
you enter the correct login and password from the `auth-vm-cluster.yml`
it will continue to ask about the username and password. It happens
because the datasource is configured to ask for data vmauth, but in the
configuration, no auth information is provided.
Added the basich auth info to the datasource provision file
Adding note for -dst config. Adding additional reference for snapshot troubleshooting for better accessibility
Making documentation easier to use following customer issues
TestClusterMultiTenantSelect passes in cluster branch but fails in master.
Integration tests must pass in any branch since they do not distinguish
between master and cluster.
The test fails because the multitenant_test.go is different in master and
cluster. This commit makes the file identical in both branches.
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
The remoteAddr and requestURI aren't sent to the client in the error response after the commit 524f58c78c .
They are logged locally instead. This increases security by not exposing potentially sensitive information (remoteAddr and requestURI) to the client.
This doesn't reduce debuggability, since remoteAddr and requestURI are logged locally together with the error message.
### Describe Your Changes
This PR adds support for parsing Unix timestamps (both integer and
float) in the format pipe using the `time:` prefix.
The timestamp precision (seconds, milliseconds, microseconds, or
nanoseconds) is automatically determined based on the value.
It might be worth creating a new prefix for this rather than reusing
`time:`, but I haven't found any compelling reason to extend the syntax.
### Checklist
The following checks are **mandatory**:
- [X] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
This should allow detecting the client, which led to the error logged via SendPrometheusError() function,
e.g. this improves troubleshooting of the logged errors.
This is a follow-up for c9bb4ddeed
This commit renames flag `remoteWrite.retryMaxTime` into `remoteWrite.retryMaxInterval`. New name aligns with corresponding `MinInterval` flag. Previous flag name still could be used, but vmagent will log warning message with suggested migration.
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/9169
While working on #8134 it has been discovered that
`/graphite/index.json` always returns an empty result. This PR adds a
test and a fix. This fix is a no-op for the master but adding it here in
order to reduce the diff in #8134.
Co-authored-by: Artem Fetishev <149964189+rtm0@users.noreply.github.com>
Co-authored-by: Artem Fetishev <rtm@victoriametrics.com>
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/#pull-request-checklist).
(cherry picked from commit dd40ed2456)
Add metrics to track the behavior of the message processor:
- `vl_insert_processors_count` tracks the current number of active log
message processors.
- `vl_insert_flush_duration_seconds{type="protocol"}` measures the time
taken to flush log data from recently parsed logs to the underlying
storage system (in-memory buffering, remote storage nodes, or local
storage).
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/9320
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Re-use existing `lib/snapshot` for snapshot operations in order to reduce code duplicates and unify supported options and error handling.
Previously, vmbackupmanager did not support `snapshot.tls*` options and did not handle errors in a same was as vmbackup does.
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/9340
When the page loads, the query field automatically receives focus, which
is expected. However, this also immediately opens the autocomplete
popup, triggering an unnecessary request and showing suggestions before
the user starts typing.
This PR commit the autocomplete popup from opening on initial page
load.
Related to https://github.com/VictoriaMetrics/VictoriaMetrics/pull/9342
This commits changes release process by introducing new intermediate
docker image tag with `-rcY` suffix. It's needed to properly test
releases for possible regressions without need to erase release docker
images. Which could be cached by some proxy and actually used in
production.
Release candidate image must re-tagged into final release via make
command.
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/9136
This commit adds the following new metric `vm_cache_eviction_bytes_total` for tsid, metricName and metricIDs caches backed by workingset cache.
The problem this is attempting to solve is doing fine grained tuning for
caches on large high churn, high query load VictoriaMetrics deployments.
Cache performance in those scenarios is critical for cluster
performance, however for stability reasons its imperative not to provide
_too much_ cache space compared to available memory or there becomes a
risk of OOM which can easily destabilize a cluster.
Given the cost of memory, especially for large deployments, there is
therefore a need to optimize cache usage to be near the minimum required
to provide acceptable performance under churn, ingestion and query
loads.
VM provides an excellent set of flags for configuring this cache
performance(size and expiry and removal percentage).
However specifically when tuning `prevCacheRemovalPercent` and
`cacheExpireDuration` it can be hard to known exactly what impact
they're having. Sometimes its very clear when theres cyclical behavior
that cacheExpireDuration is involved, but in other scenarios it can be
very unclear why cache miss rate goes up.
Specifically when attempting to debug spikes in cache miss rate there
are questions that are hard to answer:
* Did a new workload come in that was poorly cached?
* Were caches rotated due to expiry and the `prev` cache contained many
active entries due to limited cache size?
* Where the caches rotated due to miss percentage and our tuning is too
aggressive?
In many of these debugging scenarios it would be greatly helpful to have
explicit metrics on how many bytes of key caches were evicted and for
what reasons, it then becomes trivial to associate spikes in eviction
for a specific reason with a miss rate spike that causes poor
performance. The appropriate tuning them becomes much more obvious.
Related to https://github.com/VictoriaMetrics/VictoriaMetrics/pull/9293
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/#pull-request-checklist).
(cherry picked from commit aa342cea3b)
removeCounterResets was operating only using stalenessInterval (when set
by user) and didn't account for user-specified [window] in rollup
functions. Since in rollup functions MetricsQL could look behind for up
to [window] interval - it can capture those datapoints that weren't
accounted in removeCounterResets. With this change, we remove resets on
stalenessInterval+window interval to avoid this corner case.
Fixes
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8935#issuecomment-2978728661
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/#pull-request-checklist).
---------
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 2d27404639)
GA noticed requests to /victorialogs/keyConcepts/ not being served properly
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit e9571576b8)
This commit adds a new option `concurrency` for kafka producer, which adds additional workers
for publishing messages into the kafka cluster.
This setting could be useful at networks with high latency. And it increases write throughput.
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/9249
Headers sent together with queries could contain important information
about the source of the request. This could help to find sources that send specific queries.
Logging all headers could expose sensitive info in logs.
So introducing `-search.logSlowQueryStatsHeaders` to whitelist headers that need
to be logged. See test case for example.
Previously, regex `.+|^$` was optimised into `.+`. But in fact, it must
optimised into `.*`. Since this regex matches any non empty symbols and
empty value. Which combines into any input.
This commit adds a special case of isDotStar check, which covers this
expression.
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/9290
This commit introduces new component - VictoriaLogs Agent (vlagent).
It accepts logs data via any data ingestion protocol supported by
VictoriaLogs and forwards it to the provided remote storages.
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8766
It appears the `lib/mergeset.Item` violates the 3rd rule
https://pkg.go.dev/unsafe#Pointer
> (3) Conversion of a Pointer to a uintptr and back, with arithmetic.
>
> Unlike in C, it is not valid to advance a pointer just beyond the end
of its original allocation:
> ```
> // INVALID: end points outside allocated space.
> b := make([]byte, n)`
> end = unsafe.Pointer(uintptr(unsafe.Pointer(&b[0])) + uintptr(n))
> ```
`CGO_ENABLED=0 go test -trimpath -gcflags=all=-d=checkptr -run
TestTableAddItemsSerial`
```
fatal error: checkptr: pointer arithmetic result points to invalid allocation
This commit adds a special path to item encoding, that prevents invalid memory references.
See this PR for details:
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/9264
The change should solve the problem with loading manifest when vmui is behind reverse-proxy
with basic auth enabled.
(cherry picked from commit 71eef98b0e)
* reduce time wait on notifiers update
* deterministically sort notifiers() call. This improves and tests and output for users
a9b641531b
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 482ae8135f)
Besides `/select/`-prefixed requests vmauth also does `/admin/` requests
to fetch tenant IDs. Without whitelisting, such requests will fail to route.
I am adding this change only to generic vmauth configs where /admin/ access
is expected to be by default.
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 4a7b4ae852)
This is ready, since https://github.com/VictoriaMetrics/cloud/pull/3229
is merged .
This PR deletes the cloud docs folder after moving it into the private
cloud repo. The main reason is to keep things tidy and handle reviews in
a better way, as the helm charts repo is doing.
Future updates should be protected since the github actions file with
rsync command should not be messing with this folder when updating
everything.
### Checklist
The following checks are **mandatory**:
- [X] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/#pull-request-checklist).
(cherry picked from commit 65087f08c4)
### Describe Your Changes
The example in the VictoriaLogs docs (Comments section) is currently
missing an aggregate function in the `stats` pipe:
```logsql
error # find logs with `error` word
| stats by (_stream) logs # then count the number of logs per `_stream` label
| sort by (logs) desc # then sort by the found logs in descending order
| limit 5 # and show top 5 streams with the biggest number of logs
```
However, `stats by (_stream) logs` is invalid syntax - `logs` is
interpreted as a function name, which causes a parsing error.
This fix replaces it with a valid version using `count()`:
```logsql
| stats by (_stream) count() as logs
```
Without `count()`, the query fails with:
```
cannot parse 'stats' pipe: unknown stats func "logs"
```
Signed-off-by: Yury Molodov <yurymolodov@gmail.com>
(cherry picked from commit 644c7a97c8)
* split migration mods in separate sub-section. This removes conflicting
#-anchors and makes it easier to read&modify in future
* remove duplicating wording
* simplify texts
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8964
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/#pull-request-checklist).
---------
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 96b773198f)
Previously, if annotation was update on object, it could result into
duplicate targets register for dropped targets service-discovery page.
Mostly it affects endpoint annotations update. Endpoint holds
annotation with last update time. If any pod that belongs to the given
annotation changed, it causes duplication targets for all pod backed by
the endpoint. It makes service-discovery debug page hard to use.
This commit excludes `__metadata_kubernetes_*_annotation_` from key
generation for dropped targets map. Instead it updates target with new
labels value.
It may lead to some targets collision, but
since hash function already could produce collisions, it should not be a
problem.
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8626
The files with synctest usage are built only if the goexperiment.synctest build tag is set.
This allows running `go test ./lib/...` and `go vet ./lib/...` without the need to pass GOEXPERIMENT=synctest to them.
This is a follow-up for d33d7e20be
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8848
New variable should help to change docker image tag name during release prepare.
Instead of publishing an exact version, like v1.1.0, it should use suffixed version v1.1.0-rc1.
Which should be re-tagged later, when release will be promoted to stable.
Related to https://github.com/VictoriaMetrics/VictoriaMetrics/issues/9136
Signed-off-by: f41gh7 <nik@victoriametrics.com>
### Describe Your Changes
Revival and modification of original PR
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8762 after
discussion on
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7387.
`group.concurrency` is now respected if and only if
`-replay.rulesDelay=0` rather than always. This allows rules to be run
concurrently without ambiguity about rule chaining. If
`-replay.rulesDelay` is set greater than zero concurrency is still
ignored. This will be the default behavior since it defaults to 1s.
Implementation considerations:
I chose to add split some simple logic into a helper function in
preparation for adding `replay.singleRuleEvaluationConcurrency` in a
follow up PR as thats we're we can share that logic.
cc @Haleygo
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/#pull-request-checklist).
---------
Co-authored-by: Hui Wang <haley@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit b3c92540e5)
During the data retrieval, VictoriaMetrics switches to global index
search if the time range is > 40 days. Prior
ba0d7dc2fc, this switch was handled in
indexDB code. That commit moved that logic to the storage level. As the
result, indexDB will search per-day index for whatever time range that
is passed to it.
This broke the BenchmarkHeadPostingForMatchers. It now shows very slow
indexDB performance compared to v1.111.0 (the last release before that
commit). This is because now indexDB tries to search 55 years of data by
spawning a separate goroutine for each day. Even though the data was
inserted only for the current day, running that many goroutines
significantly slows down the search.
The fix is to use global index time range explicitly.
Fixes#9203
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
Similar to other storage caches `storage/metricName` can be very
important to performance, however it is not tunable independently like
other caches.
In high cardinality setups where a large amount of that cardinality is
actively queried we can see a high `metricName` miss rate.
The only way to correct this is to increase available memory either by
provisioning more or by increasing `memory.allowedPercent`, which is
often expensive or undesirable for stability reasons.
Its possible to work around this by increasing `memory.allowedPercent`
and then adjusting `storage.cacheSizeIndexDBDataBlocks` and
`storage.cacheSizeStorageTSID` down as they are the largest caches, but
this is a less ideal solution than being able to directly control this
cache size.
Related to
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8843
### Describe Your Changes
The `DataBlock` contains structs with string fields, and while the
original comment mentioned not holding references to `br`, it wasn't
immediately clear that this also applies to fields like strings within
the data.
This change clarifies that the `writeBlockResultFunc` must not retain
references to any part of `br`, including its fields. This makes it
explicit that even seemingly safe types like strings must be copied if
needed.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/#pull-request-checklist).
The 'select' HTTP request duration metric is currently implemented as a
summary, while the 'insert' HTTP request duration uses a histogram.
All time series under the same metric name must use the same metric
type. Using the summary type for this metric is preferable, as it
significantly reduces the number of unique time series generated. While
summaries have the limitation of not supporting accurate aggregation
across multiple instances or paths, this trade-off is more manageable
than dealing with a high-cardinality explosion caused by histograms.
### Describe Your Changes
- removed unneeded loop for binary field value size extraction, since it
should not be delimited by multiple `\n` symbols
- changed loop condition
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/#pull-request-checklist).
### Describe Your Changes
Related issue: #8749
Added query to requests `/field_names` and `/field_values` to get more
precision results:
- If `filterName` is `_msg` or `_stream_id`, the query cannot be
generated specifically,
so a wildcard query (`"*"`) is returned.
- If `filterName` is `_stream`, the query is generated using regexp
(`{type=~"value.*"}`).
- If `filterName` is `_time`, a simplified query is created by trimming
the value up
to the first occurrence of a delimiter such as `-` or `:`.
- For all other values of `filterName`, a prefix query is returned using
the `query` value with a `*` appended (e.g., `"value*"`).
Related issue: #8806
Enhanced autocomplete with parsed field suggestions from unpack pipe.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Co-authored-by: Nikolay <nik@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
### Describe Your Changes
This PR adds a call to optimize pipes after the `limit` pipe has been
appended.
Related: #9200
While this approach is not ideal, since it forces us to re-optimize all
pipes, but it is simpler.
An alternative would be to reapply only the relevant optimizations
specifically for this case, something like:
```go
func (q *Query) AddPipeLimit(n uint64) {
if len(q.pipes) > 0 {
ps, ok := q.pipes[len(q.pipes)-1].(*pipeSort)
if ok {
if ps.limit == 0 || n < ps.limit {
ps.limit = n
}
return
}
pu, ok := q.pipes[len(q.pipes)-1].(*pipeUniq)
if ok {
if pu.limit == 0 || n < pu.limit {
pu.limit = n
}
return
}
}
q.pipes = append(q.pipes, &pipeLimit{
limit: n,
})
}
```
### Checklist
The following checks are **mandatory**:
- [X] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/#pull-request-checklist).
vlinsert package could be used as an imported dependency. Mostly it's
needed for vlagent, but it could be a case for other applications.
This commit introduces new interface for vlinsertutil package, which
represents actual storage for log rows ingestion.
It includes CanWriteData, which also could be used to introduce
back-pressure mechanism for vlinsert clients.
Related PR: https://github.com/VictoriaMetrics/VictoriaMetrics/pull/9034
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/#pull-request-checklist).
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
Previously, ForEachRow always reset last row fields after iteration.
It makes impossible concurrent iteration with forEachRow, since
ForEachRow performed hidden mutation of LogRows.
This commit resolves this issue by removal of fields reference.
Related to
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/9076
This allows parsing unlimited number of logs in a single HTTP request,
without the need to buffer the logs in memory.
This is needed for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/9070
Thanks to @AndrewChubatiuk for the initial pull request - https://github.com/VictoriaMetrics/VictoriaMetrics/pull/9153
This commit is based on the https://github.com/VictoriaMetrics/VictoriaMetrics/pull/9153 .
It contains the following changes comparing to the original pull request:
- Remove ugly function LineReader.NextLineWithLineFn(). Instead, uglify the Journald parser a bit
with hacky calls to LineReader.NextLine() in order to parse binary-encoded field values.
This should preserve the maintainability of the LineReader, which is shared among multiple protocol parsers,
under control, while keeping the complexity of Journald parsing inside the app/vlinsert/journald package.
- Fix a typo bug inside isNameValid() - `(r < '0' && r > '9')` must be written as `(r < '0' || r > '9')`.
Rewrite isNameValid() into easier to understand code and rename it to isValidJournaldFieldName() for better readability.
Add tests for this function.
- Remove mentioning of the -journald.maxRequestSize command-line flag from VictoriaLogs docs.
- Add the description of the fix to VictoriaLogs changelog.
- Properly increment errorsTotal metric on every journald parse error.
- Add missing protoparserutil.PutUncompressedReader(reader) call, so the reader could be re-used between client requests.
- Remove improperly working code, which tries continuing parsing the request stream after parse errors.
It is impossible to recover reliably from journald parse errors related to reading the data from the request stream,
since the journald protocol format is completely braindead. So it is better to immediately return the error
to the client instead of trying to recover. The only errors, which could be recovered, are related to invalid field names / values.
Such errors are logged with the WARN level and the corresponding fields are skipped.
- Fix incorrect storage of the re-used name and value strings into fb.fields. The contents of the name and value strings
must be copied per every loop, which reads these strings from the request stream. Otherwise the contents of the previously
added Name and Value fields into fb.fields will be overwritten on the next loop.
- Ensure that LineReader.Line is set to nil after LineReader.NextLine() returns false. This should prevent from subtle bugs
when the LineReader.Line is read after LineReader.NextLine() returns false.
### Describe Your Changes
> ⚠️ still draft, don't merge even if already approved
Docs update for vmanomaly v1.24.0 release with stateful service option
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/#pull-request-checklist).
Such filters can be optimized to `*`. This avoid executing other OR filters.
For example, `foo or * or bar` is optimized to `*`, while `foo` and `bar` filters aren't executed.
Such filters are frequently generated by Grafana, so this should improve query performance there.
Use getCompoundToken() instead of getCompoundFuncArg() for obtaining regexp filter value,
since getCompoundFuncArg() skips trailing '*' chars.
This allows detecting invalid queries in the https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8582 .
`Samples` could be confusing for users, especially for alerting rules:
* we say in docs that each returned "series" will create a new alert
* we say that there are "series fetched", so both columns should be
either "series" or "samples".
Let's rename it to `series` for consistency.
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/#pull-request-checklist).
---------
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit ddd686c026)
…s `query` template, but only logging a warning
The `query` template is not supported in replay mode, because we perform
range queries on the rule’s expression, but not on the `query` template.
Previously, if user see error `query template isn't supported in replay
mode`, they need remove the `query` template from the rule for replay
mode.
Also, templating is only used for alerting rules. Replaying alerting
rules don't send notifications(rule annotations are included here) and
users mainly focus on the generated `ALERTS`, the `query` result is
trivial. This pull request shouldn't break things but simplifies the
usage of replay mode for the case.
related https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8746
(cherry picked from commit 0e8007a02b)
### Describe Your Changes
Users try to migrate from systems like Thanos or Prometheus using
`vmctl` and `promtheus` migration protocol, with questions about the
problems when they try to specify the snapshot, but `vmctl` shows `0`
series to be found for migration.
This issue happens because users specify the block folder instead of the
snapshot folder.
Added clarification about snapshot structure and its appearance with
multiple blocks inside.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/#pull-request-checklist).
(cherry picked from commit cbd76ac4dc)
* packaging/make: fix archiving release binaries for cluster
Cluster binaries for non-windows platforms did not include fips binaries, fix that by properly including binaries.
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
See #9144
VictoriaLogs was not correctly parsing timestamps from journald data
when using the default time field configuration. This caused
VictoriaLogs to look for a `_time` field instead of
`__REALTIME_TIMESTAMP` in journald data by default, resulting in
timestamps falling back to ingestion time rather than the actual log
timestamps.
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
The CommonParams.TimeFields is initialized to []{"_time"} by default. This prevent from the proper usage of the -journald.timeField
as the default field for reading log timestamps.
This bug has been introduced in the commit a1a731eb61
While at it, make sure that every parsed log entry has its own timestamp if the timestamp couldn't be read from the log entry.
This provides reliable sort order of the log fields.
### Describe Your Changes
fixes#8535
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
Storage node with large number of partitions (e.g. 150+ partitions with 12Tb of data) will take more than 30 seconds to start. This was causing vmbackupmanager restarts when running vmstorage colocated with vmbackupmanager in a single pod.
Use exponential backoff for retries and increase overall timeout for storage node healthcheck to 3 minutes to avoid vmbackupmanager restarts during storage node startup.
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Using SAS token is not required when copying data within a single
storage account. Using a plain object URL is sufficient for this case. A
single storage account is normally used to store backups so it is safe
to remove SAS tokens usage.
This fixes support of server-side copy when using managed identity as
authentication source as SAS token can only be generated when using
"shared key" type of credentials.
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/9131
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
### Describe Your Changes
removed trailing comma in ignoreFields, which led to _msg field removal,
since it is converted to empty string before filtering
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/#pull-request-checklist).
### Describe Your Changes
This pull request addresses a bug in the Less method of the tagFilter
struct. The original implementation incorrectly assigned the value of
isCompositeB by calling tf.isComposite() instead of other.isComposite().
This caused both isCompositeA and isCompositeB to always have the same
value, leading to incorrect comparisons when determining the order of
tagFilter instances.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/#pull-request-checklist).
---------
Co-authored-by: Zhu Jiekun <jiekun@victoriametrics.com>
Anonymous RSS memory usage per component type is more useful to observe
for finding anomalies.
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 9244557b6e)
### Describe Your Changes
Currently, some PRs have a failed CI due to low code coverage reported
by Codecov. However, the team typically ignore this and relies on other
quality indicators such as thorough code reviews.
This change configures Codecov to continue posting coverage reports
without marking the build as failed.
It also helps reduce confusion for external contributors, who might
otherwise feel pressured to add unnecessary tests just to satisfy
Codecov requirements (for example
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/9002#discussion_r2111651046).
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/#pull-request-checklist).
This commit adds a basic backup/restore test for vmsingle and vmcluster. A
more sophisticated was originally added to the partition index PR
(#8134) and was aimed to test backup/restore when switching back and
forth between legacy and partition index. During the code review it was
decided that it would be good to have a separate test as well since
legacy code will be removed in future and so will the test.
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
This speeds up building the Go builder image significantly (from hours to a few minutes),
since the build speed was limited by the download speed from https://musl.cc , and this speed
was extremely slow (e.g. 10kb/s and slower).
This also improves build security, since the local mirror of musl.cc is under our control.
### Describe Your Changes
This PR addresses two issues:
When tenant labels (e.g. vm_account_id, vm_project_id) are passed via
extra_filters, they were not included in the rollupCache key. This could
cause cache entries to be reused across different tenants, resulting in
incorrect query results.
If a tenant is specified only via extra_filters, and that tenant does
not exist in TenantsCached, it gets silently filtered out by
GetTenantTokensFromFilters, causing the query to fall back to a global
(non-tenant) query — which is likely unexpected and potentially unsafe.
This fix ensures correct tenant scoping and avoids unintended data
exposure or cache pollution.
Related issue
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/9001
---------
Co-authored-by: Zakhar Bessarab <me@zekker.dev>
Co-authored-by: Max Kotliar <kotlyar.maksim@gmail.com>
### Describe Your Changes
Typo? It's called "fields" pipe, not "field".
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
(cherry picked from commit b3d22403eb)
Remove the community-provided dashboard as it remains without updates
for a few years already. Recommending it may hurt user's experience.
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 9e118fe1ee)
Dual stack mode is disabled by default in order to avoid accidentally
exposing components via IPv6 networks.
vmctl does not expose any endpoints and does not allow using default Go
flags as it is using `urfave/cli` lib.
This commit enables IPv6 support by default since there is no security
risks related to network configuration and this make vmctl easier to use
with default configuration.
See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/9116
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
(cherry picked from commit 3553c60399)
### Describe Your Changes
This change adds logging of the number of skipped bytes when a log line
exceeds the configured `insert.maxLineSizeBytes`.
it helps diagnose and tune systems dealing with oversized log records by
showing how much to increase the parameter for the log to fit in
storage.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/#pull-request-checklist).
---------
Signed-off-by: Dmitry Ponomaryov <iamhalje@gmail.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
### Describe Your Changes
Added a log limit if the 200 logs per second limit is reached and a
notification for the user asking them to add a filter to the query
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/#pull-request-checklist).
### Describe Your Changes
This PR adds the explore section to the docs. It emphasizes on
explaining and linking assets for VMUI and
MetricsQL
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/#pull-request-checklist).
### Describe Your Changes
Properly check precense of `/`, previously it was
ignoring a case where "/" would be at the beginning of the string.
This is a follow-up for 00712b18
---------
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
## Problem
In vlcluster evel setups, components like vlselect can still accept and
forward /insert requests. The lack of strict endpoint control increases
the risk of human error and undermines deployment security boundaries.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/9061
## Fix
Add flags to disable the vlinsert and vlselect endpoints. The
`-insert.disable` flag also disables the internalinsert endpoint.
Similarly for vlselect.
---------
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
Previously, if metric_usage_tracker file was corrupted. It prevented
VictoriaMetrics from start and required manual action. Corruption may
happen in various reasons, such as unclean shutdown of the process.
This commit changes panic into error message, in the same way as other
caches do.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/9074
### Describe Your Changes
Previously, any case when cache returned items was skipping lookup of
tenants at vmstorage nodes. This leaded to inconsistent results for
cases when cache contained items to cover only some part of requested
time range.
Fix this by forcing a cache item to cover full requested time range.
This forces cache hits to always be "full hits".
See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/9042
Target branch for this PR is another PR related to the same issue -
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/9048, this is in
order to avoid additional rebasing/merge as this PR will conflict with
cluster branch after initial PR merge. GH will change target for this pr
to cluster brance once #9048 will be merged.
---------
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
This commits adds additional vmselect routes.
Such as `/static`, `/api/v1/status/metric_names_stats` and others.
In addition it properly redirects `/vmui` and `/vmalert` access endpoints requests. Such endpoints require to preserve trailing `/`. Previously it was omitted and redirect requests failed.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/9003
### What this PR does
log error returned by `fastcache.LoadFromFile` before falling back to
creating a new cache instance. this improves observability and helps
detect problems like file corruption or permission issues early.
this replaces `fastcache.LoadFromFileOrNew` with a custom function
`loadFromFileOrNewWithLog` that explicitly logs errors encountered
during cache restoration.
---
### Related Issue
Closes#8934
---
### Test Plan
- manually tested by simulating a missing file scenario
- ensured expected log output on cache load failure
- verified normal cache creation fallback path
---
### Changelog
log error when cache fails to restore from file during workingsetcache
initialization (#8934)
---
### Checklist
- [x] Signed commits
- [x] Follows coding and commit message conventions
- [x] Tested manually
- [x] Scope limited to relevant change
- [x] Changelog entry added
Co-authored-by: Robin Hayer <rshayer95@gmail.com>
Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>
(cherry picked from commit 8e645ea708)
Add the step for running this command after publishing Docker images during the release process.
See docs/victoriametrics/Release-Guide.md
This commit resolves the issue with the missing `latest` and `stable` tags after the https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7336
This also resolve issues with accidental publishing of incorrect Docker images under the `latest` and `stable` tags.
It is very easy to fix incorrectly published `latest` and `stable` tags by re-running the `TAG=v1.x.y make publish-latest` command,
which updates the `latest` and `stable` tags, so they point to the given TAG=v1.x.y.
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
---------
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Max Kotliar <kotlyar.maksim@gmail.com>
(cherry picked from commit 3cc8013dd9)
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
(cherry picked from commit 1209f33c6d)
### Describe Your Changes
Fixed grammatical and phrasing issues in first half of FAQ docs.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
---------
Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>
(cherry picked from commit 3c87e361ba)
This should simplify working with big number of log fields in LogsQL queries.
Examples:
- `... | keep foo*` leaves only fields starting with `foo` prefix
- `... | rm foo*` removes all the fields starting with `foo` prefix
- `... | mv foo* bar*` replaces `foo` prefix with `bar` prefix in log fields
- `... | sum(foo*)` sums all the log fields starting with `foo` prefix
The DataBlock.GetTimestamps() was returning a slice of strings, which belong to the DataBlock.
These strings are changed whenever the DataBlock is re-used for the next block.
So these strings couldn't be assigned to logRow.timestamp and to tailProcessor.lastTimestamps,
which outlive the DataBlock. The commit aa8c18fc9f5d44091d7ca92be6935eeaf3b85d7f broke this assumption,
which triggered the following bugs:
1. The bug, which could return incorrectly sorted results from /select/logsql/query when the 'limit' query arg is passed to it.
The endpoint must return the last 'limit' log entries on the selected time range in this case, and these log entries
must be sorted by _time.
2. The bug, which could return incorrect results from /select/logsql/tail (e.g. it could incorrectly skip some matching logs,
it could return the same logs multiple times and it could return out-of-order logs without proper sorting by _time).
The solution is to return parsed timestamps from the DataBlock.GetTimestamps() function, so they could be safely
used by the caller without worries that they could be changed while in use.
### Describe Your Changes
Fix https://github.com/VictoriaMetrics/VictoriaMetrics/issues/9016.
Data will carry `vm_account_id` and `vm_project_id` labels when
exporting with native export API in cluster.
These labels could be treated as normal labels and be imported to
victoriametrics cluster, hence inconsistent with the source metrics
data.
e.g.:
1. source data: `{__name__="metrics_test"}`.
2. exported data: `{__name__="metrics_test", vm_account_id="0",
vm_project_id="0"}`.
3. re-imported data: `{__name__="metrics_test", vm_account_id="0",
vm_project_id="0", vm_account_id="0", vm_project_id="0"}`.
4. query result for MetricsQL `metrics_test{}`:
`{__name__="metrics_test", vm_account_id="0", vm_project_id="0"}`.
5. expect query result: `{__name__="metrics_test"}`
In VictoriaMetrics cluster, `vm_account_id` and `vm_project_id` label
are only useful when doing multi-tenant export/import. So they should be
remove if the export URL is not for multi-tenant.
This pull request:
- properly remove tenant info when exporting data in native format.
Note:
- Commit 67514c37ef23c22b91638e80e30504be23fa8dc1 is for apptest and
need to be cherry pick to master branch cc @rtm0 .
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
---------
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
Co-authored-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
### Describe Your Changes
`tests-full` (plural) target doesn't exist, but test (singular) does
discovered while working through unrelated PR
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
Add an option to configure metadata of objects when uploading backups.
For AWS S3 also support using object tagging.
Using metadata of objects is useful in order to get extended reports
about bucket content and billing details. It is also useful when
performing queries to bucket content based on metadata.
See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8010
---------
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
Previously, address was always parsed as "host:port" and added port if
it was missing. This leaded to hard to understand errors in case address
was provided in "http://host:port" format.
Improve error validation in order to provide more precise error message
in case of invalid address format.
See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/9029
Previous error message: `cannot dial storageNode
"http://localhost:8488": dial tcp4: address http://localhost:8488: too
many colons in address` and vminsert continue running.
Current error message: `cannot normalize
-storageNode="http://localhost:8480": invalid address
"http://localhost:8480"; expected format: host:port` and vminsert exists
with error status code.
---------
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Previously, if the value of rentetionFilter was changed within the same
retention, storage didn't start background merge for historical data.
This commits changes this behaviour by writing applied
filters into metadata.json. For backward-compatibility it reads content
of appliedRetention.txt file. It should prevent from triggering
background merge on storage update. If needed, manually remove appliedRetention.txt file from
storage/data/PART folder and remove storage.
Also, it properly applies retentionFilter for data back-filling.
Previously, it was ignored and data outside of retention could be
ingested.
In addition, it changes scheduling of historical merges.
Instead 2 separate background processes, storage launches a single
thread. It reduces CPU resource and disk IO resources usage.
Related issues:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8885https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4592
### Describe Your Changes
Previously, invalid label name or value could cause a panic of vmselect
or vmsingle as it was using MustNewLabelsFromString which was added for
usage in tests only.
Fix this by properly handling and propagating error to user interface if
there is any.
See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8661
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Previously, restore mark could be create to a backup which does not exist or incomplete. This would lead to a crash when attempting to perform restore later on.
This commit adds verification of backup availability and completion to prevent such issues from happening. It also adds a verification bypass mechanism for cases when user wants to create a restore mark which is not currently available.
related issues:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5361https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8771
### Describe Your Changes
- dynamically adjusts the concurrent dial limit between 8 and 64 based
on the `-search.maxConcurrentRequests`.
- goroutines now have the chance to access available connections while
awaiting the dial limit token.
Related PR:
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8922
Skipping downsampling rules with filters based on the timestamp and offset leads to unexpected behaviour in case both rules with and without filters are present.
For example, with the following configuration: `-downsampling.period='{__name__="foo"}:60d:2m,7d:4m'`
The user would expect `foo` metrics to be downsampled only after 60d to 2m intervals. But actually pre-filter would skip scoped rule and use global rule after 7d with 4m interval.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8969
### Describe Your Changes
Fix an issue where queries were not triggered when relative time was
selected and the chart was hidden.
Related issue: #8983
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
(cherry picked from commit 51fdd885ea)
### Describe Your Changes
* UI now respects the `sort by` pipe in queries — if it's present, the
order returned by the server is preserved. Related issue: #8660.
* If no `sort by` pipe is used, logs are reversed on the client to show
the newest entries first (since VictoriaLogs returns them in ascending
time order — [see this in the
code](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vlselect/logsql/logsql.go#L1047)).
* Removed redundant client-side time-based sorting logic.
Additionally:
* Log record fields are now sorted alphabetically in UI selectors such
as **Group by field**, **Display fields**, and **Customize columns**.
Related issue: #8438.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
(cherry picked from commit a213f5a423)
Move change 53a6bbfdf8 to the actual
release. Before, it was mistakenly merged to prev release.
Re-classify change from BUGFIX to FEATURE due to following reasons:
* the risk of facing this issue is low, as it reveals itself only for short staleness intervals
* it slightly changes increase_pure logic in a good way. But it is still a change, not bugfix.
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 4375699013)
This change has effect only if one of the flags below are set:
`-search.maxLookback`, `-search.setLookbackToStep` or
`-search.maxStalenessInterval`
These flags instruct query engine to ignore data points outside of the
look-behind window if these data points are beyond the staleness
interval.
This logic is used for `removeCounterResets` function, and in functions
`increase`, `increase_pure` or `delta`. The bug described in
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8935 hit the
corner case when `removeCounterResets` detected the stale series and
`increase` did not.
The reason why staleness detection failed for `increase` is that
`removeCounterResets` calculates interval between real data points. And
`realPrevValue` (that is used by those functions) calculates the
difference between look-behind window start and previous data point.
Which, at smaller gaps or smaller staleness intervals, could affect
staleness detection and make it different to `removeCounterResets`.
This change makes `realPrevValue` to acocunt for staleness between first
data point in captured look-behind window and previous data point.
-------
While there, also updated `increase_pure` logic. It was changed in
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1381 without
good explanation. Turns out, that `increase_pure` always compared last
value on the interval with value before the interval. While other
increase or delta functions did compare it with first data point on
interval, and only if it is missing - with the realPrevValue.
This change makes `increase_pure` logic consistent with other similar
function. The reason why it is not a separate PR is because tests
started to fail once `realPrevValue` callculation logic changed and
there were no good solution to isolate this change.
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
---------
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Max Kotliar <kotlyar.maksim@gmail.com>
(cherry picked from commit 53a6bbfdf8)
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
(cherry picked from commit 8998526384)
### Describe Your Changes
When downloading archives for benchmarks, an error appears saying that
the archive was placed in a new path.
The error could have been prevented by providing the `-L (--location)`
flag that would tell curl to follow the redirect, so in addition to
updating the paths, this flag was added.
### Checklist
The following checks are **mandatory**:
- [X] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
(cherry picked from commit 8223a5235f)
### Describe Your Changes
Fix extra newline escape characters in panel descriptions.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
(cherry picked from commit fe5f2bd5d7)
1. Add `!lex.isEnd()` to prevent an infinite loop. Although the current
code doesn't trigger this bug, it's a latent issue that could occur if
someone modifies the callers or adds new code paths without proper stop
tokens.
This PR improves integrations docs in 2 areas:
- Background set to white, avoiding issues when changing to dark mode.
- Height set to avoid blank spaces
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
When multiple users run tests on the same instance, the first user
creating a folder will own the testStoragePath, which can lead to issues
accessing this folder for other users. This change will allow us to
create unique folders per user.
```
% ls -ld /usr/tmp/vmalert-unittest/
drwxr-xr-x 2 some_user users 4096 May 12 17:22 /usr/tmp/vmalert-unittest/
...
2025-05-20T13:56:16.488Z panic lib/fs/fs.go:132 FATAL: cannot create directory: mkdir /usr/tmp/vmalert-unittest/1747749376488491648: permission denied
panic: FATAL: cannot create directory: mkdir /usr/tmp/vmalert-unittest/1747749376488491648: permission denied
```
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
---------
Co-authored-by: Hui Wang <haley@victoriametrics.com>
Mention that data is flushed every 5s, not 2s.
Hence, expected pending data points is x5.
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 65e0b3b86f)
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8985
When using `AddTimeFilter`, it creates a string representation with the
exact same timestamps but doesn't transform the internal end value. This
is different from the `parseFilterTime` function, which makes the
behavior of these two paths different.
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
---------
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 6d7d22f3e6)
### Describe Your Changes
compose.yml does not exist, only compose-base.yml
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
(cherry picked from commit 6a4757ad06)
This commit re-fines the relabeling cookbok and moves all
relabeling related docs to the same page.
It also removes duplicated information from vmagent readme.
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 134501bf99)
The link to enterprise release guide now points to the doc in
enterprise-single-node branch instead of enterprise master. This is
because we don't use enterprise master.
Additionally the `Public Announcement` section has been removed because
we don't make public announcements for releases.
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
### Describe Your Changes
As planned, we are adding a more structured way of explaining
deployments and tiers.
In this PR the following changes are added:
- The previous tiering section is moved under the deployments
placeholder
- Explanations for users to pick single or cluster
- Explanations for different parameters
- Re-styling of the docs to look more appealing
- Reorder some hanging docs so the sections are more clearly presented
to the users
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
---------
Co-authored-by: Alexander Marshalov <_@marshalov.org>
Apparently, when the doc was created for vmctl the anchor conflicts
weren't accounted for or weren't a thing yet. Now, various migration modes
have conflicting anchors. This should be addressed in follow-up commits.
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 63d8d0c5ac)
Improve wording and instructions. It has been a while since the last
time we updated it (more than 4 years!).
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 6b485c4e46)
### Describe Your Changes
added command to run vmui locally backed by vm playground
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
(cherry picked from commit 0a4f7e0958)
### Describe Your Changes
Related issue: #8925
Properly escaped special characters in field values shown in
autocomplete suggestions.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
(cherry picked from commit 9eb6796bad)
The following test produces duplicate per-day index records on a system
with 1 CPU even when data inserted sequentially:
```
GOEXPERIMENT=synctest taskset -c 0 go test ./lib/storage -run=TestStorageAddRowsForVariousDataPatternsConcurrently/perDayIndexes/serial/sameBatchMetrics/sameRowMetrics/sameBatchDates/diffRowDates
```
See: #8654
Make this test pass by relaxing got and want data equality requirement
if the number of CPUs is 1. This is temporary until one insertion corner
case is fixed:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8948
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
(cherry picked from commit 1916f5be4b)
### Describe Your Changes
Related issue: #7046
Changes:
- add JSX automatic runtime import of React, you don't need to import
React anymore in JSX files.
- add unused imports eslint rule
- add headers to ProcessLiveTailRequest to enable client-side connection
setup
- refactor ExploreLogs: divided the component into several separate
components for better readability and code maintenance
- add live tailing tab to VictoriaLogs
short demo:
https://github.com/user-attachments/assets/3e5f57ee-8e72-4835-9fc6-35c6f38bc9ef
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
---------
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 231bfcf4cf)
### Describe Your Changes
Capitalize "Enterprise" in VictoriaMetrics Enterprise phrase
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
(cherry picked from commit 03ceeb7211)
### Describe Your Changes
Improve relabel bug fix changelog message
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
(cherry picked from commit 118a322aa4)
Move anchors that were kept only for backward-compatibility reasons
to the bottom of the document. So they don't take extra space in main doc.
Demote anchor level to `h6`, so docs engine will stop rendering in the
right navigation column.
In this way, we still keep the backward compatibility: old links will
continue working. And free space occupied by these link in the main doc.
Thanks to @makasim for the idea.
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 266bceaffd)
Commit 3b84f45e0a introduce a typo at `relabelConfigs.IsSet` function. It incorrectly returned value if relabeling configuration is set or not.
As a result, vmagent was not able to properly perform relabel configuration reload.
And incorrectly exposed metrics for reload configuration.
Related issue:
https://github.com/VictoriaMetrics/helm-charts/issues/2119
Previously, headers hash calculation had a typo, instead of `hash.Write` - `hash.Sum` method was used.
It discards any previous writes and as a result digest always had the same value. It prevented from proper config reload.
This commit fixes this typo and properly calculates headers digest.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8931
Previously, OTL attributes fields with KeyValue type were ingested as a single json formated field. It complicates requests and requires extra effort at query time.
This commit adds support for handling nested fields to match the behavior of
other handlers, such as `/jsonline`. KeyValue attribute will be converted into separate field.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8862
### Describe Your Changes
Added npm command `npm run start:logs:playground` to run VictoriaLogs UI
with playground env.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
(cherry picked from commit 82a4889adb)
### Describe Your Changes
Version with the latest changes from prometheus/common have been
released as v0.303.1 so builds are no longer failing.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
(cherry picked from commit f1502b7d16)
It solves the following build error:
```
duplicate menu entry with identifier "Release process guidance" in menu "docs"
```
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit b0bf7fd7d5)
### Describe Your Changes
Update /anomaly-detection/ docs according to release `v1.22.1`
P.s. The links in guides and quickstarts were not intentionally set to
v1.22.1 until we fully restore parallelization in forthcoming releases
and mark it as a go-to release everywhere
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
There are three possible cases for blockResultColumn.getValuesEncoded():
- The blockResultColumn.valuesEncoded is already set. This is the case for manually constructed blockResultColumn,
or if it is cloned via blockResult.clone().
- The blockResultColumn.chSrc is non-nil. In this case the valuesEncoded must be read from the corresponding br.bs,
by applying br.bm filter.
- The blockResultColumn.cSrc is non-nil. In this case the valuesEncoded must be read from the corresponding br.brSrc,
by applying br.bm filter.
It is better from maintainability and debuggability PoV to write this logic in a single getValuesEncoded() function
instead of indirecting it via valuesEncodedCreator.
VictoriaLogs stores min and max column values per every data block.
These values were incorrectly used by min() and max() stats functions
inside updateStatsForAllRows() function. It was assumed that this function
could use min / max values stored in the block, since all the rows in the blockResult
must be processed. But the blockResult contains _filtered_ rows,
e.g. it may have less rows than the number of rows in the original block.
In this case it is unsafe assuming that the min / max values from the original block
exist in the filtered rows inside blockResult.
Add blockResult.isFull() function, which returns true if the blockResult contains all rows
from the original block (e.g. they aren't filtered). Use this function in fast path,
while fall back to slow path, which triggers reading the column values and iterating over them.
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [X] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
* update sorting to reflect more important changes first
* use issue numbers instead of `this issue` wording
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 2c8b6fdd7a)
### Describe Your Changes
Add a link to UI on the welcome page.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
(cherry picked from commit 049768a5ae)
* actualize information
* explain alert state in more details
* re-order content by its importance
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit c492dce761)
### Describe Your Changes
Fix logs not updating after query change
Related issue: #8912
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
### Describe Your Changes
Fixes UI freeze in logs timeline after repeated zooming and panning.
The issue was caused by excessive query requests triggered during fast
timeline interactions. Added debounce to limit the frequency of
requests, similar to the approach used in vmui for metrics.
Related issue: #8655
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
(cherry picked from commit ad4e4a411e)
* tidy up wording
* mention available integrations in quickstart
* cross-reference integrations and data ingestion
* move these docs closer to each other
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 7d199d1d83)
### Describe Your Changes
Limit number of points per series and total response size (30 MiB) on
the Raw Query page to improve UI stability.
Related issue: #7895
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
(cherry picked from commit 49f23a8755)
* move example alerts out of /rules folder, since this folder should
contain only useful rules
* expose ports for vlogs and vmetrics for local debug
* add some comments explamining vmalert config
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 001dc7c985)
The new templating option could contain following options: `prometheus`,
`vlogs` or `graphite`.
It represents the datasource type this rule is evaluated against. The
new templating option
can be used in rule's annotations or for routing to a specific
destination (e.g. Grafana datasource).
---------
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 4745b0c511)
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
(cherry picked from commit e1630aab4e)
This functionality was broken in
0e313e5355
Was caught by integration tests:
```
--- FAIL: TestSingleVMAuthRouterWithInternalAddr (5.00s)
vmauth_routing_test.go:148: Could not start vmauth: could not extract some or all regexps from stderr: ["pprof handlers are exposed at http://(.*:\\d{1,5})/debug/pprof/"]
```
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit e77df5d00b)
VictoriaLogs happily accepts logs without _msg field, and there are legitimate cases where the _msg field isn't needed.
Remove the warning in order to reduce the level of confusion for VictoriaLogs users.
This feature is useful when ingesting logs, which may contain timestamps across different fields.
Then the first non-empty field from the _time_field list is used as a timestamp field.
This is needed in order to be able to find these playgrounds by `demo` word.
While at it, replace misleading `playground` word with less confusing wording across different places of the docs.
Previously `*_last_reload_successful` metrics are set to 1 even when respective
configuration is not defined, besides stream aggregation and scraping.
This commit initializes config reload related metrics only when respective
configuration is set.
Related issue:
https://github.com/VictoriaMetrics/helm-charts/issues/2119
This commit adds the following changes to the enterprise version:
- add make target for testing in FIPS mode
- disallow using OVH in FIPS mode. OVH is using SHA1 for authentication via headers and SHA1 is not allowed to be used in FIPS mode. There is no option to switch to another hashing algorithm in OVH API, so disabling it completely.
- build fips binaries together with regular ones. This will allow to make sure that FIPS builds are always up to date and compatible with regular ones.
- disable CGO in FIPS builds for vmagent, since vmagent imports Kafka library which uses CGO imports. This might lead to using OpenSSL version which is not certified for FIPS mode. Using pure Go implementation allows to avoid this and keep all validations on Go build process side.
- vm_retention_filtering_partitions_scheduled is a guage metric that shows now
many partitions the retention filtering is currently being applied to.
- vm_retention_filtering_partitions_scheduled_size_bytes is a guage metric that
shows the total size (in bytes) of partitions the retention filtering is
currently being applied to.
These metrics are similar to vm_downsampling_partitions_scheduled and
vm_downsampling_partitions_scheduled_size_bytes and can be used for observing
how much time it took for a given vmstorage instance to perform retention
filtering across this many partitions whose total size was this many bytes.
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit a23c277b04)
* {lib/backup,app/}: gracefully cancel currently running operation during graceful shutdown
Make backup/restore process interruptable by passing global context from the operation caller.
This is needed in order to reduce shutdown delays in case backup/restore cancellation is requested.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8554
Adds a hint to check for errors on the client side when a network error
occurs during the handshake.
Follow-up on commit 53170abdccd2ca3f5952a916c5f544e0e77b5596
This PR fixes two related bugs in the `replace_regexp` pipe:
1. **Infinite loop on empty matches when `limit` is not set**
[#8625](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8625)
When a regex pattern like `\d*`, `()`, or `\b` was used, the
implementation could repeatedly match the same zero-width position
without advancing the string, causing unbounded memory usage and
eventual OOM. This is now fixed by collecting all matches up front,
respecting the `limit`, and applying replacements in a single pass.
2. **Incorrect handling of anchors (`^` and `$`)**
The previous implementation applied regex matching to progressively
sliced substrings (`s = s[end:]`), which unintentionally caused anchor
patterns like `^` (start-of-string) to match at every new substring's
start. As a result, patterns that should have matched only once (e.g.,
`^|$`) ended up matching multiple times.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8625
Hopefully this should reduce the amounts of questions whether vlinsert replicates or shards the incoming logs.
The answer - it spreads the incoming log _evenly_ among the configured vlstorage nodes (e.g. it shards logs).
### Changes
Updated `lib/httpserver/httpserver.go` to include a flag that can toggle
CORS (defaults to true to keep the current behavior).
This PR relates to
[this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8680#issue-2983786438)
feature request
### Checklist
The following checks are **mandatory**:
- [x] My change does not break backwards compatibility (i.e., preserves
CORS being enabled unless specified otherwise via the
`-http.cors.disabled=true` flag & value)
---------
Co-authored-by: Jai Mehra <jai.mehra@nav-timing.safrangroup.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 84163a56eb)
### Describe Your Changes
#8842
As a quick fix, it seems like it works. checked on my failing tests
(they are not emitting this error anymore)
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>
(cherry picked from commit d85bb968c0)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
These were added while working on paritition index (#8134). Submitting
the separately in order to:
1. Make sure partition index will not break anything and
2. Be able to compare performance before and after swiching to parition
index.
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
fixed not successful rebase in PR, moved svg icons to a sprite and added
VM logo to UI
<img width="1290" alt="image"
src="https://github.com/user-attachments/assets/34cfdc61-6349-4320-b133-38965cb6c30f"
/>
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
(cherry picked from commit 6e6d0b35c8)
### Describe Your Changes
use the built-in max/min to simplify the code
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
Signed-off-by: pkucode <cssjtu@163.com>
(cherry picked from commit 41abe16adc)
The new rule should notify user when there is a job with 0 configured or
discovered targets, which is usually a sign of misconfiguration.
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit b5c9284748)
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 1bfe574401)
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8466
This pull request:
1. Add `GraphiteWrite`, `CSVImport`, `OpenTSDBHTTPImport` data import
methods for vmsingle and vminsert.
2. Add TCP `Write` method to `apptest.Client` to make it capable of
writing data in TCP-based protocol.
3. Add test cases:
1. for new import methods.
2. for corner cases placed under `app/victoria-metrics/main_test`.
4. Removed all test cases under `app/victoria-metrics/main_test`.
Signed-off-by: f41gh7 <nik@victoriametrics.com>
Co-authored-by: f41gh7 <nik@victoriametrics.com>
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
### Describe Your Changes
Updated "> note" sections in /anomaly-detection/ doc path for new
formatting and improved clarity
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
This is needed because Go interprets backticked string as a measurement unit for the given command-line flag:
The listed type, here int, can be changed by placing a back-quoted name in the flag's usage string;
the first such item in the message is taken to be a parameter name to show in the message and the back quotes
are stripped from the message when displayed
See https://pkg.go.dev/flag#PrintDefaults
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 1cca22f88d)
Here a description of important terms related to Organizations is
provided. The intention is to leverage on a good UI design rather than
explaining all steps needed to perfomr certain actions.
However, an effort on explaining the meaning of internal states and
terms has been done.
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
### Describe Your Changes
1. Fixed log entry sorting in the "Group" view: newest entries are now
displayed at the top again. In v1.18.0, log entries were incorrectly
sorted (oldest first) due to changes related to nanosecond precision
support.
Related issue: #8726
2. Added alphabetical sorting of fields by key in the "Group" view for
easier navigation.
Related issue: #8438
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 492def6d57)
While there, remove version requirements for VM migration as it
is almost 4y old by now.
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit c1b9b2bd8b)
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
(cherry picked from commit 5d12ba702e)
### Describe Your Changes
Related issue: #8697
Renamed `retentionFilters` to `retentionFilter`.
🛑 before merge this pull request, need to merge API changes in
enterprise version.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
(cherry picked from commit f0f548231d)
* set Y-min to 0. This helps to see real spikes in resource usage;
* add text panel explaining how to use query hash;
* support filtering by tenant;
* use `\tvm_slow_query_stats` prefix filter to filter out logs
that mention failed queries with `vm_slow_query_stats` word;
* add panel to show the minimum queried timestamp - see https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8828
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 5147846d18)
Based on the code, an active time series is one that has received at
least one sample during the last hour. It does not includes querying.
The first time the docs were updated with this was in
d06aae9454. But even at that time, the
code shows that querying a metric did not make it active.
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
* put important changes on top;
* replace `this issue` with corresponding issue number to make it more sense;
* consistently format components names.
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 3f39bd08cf)
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Co-authored-by: Jose Gómez-Sellés <14234281+jgomezselles@users.noreply.github.com>
(cherry picked from commit cddf36af43)
* display max values instead of cumulative values
to outline anomalies;
* remove unnecessary overrides on column width;
* add corresponding formatting units to displayed values.
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 0e555b421d)
datadb.rb contains logRows shards, which weren't freed up after the data ingestion
for the given per-day datadb is stopped. This leads to slow memory leak when VictoriaLogs runs
for multiple days without restarts. Avoid this memory leak by freeing up the logRows shards
after converting them to in-memory parts. Re-use the freed up logRows shards via a pool in order
to reduce the pressure on GC.
(cherry picked from commit ec6f33f526)
This commit moved all the docs/*.md files to docs/victoriametrics/ folder. This broke direct links to these files
such as https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/docs/vmagent.md .
Add the missing `/victoriametrics/` part here, so the link becomes https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/docs/victoriametrics/vmagent.md .
The move of all the docs/*.md files into docs/victoriametrics/ folder is dubious because of the following reasons:
- It breaks direct links to *.md files like mentioned above. It is impossible to fix such links all over the Internet,
so they will remain broken :(
The best thing we can do is to fix them on the resources we control such as VictoriaMetrics repository.
- It breaks Google indexing of VictoriaMetrics docs. Google index contains old links such as https://docs.victoriametrics.com/vmagent/#features .
Now such links are automatically redirected to https://docs.victoriametrics.com/victoriametrics/vmagent/#features by a javascript on the https://docs.victoriametrics.com/vmagent/ page.
Google doesn't like redirects at javascript, since they are frequently used by black hat SEO purposes. This leads to pessimization of VictoriaMetrics docs
in Google search result :( We cannot update the old links all over the Internet in order to avoid the redirect by javascript :(
The best thing we can do is to add <meta rel="canonical"> header with the new location of the page at the old url,
and hope Google won't remove VictoriaMetrics docs from its search results. @AndrewChubatiuk , please do this.
- It breaks backwards navigation. When you click the https://docs.victoriametrics.com/vmagent/ link on some page and then press `back` button
in the web browser, it won't return back you to the original page because of the intermediate redirect :( The broken navigation cannot be fixed
for old links located all over the Internet.
- It increases chances of breaking old links left on the Internet in the future, which will lead to 404 Not Found pages
and angry users :(
The sad thing is that we hit the same wall with harmful redirects again :( In the beginning the VictoriaMetrics docs links had .html suffix,
and they were case-sensitive. For example, http://docs.victoriametrics.com/MetricsQL.html#rate . It has been decided for some unknown reason
that it is a good idea to remove the .html suffix and to make all the links lowercase. So now such links are automatically redirected by javascript
to https://docs.victoriametrics.com/metricsql/#rate and then redirected again by another javasript to https://docs.victoriametrics.com/victoriametrics/metricsql/#rate :(
There are many old links all over the Internet (for example, at Reddit, StackOverflow, some internal Wiki pages, etc.). We cannot fix all of them,
so we need to pray these links won't break in the future.
@hagen1778, @tenmozes, @makasim , please make sure we won't hit the same wall in a third time.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8595
(cherry picked from commit e316ab878c)
VictoriaMetrics source code depends on Go 1.24, so it is better to run `go mod tidy` in Go 1.24 compatibility mode.
(cherry picked from commit 3ec70a4b6d)
Use multiple independent logRows shards for storing the pending log entries before converting them to searchable parts.
Every shard is protected by its own mutex, so multiple CPU cores may add multiple log rows into datadb at the same time.
This increases the performance of BenchmarkStorageMustAddRows/rowsPerInsert-1, which ingests log rows own-by-one
from concurrently running goroutines, by 2x.
(cherry picked from commit 8ad81220d3)
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
(cherry picked from commit 22c85a149e)
This commit modifies the logging behavior for client network errors
(e.g., EOFs, timeouts) during the handshake process. They are now logged
as warnings instead of errors, as they are not actionable from the
server’s perspective. Here's some examples of such errors.
Timeouts during the initial read phase:
2025-04-09T07:08:59.323Z error
VictoriaMetrics/lib/vmselectapi/server.go:204 cannot perform
vmselect handshake with client "<REDACTED>": cannot read hello: cannot
read message with size 11: read tcp4 <REDACTED>-><REDACTED>: i/o
timeout; read only 0 bytes
EOFs occurring later in the handshake process:
2025-04-08T18:01:30.783Z error
VictoriaMetrics/lib/vmselectapi/server.go:204 cannot perform
vmselect handshake with client "<REDACTED>": cannot read isCompressed
flag: cannot read message with size 1: EOF; read only 0 bytes
By logging these as warnings, we reduce noise in error logs while
preserving valuble information for debug.
Previously, if `cpu.max` file has only `max` resource defined without
`period`, it was parsed incorrectly and silently drop error. While this
syntax is valid and actually used by some container runtimes. If period
is not defined, default value for it 100_000 must be used.
This commit fixes parsing function by using default value for period.
In addition, it adds zero value check, which fixes possible panic if
period has 0 value.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8808
This reverts commit fa6a32a39d.
Reason for revert: the broken tests were fixed on GOARCH=386 by skipping the check for the state size
after improting the state of stats function, since the state size depends on the hardware architecture.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8710
### Describe Your Changes
Two new columns were added to the **Cardinality Explorer** table:
`Requests count`
- Displays how many times a metric was queried since stats collection
began
- Highlighted **red** when the value is `0` or missing
`Last request`
- Shows when the metric was last queried
- Highlighted:
- **Red** if older than **30 days**
- **Yellow** if between **7 and 30 days**
- Shows exact timestamp on hover in `YYYY-MM-DD HH:mm:ss` format
Related issue: #6145
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 1c39853164)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
(cherry picked from commit e9c577132d)
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
Co-authored-by: Hui Wang <haley@victoriametrics.com>
(cherry picked from commit 7390f99897)
* downsampling: allow using `-downsampling.period=filter:0s:0s` to skip downsampling for time series that match the specified `filter`
* doc minor fix
* lib/storage/downsampling: fix a typo in error message
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
---------
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
### Describe Your Changes
Fixes incorrect /flags request in vmui when using URL prefixes.
Related issue: #8641
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
(cherry picked from commit 899d70a93e)
### Describe Your Changes
Hide identical series in the legend on the Raw Query page when
deduplication is disabled.
Related issue: #8688
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
(cherry picked from commit fa4708178f)
* update wording;
* add more details about returned response;
* cover details on counter increments;
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 19025d5a19)
Before, this doc section was related to vmui, by mistake.
Moved it closer to tsdb stats, where it makes more sense to be.
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 1f8c115428)
Badges is more a decorative element rather than source of useful information.
Some badges break from time to time, giving misleading impression to users.
It is better to remove them to reduce visual load and confusion.
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 1863c5b95b)
### Describe Your Changes
HTTP/2 support is used by some S3-compatible storage providers, so
disabling it default leads to unexpected errors when trying to connect
to S3 endpoint.
For example, using MinIO as S3 storage backend: `net/http: HTTP/1.x
transport connection broken: malformed HTTP response`.
HTTP/2 was enabled by default previously, but while fixing inconsistency
e5f4826 commit disabled this by default.
cc: @valyala
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
### Describe Your Changes
Fixes which are required in order to build FIPS-compliant binaries.
These changes were originally added for enterprise version and synced to
opensource for consistency and easier maintenance.
- consistently use `hash/fnv` at `app/vmalert` when calculating
checksums. Usage of md5 is not allowed in FIPS mode.
- increase encryption keys size used in testing in order to allow tests
to successfully run in FIPS mode
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
The suffix after the ID may change in dashboard settings.
If it changes, the link becomes broken (dubious decision at grafana.com/grafana/dashboards/ ).
That's why it is better to drop all the suffixes and use links for Grafana dashbooards ending with IDs.
In this case they are automatically redirected to the url with the proper suffix.
This is a follow-up for 3e4c38c56c
See also the previous commits 9c0863babc and 0a5ffb3bc1
This reduces the overhead needed for converting the ingested log entries to searchable in-memory parts
when small number of log entries are passed to Storage.MustAddRows().
The BenchmarkStorageMustAddRows shows up to 10x performance increase for rowsPerInsert=1,
up to 5x performance increase for rowsPerInsert=10 and up to 2x performance increase for rowsPerInsert=100.
This should reduce CPU usage during data ingestion when every request contains small number of rows.
(cherry picked from commit 5491d54c11)
…te panel
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit cd4ec0e739)
On shutdown, rw client printed two log messages about shutting down and
how many series it flushed on shut down.
This commit removes these log messages for the following reasons:
1. These messages were printed for each `client.cc`, which is set to x2
of available CPUs. This behavior generated a lot of useless logs on
shutdown.
2. Number of flushed series on shutdown doesn't matter to the user.
Instead, it now prints only two log messages: when shutdown starts and
when it ends:
```
^C2025-04-22T07:37:11.932Z info VictoriaMetrics/app/vmalert/main.go:189 service received signal interrupt
2025-04-22T07:37:11.932Z info VictoriaMetrics/app/vmalert/remotewrite/client.go:152 shutting down remote write client: flushing remained series
2025-04-22T07:37:11.933Z info VictoriaMetrics/app/vmalert/remotewrite/client.go:155 shutting down remote write client: finished in 210.917µs
```
---------
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Previously, metric names stats API had a false assumption, that max
size of metric name is 256 byte. But this is configurable parameter with
4096 bytes max size. It triggered errors during API requests.
This commit replaces hard-coded 256 byte limit with common constant:
maxLabelValueSize. It has 16 MB limit.
In addition, this commit adds check for metric name stats tracker,
if metric name size exceeds default buffer limit, it will be allocated
directly on heap. It must be rare case, since most metric names has
16-64 byte size.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8759
Throttle remote write retry warning logs to one message per 5 seconds.
This reduces log noise during connectivity issues.
Users can monitor `vmagent_remotewrite_retries_count_total` and
`vmagent_remotewrite_errors_total` metrics for detailed retry rates per
destination if needed. This change prioritizes reducing log volume over
immediate destination-specific error logging in the retry warnings.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8723
VictoriaMetrics executables need the latest available stable release of Go (1.24 currently),
since they use its features. See, for example, GOEXPERIMENT=synctest from the commit 06c26315df .
There is no need in specifying the minimum supported Go version for building VictoriaMetrics products,
since Go automatically downloads and uses the version specified at `go ...` directive inside go.mod
starting from Go1.21. See https://go.dev/doc/toolchain for details.
So the actual minimum needed Go version is Go1.21, which has been released 1.5 years ago. It should automatically
install newer Go versions specified at `go ...` directive inside go.mod.
Do not reset wc.labels in order to properly keep track of the number of used labels for the scrape,
and properly re-use the same number of wc.labels on subsequent scrapes.
See 12f26668a6 (r155481168)
Before the change, the vmagent integration tests created their directory
and files inside apptest/tests.
After the change, vmagent is instructed to store all files in a real
temporary directory, which is automatically deleted after the tests
complete.
Fix tests by adding accountID and projectID.
The tests were cherry-picked from master and were failing to build because
cluster version requires accountID and projectID.
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
Using this package lets to manipulate time. In this particular case, it
lets to advance the time 61 second forward instantly.
A few side changes were necessary:
- Do not use fasttime in unit tests. The fasttime package starts a
goroutine outside the test bubble which causes the clock to be real, not
fake.
- Stop the time.Ticker explicitly and also stop idbNext. These two
create goroutines with infinite loops which causes the unit tests that
use synctest to hang forever. All goroutines created inside the bubble
must exit in order for the syntest to finish.
- synctest is an experimental package and requires an environment
variable to be set. The Makefile was changed to set it.
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
### Describe Your Changes
The write concurrency limiter in ReadUncompressedData was previously
removed in
22d1b916bf
to avoid suboptimal behavior in certain scenarios. However, follow-up
reports—including issue
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8674 and
production feedback from VictoriaMetrics Cloud—indicated a noticeable
degradation in performance after its removal.
To mitigate these regressions, this commit reintroduces the concurrency
limiter. A long-term, more optimal solution will be explored separately
in issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8728.
TODO:
* [x] Changelog
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 231810fe49)
The doc was incorrectly ported into wrong directory after
a113516588
This change moves it to the victoriametrics dir.
While there, updated the order of some pages to couple them by meaning.
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 3d9b160fce)
### Describe Your Changes
Fixed table sorting logic and added unit tests for descendingComparator.
Values are now correctly sorted by type: number, date, or string.
Related issue: #8606
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 5fa40ee631)
This commit adds new fields - `requestsCount` and `lastRequestTimestamp`
to series count be metric names stats.
It allows to display an additional stats at explore cardinality page.
Stats will only be added if `storage.trackMetricNameStats` flag is set.
This change requires an update to RPC protocol in order to properly
marshal data.
In addition, this commit adds integration tests to TSDB stats API.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6145
This commit adds `search.logSlowQueryStats=<duration>` cmd-line flag on vmselect.
It reads stats from eval function, and doesn't slow down the query execution.
Log line has the following structure:
vm_slow_query_stats type=%s query=%q query_hash=%d start_ms=%d end_ms=%d step_ms=%d range_ms=%d tenant=%q execution_duration_ms=%v series_fetched=%d samples_fetched=%d bytes=%d memory_estimated_bytes=%d
This feature is only available for enterprise version.
### Describe Your Changes
Changelog note about experimental v1.22.0 release that solves deadlock
issue on multicore systems by complete parallelization turned off until
proper refactor is made to return it back w/o reintroducing the risk of
the deadlock in child processes.
P.s. other references and guides were not updated to experimental tag,
as long as it has downsides of dropped speed. The links will be updated
once we have parallelization properly turned on.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
Before filling a bug report it would be great to [upgrade](https://docs.victoriametrics.com/#how-to-upgrade)
Before filling a bug report it would be great to [upgrade](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#how-to-upgrade)
to [the latest available release](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest)
and verify whether the bug is reproducible there.
It's also recommended to read the [troubleshooting docs](https://docs.victoriametrics.com/troubleshooting/) first.
It's also recommended to read the [troubleshooting docs](https://docs.victoriametrics.com/victoriametrics/troubleshooting/) first.
- type:textarea
id:describe-the-bug
attributes:
@@ -64,8 +64,8 @@ body:
*[Grafana dashboard for VictoriaMetrics cluster](https://grafana.com/grafana/dashboards/11176)
See how to setup monitoring here:
*[monitoring for single-node VictoriaMetrics](https://docs.victoriametrics.com/#monitoring)
*[monitoring for VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#monitoring)
*[monitoring for single-node VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#monitoring)
*[monitoring for VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#monitoring)
VictoriaMetrics is a fast, cost-saving, and scalable solution for monitoring and managing time series data. It delivers high performance and reliability, making it an ideal choice for businesses of all sizes.
## Folder Structure
-`/app`: Contains the compilable binaries.
-`/lib`: Contains the golang reusable libraries
-`/docs/victoriametrics`: Contains documentation for the project.
-`/apptest/tests`: Contains integration tests.
## Libraries and Frameworks
- Backend: Golang, no framework. Use third-party libraries sparingly.
- Frontend: React.
## Code review guidelines
Ensure the feature or bugfix includes a changelog entry in /docs/victoriametrics/changelog/CHANGELOG.md.
Verify the entry is under the ## tip section and matches the structure and style of existing entries.
Chore-only changes may be omitted from the changelog.
@@ -6,4 +6,5 @@ Please provide a brief description of the changes you made. Be as specific as po
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/).
- [ ] My change adheres to [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/#pull-request-checklist).
- [ ] My change adheres to [VictoriaMetrics development goals](https://docs.victoriametrics.com/victoriametrics/goals/).
VictoriaMetrics is a fast, cost-saving, and scalable solution for monitoring and managing time series data. It delivers high performance and reliability, making it an ideal choice for businesses of all sizes.
@@ -21,10 +21,10 @@ VictoriaMetrics is a fast, cost-saving, and scalable solution for monitoring and
Here are some resources and information about VictoriaMetrics:
- Deployment types: [Single-node version](https://docs.victoriametrics.com/), [Cluster version](https://docs.victoriametrics.com/cluster-victoriametrics/), and [Enterprise version](https://docs.victoriametrics.com/enterprise/)
- Changelog: [CHANGELOG](https://docs.victoriametrics.com/changelog/), and [How to upgrade](https://docs.victoriametrics.com/#how-to-upgrade-victoriametrics)
- Deployment types: [Single-node version](https://docs.victoriametrics.com/), [Cluster version](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/), and [Enterprise version](https://docs.victoriametrics.com/victoriametrics/enterprise/)
- Changelog: [CHANGELOG](https://docs.victoriametrics.com/victoriametrics/changelog/), and [How to upgrade](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#how-to-upgrade-victoriametrics)
Yes, we open-source both the single-node VictoriaMetrics and the cluster version.
@@ -35,22 +35,22 @@ VictoriaMetrics is optimized for timeseries data, even when old time series are
* **Long-term storage for Prometheus** or as a drop-in replacement for Prometheus and Graphite in Grafana.
* **Powerful stream aggregation**: Can be used as a StatsD alternative.
* **Ideal for big data**: Works well with large amounts of time series data from APM, Kubernetes, IoT sensors, connected cars, industrial telemetry, financial data and various [Enterprise workloads](https://docs.victoriametrics.com/enterprise/).
* **Ideal for big data**: Works well with large amounts of time series data from APM, Kubernetes, IoT sensors, connected cars, industrial telemetry, financial data and various [Enterprise workloads](https://docs.victoriametrics.com/victoriametrics/enterprise/).
* **Query language**: Supports both PromQL and the more performant MetricsQL.
* **Easy to setup**: No dependencies, single [small binary](https://medium.com/@valyala/stripping-dependency-bloat-in-victoriametrics-docker-image-983fb5912b0d), configuration through command-line flags, but the default is also fine-tuned; backup and restore with [instant snapshots](https://medium.com/@valyala/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282).
* **Global query view**: Multiple Prometheus instances or any other data sources may ingest data into VictoriaMetrics and queried via a single query.
* **Various Protocols**: Support metric scraping, ingestion and backfilling in various protocol.
* [InfluxDB line protocol](https://docs.victoriametrics.com/#how-to-send-data-from-influxdb-compatible-agents-such-as-telegraf) over HTTP, TCP and UDP.
* [Graphite plaintext protocol](https://docs.victoriametrics.com/#how-to-send-data-from-graphite-compatible-agents-such-as-statsd) with [tags](https://graphite.readthedocs.io/en/latest/tags.html#carbon).
* [OpenTSDB put message](https://docs.victoriametrics.com/#sending-data-via-telnet-put-protocol).
- **Multiple retentions**: Reducing storage costs by specifying different retentions for different datasets.
- **Downsampling**: Reducing storage costs and increasing performance for queries over historical data.
- **Stable releases** with long-term support lines ([LTS](https://docs.victoriametrics.com/lts-releases/)).
- **Stable releases** with long-term support lines ([LTS](https://docs.victoriametrics.com/victoriametrics/lts-releases/)).
- **Comprehensive support**: First-class consulting, feature requests and technical support provided by the core VictoriaMetrics dev team.
- Many other features, which you can read about on [the Enterprise page](https://docs.victoriametrics.com/enterprise/).
- Many other features, which you can read about on [the Enterprise page](https://docs.victoriametrics.com/victoriametrics/enterprise/).
[Contact us](mailto:info@victoriametrics.com) if you need enterprise support for VictoriaMetrics. Or you can request a free trial license [here](https://victoriametrics.com/products/enterprise/trial/), downloaded Enterprise binaries are available at [Github Releases](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest).
@@ -77,7 +77,7 @@ Some good benchmarks VictoriaMetrics achieved:
* **Minimal memory footprint**: handling millions of unique timeseries with [10x less RAM](https://medium.com/@valyala/insert-benchmarks-with-inch-influxdb-vs-victoriametrics-e31a41ae2893) than InfluxDB, up to [7x less RAM](https://valyala.medium.com/prometheus-vs-victoriametrics-benchmark-on-node-exporter-metrics-4ca29c75590f) than Prometheus, Thanos or Cortex.
* **Highly scalable and performance** for [data ingestion](https://medium.com/@valyala/high-cardinality-tsdb-benchmarks-victoriametrics-vs-timescaledb-vs-influxdb-13e6ee64dd6b) and [querying](https://medium.com/@valyala/when-size-matters-benchmarking-victoriametrics-vs-timescale-and-influxdb-6035811952d4), [20x outperforms](https://medium.com/@valyala/insert-benchmarks-with-inch-influxdb-vs-victoriametrics-e31a41ae2893) InfluxDB and TimescaleDB.
* **High data compression**: [70x more data points](https://medium.com/@valyala/when-size-matters-benchmarking-victoriametrics-vs-timescale-and-influxdb-6035811952d4) may be stored into limited storage than TimescaleDB, [7x less storage](https://valyala.medium.com/prometheus-vs-victoriametrics-benchmark-on-node-exporter-metrics-4ca29c75590f) space is required than Prometheus, Thanos or Cortex.
* **Reducing storage costs**: [10x more effective](https://docs.victoriametrics.com/casestudies/#grammarly) than Graphite according to the Grammarly case study.
* **Reducing storage costs**: [10x more effective](https://docs.victoriametrics.com/victoriametrics/casestudies/#grammarly) than Graphite according to the Grammarly case study.
* **A single-node VictoriaMetrics** can replace medium-sized clusters built with competing solutions such as Thanos, M3DB, Cortex, InfluxDB or TimescaleDB. See [VictoriaMetrics vs Thanos](https://medium.com/@valyala/comparing-thanos-to-victoriametrics-cluster-b193bea1683), [Measuring vertical scalability](https://medium.com/@valyala/measuring-vertical-scalability-for-time-series-databases-in-google-cloud-92550d78d8ae), [Remote write storage wars - PromCon 2019](https://promcon.io/2019-munich/talks/remote-write-storage-wars/).
* **Optimized for storage**: [Works well with high-latency IO](https://medium.com/@valyala/high-cardinality-tsdb-benchmarks-victoriametrics-vs-timescaledb-vs-influxdb-13e6ee64dd6b) and low IOPS (HDD and network storage in AWS, Google Cloud, Microsoft Azure, etc.).
httpListenAddrs=flagutil.NewArrayString("httpListenAddr","TCP address to listen for incoming http requests. See also -httpListenAddr.useProxyProtocol")
useProxyProtocol=flagutil.NewArrayBool("httpListenAddr.useProxyProtocol","Whether to use proxy protocol for connections accepted at the given -httpListenAddr . "+
"See https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt . "+
"With enabled proxy protocol http server cannot serve regular /metrics endpoint. Use -pushmetrics.url for metrics pushing")
)
funcmain(){
// Write flags and help message to stdout, since it is easier to grep or pipe.
flag.CommandLine.SetOutput(os.Stdout)
flag.Usage=usage
envflag.Parse()
buildinfo.Init()
logger.Init()
listenAddrs:=*httpListenAddrs
iflen(listenAddrs)==0{
listenAddrs=[]string{":9428"}
}
logger.Infof("starting VictoriaLogs at %q...",listenAddrs)
datadogStreamFields=flagutil.NewArrayString("datadog.streamFields","Comma-separated list of fields to use as log stream fields for logs ingested via DataDog protocol. "+
"See https://docs.victoriametrics.com/victorialogs/data-ingestion/datadog-agent/#stream-fields")
datadogIgnoreFields=flagutil.NewArrayString("datadog.ignoreFields","Comma-separated list of fields to ignore for logs ingested via DataDog protocol. "+
"See https://docs.victoriametrics.com/victorialogs/data-ingestion/datadog-agent/#dropping-fields")
maxRequestSize=flagutil.NewBytes("datadog.maxRequestSize",64*1024*1024,"The maximum size in bytes of a single DataDog request")
defaultMsgValue=flag.String("defaultMsgValue","missing _msg field; see https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field",
"Default value for _msg field if the ingested log entry doesn't contain it; see https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field")
)
// CommonParams contains common HTTP parameters used by log ingestion APIs.
//
// See https://docs.victoriametrics.com/victorialogs/data-ingestion/#http-parameters
// ParseUnixTimestamp parses s as unix timestamp in seconds, milliseconds, microseconds or nanoseconds and returns the parsed timestamp in nanoseconds.
funcParseUnixTimestamp(sstring)(int64,error){
ifstrings.IndexByte(s,'.')>=0{
// Parse timestamp as floating-point value
f,err:=strconv.ParseFloat(s,64)
iferr!=nil{
return0,fmt.Errorf("cannot parse unix timestamp from %q: %w",s,err)
}
iff<(1<<31)&&f>=(-1<<31){
// The timestamp is in seconds.
returnint64(f*1e9),nil
}
iff<1e3*(1<<31)&&f>=1e3*(-1<<31){
// The timestamp is in milliseconds.
returnint64(f*1e6),nil
}
iff<1e6*(1<<31)&&f>=1e6*(-1<<31){
// The timestamp is in microseconds.
returnint64(f*1e3),nil
}
// The timestamp is in nanoseconds
iff>math.MaxInt64{
return0,fmt.Errorf("too big timestamp in nanoseconds: %v; mustn't exceed %v",f,int64(math.MaxInt64))
}
iff<math.MinInt64{
return0,fmt.Errorf("too small timestamp in nanoseconds: %v; must be bigger or equal to %v",f,int64(math.MinInt64))
}
returnint64(f),nil
}
// Parse timestamp as integer
n,err:=strconv.ParseInt(s,10,64)
iferr!=nil{
return0,fmt.Errorf("cannot parse unix timestamp from %q: %w",s,err)
disableInsert=flag.Bool("internalinsert.disable",false,"Whether to disable /internal/insert HTTP endpoint")
maxRequestSize=flagutil.NewBytes("internalinsert.maxRequestSize",64*1024*1024,"The maximum size in bytes of a single request, which can be accepted at /internal/insert HTTP endpoint")
journaldStreamFields=flagutil.NewArrayString("journald.streamFields","Comma-separated list of fields to use as log stream fields for logs ingested over journald protocol. "+
"See https://docs.victoriametrics.com/victorialogs/data-ingestion/journald/#stream-fields")
journaldIgnoreFields=flagutil.NewArrayString("journald.ignoreFields","Comma-separated list of fields to ignore for logs ingested over journald protocol. "+
"See https://docs.victoriametrics.com/victorialogs/data-ingestion/journald/#dropping-fields")
journaldTimeField=flag.String("journald.timeField","__REALTIME_TIMESTAMP","Field to use as a log timestamp for logs ingested via journald protocol. "+
"See https://docs.victoriametrics.com/victorialogs/data-ingestion/journald/#time-field")
journaldTenantID=flag.String("journald.tenantID","0:0","TenantID for logs ingested via the Journald endpoint. "+
"See https://docs.victoriametrics.com/victorialogs/data-ingestion/journald/#multitenancy")
journaldIncludeEntryMetadata=flag.Bool("journald.includeEntryMetadata",false,"Include journal entry fields, which with double underscores.")
maxRequestSize=flagutil.NewBytes("journald.maxRequestSize",64*1024*1024,"The maximum size in bytes of a single journald request")
syslogTimezone=flag.String("syslog.timezone","Local","Timezone to use when parsing timestamps in RFC3164 syslog messages. Timezone must be a valid IANA Time Zone. "+
"For example: America/New_York, Europe/Berlin, Etc/GMT+3 . See https://docs.victoriametrics.com/victorialogs/data-ingestion/syslog/")
streamFieldsTCP=flagutil.NewArrayString("syslog.streamFields.tcp","Fields to use as log stream labels for logs ingested via the corresponding -syslog.listenAddr.tcp. "+
streamFieldsUDP=flagutil.NewArrayString("syslog.streamFields.udp","Fields to use as log stream labels for logs ingested via the corresponding -syslog.listenAddr.udp. "+
tenantIDTCP=flagutil.NewArrayString("syslog.tenantID.tcp","TenantID for logs ingested via the corresponding -syslog.listenAddr.tcp. "+
"See https://docs.victoriametrics.com/victorialogs/data-ingestion/syslog/#multitenancy")
tenantIDUDP=flagutil.NewArrayString("syslog.tenantID.udp","TenantID for logs ingested via the corresponding -syslog.listenAddr.udp. "+
"See https://docs.victoriametrics.com/victorialogs/data-ingestion/syslog/#multitenancy")
listenAddrTCP=flagutil.NewArrayString("syslog.listenAddr.tcp","Comma-separated list of TCP addresses to listen to for Syslog messages. "+
"See https://docs.victoriametrics.com/victorialogs/data-ingestion/syslog/")
listenAddrUDP=flagutil.NewArrayString("syslog.listenAddr.udp","Comma-separated list of UDP address to listen to for Syslog messages. "+
"See https://docs.victoriametrics.com/victorialogs/data-ingestion/syslog/")
tlsEnable=flagutil.NewArrayBool("syslog.tls","Whether to enable TLS for receiving syslog messages at the corresponding -syslog.listenAddr.tcp. "+
"The corresponding -syslog.tlsCertFile and -syslog.tlsKeyFile must be set if -syslog.tls is set. See https://docs.victoriametrics.com/victorialogs/data-ingestion/syslog/#security")
tlsCertFile=flagutil.NewArrayString("syslog.tlsCertFile","Path to file with TLS certificate for the corresponding -syslog.listenAddr.tcp if the corresponding -syslog.tls is set. "+
"Prefer ECDSA certs instead of RSA certs as RSA certs are slower. The provided certificate file is automatically re-read every second, so it can be dynamically updated. "+
"See https://docs.victoriametrics.com/victorialogs/data-ingestion/syslog/#security")
tlsKeyFile=flagutil.NewArrayString("syslog.tlsKeyFile","Path to file with TLS key for the corresponding -syslog.listenAddr.tcp if the corresponding -syslog.tls is set. "+
"The provided key file is automatically re-read every second, so it can be dynamically updated. "+
"See https://docs.victoriametrics.com/victorialogs/data-ingestion/syslog/#security")
tlsCipherSuites=flagutil.NewArrayString("syslog.tlsCipherSuites","Optional list of TLS cipher suites for -syslog.listenAddr.tcp if -syslog.tls is set. "+
"See the list of supported cipher suites at https://pkg.go.dev/crypto/tls#pkg-constants . "+
"See also https://docs.victoriametrics.com/victorialogs/data-ingestion/syslog/#security")
tlsMinVersion=flag.String("syslog.tlsMinVersion","TLS13","The minimum TLS version to use for -syslog.listenAddr.tcp if -syslog.tls is set. "+
"Supported values: TLS10, TLS11, TLS12, TLS13. "+
"See https://docs.victoriametrics.com/victorialogs/data-ingestion/syslog/#security")
compressMethodTCP=flagutil.NewArrayString("syslog.compressMethod.tcp","Compression method for syslog messages received at the corresponding -syslog.listenAddr.tcp. "+
"Supported values: none, gzip, deflate. See https://docs.victoriametrics.com/victorialogs/data-ingestion/syslog/#compression")
compressMethodUDP=flagutil.NewArrayString("syslog.compressMethod.udp","Compression method for syslog messages received at the corresponding -syslog.listenAddr.udp. "+
"Supported values: none, gzip, deflate. See https://docs.victoriametrics.com/victorialogs/data-ingestion/syslog/#compression")
useLocalTimestampTCP=flagutil.NewArrayBool("syslog.useLocalTimestamp.tcp","Whether to use local timestamp instead of the original timestamp for the ingested syslog messages "+
"at the corresponding -syslog.listenAddr.tcp. See https://docs.victoriametrics.com/victorialogs/data-ingestion/syslog/#log-timestamps")
useLocalTimestampUDP=flagutil.NewArrayBool("syslog.useLocalTimestamp.udp","Whether to use local timestamp instead of the original timestamp for the ingested syslog messages "+
"at the corresponding -syslog.listenAddr.udp. See https://docs.victoriametrics.com/victorialogs/data-ingestion/syslog/#log-timestamps")
)
// MustInit initializes syslog parser at the given -syslog.listenAddr.tcp and -syslog.listenAddr.udp ports
//
// This function must be called after flag.Parse().
//
// MustStop() must be called in order to free up resources occupied by the initialized syslog parser.
funcMustInit(){
ifworkersStopCh!=nil{
logger.Panicf("BUG: MustInit() called twice without MustStop() call")
`<123>1 2023-06-03T17:42:12.345Z mymachine.example.com appname 12345 ID47 [exampleSDID@32473 iut="3" eventSource="Application 123 = ] 56" eventID="11211"] This is a test message with structured data.`,
})
}
funcTestSyslogLineReader_Failure(t*testing.T){
f:=func(datastring){
t.Helper()
r:=bytes.NewBufferString(data)
slr:=getSyslogLineReader(r)
deferputSyslogLineReader(slr)
ifslr.nextLine(){
t.Fatalf("expecting failure to read the first line")
{"priority":"123","facility":"15","severity":"3","format":"rfc5424","hostname":"mymachine.example.com","app_name":"appname","proc_id":"12345","msg_id":"ID47","exampleSDID@32473.iut":"3","exampleSDID@32473.eventSource":"Application 123 = ] 56","exampleSDID@32473.eventID":"11211","_msg":"This is a test message with structured data."}`
datasourceURL=flag.String("datasource.url","http://localhost:9428/select/logsql/query","URL for querying VictoriaLogs; "+
"see https://docs.victoriametrics.com/victorialogs/querying/#querying-logs . See also -tail.url")
tailURL=flag.String("tail.url","","URL for live tailing queries to VictoriaLogs; see https://docs.victoriametrics.com/victorialogs/querying/#live-tailing ."+
"The url is automatically detected from -datasource.url by replacing /query with /tail at the end if -tail.url is empty")
historyFile=flag.String("historyFile","vlogscli-history","Path to file with command history")
header=flagutil.NewArrayString("header","Optional header to pass in request -datasource.url in the form 'HeaderName: value'")
accountID=flag.Int("accountID",0,"Account ID to query; see https://docs.victoriametrics.com/victorialogs/#multitenancy")
projectID=flag.Int("projectID",0,"Project ID to query; see https://docs.victoriametrics.com/victorialogs/#multitenancy")
)
const(
firstLinePrompt=";> "
nextLinePrompt=""
)
funcmain(){
// Write flags and help message to stdout, since it is easier to grep or pipe.
The `run_id` field uniquely identifies every `vlogsgenerator` invocation.
### How to write logs to VictoriaLogs?
The generated logs can be written directly to VictoriaLogs by passing the address of [`/insert/jsonline` endpoint](https://docs.victoriametrics.com/victorialogs/data-ingestion/#json-stream-api)
to `-addr` command-line flag. For example, the following command writes the generated logs to VictoriaLogs running at `localhost`:
`vlogsgenerator` accepts various command-line flags, which can be used for configuring the number and the shape of the generated logs.
These flags can be inspected by running `vlogsgenerator -help`. Below are the most interesting flags:
*`-start` - starting timestamp for generating logs. Logs are evenly generated on the [`-start` ... `-end`] interval.
*`-end` - ending timestamp for generating logs. Logs are evenly generated on the [`-start` ... `-end`] interval.
*`-activeStreams` - the number of active [log streams](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields) to generate.
*`-logsPerStream` - the number of log entries to generate per each log stream. Log entries are evenly distributed on the [`-start` ... `-end`] interval.
The total number of generated logs can be calculated as `-activeStreams` * `-logsPerStream`.
For example, the following command generates `1_000_000` log entries on the time range `[2024-01-01 - 2024-02-01]` across `100`
[log streams](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields), where every logs stream contains `10_000` log entries,
and writes them to `http://localhost:9428/insert/jsonline`:
```
bin/vlogsgenerator \
-start=2024-01-01 -end=2024-02-01 \
-activeStreams=100 \
-logsPerStream=10_000 \
-addr=http://localhost:9428/insert/jsonline
```
### Churn rate
It is possible to generate churn rate for active [log streams](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields)
by specifying `-totalStreams` command-line flag bigger than `-activeStreams`. For example, the following command generates
logs for `1000` total streams, while the number of active streams equals to `100`. This means that at every time there are logs for `100` streams,
but these streams change over the given [`-start` ... `-end`] time range, so the total number of streams on the given time range becomes `1000`:
```
bin/vlogsgenerator \
-start=2024-01-01 -end=2024-02-01 \
-activeStreams=100 \
-totalStreams=1_000 \
-logsPerStream=10_000 \
-addr=http://localhost:9428/insert/jsonline
```
In this case the total number of generated logs equals to `-totalStreams` * `-logsPerStream` = `10_000_000`.
### Benchmark tuning
By default `vlogsgenerator` generates and writes logs by a single worker. This may limit the maximum data ingestion rate during benchmarks.
The number of workers can be changed via `-workers` command-line flag. For example, the following command generates and writes logs with `16` workers:
```
bin/vlogsgenerator \
-start=2024-01-01 -end=2024-02-01 \
-activeStreams=100 \
-logsPerStream=10_000 \
-addr=http://localhost:9428/insert/jsonline \
-workers=16
```
### Output statistics
Every 10 seconds `vlogsgenerator` writes statistics about the generated logs into `stderr`. The frequency of the generated statistics can be adjusted via `-statInterval` command-line flag.
For example, the following command writes statistics every 2 seconds:
```
bin/vlogsgenerator \
-start=2024-01-01 -end=2024-02-01 \
-activeStreams=100 \
-logsPerStream=10_000 \
-addr=http://localhost:9428/insert/jsonline \
-statInterval=2s
```
VictoriaLogs source code has been moved to [github.com/VictoriaMetrics/VictoriaLogs](https://github.com/VictoriaMetrics/VictoriaLogs/).
addr=flag.String("addr","stdout","HTTP address to push the generated logs to; if it is set to stdout, then logs are generated to stdout")
workers=flag.Int("workers",1,"The number of workers to use to push logs to -addr")
start=newTimeFlag("start","-1d","Generated logs start from this time; see https://docs.victoriametrics.com/#timestamp-formats")
end=newTimeFlag("end","0s","Generated logs end at this time; see https://docs.victoriametrics.com/#timestamp-formats")
activeStreams=flag.Int("activeStreams",100,"The number of active log streams to generate; see https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields")
totalStreams=flag.Int("totalStreams",0,"The number of total log streams; if -totalStreams > -activeStreams, then some active streams are substituted with new streams "+
"during data generation")
logsPerStream=flag.Int64("logsPerStream",1_000,"The number of log entries to generate per each log stream. Log entries are evenly distributed between -start and -end")
constFieldsPerLog=flag.Int("constFieldsPerLog",3,"The number of fields with constant values to generate per each log entry; "+
"see https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model")
varFieldsPerLog=flag.Int("varFieldsPerLog",1,"The number of fields with variable values to generate per each log entry; "+
"see https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model")
dictFieldsPerLog=flag.Int("dictFieldsPerLog",2,"The number of fields with up to 8 different values to generate per each log entry; "+
"see https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model")
u8FieldsPerLog=flag.Int("u8FieldsPerLog",1,"The number of fields with uint8 values to generate per each log entry; "+
"see https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model")
u16FieldsPerLog=flag.Int("u16FieldsPerLog",1,"The number of fields with uint16 values to generate per each log entry; "+
"see https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model")
u32FieldsPerLog=flag.Int("u32FieldsPerLog",1,"The number of fields with uint32 values to generate per each log entry; "+
"see https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model")
u64FieldsPerLog=flag.Int("u64FieldsPerLog",1,"The number of fields with uint64 values to generate per each log entry; "+
"see https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model")
i64FieldsPerLog=flag.Int("i64FieldsPerLog",1,"The number of fields with int64 values to generate per each log entry; "+
"see https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model")
floatFieldsPerLog=flag.Int("floatFieldsPerLog",1,"The number of fields with float64 values to generate per each log entry; "+
"see https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model")
ipFieldsPerLog=flag.Int("ipFieldsPerLog",1,"The number of fields with IPv4 values to generate per each log entry; "+
"see https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model")
timestampFieldsPerLog=flag.Int("timestampFieldsPerLog",1,"The number of fields with ISO8601 timestamps per each log entry; "+
"see https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model")
jsonFieldsPerLog=flag.Int("jsonFieldsPerLog",1,"The number of JSON fields to generate per each log entry; "+
"see https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model")
statInterval=flag.Duration("statInterval",10*time.Second,"The interval between publishing the stats")
)
funcmain(){
// Write flags and help message to stdout, since it is easier to grep or pipe.
logger.Infof("start -workers=%d workers for ingesting -logsPerStream=%d log entries per each -totalStreams=%d (-activeStreams=%d) on a time range -start=%s, -end=%s to -addr=%s",
maxConcurrentRequests=flag.Int("search.maxConcurrentRequests",getDefaultMaxConcurrentRequests(),"The maximum number of concurrent search requests. "+
"It shouldn't be high, since a single request can saturate all the CPU cores, while many concurrently executed requests may require high amounts of memory. "+
"See also -search.maxQueueDuration")
maxQueueDuration=flag.Duration("search.maxQueueDuration",10*time.Second,"The maximum time the search request waits for execution when -search.maxConcurrentRequests "+
"limit is reached; see also -search.maxQueryDuration")
maxQueryDuration=flag.Duration("search.maxQueryDuration",time.Second*30,"The maximum duration for query execution. It can be overridden to a smaller value on a per-query basis via 'timeout' query arg")
)
funcgetDefaultMaxConcurrentRequests()int{
n:=cgroup.AvailableCPUs()
ifn<=4{
n*=2
}
ifn>16{
// A single request can saturate all the CPU cores, so there is no sense
// in allowing higher number of concurrent requests - they will just contend
.uplot,.uplot*,.uplot*:before,.uplot*:after{box-sizing:border-box}.uplot{font-family:system-ui,-apple-system,SegoeUI,Roboto,HelveticaNeue,Arial,NotoSans,sans-serif,"Apple Color Emoji","Segoe UI Emoji",SegoeUISymbol,"Noto Color Emoji";line-height:1.5;width:min-content}.u-title{text-align:center;font-size:18px;font-weight:700}.u-wrap{position:relative;-webkit-user-select:none;user-select:none}.u-over,.u-under{position:absolute}.u-under{overflow:hidden}.uplotcanvas{display:block;position:relative;width:100%;height:100%}.u-axis{position:absolute}.u-legend{font-size:14px;margin:auto;text-align:center}.u-inline{display:block}.u-inline*{display:inline-block}.u-inlinetr{margin-right:16px}.u-legendth{font-weight:600}.u-legendth>*{vertical-align:middle;display:inline-block}.u-legend.u-marker{width:1em;height:1em;margin-right:4px;background-clip:padding-box!important}.u-inline.u-liveth:after{content:":";vertical-align:middle}.u-inline:not(.u-live).u-value{display:none}.u-series>*{padding:4px}.u-seriesth{cursor:pointer}.u-legend.u-off>*{opacity:.3}.u-select{background:#00000012;position:absolute;pointer-events:none}.u-cursor-x,.u-cursor-y{position:absolute;left:0;top:0;pointer-events:none;will-change:transform}.u-hz.u-cursor-x,.u-vt.u-cursor-y{height:100%;border-right:1pxdashed#607D8B}.u-hz.u-cursor-y,.u-vt.u-cursor-x{width:100%;border-bottom:1pxdashed#607D8B}.u-cursor-pt{position:absolute;top:0;left:0;border-radius:50%;border:0solid;pointer-events:none;will-change:transform;background-clip:padding-box!important}.u-axis.u-off,.u-select.u-off,.u-cursor-x.u-off,.u-cursor-y.u-off,.u-cursor-pt.u-off{display:none}
retentionPeriod=flagutil.NewRetentionDuration("retentionPeriod","7d","Log entries with timestamps older than now-retentionPeriod are automatically deleted; "+
"log entries with timestamps outside the retention are also rejected during data ingestion; the minimum supported retention is 1d (one day); "+
"see https://docs.victoriametrics.com/victorialogs/#retention ; see also -retention.maxDiskSpaceUsageBytes")
maxDiskSpaceUsageBytes=flagutil.NewBytes("retention.maxDiskSpaceUsageBytes",0,"The maximum disk space usage at -storageDataPath before older per-day "+
"partitions are automatically dropped; see https://docs.victoriametrics.com/victorialogs/#retention-by-disk-space-usage ; see also -retentionPeriod")
futureRetention=flagutil.NewRetentionDuration("futureRetention","2d","Log entries with timestamps bigger than now+futureRetention are rejected during data ingestion; "+
"see https://docs.victoriametrics.com/victorialogs/#retention")
storageDataPath=flag.String("storageDataPath","victoria-logs-data","Path to directory where to store VictoriaLogs data; "+
"see https://docs.victoriametrics.com/victorialogs/#storage")
inmemoryDataFlushInterval=flag.Duration("inmemoryDataFlushInterval",5*time.Second,"The interval for guaranteed saving of in-memory data to disk. "+
"The saved data survives unclean shutdowns such as OOM crash, hardware reset, SIGKILL, etc. "+
"Bigger intervals may help increase the lifetime of flash storage with limited write cycles (e.g. Raspberry PI). "+
"Smaller intervals increase disk IO load. Minimum supported value is 1s")
logNewStreams=flag.Bool("logNewStreams",false,"Whether to log creation of new streams; this can be useful for debugging of high cardinality issues with log streams; "+
"see https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields ; see also -logIngestedRows")
logIngestedRows=flag.Bool("logIngestedRows",false,"Whether to log all the ingested log entries; this can be useful for debugging of data ingestion; "+
"see https://docs.victoriametrics.com/victorialogs/data-ingestion/ ; see also -logNewStreams")
minFreeDiskSpaceBytes=flagutil.NewBytes("storage.minFreeDiskSpaceBytes",10e6,"The minimum free disk space at -storageDataPath after which "+
"the storage stops accepting new data")
forceMergeAuthKey=flagutil.NewPassword("forceMergeAuthKey","authKey, which must be passed in query string to /internal/force_merge pages. It overrides -httpAuth.*")
storageNodeAddrs=flagutil.NewArrayString("storageNode","Comma-separated list of TCP addresses for storage nodes to route the ingested logs to and to send select queries to. "+
"If the list is empty, then the ingested logs are stored and queried locally from -storageDataPath")
insertConcurrency=flag.Int("insert.concurrency",2,"The average number of concurrent data ingestion requests, which can be sent to every -storageNode")
insertDisableCompression=flag.Bool("insert.disableCompression",false,"Whether to disable compression when sending the ingested data to -storageNode nodes. "+
"Disabled compression reduces CPU usage at the cost of higher network usage")
selectDisableCompression=flag.Bool("select.disableCompression",false,"Whether to disable compression for select query responses received from -storageNode nodes. "+
"Disabled compression reduces CPU usage at the cost of higher network usage")
storageNodeUsername=flagutil.NewArrayString("storageNode.username","Optional basic auth username to use for the corresponding -storageNode")
storageNodePassword=flagutil.NewArrayString("storageNode.password","Optional basic auth password to use for the corresponding -storageNode")
storageNodePasswordFile=flagutil.NewArrayString("storageNode.passwordFile","Optional path to basic auth password to use for the corresponding -storageNode. "+
"The file is re-read every second")
storageNodeBearerToken=flagutil.NewArrayString("storageNode.bearerToken","Optional bearer auth token to use for the corresponding -storageNode")
storageNodeBearerTokenFile=flagutil.NewArrayString("storageNode.bearerTokenFile","Optional path to bearer token file to use for the corresponding -storageNode. "+
"The token is re-read from the file every second")
storageNodeTLS=flagutil.NewArrayBool("storageNode.tls","Whether to use TLS (HTTPS) protocol for communicating with the corresponding -storageNode. "+
"By default communication is performed via HTTP")
storageNodeTLSCAFile=flagutil.NewArrayString("storageNode.tlsCAFile","Optional path to TLS CA file to use for verifying connections to the corresponding -storageNode. "+
"By default, system CA is used")
storageNodeTLSCertFile=flagutil.NewArrayString("storageNode.tlsCertFile","Optional path to client-side TLS certificate file to use when connecting "+
"to the corresponding -storageNode")
storageNodeTLSKeyFile=flagutil.NewArrayString("storageNode.tlsKeyFile","Optional path to client-side TLS certificate key to use when connecting to the corresponding -storageNode")
storageNodeTLSServerName=flagutil.NewArrayString("storageNode.tlsServerName","Optional TLS server name to use for connections to the corresponding -storageNode. "+
"By default, the server name from -storageNode is used")
storageNodeTLSInsecureSkipVerify=flagutil.NewArrayBool("storageNode.tlsInsecureSkipVerify","Whether to skip tls verification when connecting to the corresponding -storageNode")
)
varlocalStorage*logstorage.Storage
varlocalStorageMetrics*metrics.Set
varnetstorageInsert*netinsert.Storage
varnetstorageSelect*netselect.Storage
// Init initializes vlstorage.
//
// Stop must be called when vlstorage is no longer needed
funcInit(){
iflen(*storageNodeAddrs)==0{
initLocalStorage()
}else{
initNetworkStorage()
}
}
funcinitLocalStorage(){
iflocalStorage!=nil{
logger.Panicf("BUG: initLocalStorage() has been already called")
}
ifretentionPeriod.Duration()<24*time.Hour{
logger.Fatalf("-retentionPeriod cannot be smaller than a day; got %s",retentionPeriod)
}
cfg:=&logstorage.StorageConfig{
Retention:retentionPeriod.Duration(),
MaxDiskSpaceUsageBytes:maxDiskSpaceUsageBytes.N,
FlushInterval:*inmemoryDataFlushInterval,
FutureRetention:futureRetention.Duration(),
LogNewStreams:*logNewStreams,
LogIngestedRows:*logIngestedRows,
MinFreeDiskSpaceBytes:minFreeDiskSpaceBytes.N,
}
logger.Infof("opening storage at -storageDataPath=%s",*storageDataPath)
logger.Infof("forced merge for partition_prefix=%q has been started",partitionNamePrefix)
startTime:=time.Now()
localStorage.MustForceMerge(partitionNamePrefix)
logger.Infof("forced merge for partition_prefix=%q has been successfully finished in %.3f seconds",partitionNamePrefix,time.Since(startTime).Seconds())
}()
returntrue
}
// CanWriteData returns non-nil error if it cannot write data to vlstorage
funcCanWriteData()error{
iflocalStorage==nil{
// The data can be always written in non-local mode.
returnnil
}
iflocalStorage.IsReadOnly(){
return&httpserver.ErrorWithStatusCode{
Err:fmt.Errorf("cannot add rows into storage in read-only mode; the storage can be in read-only mode "+
"because of lack of free disk space at -storageDataPath=%s",*storageDataPath),
StatusCode:http.StatusTooManyRequests,
}
}
returnnil
}
// MustAddRows adds lr to vlstorage
//
// It is advised to call CanWriteData() before calling MustAddRows()
funcMustAddRows(lr*logstorage.LogRows){
iflocalStorage!=nil{
// Store lr in the local storage.
localStorage.MustAddRows(lr)
}else{
// Store lr across the remote storage nodes.
lr.ForEachRow(netstorageInsert.AddRow)
}
}
// RunQuery runs the given q and calls writeBlock for the returned data blocks
logger.Warnf("skipping too long log entry, since its length exceeds %d bytes; the actual log entry length is %d bytes; log entry contents: %s",maxInsertBlockSize,len(b),b)
// WriteRequest contains the WriteRequest, which must be pushed later to remote storage.
//
// The actual labels and samples for the time series are stored in Labels and Samples fields.
WriteRequestprompbmarshal.WriteRequest
WriteRequestprompb.WriteRequest
// Labels contains flat list of all the labels used in WriteRequest.
Labels[]prompbmarshal.Label
Labels[]prompb.Label
// Samples contains flat list of all the samples used in WriteRequest.
Samples[]prompbmarshal.Sample
Samples[]prompb.Sample
}
// Reset resets ctx.
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
