Introduce new query filter option matchNames for metricnamestats query Requests.
It allows to fetch stats for exact metric names. Which is useful for Explore Cardinality page.
It's also allows 3rd party tool to check if application metrics are used
for query requests.
This commit adds the following changes:
* replaces MetricNamesUsageStats inline func query args with dedicated struct.
* bumps cluster RPC metricNamesUsageStats_v1 to metricNamesUsageStats_v2
in order to properly encode new RequestQuery params
* adds new methods for MetricNameTracker
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6145
### Describe Your Changes
Follow-up to
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8462
Addressed review comments:
- Log panic with FATAL prefix to indicate possible on-disk data
corruption
- Moved version bump line to the tip block (v1.114.0 has already been
released) in changelog
- Removed duplicate vmagent entry from targets list from Makefile
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
### Describe Your Changes
Under heavy load, vmagent's wirte concurrency limiter
(2ab53acce4/lib/writeconcurrencylimiter/concurrencylimiter.go (L111))
queues incoming requests. If a client's timeout is shorter than the wait
time in the
queue, the client may close the connection before vmagent starts
processing it. When vmagent then tries to read the request body, it
encounters an ambiguous `unexpected EOF` error
(https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8675).
This commit adds more context to such errors to help users diagnose and
resolve
the issue when it's related to vmagent's own load and queuing behavior.
Possible user actions include:
- Lowering `-insert.maxQueueDuration` below the client's timeout.
- Increasing the client-side timeout, if applicable.
- Scaling up vmagent (e.g., adding more CPU resources).
- Increasing `-maxConcurrentInserts` if CPU capacity allows.
Steps to reproduce:
https://gist.github.com/makasim/6984e20f57bfd944411f56a7ebe5b6bf
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
This commit improves how vmagent selects the remote write protocol.
Previously, vmagent [performed a handshake
probe](0ff1a3b154/lib/protoparser/protoparserutil/vmproto_handshake.go (L11))
at
[startup](0ff1a3b154/app/vmagent/remotewrite/client.go (L173)):
- If the probe succeeded, it used the VictoriaMetrics (VM) protocol.
- If the probe failed, it downgraded to the Prometheus protocol.
- No protocol changes occurred after the initial probe at runtime.
However, this approach had limitations:
- If vmstorage was unavailable during vmagent startup, vmagent would
immediately downgrade to the Prometheus protocol, leading to higher
network usage unitl vmagent restarted. This case has been reported in
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7615.
- If the remote write server was updated or downgraded (e.g., during a
fallback or migration), vmagent would not detect the protocol change. It
would continue retrying failed requests and eventually drop them.
Require a restart of vmagent to pick up the new protocol.
This commit introduces a more adaptive mechanism.
vmagent always starts with the VM protocol and downgrades to the
Prometheus protocol only if an unsupported media type or bad request
response is received.
When this happens, the protocol is downgraded for all future requests.
In-flight requests are re-packed from Zstd to Snappy and retried
immediately.
Snappy-encoded requests are dropped if an unsupported media type or bad
request is received (no retrying).
Additionally, the in-memory and persisted queues could mix snappy and
zstd encoded blocks. The proper encoding is decided before sending by
encoding.IsZstd function.
TODO:
* [x] Add tests
* [x] Update documentation
* [x] Changelog
* [x] Research on
[content-type](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8462#issuecomment-2786918054),
[accept-encoding](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8462#issuecomment-2786923382)
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7615#top
issue.
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
Previously, if vmagent was built with CGO_ENABLED=0, vmagent cannot start and reported runtime error:
`Kafka client is not supported at systems without CGO`
This error was trigger even if `-kafka.consumer.topic` was not
provided. CGO_ENABLED=0 is default build option for linux/arm and some other archs.
This commit properly inits kafka consumer by checking if `-kafka.consumer.topic` is set.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6019
Missing cherry-picked commits:
- 30b61c6d8a
- a335ed23c7
These commits cannot be cherry-picked now because of too big churn in the docs/anomaly-detection after these commits.
'authKey' is well-known url and form param for VictoriaMetrics
components authorization. Previously, it could be printed into stdout
via httpserver error logger. It makes this authKey insecure and hard to
use.
This commit prevents from logging authKey defined at PostForm or as part
of url.Query.
It's recommneded to transfer authKey via PostForm and it should be
implemented at separate PRs.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5973
---------
Signed-off-by: f41gh7 <nik@victoriametrics.com>
Previously, if ProfileName is set to empty value (as default). AWS s3
lib ignored any profile config defined with `-configProfilePath`.
This commit correctly configure client options and set profile name only
if it's set to non-empty value.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8668
Currently, requests failing due to network timeout would receive "200
OK" while producing a warning log message about the timeout. This
behaviour is confusing and might produce unexpected issues as it is not
possible to retry errors properly.
Change this to return "502 Bad Gateway" response so that error can be
handled by the client.
See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8621
Config for testing:
```
unauthorized_user:
url_prefix: "http://example.com:9800"
```
Before the change:
```
* Trying 127.0.0.1:8427...
* Connected to 127.0.0.1 (127.0.0.1) port 8427
* using HTTP/1.x
> HEAD /api/v1/query HTTP/1.1
> Host: 127.0.0.1:8427
> User-Agent: curl/8.12.1
> Accept: */*
>
* Request completely sent off
/* NOTE: 30 seconds timeout passes */
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Vary: Accept-Encoding
Vary: Accept-Encoding
< X-Server-Hostname: pc
X-Server-Hostname: pc
< Date: Tue, 01 Apr 2025 08:54:05 GMT
Date: Tue, 01 Apr 2025 08:54:05 GMT
<
* Connection #0 to host 127.0.0.1 left intact
```
After:
```
* Trying 127.0.0.1:8427...
* Connected to 127.0.0.1 (127.0.0.1) port 8427
* using HTTP/1.x
> HEAD /api/v1/query HTTP/1.1
> Host: 127.0.0.1:8427
> User-Agent: curl/8.12.1
> Accept: */*
>
* Request completely sent off
< HTTP/1.1 502 Bad Gateway
HTTP/1.1 502 Bad Gateway
< Content-Type: text/plain; charset=utf-8
Content-Type: text/plain; charset=utf-8
< Vary: Accept-Encoding
Vary: Accept-Encoding
< X-Content-Type-Options: nosniff
X-Content-Type-Options: nosniff
< X-Server-Hostname: pc
X-Server-Hostname: pc
< Date: Tue, 01 Apr 2025 09:13:57 GMT
Date: Tue, 01 Apr 2025 09:13:57 GMT
< Content-Length: 109
Content-Length: 109
<
* Connection #0 to host 127.0.0.1 left intact
```
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
### Describe Your Changes
Log when the data response from vmselect is partial during
rule(recording, alertingrule) evaluations.
vmselect returns `isPartial: true` in case data is not fully fetched
from scattered vmstorages. At the time of rule evals, it may be drifting
apart from real values due to missing points. This is an important event
that should be logged to inform users to see how often that happens as
it may lead to false positive alerts.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Signed-off-by: emreya <emre.yazici@adyen.com>
Signed-off-by: emreya <e.yazici1990@gmail.com>
Signed-off-by: Emre Yazici <e.yazici1990@gmail.com>
When creating and listing snapshots, panic instead of returning an error
since errors are not recoverable anyway.
Also do not cleanup the filesystem on panic. Leave as is for further
manual inspection.
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
This reduces CPU usage by up to 30% in exchange of the increased RAM usage by 10%
when scraping thousands of targets, which expose millions of metrics in summary.
This looks like a good tradeoff after the commit edac875179 ,
which reduced RAM usage by more than 10%, so the final RAM usage for vmagent
is still lower than the RAM usage at v1.114.0 by ~15%, while CPU usage drops by 30%.
This reduces memory usage when reading large response bodies because the underlying buffer
doesn't need to be re-allocated during the read of large response body in the buffer.
Also decompress response body under the processScrapedDataConcurrencyLimitCh .
This reduces CPU usage and RAM usage a bit when scraping thousands of targets.
This reduces memory usage for vmagent when scraping big number of targets at the cost of slightly higher CPU usage.
The increased CPU usage can be decreased by disabling tracking of stale markers either via -promscrape.noStaleMarkers
command-line flag or via `no_stale_markers: true` option at the scrape config pointed by -promscrape.config command-line flag.
See https://docs.victoriametrics.com/vmagent/#prometheus-staleness-markers
The pool is used mostly for obtaining byte buffers for responses from scrape targets.
There are no responses smaller than 256 bytes in practice, so there is no sense in maintaining
pools for byte slices up to 64 and 128 bytes.
Previously the maxLabelsLen could be updated with smaller value after it is updated to bigger value by concurrently running goroutines.
Prevent this by loading the latest maxLabelsLen value and updating it only if it is smaller than the current len(wc.labels)
before the exit from callback passed to stream.Parse.
While at it, return early from the callback on the sample_limit exceeding error,
since the rest of the code in the callback becomes no-op after wc.reset().
This simplifies following the logic in the code a bit.
Also remove outdated misleading comment in front of sw.pushData() call inside callbacks passed to stream.Parse.
This comment has no sense after every callback start working with its own goroutine-local wc.
Start with writeRequestCtx containing up to 256 labels instead of 8 labels,
since a typical response from scrape target contains much more than 8 labels across all the exposed metrics.
Do not pre-allocate labels at writeRequestCtx, since they are pre-allocated inside writeRequestCtx.addRows(),
together with the pre-allocation of samples and writeRequest.Timeseries.
The applySeriesLimit applies the limit to samples stored at writeRequestCtx,
while the scrapeWork is used as read-only configuration source.
That's why it is better from maintainability PoV to attach the applySeriesLimit
method to writeRequestCtx.
While at it, clarify docs for the applySeriesLimit function.
The rows are added to writeRequestCtx, while the scrapeWork is used only as a read-only configuration source.
So it is better from maintainability PoV to attach addRows function to writeRequestCtx instead of scrapeWork.
Also attach addAutoMetrics to writeRequestCtx instead of scrapeWork due to the same reason:
addAutoMetrics adds metrics to the writeRequestCtx, while using scrapeWork as a read-only configuration source.
While at it, remove tmpRow from scrapeWork struct in order to reduce the complexity of this struct.
areIdenticalSeries doesn't access scrapeWork members except of sw.Config of *ScrapeWork type.
It is better from maintainability PoV to attach this methos to ScrapeWork then.
While at it, replace sw.Config with cfg shortcut at scrapeWork.processDataOneShot()
and scrapeWork.processDataInStreamMode().
Checklist is a more practical list of actions than a full Contributing doc.
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit b277a62e94)
Having `victoriametrics` or `victorialogs` tags should be enough for
filtering dashboards related to VictoriaMetrics components.
Related ticket
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8618
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 66e7b908ec)
Support of the latest prometheus/common is not released yet so pin to previous version.
Related commit at prometheus/prometheus: 95f49dd84b
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
This allows verifying how the benchmark performance scales with the number of available CPU cores
and makes the results of the benchmark consistent with other BenchmarkScrapeWorkScrapeInternal* benchmarks.
Also reduce the amounts of memory allocations inside generateScrape() function in order to reduce
measurement noise during the BenchmarkScrapeWorkScrapeInternalStreamBigData run.
This is a follow-up after c05ffa906d
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8515
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8159
It is always better to run benchmarks in parallel on all the available CPU cores
in order to see how their performance scales with the number of CPU cores (GOMAXPROCS).
The commit also performs the following modifications:
- Removes the dependency of on the scrapeWork from getLabelsHash() function.
- Makes sure that the benchmark cannot be optimized out by the compiler, by introducing a dependency
on a global Sink variable. Previously the getLabelsHash() function call could be optimized out
by the compiler, since this call has no side effects, and the returned result is ignored.
- Reduces the amounts of memory allocations inside the BenchmarkScrapeWorkGetLabelsHash
when preparing the labels for the benchmark. This should reduce measurements' noise during the benchmark.
This is a follow-up for c05ffa906d
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8515
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8159
* vmgateway: properly set the `Host` header when routing requests to `-write.url` and `-read.url`
* Update docs/victoriametrics/changelog/CHANGELOG.md
---------
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit b9c777a578)
* vmalert: properly attach tenant labels `vm_account_id` and `vm_project_id` to alerting rules when enabling `-clusterMode`
Previously, these labels were lost in alert messages to Alertmanager. Bug was introduced in [v1.112.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.112.0).
(cherry picked from commit d8fe739aba)
Previously the buffer was increased by 30% after it became 50% full.
For example, if more than 5MB of data is read into 10MB buffer, then its' size
was increased to 13MB, leading to 13MB-5MB = 8MB of waste.
This translates to 8MB/5MB = 160% waste in the worst case.
The updated algorithm increases the buffer by 30% after it becomes ~94% full.
This means that if more than 9.4MB of data is read into 10MB buffer,
then its' size is increased to 13MB, leading to 13MB-9.4MB = 3.6MB of waste.
This translates to 3.6MB / 9.4MB = ~38% waste in the worst case.
This should reduce memory usage when vmagent reads big responses from scrape targets.
While at it, properly append the data to buffer if it already has more than 4KiB of data.
Previously the data over 4KiB in the buffer was lost after ReadFrom call.
This is a follow-up for f28f496a9d
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6761
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6759
It is faster to read the whole data and then decompress it in one go for zstd and snappy encodings.
This reduces the number of potential read() syscalls and decompress CGO calls needed
for reading and decompressing the data.
### Describe Your Changes
replace /VictoriaLogs aliases to /victorialogs as generated directories
are anyway renamed to victorialogs before deployment
need to merge this PR immediately after
https://github.com/VictoriaMetrics/vmdocs/pull/116
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
(cherry picked from commit 0e142e4e11)
The panel was producing wrong predictions as it is almost impossible,
without making too expensive queries, to make a precise predictions.
More details on reasoning why it is better to remove it than fix it
is here https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8492.
This change also removes ETA panels from alerting rules annotations.
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit ef16681dbf)
### Describe Your Changes
remove unused files from docs
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
(cherry picked from commit 45df1e1142)
### Describe Your Changes
`its'` -> `its`
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
(cherry picked from commit 0a49d8c930)
### Describe Your Changes
Update dependencies to latest versions
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
(cherry picked from commit 53a1c6162d)
### Describe Your Changes
- add smaller search weights for changelog content
- remove replace `<details>` tag with collapse shortcode
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
(cherry picked from commit 63222a512e)
related PR https://github.com/VictoriaMetrics/vmdocs/pull/115
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
(cherry picked from commit f152021521)
- Move lib/httputil.Transport to lib/promauth.NewTLSTransport. Remove the first arg to this function (URL),
since it has zero relation to the created transport.
- Move lib/httputil.TLSConfig to lib/promauth.NewTLSConfig. Re-use the existing functionality
from lib/promauth.Config for creating TLS config. This enables the following features:
- Ability to load key, cert and CA files from http urls.
- Ability to change the key, cert and CA files without the need to restart the service.
It automatically re-loads the new files after they change.
- Avoid a data race when multiple goroutines access and update roundTripper.trBase inside roundTripper.getTransport().
The way to go is to make sure the roundTripper.trBase is updated only during roundTripper creation,
and then can be only read without updating.
- Use the http.DefaultTransport for http2 client connections at Kubernetes service discovery.
Previously golang.org/x/net/http2.Transport was used there. This had the following issues:
- An additional dependency on golang.org/x/net/http2.
- Missing initialization of Transport.DialContext with netutil.Dialer.DialContext for http2 client.
- Missing initialization of Transport.TLSHandshakeTimeout for http2 client.
- Introduction of the lib/promauth.Config.NewRoundTripperFromGetter() method, which is hard to use properly.
- Unnecessary complications of the lib/promauth.roundTripper, which led to the data race described above.
- Avoid a data race when multiple goroutines access and update tls config shared between multiple
net/http.Transport instances at the TLSClientConfig field. The way to go is to always make a copy of the tls config
before assigning it to the net/http.Transport.TLSClientConfig field.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5971
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7114
* update wording
* rm extra verbosity
* typo fixes
* add extra info about cluster sizing
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit e09c3f7938)
It is much better to panic instead of returning an error on programming error (aka BUG),
since this significantly increases chances that the bug will be noticed, reported and fixed ASAP.
The returned error can be ignored, even if it is logged, while panic is much harder to ignore.
The code must always panic instead of returning errors when any programming error (aka unexpected state) is detected.
This is a follow-up for the commit 9feee15493
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6783
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6771
- update references to the latest version
- port LTS changelog
- revert changes from 47201ace96 and overwrite by an actual latest enterprise release version instead of latest LTS release
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
(cherry picked from commit 34e1e18bcc)
### Describe Your Changes
PR updates docs to release v1.21.0, in particular, adjust docs and its
structure to High Availability (HA) and horizontal scalability (HS)
capabilities.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
Previously, performance of stream.Parse could be limited by mutex.Lock on callback function. It used shared writeContext. With complicated relabeling rules and any slowness at pushData function, it could significantly decrease parsed rows processing performance.
This commit removes locks and makes parsed rows processing lock-free in the same manner as `stream.Parse` processing implemented at push ingestion processing.
Implementation details:
- Removing global lock around stream.Parse callback.
- Using atomic operations for counters
- Creating write contexts per callback instead of sharing
- Improving series limit checking with sync.Once
- Optimizing labels hash calculation with buffer pooling
- Adding comprehensive tests for concurrency correctness
Benchmark performance:
```
# before
BenchmarkScrapeWorkScrapeInternalStreamBigData-10 13 81973945 ns/op 37.68 MB/s 18947868 B/op 197 allocs/op
# after
goos: darwin
goarch: arm64
pkg: github.com/VictoriaMetrics/VictoriaMetrics/lib/promscrape
cpu: Apple M1 Pro
BenchmarkScrapeWorkScrapeInternalStreamBigData-10 74 15761331 ns/op 195.98 MB/s 15487399 B/op 148 allocs/op
PASS
ok github.com/VictoriaMetrics/VictoriaMetrics/lib/promscrape 1.806s
```
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8159
---------
Signed-off-by: Maksim Kotlyar <kotlyar.maksim@gmail.com>
Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>
Previously, `getBackupsList` was appending `latest` backup in all cases without checking if it actually exists.
This lead to `vm_backup_last_run_failed` metric being set to `1` since folder did not contain successful completion marker.
This commit adds a check to handle a case when remote storage does not contain any backups yet.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8490
---
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
These errors could be caused by intermittent network issues, especially
in case of using proxies when accessing S3 storage. Previously, such
error would abort backup/restore process and require manual intervention
to ensure backups consistency.
This commit adds automatic retries to handle this to improve backups
reliability and resilience to network issues.
1. remove "vmalert" word from vmgateway doc and exposed metrics;
2. remove unrelated flags like -datasource.roundDigits, remoteRead.disablePathAppend, -datasource.disableStepParam
vmbackupmanager uses `runC` channel for inter-goroutine communication between `scheduler` and `execute` goroutines.
Previously, `runC` wasn't closed during graceful shutdown. And vmbackupmanager process couldn't gracefully stop. It could only be killed with-in configured timeout.
This commit properly closes `runC` by `scheduler` when stopping vmbackupmanager in order to avoid shutdown delay.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8554
Previously, backup was first scheduled at 00:00:00 and `getSleepDuration` was immediately executed to get the sleep duration for the next backup. Since it was returning `1 * time.Second` the next backup was attempted and failed to be scheduled.
Update logic to wait for full backup interval in this case so that there will be no attempt to schedule an unneeded backup.
It also adds the following changes:
* fix error log entry reference to type of policy
* add a message about retention completion similar to existing message for backups to make it more consistent
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8499
This is a follow-up for fb6d2e92e3a1cf412d1f7dee64a4852941a8aa1b
### Describe Your Changes
during initial flush with deduplication and windows enabled lower
timestamps threshold is set to an upper bound of the next deduplication
interval, which leads to ignoring all samples on subsequent intervals
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 511517f491)
### Describe Your Changes
This PR adds nanosecond precision to log sorting, ensuring accurate
ordering of entries with sub-millisecond differences.
Related issue: #8346
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
(cherry picked from commit 31f662a0f7)
The mark is important for readers to understand the type of release.
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 90d547dec0)
Commit 9ca74d1fff introduced an issue with metrics registration. Due to metrics.Summary type always registered at the global state of metrics package, vmalert had increased memory and CPU usage after multiple configuration reloads.
This commit addresses this issue and properly registers metrics.Summary metric. Now metrics for group and rules must be explicitly registered before group.Start with group.Init method. It simplifies metrics usage an ensures that all needed metrics were registered and group is ready to start.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8532
1. fix possible data race on group checksum when reload is called
concurrently. Before, it didn't affect much but might update the group
one more time.
2. remove the unnecessary g.mu.RLock() and compute group.id at newGroup creation. Changes to group.ID()
indicate that type and interval have changed, and the group is new.
Related PR:
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8540
Previous commit 9ca74d1fff added a regression for notifier's metrics exposed by vmalert. vmalert returned new notifier instances for the blackhole notifier type. And it registered new metrics each get notifiers function was called. It registered duplicate metrics and lead to OOM crash.
This commit properly init blachole notifier instances and add metrics for it only once, during application start.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8532
Previously, vmagent may incorrectly store partial scrape response
in case of scrapping error. It may happen if `sw.ReadData` call fetched
some chunked response and store it at buffer. And later context deadline
exceed error happened.
As a result, at the next scrape iteration this partial response could
be forwarded to the `sw.sendStaleSeries(lastScrape...)` function call
and lead to `Prometheus line` parsing error.
This commit properly set response body to the empty value in case of
scrapping error. It prevents storing partial scrape response body. And
it no longer sends partial staleness markers to the remote storage.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8528
This panel should have help us to identify
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8501 during
release checks. While we already track the GC pressure, its value is
relative and change wasn't noticeable for the workloads that we
observed.
The absolute values of allocations rate could have helped to see the
anomaly.
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 3511e2e6af)
### Describe Your Changes
`prasedRelabelConfig` -> `parsedRelabelConfig`
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
(cherry picked from commit 127d4f37b8)
### Describe Your Changes
Fix endless group expansion loop.
Related issue: #8347
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
(cherry picked from commit 44a54e4590)
### Describe Your Changes
Fix many spelling errors and some grammar, including misspellings in
filenames.
The change also fixes a typo in metric `vm_mmaped_files` to `vm_mmapped_files`.
While this is a breaking change, this metric isn't used in alerts or dashboards.
So it seems to have low impact on users.
The change also deprecates `cspell` as it is much heavier and less usable.
---------
Co-authored-by: Andrii Chubatiuk <achubatiuk@victoriametrics.com>
Co-authored-by: Andrii Chubatiuk <andrew.chubatiuk@gmail.com>
(cherry picked from commit 76d205feae)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
### Describe Your Changes
This PR adds the dedicated FAQ for VictoriaMetrics Cloud
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
(cherry picked from commit d852e5e0b4)
This reduces the size of LogRows.streamTagCanonicals by 1/3 because of the eliminated `cap` field
in the slice header (reflect.SliceHeader) compared to the string header (reflect.StringHeader).
### Describe Your Changes
Doc updates to a patch release v1.20.1, fixing a bug in
`PeriodicScheduler` that may affect some of the customers' deployments
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
This commit adds lib/chunkedbuffer.Buffer - an in-memory chunked buffer
optimized for random access via MustReadAt() function.
It is better than bytesutil.ByteBuffer for storing large volumes of data,
since it stores the data in chunks of a fixed size (4KiB at the moment)
instead of using a contiguous memory region. This has the following benefits over bytesutil.ByteBuffer:
- reduced memory fragmentation
- reduced memory re-allocations when new data is written to the buffer
- reduced memory usage, since the allocated chunks can be re-used
by other Buffer instances after Buffer.Reset() call
Performance tests show up to 2x memory reduction for VictoriaLogs
when ingesting logs with big number of fields (aka wide events) under high speed.
Pre-allocate the needed slice of strings and then assign items to it by index
instead of appending them. This reduces the number of memory allocations
and improves performance a bit.
…an 64KB at Reset
This commit reverts
b58e2ab214
as it has negative impacts when ByteBuffer is used for workloads that
always exceed 64KiB size. This significantly slows down affected
components because:
* buffers aren't beign reused;
* growing new buffers to >64KiB is very slow.
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8501
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
Signed-off-by: hagen1778 <roman@victoriametrics.com>
- Properly decode protobuf-encoded Loki request if it has no Content-Encoding header.
Protobuf Loki message is snappy-encoded by default, so snappy decoding must be used
when Content-Encoding header is missing.
- Return back the previous signatures of parseJSONRequest and parseProtobufRequest functions.
This eliminates the churn in tests for these functions. This also fixes broken
benchmarks BenchmarkParseJSONRequest and BenchmarkParseProtobufRequest, which consume
the whole request body on the first iteration and do nothing on subsequent iterations.
- Put the CHANGELOG entries into correct places, since they were incorrectly put into already released
versions of VictoriaMetrics and VictoriaLogs.
- Add support for reading zstd-compressed data ingestion requests into the remaining protocols
at VictoriaLogs and VictoriaMetrics.
- Remove the `encoding` arg from PutUncompressedReader() - it has enough information about
the passed reader arg in order to properly deal with it.
- Add ReadUncompressedData to lib/protoparser/common for reading uncompressed data from the reader until EOF.
This allows removing repeated code across request-based protocol parsers without streaming mode.
- Consistently limit data ingestion request sizes, which can be read by ReadUncompressedData function.
Previously this wasn't the case for all the supported protocols.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8416
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8380
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8300
The `ignore_fields` HTTTP query args can contain prefixes ending with '*'.
For example, `ignore_fields=foo.*,bar` skips all the fields starting with `foo.`
during data ingestion.
The optimization touches 2 things:
1. Reduces amount of allocations when comparing canonical metric names
between left and right parts of expressions.
2. Adds fast path for cases when right part of expression returns
scalar: `series_selector or on() vector(1)`, which is a typical
expression.
```
benchcmp old.txt new.txt
benchcmp is deprecated in favor of benchstat: https://pkg.go.dev/golang.org/x/perf/cmd/benchstat
benchmark old ns/op new ns/op delta
BenchmarkBinaryOpOr/tss:1_or_tss:1-14 291 272 -6.56%
BenchmarkBinaryOpOr/tss:1_or_tss:1000-14 44590 28592 -35.88%
BenchmarkBinaryOpOr/tss:1000_or_tss:1-14 103124 39563 -61.64%
BenchmarkBinaryOpOr/tss:1000_or_tss:1000-14 20386150 1859335 -90.88%
BenchmarkBinaryOpOr/tss:1000_or_on()_vector(0)-14 91382 36805 -59.72%
```
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8382
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit dc1f7ef0d0)
* tie relevant functionality together
* change hierarchy of related options to visually group it
No breaking changes to links.
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 8b0129f29b)
* move related sections clother to each other
* group related sections within the same section
The intention of the change is to tie related documentation together.
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 85f1bd172b)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Long constant fields cannot be stored in columnsHeader as a const column,
because their size exceeds maxConstColumnValueSize, so they are stored as regular values.
This commit optimizes storing such fields by storing only a single value
across the field values in a block instead of storing multiple values.
This should improve data ingestion performance a bit. This also should improve query
performance when the query accesses such fields because of better cache locality.
Also improve persisting of constant string lengths by storing them only once.
Previously, vmselect didn't stop multitenant query execution if it
receives error from vmstorage. Such as limit error or any other. It
continued to execute queries until it did it for all tenants. It leads
to the potential waste of resources.
In addition, callback error was incorrectly reference and can be updated by
subsequent callback call.
This commit returns error earlier, cancels sub-sequent requests for
tenants and properly return storageNode request error.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8461
Previously, if the command-line flag value `-remoteWrite.showURL` changed, vmagent dropped content of persistent queues. It's not expected behavior and may lead to data-loss at queue.
Further more if command-line flag value `-remoteWrite.showURL` is set to `true`, any changes to url query arguments will lead to persistent queue drop. The most common uses is kafka and gcp pub-sub integration. It uses url query arguments for client configuration.
Also, it complicates copy content of persistent queue between vmagents. Since it requires to properly change name inside metainfo.json.
This commit removes persistent queue name equality check from `lib/persistentqueue`. This check was added as an additional protection from on-disk data corruption.
It's safe to skip this check for vmagent, because vmagent encodes remoteWrite.url as part of path to the queue. It guarantees that there will be no collision.
related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8477.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Signed-off-by: f41gh7 <nik@victoriametrics.com>
Co-authored-by: f41gh7 <nik@victoriametrics.com>
Previously, TLS config was only created for URLs with `https` scheme.
This could lead to unexpected errors when original URL was redirecting
to `https` one as TLS config is not applied.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8494
Almost all storage API operations, both ingestion and retrieval, involve
writing and/or reading the indexdb. However, during these operations,
the indexdb refcount is not incremented. This may lead to panics if
indexdb is rotated more than once during these operations.
This commit increments the refcount before using indexdb and decrements it
after use.
Note that rotating indexdb more than once during some operation is an
impossible case under normal circumstances as the min retention period
is 1 day (i.e. the indexdb will be rotated once per day). However, we
want the storage to behave correctly in all cases.
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
Loki doesn't support well high-cardinality log fields (e.g. fields with big number of unique values).
That's why Promtail, Grafana Agent and Grafana Alloy encode such fields into a JSON and push them
as a plaintext log message to the remote storage. This isn't an efficient way to store high-cardinality
log fields in VictoriaLogs, since it is optimized for storing and querying such fields when they are stored
distinctly as a regular log fields according to VictoriaLogs data model ( https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model ).
This commit enables automatic parsing of JSON-encoded log fields at plaintext log message received over Loki protocol
and storing them as a separate log fields. This should improve data compression ratio and reduce disk space usage.
This should also improve query performance when the parsed log fields are used in queries for filtering and aggregation.
The old behaviour can be restored by passing -loki.disableMessageParsing command-line flag to VictoriaLogs.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8486
Fix metric that shows number of active time series when per-day index is disabled. Previously, once per-day index was disabled, the active time series metric would stop being populated and the `Active time series` chart would show 0.
See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8411.
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
- Properly handle negative timestamps (e.g. timestamps before 1970-01-01)
- Optimize parsing floating-point timestamps by eliminating the memory allocation
needed for returning an error from strconv.ParseInt. Instead, check whether the string contains a dot,
and then parse it as a floating-point number.
- Add tests for ParseUnixTimestamp function.
- Make the code easier to understand and maintain by removing unneeded generic function toNano().
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8470
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8472
### Describe Your Changes
Add `vmalert_alerts_send_latency_seconds` metric for
alertmanager.notifier.
To measure the time for alertmanager calls to send alerts per notifier.
This is needed to see the latency for each notifier from vmalert calls.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Signed-off-by: emreya <e.yazici1990@gmail.com>
Co-authored-by: Hui Wang <haley@victoriametrics.com>
Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>
(cherry picked from commit cfd2c6e5e7)
### Describe Your Changes
Implement changes mentioned in #6898
Allow the administrator to specify the limit of returned TSDB series in
`/api/v1/status/tsdb` by making a TopN limit configurable from CLI.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Signed-off-by: alicja-karasiewicz <alicja.karasiewicz@allegro.com>
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
* add version since feature is available
* add cluster endpoint paths
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
Signed-off-by: f41gh7 <nik@victoriametrics.com>
(cherry picked from commit 11436d5f00)
### Fix scrapePool name
If in the scrape file, I do some magic and manipulate the job name then
Prometheus will show scrapePool as the original job name in the targets
API, but vmagent will set it to the final value which is wrong.
example
```
job: consul-targets
...
- source_labels: [ __meta_consul_service ]
regex: (\w+)[_-]exporter
target_label: job
replacement: $1
```
curl to prom API will show
`"scrapePool": "consul-targets",`
vmagent:
`""scrapePool": "node",`
before changes:
```
curl -s 'http://localhost:8429/api/v1/targets' | jq -r '.data.activeTargets[].scrapePool'| sort|uniq
blackbox
pgbackrest
postgres
```
after changes
```
curl -s 'http://localhost:8429/api/v1/targets' | jq -r '.data.activeTargets[].scrapePool'| sort|uniq
blackbox
consul-targets
```
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 486b9e1c64)
Ordering changes by release versions enhances the searchability
of the documentation. For example, tracking which release got
the bugfix becomes easier if releases are already sorted.
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 6a1c70115a)
### Describe Your Changes
This PR updates the documentation by removing old assets and adding the
user management chapter, divided in different sections.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
(cherry picked from commit 0873d1d8ab)
### Describe Your Changes
Fixed the typo in the documentation. Updated `Ir provides` to `It
provides`
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
(cherry picked from commit 179c530095)
### Describe Your Changes
fixes#8469
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
(cherry picked from commit c174a046e2)
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
(cherry picked from commit fd7b016c5b)
This feature allows to track query requests by metric names. Tracker
state is stored in-memory, capped by 1/100 of allocated memory to the
storage. If cap exceeds, tracker rejects any new items add and instead
registers query requests for already observed metric names.
This feature is disable by default and new flag:
`-storage.trackMetricNamesStats` enables it.
New API added to the select component:
* /api/v1/status/metric_names_stats - which returns a JSON
object
with usage statistics.
* /admin/api/v1/status/metric_names_stats/reset - which resets internal
state of the tracker and reset tsid/cache.
New metrics were added for this feature:
* vm_cache_size_bytes{type="storage/metricNamesUsageTracker"}
* vm_cache_size{type="storage/metricNamesUsageTracker"}
* vm_cache_size_max_bytes{type="storage/metricNamesUsageTracker"}
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4458
---------
Signed-off-by: f41gh7 <nik@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
### Describe Your Changes
Previously, "selector @ another_selector" assumed that
"another_selector" metric is supposed to exist since "start" used in the
query.
If the query was evaluated in the following case (timestamps):
- start - 2, end - 10
- "another_selector" 5,6,7,8,9,10
- "selector" The resulting "at" timestamp would be taken from NaN (as
`int64(NaN * 1000)`), causing a panic or invalid behavior later.
Note that type cast of `NaN` to int64 is also platform-dependent, so
value of `int64(math.NaN() * 1000)` can produce `0` or max int64 on
different platforms and versions of Go.
This commit changes this and checks for the first non-NaN value. This
makes it easier to use for users as series are not always aligned and
returning an error in this case would disallow using this for some time
ranges.
See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8444
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 7dfaef9088)
Previously, the trailing slash was removed and caused an incorrect redirect path when visiting VMUI.
This commit leaves it as is. Also it applies minor refactoring to url formatting.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8439
Previously, opentelemetry attribute parsed added extra field names according to
golang JSON parser spec for structs:
```
struct AnyValue{
StringValue string
}
```
Was serialized into:
```
{"StringValue": "some-string"}
```
While opentelemetry-collector serializes it as
```
"some-string"
```
This commit changes this behaviour it makes parses compatible with opentelemetry-collector format. See test cases for examples.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8384
### Describe Your Changes
Fix available from version hint. The feature was introduced in
[v1.91.0](https://docs.victoriametrics.com/changelog/changelog_2023/#v1910).
I noticed that the sentence uses both `{{% available_from "v1.91.0" %}}`
and a manual reference like `starting from
[v1.91](https://docs.victoriametrics.com/changelog/#v1910)`. Does {{%
available_from %}} fully supersede the manual changelog reference, and
should the later be removed? Or should\could both be used together?
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
(cherry picked from commit e11f5eda1c)
### Describe Your Changes
Removed VictoriaMetrics prefix from anomaly detection menu items
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
(cherry picked from commit bbdb650f2f)
### Describe Your Changes
This PR adds the remaining subsections for the get started part. Some
content is taken from the product page.
Ideally, the get-started page would have some cards instead of raw
links. We should explore that.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
(cherry picked from commit 400101c674)
Commit cd39df1 introduced regression, which caused any write path related limits to be ignored.
This commit fixes match typo and adds check to prevent such kind of regression in future.
### Describe Your Changes
> ⚠️ Even if approved, please don't merge it on my behalf, I
still may apply a couple of re-phrasings before merging it on Monday
03.03.2025
- Aligned `vmanomaly` docs with release v1.20.0
- Re-structured root page of anomaly detection docs for clarity
- Added several sections to FAQ, e.g. on how to incorporate domain
knowledge into anomaly detection configs
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
### Describe Your Changes
This PR fixes an issue where the Downsampling filters debug page would
get stuck in an infinite loading state when labels had no matches. Now,
the case is properly handled.
Related issue: #8339
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 974c094a52)
### Describe Your Changes
I guess a "downsampling" term is more appropriate than "deduplication"
in the downsampling paragraph.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
(cherry picked from commit 649f4d5e00)
VictoriaLogs inserts `_time` field as a label in result when query with
[time buckets stats
pipe](https://docs.victoriametrics.com/victorialogs/logsql/#stats-by-time-buckets),
making the result meaningless and may lead to cardinality issues.
>curl --location --request POST
'https://play-vmlogs.victoriametrics.com/select/logsql/stats_query?query=_time%3A1m%20%7C%20stats%20by%20(_time%3A10s)%20count%20()%20as%20total'
>{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"total","_time":"2025-01-24T12:31:30Z"},"value":[1737721904.4476516,"12"]},{"metric":{"__name__":"total","_time":"2025-01-24T12:31:10Z"},"value":[1737721904.4476516,"10"]},{"metric":{"__name__":"total","_time":"2025-01-24T12:31:00Z"},"value":[1737721904.4476516,"10"]},{"metric":{"__name__":"total","_time":"2025-01-24T12:31:20Z"},"value":[1737721904.4476516,"12"]},{"metric":{"__name__":"total","_time":"2025-01-24T12:30:50Z"},"value":[1737721904.4476516,"10"]},{"metric":{"__name__":"total","_time":"2025-01-24T12:30:40Z"},"value":[1737721904.4476516,"9"]}]}}%
---------
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 52988ebdc8)
* use unified numeric list;
* apply line width limits;
* remove time filter from quantile examples, as we suggest to
not use time filters.
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 065a3d068c)
### Describe Your Changes
Just a small typo in the docs
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
(cherry picked from commit 7451a3631a)
Since funcs `ParseDuration` and `ParseTimeMsec` are used in vlogs,
vmalert, victoriametrics and other components, importing promutils only
for this reason makes them to export irrelevant
`vm_rows_invalid_total{type="prometheus"}` metric.
This change removes `vm_rows_invalid_total{type="prometheus"}` metric
from /metrics page for these components.
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 63f6ac3ff8)
### Describe Your Changes
Previous link was incorrectly created by adding repository name to
organization link.
All other links are using proper schema.
This is a follow-up for 6ff61a1c.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
(cherry picked from commit 7f6156e46d)
### Describe Your Changes
#8342
fix negative rate result when the lookbehind window is longer than
`-search.maxLookback` or `-search.maxStalenessInterval` and data
contains gap.
This issue was introduced since
[v1.110.0](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8072).
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
`MustParsePromMetrics` imports `lib/protoparser/prometheus`, and this
package exposes the following metrics:
```
vm_protoparser_rows_read_total{type="promscrape"}
vm_rows_invalid_total{type="prometheus"}
```
It means every package that uses `lib/prompbmarshal` will start exposing
these metrics. For example, vlogs imports `lib/protoparser/common` which
uses `lib/prompbmarshal.Label`. And only because of this vlogs starts
exposing unrelated prometheus metrics on /metrics page.
Moving `MustParsePromMetrics` to `lib/protoparser/prometheus` seems like
the leas intrusive change.
-----------
Depends on another change
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8403
Signed-off-by: hagen1778 <roman@victoriametrics.com>
This change would help to track slow queries via alerts or dashboards.
---------
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit ba2bf9e73a)
### Describe Your Changes
Updated `publish-via-docker` task to push images to multiple registries
as defined by `DOCKER_REGISTRIES`. By default, publish pushes images to
both docker.io and quay.io. It is possible to choose a single repo by
overriding `DOCKER_REGISTRIES`, for example: `DOCKER_REGISTRIES=quay.io
make publish-victoria-logs`
Note that `package-via-docker` task is not using multiple registries as
there is usually little sense in building same image with different tags
for local use.
See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4116
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Previously, if indexDB search failed for some reason during search at previous indexDB (aka extDB), VictoriaMetrics stored empty search result at cache. It could cause incorrect search results at subsequent requests.
This commit checks search error and stores request results only on success.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8345
For example, `field:~".+"`, `field:~".*"` or `field:""`
Replace such filters to faster ones. For example, `field:~".*"` is replaced with `*`,
while `field:~".+"` is replaced with `field:*`.
These filters can be used for selecting logs where one field value is less than another field value.
These filter complement `<=` and `<` filters for constant literals.
(cherry picked from commit 30974e7f3f)
TooHighQueryLoad should trigger when vmsingle or vmselect can't start
processing read queries for last 15min.
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 8f87427c81)
The version tooltips were incorrectly set to `#tip` instead of `#`,
so ` make docs-update-version` didn't catch them.
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 3828c01540)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
This allows using different intervals for flushing in-memory data among different mergeset.Table instances.
The initial user of this feature is lib/logstorage.Storage, which explicitly passes Storage.flushInterval
to every created mereset.Table instance. Previously mergeset.Table instances were using 5 seconds
flush interval, which didn't depend on the Storage.flushInterval.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4775
The new version of github.com/VictoriaMetrics/metricsql handles $__interval and $__rate_interval
inside rollup functions in more correct way - it drops square brakets, so VictoriaMetrics
could automatically detect the needed lookbehind window depending on the time distance between real samples.
For example, rate(m[$__rate_interval]) is parsed and processed as rate(m) now.
Make sure that the maximum log field name, which can be generated by JSONParser.ParseLogMessage,
doesn't exceed the hardcoded limit maxFieldNameSize. Stop flattening of nested JSON objects
when the resulting field name becomes longer than maxFieldNameSize, and return the nested JSON object
as a string instead.
This should prevent from parse errors when ingesting deeply nested JSON logs with long field names.
- `contains_any` selects logs with fields containing at least one word/phrase from the provided list.
The provided list can be generated by a subquery.
- `contains_all` selects logs with fields containing all the words and phrases from the provided list.
The provided list can be generated by a subquery.
### Describe Your Changes
- Fixed 404 links to anomaly score dashboard `.json` file on Github.
- Fixed broken markdown on QuickStart page.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
### Describe Your Changes
> (currently in DRAFT) This dashboard and the docs may look differently
after recent review and the feedback received.
As requested by our customers, this PR introduces `anomaly score`-based
dashboard for `vmanomaly` to improve the visual experience and ease the
drill down debugging process for respective anomaly detection setups.
Accompanying guide (under `default` preset mode) is provided as well.
- [For additional
benefits](https://docs.victoriametrics.com/victoriametrics-datasource/#motivation),
the dashboard is based on [VictoriaMetrics
datasource](https://docs.victoriametrics.com/victoriametrics-datasource/)
rather than on `Prometheus` datasource
- Tested locally and on our https://play.victoriametrics.com/
`anomaly_score` data (tenant ID 0)
- To check the guide, build the docs locally (`vmdocs`, `make run
local`) and follow to the
http://localhost:1313/anomaly-detection/presets/#default section
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
Previously if rule group parameters were changed, alerting rules related metrics could be deleted due to bug at `utils/metrics` package.
This commit introduces `metrics.Set` per rule group. It holds group and alerting rules metrics. It properly unregister alerting rules metrics and addresses issue.
In addition:
- expose group metrics only once group is started - this helps to avoid
exposing metrics for groups which are created during YAML unmarshaling
and only used to update existing group.
- properly close rules which are discarded after updating existing rules
so that metrics are also correctly closed.
- detect file renames and properly recreate groups "moved" between files.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8229
Examples:
- `top 5 x, y` is equivalent to `top 5 by (x, y)`
- `uniq foo, bar` is equivalent to `uniq by (foo, bar)`
- `unroll foo, bar` is equivalent to `unroll (foo, bar)`
_time:<=max_time filter must include logs with timestamps matching max_time.
For example, _time:<=2025-02-24Z must include logs with timestamps until the end of February 24, 2025.
Examples:
_time:>=2025-02-24Z selects logs with timestamps bigger or equal to 2025-02-24 UTC
_time:>1d selects logs with timestamps older than one day comparing to the current time
This simplifies writing queries with _time filters.
See https://docs.victoriametrics.com/victorialogs/logsql/#time-filter
It looks like the version deployed works differently than in local, and
some paths are not taken correctly. This commit fixes that by adding
absolute paths to the 2 images included in this section.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
It looks like when using html image insertion, the path is changed. This
commit fixes that by redirecting to the previous dir in path.
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
This guide was outdated. the intention of these changes is to:
* Ease maintainability
* Ease user experience
* Encourage the user to quickly set up the product
Some images are removed because they were obsolete or to improve
readability
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
### Describe Your Changes
apply proper syntax for code blocks
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
follow-up for
855dfb324d
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 85e2307e7c)
Legend settings have been added to **Graph Settings**.
#### **New Features:**
1. **Table View** – Toggle to display the legend in a table format.
2. **Hide Common Values** – Option to hide fields with identical values
across all series.
3. **Hide Min/Medium/Max** – Ability to hide min, median, and max values
from the legend.
4. **Custom Label Format** – Set a custom format for series labels
(applies only to the legend).
5. **Group by Label** – Group legend entries based on a selected label.
Related Issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8031
Currently, vmalert uses a fixed 10-second client timeout for notifiers,
which can prevent large sets of alerts from being sent successfully.
This introduces `-notifier.sendTimeout` flag to vmalert to control the
client timeout duration for the notifiers.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8287
Inside PARAM-VALUE, the characters '"' (ABNF %d34), '\' (ABNF %d92),
and ']' (ABNF %d93) MUST be escaped. This is necessary to avoid
parsing errors. Escaping ']' would not strictly be necessary but is
REQUIRED by this specification to avoid syslog application
implementation errors. Each of these three characters MUST be
escaped as '\"', '\\', and '\]' respectively. The backslash is used
for control character escaping for consistency with its use for
escaping in other parts of the syslog message as well as in traditional syslog.
Related RFC:
https://datatracker.ietf.org/doc/html/rfc5424#section-6.3.3
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8282
This commit properly validate retention flag value input for the enterprise debug UI.
API properly checks:
- presence of global retention configuration
- validate that retention is at least 1 day, same as vmstorage enforcement
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8343
---------
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Use `rate_sum` instead of `total` output for the following reasons:
* `rate_sum` is far less sensitive for data delays than `total`, since
it represents the speed of change instead of absolute values.
* `rate_sum` remove need in using `rate` function for calculating final
results on query time.
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
Signed-off-by: hagen1778 <roman@victoriametrics.com>
* make QuickStart first, similarly to vlogs docs
* push release guide to bottom, as it is not for users
Signed-off-by: hagen1778 <roman@victoriametrics.com>
### Describe Your Changes
By default, stream aggregation and deduplication stores a single state
per each aggregation output result.
The data for each aggregator is flushed independently once per
aggregation interval. But there's no guarantee that
incoming samples with timestamps close to the aggregation interval's end
will get into it. For example, when aggregating
with `interval: 1m` a data sample with timestamp 1739473078 (18:57:59)
can fall into aggregation round `18:58:00` or `18:59:00`.
It depends on network lag, load, clock synchronization, etc. In most
scenarios it doesn't impact aggregation or
deduplication results, which are consistent within margin of error. But
for metrics represented as a collection of series,
like
[histograms](https://docs.victoriametrics.com/keyconcepts/#histogram),
such inaccuracy leads to invalid aggregation results.
For this case, streaming aggregation and deduplication support mode with
aggregation windows for current and previous state. With this mode,
flush doesn't happen immediately but is shifted by a calculated samples
lag that improves correctness for delayed data.
Enabling of this mode has increased resource usage: memory usage is
expected to double as aggregation will store two states
instead of one. However, this significantly improves accuracy of
calculations. Aggregation windows can be enabled via
the following settings:
- `-streamAggr.enableWindows` at [single-node
VictoriaMetrics](https://docs.victoriametrics.com/single-server-victoriametrics/)
and [vmagent](https://docs.victoriametrics.com/vmagent/). At
[vmagent](https://docs.victoriametrics.com/vmagent/)
`-remoteWrite.streamAggr.enableWindows` flag can be specified
individually per each `-remoteWrite.url`.
If one of these flags is set, then all aggregators will be using fixed
windows. In conjunction with `-remoteWrite.streamAggr.dedupInterval` or
`-streamAggr.dedupInterval` fixed aggregation windows are enabled on
deduplicator as well.
- `enable_windows` option in [aggregation
config](https://docs.victoriametrics.com/stream-aggregation/#stream-aggregation-config).
It allows enabling aggregation windows for a specific aggregator.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit c8fc903669)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
A follow-up after 9bb5ba5d2f
that impacted compression ratio for data compressed with native GO zstd lib (`make test-pure`).
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 38bded4e58)
### Describe Your Changes
This PR introduces several enhancements and optimizations for the Group
view:
1. **Disable hover effect:**
Add the option to disable the hover effect. This can help reduce CPU
load when viewing a large number of logs.
2. **Limit entries per Group:**
Add the ability to limit the number of records displayed per group. When
a limit is set, groups are rendered sequentially – the next group starts
only after the current group has finished. By default, there is no
limit.
3. **Display group info:**
Include the group number in the title along with the total count of
groups to improve clarity and navigation.
4. **Performance improvement:**
In addition to the features above, separate optimizations reduce CPU
load during hover interactions by approximately 5-10%.
Related issue: #8135
<details>
<summary>Demo UI</summary>
<img
src="https://github.com/user-attachments/assets/9c89066e-28af-4df2-b368-2380412b3c3f"/>
<img
src="https://github.com/user-attachments/assets/a2338c8d-558c-437c-969e-f825043eb35b"/>
</details>
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
(cherry picked from commit b2a99d7c6f)
It has been appeared these optimizatios do not give measurable performance improvements,
while they complicate the code too much and may result in slowdown when the ingested logs have
different sets of fields.
This is a follow-up for 630601488e
(cherry picked from commit dce5eb88d3)
It has been appeared that 256 files increase RAM usage too much comparing to 128 files
when ingesting logs with hundreds of fields (aka wide events). So let's return back 128 files
limit for now.
This is a follow-up for 9bb5ba5d2f
(cherry picked from commit a50ab10998)
There is little sense in keeping too big buffers - they just waste RAM and do not reduce
the load on GC too much. So it is better dropping such buffers at Reset instead of keeping them around.
(cherry picked from commit b58e2ab214)
The number of filestream readers is proportional to the number of parts to be merged,
while the number of filestream writers is proportional to the number of concurrent merges.
Usually around 4-16 parts are merged at once, so the number of active filestream readers is ~8x
bigger than the number of active filestream writers.
So it is a good idea to use smaller size of read buffers comparing to the size of write buffers.
Limit read buffer size by 64Kb, while write buffer size is limited by 128Kb.
This should reduce the overall memory usage when merging parts with big number of files.
This is the case for VictoriaLogs, which works with logs containing hundreds of fields (aka wide events).
(cherry picked from commit 659251beaa)
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
(cherry picked from commit 616fed47a0)
This should improve query performance for logs with hundreds of fields (aka wide events).
Previously there was a high chance that the data for multiple log fields is stored in the same file.
This could result in query performance slowdown and/or increased disk read IO,
since the operating system could read unnecessary data for the fields, which aren't used in the query.
Now log fields are guaranteed to be stored in separate files until the number of fields exceeds 256.
After that multiple log fields start sharing files.
(cherry picked from commit 9bb5ba5d2f)
This reduces memory usage when many filestreams are processed simultaneously.
This is the case for VictoriaLogs when it processes logs with hundreds of fields.
(cherry picked from commit 2a681f2e8d)
- Re-use column names and values from the previously added rows if possible.
This increases locality of reference for field names and values, while improving
access speed for the field names and values.
- Postpone sorting fields in the added rows until creating inmemory part from them.
This allows optimizing the sorting for log fields with the same set of fields.
This is usually the case for logs, which belong to the same logs stream.
(cherry picked from commit 630601488e)
In this first step, content is moved from cloud docs root to a new
folder: get started. This one will allocate: overview, key features and
benefits, quickstart and guides and best practices.
### Changes Description
- In order to reuse content, the overview and README sections are
merged.
- A dummy Get started section is created, but needs further work.
- Next steps will also include reworking the quick start section and
creating the 2 new docs
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
(cherry picked from commit 95a42717b6)
Add missing sudo's to binary install snippets
### Describe Your Changes
Some of the commands the need to be done as root are missing invocation
via sudo. This commit adds missing sudo invocations.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
(cherry picked from commit 13fa70a03a)
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
(cherry picked from commit 081292b897)
### Describe Your Changes
trigger docs pipeline on docs workflow change
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
(cherry picked from commit 10e0c03fef)
### Describe Your Changes
exclude operator and helm charts docs sync as these repos will be synced
with vmdocs directly
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
(cherry picked from commit 5bdbafb9b6)
Commit 68791f9ccc21548d27f1cf04d0b3270be4146b82 introduced regression.
It performed basicAuth check before built-in routes. It made impossible
to bypass basic authorization with `authKey` param.
This commit fixeds that issue and removes unneeded check. It also adds
integration tests for this case.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7345
---------
Signed-off-by: f41gh7 <nik@victoriametrics.com>
This commit changes journald ingestion validation regex:
from `^[A-Z_][A-Z0-9_]+` to `^[A-Z_][A-Z0-9_]*`.
It's needed to properly support entities with single-character
names.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8314
Some of the test cases were failing because of the changes in data sharding
caused by changes in vmstorage ports caused by cluster restarts required by the
test.
This commit configures the cluster under test to use static ports for vmstorage
replicas. This required to make vmcluster type to become visible to tests.
We normally prefer ports assigned dynamically to avoid any conflicts with other
apps running locally. Should the conflict happen, please re-run the tests.
In CI/CD environment, no conflicts should happen because the tests are run
within a container and the components of the vmcluster will be the only servers
listening for incoming connections.
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
### Describe Your Changes
Just an extremely nit-picky grammatical fix.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
This eliminates a class of potential bugs with incorrect stats calculations when an additional filter
is applied to the blockResult before passing it to the stats function, and this filter removes
all the rows from blockResult.
Allow disabling the per-day index using the `-disablePerDayIndex` flag.
This should significantly improve the ingestion rate and decrease the
disk space usage for the use cases that assume small or no churn rate.
See the docs added to `docs/README.md` for details.
Both improvements are due to no data written to the per-day index.
Benchmark results:
```shell
rm -Rf ./lib/storage/Benchmark*; go test ./lib/storage -run=NONE -bench=BenchmarkStorageInsertWithAndWithoutPerDayIndex --loggerLevel=ERROR
goos: linux
goarch: amd64
pkg: github.com/VictoriaMetrics/VictoriaMetrics/lib/storage
cpu: 13th Gen Intel(R) Core(TM) i7-1355U
BenchmarkStorageInsertWithAndWithoutPerDayIndex/HighChurnRate/perDayIndexes-12 1 3850268120 ns/op 39.56 data-MiB 28.20 indexdb-MiB 259722 rows/s
BenchmarkStorageInsertWithAndWithoutPerDayIndex/HighChurnRate/noPerDayIndexes-12 1 2916865725 ns/op 39.57 data-MiB 25.73 indexdb-MiB 342834 rows/s
BenchmarkStorageInsertWithAndWithoutPerDayIndex/NoChurnRate/perDayIndexes-12 1 2218073474 ns/op 9.772 data-MiB 13.73 indexdb-MiB 450842 rows/s
BenchmarkStorageInsertWithAndWithoutPerDayIndex/NoChurnRate/noPerDayIndexes-12 1 1295140898 ns/op 9.771 data-MiB 0.3566 indexdb-MiB 772119 rows/s
PASS
ok github.com/VictoriaMetrics/VictoriaMetrics/lib/storage 11.421s
```
Signed-off-by: Artem Fetishev <wwctrsrx@gmail.com>
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
There are many subtle issues while ingesting logs from popular log shippers into VictoriaLogs via Loki JSON protocol.
For example, the common issue is that structured logs are ingested as a JSON string at _msg field.
This is not what most users expect - they expect that fields in structured logs are ingested as separate log fields.
It is better removing examples with configs for Loki JSON protocol, since other supported protocols work much better
without any issues. This should reduce the confusion level for new users, who try Loki protocol and hit its issues.
- Elasticsearch ( https://docs.victoriametrics.com/victorialogs/data-ingestion/#elasticsearch-bulk-api )
- JSON stream ( https://docs.victoriametrics.com/victorialogs/data-ingestion/#json-stream-api )
MustOpenStorage function may accept variable number of optional
arguments. This commit combines optional arguments into dedicated OpenOptions
struct. It reduces complexity of adding new optional arguments.
Related PR:
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8118
1. Use distinct code paths for blockResult.getValues() and blockResult.getValuesBucketed().
This should simplify debugging and maintenance of the resulting code.
2. Do not load column values if all the values in the block fit the same bucket.
Use blockResultColumn.minValue and blockResultColumn.maxValue for determining whether
column values must be loaded via blockResultColumn.getValuesEncoded().
This signiciantly improves performance for big buckets, which cover all the column
values in a block.
3. Properly calculate buckets for negative values.
4. Properly adjust weekly buckets by Monday.
### Describe Your Changes
Fix autocomplete not passing `AccountID` and `ProjectID` headers when
fetching suggestions in VictoriaLogs UI.
Related: #8042
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 778bcea30d)
### Describe Your Changes
fix typo for description of flag -pprofAuthKey
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
(cherry picked from commit 5fad3c8492)
See https://github.com/codecov/codecov-action?tab=readme-ov-file#usage
```
- uses: codecov/codecov-action@v5
with:
files: ./coverage1.xml,./coverage2.xml # optional
```
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 91ce900588)
Adding Accept-Encoding response header to support content negotiation,
which was introduced in [this
PR](https://github.com/systemd/systemd/pull/34822) and enables
compression on journald
(cherry picked from commit 3c9f9f49b0)
Previously the `callbackErr` is silently ignored in clusternative parser, which is used at vminsert for parsing clusterNative requests and at vmstorage for parsing vminsert requests.
This commit fixes that by properly return callbackError after reading all block metrics. This aligns
with other parsers in `lib/protoparser`.
### Describe Your Changes
Updated default `step` calculation for `Table` and `JSON` views.
- When switching to the `Table` or `JSON` view, the default step is now
automatically set to `end - start`, where `end` and `start` correspond
to the selected time range.
- If the user has manually set a step value, it will remain unchanged
when switching views or changing the time range.
- The UI now explicitly indicates when the step is automatically
calculated.
<details> <summary>Demo</summary>
https://github.com/user-attachments/assets/2540de24-36ed-4764-a047-1c6b48a80ed4
</details>
**Note:** These views use the
[`/api/v1/query`](https://docs.victoriametrics.com/keyconcepts/#instant-query)
endpoint for instant queries.
Related issue: #8240
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 71160e64b1)
This improves performance for queries, which use `sort by (...) limit N` without mentioning _time field.
For example, the following query must work faster now
_time:1d | rm _time | sort by (request_duration desc) limit 10
(cherry picked from commit 422caf6bd7)
- Add a fast path for timestamps ending with 'Z'
- Use strings.LastIndexAny instead of strings.IndexAny for searching
for timezone offset at the end of the string. This works faster
for timestamps with sub-second precision.
(cherry picked from commit 335071cf3d)
The 2025 changelog was in the parent directory - a default page
that opens for /changelog.
But it seems like it was confusing for users, so add 2025 that mirrors
/changelog page.
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 5f9257a6d2)
If test is using binaries from `master` branch, then test name should be prefixed
with `TestVmsingle` word.
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit c8e3737a93)
Update the pipe state only once per each series of repeated strings, uint8 values and tuples.
This improves performance a bit for the following `top` pipes:
- top (string_field)
- top (uint8_field)
- top (field1, ..., fieldN)
Do not apply the optimization for uint16, uint32, uint64 and int64 fields, since they
usually contain big number of unique values, which do not repeat most of the time.
### Describe Your Changes
There is an issue described in #8040 this should fix it
- The alerts slice is shared across multiple goroutines (since send() is
called concurrently).
- `alerts[:0]` creates a new slice header, but it still references the
same underlying array.
- Appending (append(alertsToSend, a)) modifies the underlying array,
which may also be used by another goroutine.
Solution: Use a separate slice copy for each goroutine.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Co-authored-by: Evgeny Kuzin <evgeny@hudson-trading.com>
Co-authored-by: Hui Wang <haley@victoriametrics.com>
(cherry picked from commit 4e9fb93acc)
### Describe Your Changes
- Update references to the latest release - v1.110.1
- Update links to the latest LTS releases
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
(cherry picked from commit 04943907e3)
Previously RFC3339 timestamps with sub-second precision could be incorrectly compared by lessString().
For example, 2025-01-20T10:20:30.1Z was incorrectly treated as smaller than 2025-01-20T10:20:30.09Z,
because the first timestamp has smaller decimal number after the last dot than the second timestamp.
(cherry picked from commit 81d359507d)
Previosly the parsing of the input stream was stopped after the first parse error.
This isn't what most users expect when ingesting JSON lines in a stream where some JSON lines may be invalid.
(cherry picked from commit ebac07bcf6)
### Describe Your Changes
- Migrated build process from Webpack (CRA) to Vite
Reason for migration: `create-react-app` has been
[deprecated](b532a58792)
and contains outdated dependencies that haven’t been updated for over
two years, leading to security vulnerabilities. Additionally, build
speed improved by more than 2x.
- Updated dependencies and fixed TypeScript typings in accordance with
the updates
b532a58792
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
(cherry picked from commit 6a8fa799c6)
* add troubleshooting link
* add panels for CPU and mem usage
* rm unnecessary links from panels
* update to grafana v11
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 5dbca072bb)
Split unique values (groups) into shards according to the configured concurrency
during processing of the matching rows if the number of unique values exceeds the hardcoded threshold.
Previously this splitting was performed unconditionally at the merge stage when merging independently
calculated per-CPU states into a single state. It is faster to perform the split during rows processing
if the number of unique values is big.
This gives up to 30% perfromance improvements when these pipes are applied to big number of unique values (groups).
(cherry picked from commit 48602a1ae8)
The number of worker shards per each pipe processor is created during query initialization.
This number equals to the `options(concurrency=N)` if this option is set or to the number of available CPU cores.
This means that all the pipes must adhere the given concurrency when passing data blocks
to the next pipe.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8201
The bug has been introduced in 0214aa328e
vmauth uses 'lib/httpserver' for serving HTTP requests. This server
unconditionally defines built-in routes (such as '/metrics',
'/health', etc). It makes impossible to proxy `HTTP` requests to backends with the same routes.
Since vmauth's httpserver matches built-in route and return local
response.
This commit adds new flag `httpInternalListenAddr` with
default empty value. Which removes internal API routes from public
router and exposes it at separate http server.
For example given configuration disables private routes at `0.0.0.0:8427` address and serves it at `0.0.0.0:8426`:
`./bin/vmauth --auth.config=config.yaml --httpListenAddr=:8427 --httpInternalListenAddr=127.0.0.1:8426`
Related issues:
- https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6468
- https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7345
This commit improves integration with third-party solutions who rely on non-root
endpoints (i.e. MinIO) when the vmselect path has been specified in the
configured Prometheus URL like:
`http://vmselect.:8481/select/0/prometheus`
Comparable change has been done before
(b885a3b6e9), however only takes care of
the root path. This means endpoints `-/healthy` and `-/ready` are still
not available on full vmselect Prometheus paths, resulting in
unsupported path requests.
This change makes these endpoints available on the full paths like:
`/select/0/prometheus/-/healthy` and `/select/0/prometheus/-/ready`,
thus achieving full Prometheus compatibility for external dependencies.
Related issues:
- https://github.com/minio/console/issues/2829
- https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1833
---
Signed-off-by: Joost Buskermolen <j.buskermolen@cloudmeesters.com>
### Describe Your Changes
The cardinality limiter in this case does not receive the actual
metricID but some other value found in r.TSID.MetricID and is not
initialized. Depending on the system and/or go runtime implementation,
this value can be 0 or some garbage value (which shouldn't have too wide
a range). Thus, there basically no limit for inserted metricIDs.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 631b736bc2)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
### Describe Your Changes
- fixed recently found 404 under `/anomaly-detection/` docs section
- get rid of remaining .html / .index.html endings in links under
`/anomaly-detection/` docs section
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
(cherry picked from commit ceefa406cc)
### Describe Your Changes
Explicitly mention that `--vm-rate-limit` is used for each individual
worker defined by `--vm-concurrency`.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
(cherry picked from commit 2748681f40)
### Describe Your Changes
Fix metric name labels set not being "closed" by `}`.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
(cherry picked from commit e2cfd351c0)
At v1.107.0 release with commit 21a7f06fc4beeb6ad32b9f7fd88704ed33674905 was introduced regression.
It prevents kafka producer client from graceful shutdown. This change also removed kafka message headers.
This commit properly closes client.
It also addresses the following issues:
* properly add headers to kafka message
* add url_id to the exported metrics, which prevents possible metrics clash if multiple remote write targets have
the same url and topic name.
Signed-off-by: f41gh7 <nik@victoriametrics.com>
### Describe Your Changes
Currently the alert descrption considers only one end of the connection
(vmagent). While saturation can also be caused by slowness of the
receiving components (vminsert, vmstorage). Update the alert description
with a brief suggestion to also check the dashboards of these
components.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 30af662d84)
metricsql: bump to v0.83.0
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7703
The update also returns an error if metric name is specified twice in
metrics selector.
For example, `foo{__name__="bar"}` is not allowed anymore. It would
successfully parse before
this change, but it won't satisfy the search filter any way. So it had
no sense in supporting this. This is why some test cases were removed.
Signed-off-by: hagen1778 <roman@victoriametrics.com>
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit fa2107bbec)
### Describe Your Changes
Fixed misspelling of the word "components" from "componetns"
### Checklist
The following checks are **mandatory**:
- [X] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
(cherry picked from commit 1427ea3ca9)
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
(cherry picked from commit 6166149474)
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
(cherry picked from commit 071db473a7)
### Describe Your Changes
fixed some cloud docs links
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
(cherry picked from commit 94ac051647)
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
(cherry picked from commit 3419c56eef)
* use signed datasource
* bump Grafana because on v10 pre-installing+provisioning didn't work
* consistently rename provisioning folder
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 63a1baf10f)
### Describe Your Changes
Fixed an issue where dropdown menus were not visible in the Group View
settings.
Ref issue: #8153
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
At this time `bufferedwriter` [silently ignores connection close
errors](78eaa056c0/lib/bufferedwriter/bufferedwriter.go (L67)).
It may be very convenient in some situations (to not log such
unimportant errors), but it's too implicit and unsafe for the others.
For example, if you close [export
API](https://docs.victoriametrics.com/#how-to-export-time-series) client
connection in the middle of communication, VictoriaMetrics won't notice
it and will start to hog CPU by exporting all the data into nowhere
until it process all of them. If you'll make a few retries, it will be
effectively a DoS on the server.
This commit replaces this implicit error suppressing with explicit error
handling which fixes the issue with export API.
Issue was introduced at e78f3ac8ac
It is better to use the AppendFieldsToJSON function directly
instead of hiding it under RowsFormatter abstraction.
(cherry picked from commit 95f182053b)
### Describe Your Changes
docs/vmanomaly: release v1.19.2 (patch that addresses some of the bugs
found in 1.19.0)
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Co-authored-by: Mathias Palmersheim <mathias@victoriametrics.com>
- Deployment types: [Single-node version](https://docs.victoriametrics.com/), [Cluster version](https://docs.victoriametrics.com/cluster-victoriametrics/), and [Enterprise version](https://docs.victoriametrics.com/enterprise/)
- Changelog: [CHANGELOG](https://docs.victoriametrics.com/changelog/), and [How to upgrade](https://docs.victoriametrics.com/#how-to-upgrade-victoriametrics)
datadogStreamFields=flagutil.NewArrayString("datadog.streamFields","Datadog tags to be used as stream fields.")
datadogIgnoreFields=flagutil.NewArrayString("datadog.ignoreFields","Datadog tags to ignore.")
datadogStreamFields=flagutil.NewArrayString("datadog.streamFields","Comma-separated list of fields to use as log stream fields for logs ingested via DataDog protocol. "+
"See https://docs.victoriametrics.com/victorialogs/data-ingestion/datadog-agent/#stream-fields")
datadogIgnoreFields=flagutil.NewArrayString("datadog.ignoreFields","Comma-separated list of fields to ignore for logs ingested via DataDog protocol. "+
"See https://docs.victoriametrics.com/victorialogs/data-ingestion/datadog-agent/#dropping-fields")
maxRequestSize=flagutil.NewBytes("datadog.maxRequestSize",64*1024*1024,"The maximum size in bytes of a single DataDog request")
// ParseUnixTimestamp parses s as unix timestamp in seconds, milliseconds, microseconds or nanoseconds and returns the parsed timestamp in nanoseconds.
funcParseUnixTimestamp(sstring)(int64,error){
ifstrings.IndexByte(s,'.')>=0{
// Parse timestamp as floating-point value
f,err:=strconv.ParseFloat(s,64)
iferr!=nil{
return0,fmt.Errorf("cannot parse unix timestamp from %q: %w",s,err)
}
iff<(1<<31)&&f>=(-1<<31){
// The timestamp is in seconds.
returnint64(f*1e9),nil
}
iff<1e3*(1<<31)&&f>=1e3*(-1<<31){
// The timestamp is in milliseconds.
returnint64(f*1e6),nil
}
iff<1e6*(1<<31)&&f>=1e6*(-1<<31){
// The timestamp is in microseconds.
returnint64(f*1e3),nil
}
// The timestamp is in nanoseconds
iff>math.MaxInt64{
return0,fmt.Errorf("too big timestamp in nanoseconds: %v; mustn't exceed %v",f,int64(math.MaxInt64))
}
iff<math.MinInt64{
return0,fmt.Errorf("too small timestamp in nanoseconds: %v; must be bigger or equal to %v",f,int64(math.MinInt64))
}
returnint64(f),nil
}
// Parse timestamp as integer
n,err:=strconv.ParseInt(s,10,64)
iferr!=nil{
return0,fmt.Errorf("cannot parse unix timestamp from %q: %w",s,err)
disableInsert=flag.Bool("internalinsert.disable",false,"Whether to disable /internal/insert HTTP endpoint")
maxRequestSize=flagutil.NewBytes("internalinsert.maxRequestSize",64*1024*1024,"The maximum size in bytes of a single request, which can be accepted at /internal/insert HTTP endpoint")
journaldStreamFields=flagutil.NewArrayString("journald.streamFields","Journal fields to be used as stream fields. "+
"See the list of allowed fields at https://www.freedesktop.org/software/systemd/man/latest/systemd.journal-fields.html.")
journaldIgnoreFields=flagutil.NewArrayString("journald.ignoreFields","Journal fields to ignore. "+
"See the list of allowed fields at https://www.freedesktop.org/software/systemd/man/latest/systemd.journal-fields.html.")
journaldTimeField=flag.String("journald.timeField","__REALTIME_TIMESTAMP","Journal field to be used as time field. "+
"See the list of allowed fields at https://www.freedesktop.org/software/systemd/man/latest/systemd.journal-fields.html.")
journaldTenantID=flag.String("journald.tenantID","0:0","TenantID for logs ingested via the Journald endpoint.")
journaldStreamFields=flagutil.NewArrayString("journald.streamFields","Comma-separated list of fields to use as log stream fields for logs ingested over journald protocol. "+
"See https://docs.victoriametrics.com/victorialogs/data-ingestion/journald/#stream-fields")
journaldIgnoreFields=flagutil.NewArrayString("journald.ignoreFields","Comma-separated list of fields to ignore for logs ingested over journald protocol. "+
"See https://docs.victoriametrics.com/victorialogs/data-ingestion/journald/#dropping-fields")
journaldTimeField=flag.String("journald.timeField","__REALTIME_TIMESTAMP","Field to use as a log timestamp for logs ingested via journald protocol. "+
"See https://docs.victoriametrics.com/victorialogs/data-ingestion/journald/#time-field")
journaldTenantID=flag.String("journald.tenantID","0:0","TenantID for logs ingested via the Journald endpoint. "+
"See https://docs.victoriametrics.com/victorialogs/data-ingestion/journald/#multitenancy")
journaldIncludeEntryMetadata=flag.Bool("journald.includeEntryMetadata",false,"Include journal entry fields, which with double underscores.")
maxRequestSize=flagutil.NewBytes("journald.maxRequestSize",64*1024*1024,"The maximum size in bytes of a single journald request")
totalStreams=flag.Int("totalStreams",0,"The number of total log streams; if -totalStreams > -activeStreams, then some active streams are substituted with new streams "+
"during data generation")
logsPerStream=flag.Int64("logsPerStream",1_000,"The number of log entries to generate per each log stream. Log entries are evenly distributed between -start and -end")
constFieldsPerLog=flag.Int("constFieldsPerLog",3,"The number of fields with constaint values to generate per each log entry; "+
constFieldsPerLog=flag.Int("constFieldsPerLog",3,"The number of fields with constant values to generate per each log entry; "+
"see https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model")
varFieldsPerLog=flag.Int("varFieldsPerLog",1,"The number of fields with variable values to generate per each log entry; "+
"see https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model")
.uplot,.uplot*,.uplot*:before,.uplot*:after{box-sizing:border-box}.uplot{font-family:system-ui,-apple-system,SegoeUI,Roboto,HelveticaNeue,Arial,NotoSans,sans-serif,"Apple Color Emoji","Segoe UI Emoji",SegoeUISymbol,"Noto Color Emoji";line-height:1.5;width:min-content}.u-title{text-align:center;font-size:18px;font-weight:700}.u-wrap{position:relative;-webkit-user-select:none;user-select:none}.u-over,.u-under{position:absolute}.u-under{overflow:hidden}.uplotcanvas{display:block;position:relative;width:100%;height:100%}.u-axis{position:absolute}.u-legend{font-size:14px;margin:auto;text-align:center}.u-inline{display:block}.u-inline*{display:inline-block}.u-inlinetr{margin-right:16px}.u-legendth{font-weight:600}.u-legendth>*{vertical-align:middle;display:inline-block}.u-legend.u-marker{width:1em;height:1em;margin-right:4px;background-clip:padding-box!important}.u-inline.u-liveth:after{content:":";vertical-align:middle}.u-inline:not(.u-live).u-value{display:none}.u-series>*{padding:4px}.u-seriesth{cursor:pointer}.u-legend.u-off>*{opacity:.3}.u-select{background:#00000012;position:absolute;pointer-events:none}.u-cursor-x,.u-cursor-y{position:absolute;left:0;top:0;pointer-events:none;will-change:transform}.u-hz.u-cursor-x,.u-vt.u-cursor-y{height:100%;border-right:1pxdashed#607D8B}.u-hz.u-cursor-y,.u-vt.u-cursor-x{width:100%;border-bottom:1pxdashed#607D8B}.u-cursor-pt{position:absolute;top:0;left:0;border-radius:50%;border:0solid;pointer-events:none;will-change:transform;background-clip:padding-box!important}.u-axis.u-off,.u-select.u-off,.u-cursor-x.u-off,.u-cursor-y.u-off,.u-cursor-pt.u-off{display:none}
<!doctype html><htmllang="en"><head><metacharset="utf-8"/><linkrel="icon"href="./favicon.svg"/><linkrel="apple-touch-icon"href="./favicon.svg"/><linkrel="mask-icon"href="./favicon.svg"color="#000000"><metaname="viewport"content="width=device-width,initial-scale=1,maximum-scale=5"/><metaname="theme-color"content="#000000"/><metaname="description"content="Explore your log data with VictoriaLogs UI"/><linkrel="manifest"href="./manifest.json"/><title>UI for VictoriaLogs</title><metaname="twitter:card"content="summary"><metaname="twitter:title"content="UI for VictoriaLogs"><metaname="twitter:site"content="@https://victoriametrics.com/products/victorialogs/"><metaname="twitter:description"content="Explore your log data with VictoriaLogs UI"><metaname="twitter:image"content="./preview.jpg"><metaproperty="og:type"content="website"><metaproperty="og:title"content="UI for VictoriaLogs"><metaproperty="og:url"content="https://victoriametrics.com/products/victorialogs/"><metaproperty="og:description"content="Explore your log data with VictoriaLogs UI"><scriptdefer="defer"src="./static/js/main.55c8060b.js"></script><linkhref="./static/css/main.02a1c6cb.css"rel="stylesheet"></head><body><noscript>You need to enable JavaScript to run this app.</noscript><divid="root"></div></body></html>
forceMergeAuthKey=flagutil.NewPassword("forceMergeAuthKey","authKey, which must be passed in query string to /internal/force_merge pages. It overrides -httpAuth.*")
storageNodeAddrs=flagutil.NewArrayString("storageNode","Comma-separated list of TCP addresses for storage nodes to route the ingested logs to and to send select queries to. "+
"If the list is empty, then the ingested logs are stored and queried locally from -storageDataPath")
insertConcurrency=flag.Int("insert.concurrency",2,"The average number of concurrent data ingestion requests, which can be sent to every -storageNode")
insertDisableCompression=flag.Bool("insert.disableCompression",false,"Whether to disable compression when sending the ingested data to -storageNode nodes. "+
"Disabled compression reduces CPU usage at the cost of higher network usage")
selectDisableCompression=flag.Bool("select.disableCompression",false,"Whether to disable compression for select query responses received from -storageNode nodes. "+
"Disabled compression reduces CPU usage at the cost of higher network usage")
storageNodeUsername=flagutil.NewArrayString("storageNode.username","Optional basic auth username to use for the corresponding -storageNode")
storageNodePassword=flagutil.NewArrayString("storageNode.password","Optional basic auth password to use for the corresponding -storageNode")
storageNodePasswordFile=flagutil.NewArrayString("storageNode.passwordFile","Optional path to basic auth password to use for the corresponding -storageNode. "+
"The file is re-read every second")
storageNodeBearerToken=flagutil.NewArrayString("storageNode.bearerToken","Optional bearer auth token to use for the corresponding -storageNode")
storageNodeBearerTokenFile=flagutil.NewArrayString("storageNode.bearerTokenFile","Optional path to bearer token file to use for the corresponding -storageNode. "+
"The token is re-read from the file every second")
storageNodeTLS=flagutil.NewArrayBool("storageNode.tls","Whether to use TLS (HTTPS) protocol for communicating with the corresponding -storageNode. "+
"By default communication is performed via HTTP")
storageNodeTLSCAFile=flagutil.NewArrayString("storageNode.tlsCAFile","Optional path to TLS CA file to use for verifying connections to the corresponding -storageNode. "+
"By default, system CA is used")
storageNodeTLSCertFile=flagutil.NewArrayString("storageNode.tlsCertFile","Optional path to client-side TLS certificate file to use when connecting "+
"to the corresponding -storageNode")
storageNodeTLSKeyFile=flagutil.NewArrayString("storageNode.tlsKeyFile","Optional path to client-side TLS certificate key to use when connecting to the corresponding -storageNode")
storageNodeTLSServerName=flagutil.NewArrayString("storageNode.tlsServerName","Optional TLS server name to use for connections to the corresponding -storageNode. "+
"By default, the server name from -storageNode is used")
storageNodeTLSInsecureSkipVerify=flagutil.NewArrayBool("storageNode.tlsInsecureSkipVerify","Whether to skip tls verification when connecting to the corresponding -storageNode")
)
varlocalStorage*logstorage.Storage
varlocalStorageMetrics*metrics.Set
varnetstorageInsert*netinsert.Storage
varnetstorageSelect*netselect.Storage
// Init initializes vlstorage.
//
// Stop must be called when vlstorage is no longer needed
funcInit(){
ifstrg!=nil{
logger.Panicf("BUG: Init() has been already called")
iflen(*storageNodeAddrs)==0{
initLocalStorage()
}else{
initNetworkStorage()
}
}
funcinitLocalStorage(){
iflocalStorage!=nil{
logger.Panicf("BUG: initLocalStorage() has been already called")
}
ifretentionPeriod.Duration()<24*time.Hour{
@@ -63,60 +107,139 @@ func Init() {
}
logger.Infof("opening storage at -storageDataPath=%s",*storageDataPath)
logger.Infof("forced merge for partition_prefix=%q has been started",partitionNamePrefix)
startTime:=time.Now()
strg.MustForceMerge(partitionNamePrefix)
logger.Infof("forced merge for partition_prefix=%q has been successfully finished in %.3f seconds",partitionNamePrefix,time.Since(startTime).Seconds())
logger.Infof("forced merge for partition_prefix=%q has been started",partitionNamePrefix)
startTime:=time.Now()
localStorage.MustForceMerge(partitionNamePrefix)
logger.Infof("forced merge for partition_prefix=%q has been successfully finished in %.3f seconds",partitionNamePrefix,time.Since(startTime).Seconds())
}()
returntrue
}
// CanWriteData returns non-nil error if it cannot write data to vlstorage
funcCanWriteData()error{
ifstrg.IsReadOnly(){
iflocalStorage==nil{
// The data can be always written in non-local mode.
returnnil
}
iflocalStorage.IsReadOnly(){
return&httpserver.ErrorWithStatusCode{
Err:fmt.Errorf("cannot add rows into storage in read-only mode; the storage can be in read-only mode "+
"because of lack of free disk space at -storageDataPath=%s",*storageDataPath),
@@ -130,50 +253,77 @@ func CanWriteData() error {
//
// It is advised to call CanWriteData() before calling MustAddRows()
funcMustAddRows(lr*logstorage.LogRows){
strg.MustAddRows(lr)
iflocalStorage!=nil{
// Store lr in the local storage.
localStorage.MustAddRows(lr)
}else{
// Store lr across the remote storage nodes.
lr.ForEachRow(netstorageInsert.AddRow)
}
}
// RunQuery runs the given q and calls writeBlock for the returned data blocks
logger.Warnf("skipping too long log entry, since its length exceeds %d bytes; the actual log entry length is %d bytes; log entry contents: %s",maxInsertBlockSize,len(b),b)
// Just drop block on 409 and 400 status codes like Prometheus does.
ifstatusCode==409{
logBlockRejected(block,c.sanitizedURL,resp)
// Just drop block on 409 status code like Prometheus does.
// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/873
// and https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1149
_=resp.Body.Close()
c.packetsDropped.Inc()
returntrue
// - Remote Write v1 specification implicitly expects a `400 Bad Request` when the encoding is not supported.
// - Remote Write v2 specification explicitly specifies a `415 Unsupported Media Type` for unsupported encodings.
// - Real-world implementations of v1 use both 400 and 415 status codes.
// See more in research: https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8462#issuecomment-2786918054
}elseifstatusCode==415||statusCode==400{
ifc.canDowngradeVMProto.Swap(false){
logger.Infof("received unsupported media type or bad request from remote storage at %q. Downgrading protocol from VictoriaMetrics to Prometheus remote write for all future requests. "+
"See https://docs.victoriametrics.com/vmagent/#victoriametrics-remote-write-protocol",c.sanitizedURL)
c.useVMProto.Store(false)
}
ifencoding.IsZstd(block){
logger.Infof("received unsupported media type or bad request from remote storage at %q. Re-packing the block to Prometheus remote write and retrying."+
"See https://docs.victoriametrics.com/vmagent/#victoriametrics-remote-write-protocol",c.sanitizedURL)
block=mustRepackBlockFromZstdToSnappy(block)
c.retriesCount.Inc()
_=resp.Body.Close()
gotoagain
}
// Just drop snappy blocks on 400 or 415 status codes like Prometheus does.
// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/873
// and https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1149
"clients pushing data into the vmagent. See https://docs.victoriametrics.com/stream-aggregation/#ignore-aggregation-intervals-on-start")
streamAggrGlobalDropInputLabels=flagutil.NewArrayString("streamAggr.dropInputLabels","An optional list of labels to drop from samples for aggregator "+
"before stream de-duplication and aggregation . See https://docs.victoriametrics.com/stream-aggregation/#dropping-unneeded-labels")
streamAggrGlobalEnableWindows=flag.Bool("streamAggr.enableWindows",false,"Enables aggregation within fixed windows for all global aggregators. "+
"This allows to get more precise results, but impacts resource usage as it requires twice more memory to store two states. "+
"See https://docs.victoriametrics.com/stream-aggregation/#aggregation-windows.")
// Per URL config
streamAggrConfig=flagutil.NewArrayString("remoteWrite.streamAggr.config","Optional path to file with stream aggregation config for the corresponding -remoteWrite.url. "+
@@ -53,12 +56,15 @@ var (
"See https://docs.victoriametrics.com/stream-aggregation/#ignoring-old-samples")
streamAggrIgnoreFirstIntervals=flagutil.NewArrayInt("remoteWrite.streamAggr.ignoreFirstIntervals",0,"Number of aggregation intervals to skip after the start "+
"for the corresponding -remoteWrite.streamAggr.config at the corresponding -remoteWrite.url. Increase this value if "+
"you observe incorrect aggregation results after vmagent restarts. It could be caused by receiving bufferred delayed data from clients pushing data into the vmagent. "+
"you observe incorrect aggregation results after vmagent restarts. It could be caused by receiving buffered delayed data from clients pushing data into the vmagent. "+
"See https://docs.victoriametrics.com/stream-aggregation/#ignore-aggregation-intervals-on-start")
streamAggrDropInputLabels=flagutil.NewArrayString("remoteWrite.streamAggr.dropInputLabels","An optional list of labels to drop from samples "+
"before stream de-duplication and aggregation with -remoteWrite.streamAggr.config and -remoteWrite.streamAggr.dedupInterval at the corresponding -remoteWrite.url. "+
"Multiple labels per remoteWrite.url must be delimited by '^^': -remoteWrite.streamAggr.dropInputLabels='replica^^az,replica'. "+
"See https://docs.victoriametrics.com/stream-aggregation/#dropping-unneeded-labels")
streamAggrEnableWindows=flagutil.NewArrayBool("remoteWrite.streamAggr.enableWindows","Enables aggregation within fixed windows for all remote write's aggregators. "+
"This allows to get more precise results, but impacts resource usage as it requires twice more memory to store two states. "+
"See https://docs.victoriametrics.com/stream-aggregation/#aggregation-windows.")
)
// CheckStreamAggrConfigs checks -remoteWrite.streamAggr.config and -streamAggr.config.
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
