The files with synctest usage are built only if the goexperiment.synctest build tag is set.
This allows running `go test ./lib/...` and `go vet ./lib/...` without the need to pass GOEXPERIMENT=synctest to them.
This is a follow-up for d33d7e20be
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8848
New variable should help to change docker image tag name during release prepare.
Instead of publishing an exact version, like v1.1.0, it should use suffixed version v1.1.0-rc1.
Which should be re-tagged later, when release will be promoted to stable.
Related to https://github.com/VictoriaMetrics/VictoriaMetrics/issues/9136
Signed-off-by: f41gh7 <nik@victoriametrics.com>
### Describe Your Changes
Revival and modification of original PR
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8762 after
discussion on
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7387.
`group.concurrency` is now respected if and only if
`-replay.rulesDelay=0` rather than always. This allows rules to be run
concurrently without ambiguity about rule chaining. If
`-replay.rulesDelay` is set greater than zero concurrency is still
ignored. This will be the default behavior since it defaults to 1s.
Implementation considerations:
I chose to add split some simple logic into a helper function in
preparation for adding `replay.singleRuleEvaluationConcurrency` in a
follow up PR as thats we're we can share that logic.
cc @Haleygo
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/#pull-request-checklist).
---------
Co-authored-by: Hui Wang <haley@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
During the data retrieval, VictoriaMetrics switches to global index
search if the time range is > 40 days. Prior
ba0d7dc2fc, this switch was handled in
indexDB code. That commit moved that logic to the storage level. As the
result, indexDB will search per-day index for whatever time range that
is passed to it.
This broke the BenchmarkHeadPostingForMatchers. It now shows very slow
indexDB performance compared to v1.111.0 (the last release before that
commit). This is because now indexDB tries to search 55 years of data by
spawning a separate goroutine for each day. Even though the data was
inserted only for the current day, running that many goroutines
significantly slows down the search.
The fix is to use global index time range explicitly.
Fixes#9203
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
Similar to other storage caches `storage/metricName` can be very
important to performance, however it is not tunable independently like
other caches.
In high cardinality setups where a large amount of that cardinality is
actively queried we can see a high `metricName` miss rate.
The only way to correct this is to increase available memory either by
provisioning more or by increasing `memory.allowedPercent`, which is
often expensive or undesirable for stability reasons.
Its possible to work around this by increasing `memory.allowedPercent`
and then adjusting `storage.cacheSizeIndexDBDataBlocks` and
`storage.cacheSizeStorageTSID` down as they are the largest caches, but
this is a less ideal solution than being able to directly control this
cache size.
Related to
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8843
### Describe Your Changes
The `DataBlock` contains structs with string fields, and while the
original comment mentioned not holding references to `br`, it wasn't
immediately clear that this also applies to fields like strings within
the data.
This change clarifies that the `writeBlockResultFunc` must not retain
references to any part of `br`, including its fields. This makes it
explicit that even seemingly safe types like strings must be copied if
needed.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/#pull-request-checklist).
The 'select' HTTP request duration metric is currently implemented as a
summary, while the 'insert' HTTP request duration uses a histogram.
All time series under the same metric name must use the same metric
type. Using the summary type for this metric is preferable, as it
significantly reduces the number of unique time series generated. While
summaries have the limitation of not supporting accurate aggregation
across multiple instances or paths, this trade-off is more manageable
than dealing with a high-cardinality explosion caused by histograms.
### Describe Your Changes
- removed unneeded loop for binary field value size extraction, since it
should not be delimited by multiple `\n` symbols
- changed loop condition
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/#pull-request-checklist).
### Describe Your Changes
Related issue: #8749
Added query to requests `/field_names` and `/field_values` to get more
precision results:
- If `filterName` is `_msg` or `_stream_id`, the query cannot be
generated specifically,
so a wildcard query (`"*"`) is returned.
- If `filterName` is `_stream`, the query is generated using regexp
(`{type=~"value.*"}`).
- If `filterName` is `_time`, a simplified query is created by trimming
the value up
to the first occurrence of a delimiter such as `-` or `:`.
- For all other values of `filterName`, a prefix query is returned using
the `query` value with a `*` appended (e.g., `"value*"`).
Related issue: #8806
Enhanced autocomplete with parsed field suggestions from unpack pipe.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Co-authored-by: Nikolay <nik@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
### Describe Your Changes
This PR adds a call to optimize pipes after the `limit` pipe has been
appended.
Related: #9200
While this approach is not ideal, since it forces us to re-optimize all
pipes, but it is simpler.
An alternative would be to reapply only the relevant optimizations
specifically for this case, something like:
```go
func (q *Query) AddPipeLimit(n uint64) {
if len(q.pipes) > 0 {
ps, ok := q.pipes[len(q.pipes)-1].(*pipeSort)
if ok {
if ps.limit == 0 || n < ps.limit {
ps.limit = n
}
return
}
pu, ok := q.pipes[len(q.pipes)-1].(*pipeUniq)
if ok {
if pu.limit == 0 || n < pu.limit {
pu.limit = n
}
return
}
}
q.pipes = append(q.pipes, &pipeLimit{
limit: n,
})
}
```
### Checklist
The following checks are **mandatory**:
- [X] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/#pull-request-checklist).
vlinsert package could be used as an imported dependency. Mostly it's
needed for vlagent, but it could be a case for other applications.
This commit introduces new interface for vlinsertutil package, which
represents actual storage for log rows ingestion.
It includes CanWriteData, which also could be used to introduce
back-pressure mechanism for vlinsert clients.
Related PR: https://github.com/VictoriaMetrics/VictoriaMetrics/pull/9034
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/#pull-request-checklist).
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
Previously, ForEachRow always reset last row fields after iteration.
It makes impossible concurrent iteration with forEachRow, since
ForEachRow performed hidden mutation of LogRows.
This commit resolves this issue by removal of fields reference.
Related to
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/9076
This allows parsing unlimited number of logs in a single HTTP request,
without the need to buffer the logs in memory.
This is needed for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/9070
Thanks to @AndrewChubatiuk for the initial pull request - https://github.com/VictoriaMetrics/VictoriaMetrics/pull/9153
This commit is based on the https://github.com/VictoriaMetrics/VictoriaMetrics/pull/9153 .
It contains the following changes comparing to the original pull request:
- Remove ugly function LineReader.NextLineWithLineFn(). Instead, uglify the Journald parser a bit
with hacky calls to LineReader.NextLine() in order to parse binary-encoded field values.
This should preserve the maintainability of the LineReader, which is shared among multiple protocol parsers,
under control, while keeping the complexity of Journald parsing inside the app/vlinsert/journald package.
- Fix a typo bug inside isNameValid() - `(r < '0' && r > '9')` must be written as `(r < '0' || r > '9')`.
Rewrite isNameValid() into easier to understand code and rename it to isValidJournaldFieldName() for better readability.
Add tests for this function.
- Remove mentioning of the -journald.maxRequestSize command-line flag from VictoriaLogs docs.
- Add the description of the fix to VictoriaLogs changelog.
- Properly increment errorsTotal metric on every journald parse error.
- Add missing protoparserutil.PutUncompressedReader(reader) call, so the reader could be re-used between client requests.
- Remove improperly working code, which tries continuing parsing the request stream after parse errors.
It is impossible to recover reliably from journald parse errors related to reading the data from the request stream,
since the journald protocol format is completely braindead. So it is better to immediately return the error
to the client instead of trying to recover. The only errors, which could be recovered, are related to invalid field names / values.
Such errors are logged with the WARN level and the corresponding fields are skipped.
- Fix incorrect storage of the re-used name and value strings into fb.fields. The contents of the name and value strings
must be copied per every loop, which reads these strings from the request stream. Otherwise the contents of the previously
added Name and Value fields into fb.fields will be overwritten on the next loop.
- Ensure that LineReader.Line is set to nil after LineReader.NextLine() returns false. This should prevent from subtle bugs
when the LineReader.Line is read after LineReader.NextLine() returns false.
### Describe Your Changes
> ⚠️ still draft, don't merge even if already approved
Docs update for vmanomaly v1.24.0 release with stateful service option
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/#pull-request-checklist).
Such filters can be optimized to `*`. This avoid executing other OR filters.
For example, `foo or * or bar` is optimized to `*`, while `foo` and `bar` filters aren't executed.
Such filters are frequently generated by Grafana, so this should improve query performance there.
Use getCompoundToken() instead of getCompoundFuncArg() for obtaining regexp filter value,
since getCompoundFuncArg() skips trailing '*' chars.
This allows detecting invalid queries in the https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8582 .
`Samples` could be confusing for users, especially for alerting rules:
* we say in docs that each returned "series" will create a new alert
* we say that there are "series fetched", so both columns should be
either "series" or "samples".
Let's rename it to `series` for consistency.
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/#pull-request-checklist).
---------
Signed-off-by: hagen1778 <roman@victoriametrics.com>
…s `query` template, but only logging a warning
The `query` template is not supported in replay mode, because we perform
range queries on the rule’s expression, but not on the `query` template.
Previously, if user see error `query template isn't supported in replay
mode`, they need remove the `query` template from the rule for replay
mode.
Also, templating is only used for alerting rules. Replaying alerting
rules don't send notifications(rule annotations are included here) and
users mainly focus on the generated `ALERTS`, the `query` result is
trivial. This pull request shouldn't break things but simplifies the
usage of replay mode for the case.
related https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8746
### Describe Your Changes
Users try to migrate from systems like Thanos or Prometheus using
`vmctl` and `promtheus` migration protocol, with questions about the
problems when they try to specify the snapshot, but `vmctl` shows `0`
series to be found for migration.
This issue happens because users specify the block folder instead of the
snapshot folder.
Added clarification about snapshot structure and its appearance with
multiple blocks inside.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/#pull-request-checklist).
* packaging/make: fix archiving release binaries for cluster
Cluster binaries for non-windows platforms did not include fips binaries, fix that by properly including binaries.
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
See #9144
VictoriaLogs was not correctly parsing timestamps from journald data
when using the default time field configuration. This caused
VictoriaLogs to look for a `_time` field instead of
`__REALTIME_TIMESTAMP` in journald data by default, resulting in
timestamps falling back to ingestion time rather than the actual log
timestamps.
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
The CommonParams.TimeFields is initialized to []{"_time"} by default. This prevent from the proper usage of the -journald.timeField
as the default field for reading log timestamps.
This bug has been introduced in the commit a1a731eb61
While at it, make sure that every parsed log entry has its own timestamp if the timestamp couldn't be read from the log entry.
This provides reliable sort order of the log fields.
### Describe Your Changes
fixes#8535
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
Storage node with large number of partitions (e.g. 150+ partitions with 12Tb of data) will take more than 30 seconds to start. This was causing vmbackupmanager restarts when running vmstorage colocated with vmbackupmanager in a single pod.
Use exponential backoff for retries and increase overall timeout for storage node healthcheck to 3 minutes to avoid vmbackupmanager restarts during storage node startup.
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Using SAS token is not required when copying data within a single
storage account. Using a plain object URL is sufficient for this case. A
single storage account is normally used to store backups so it is safe
to remove SAS tokens usage.
This fixes support of server-side copy when using managed identity as
authentication source as SAS token can only be generated when using
"shared key" type of credentials.
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/9131
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
### Describe Your Changes
removed trailing comma in ignoreFields, which led to _msg field removal,
since it is converted to empty string before filtering
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/#pull-request-checklist).
### Describe Your Changes
This pull request addresses a bug in the Less method of the tagFilter
struct. The original implementation incorrectly assigned the value of
isCompositeB by calling tf.isComposite() instead of other.isComposite().
This caused both isCompositeA and isCompositeB to always have the same
value, leading to incorrect comparisons when determining the order of
tagFilter instances.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/#pull-request-checklist).
---------
Co-authored-by: Zhu Jiekun <jiekun@victoriametrics.com>
### Describe Your Changes
Currently, some PRs have a failed CI due to low code coverage reported
by Codecov. However, the team typically ignore this and relies on other
quality indicators such as thorough code reviews.
This change configures Codecov to continue posting coverage reports
without marking the build as failed.
It also helps reduce confusion for external contributors, who might
otherwise feel pressured to add unnecessary tests just to satisfy
Codecov requirements (for example
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/9002#discussion_r2111651046).
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/#pull-request-checklist).
This commit adds a basic backup/restore test for vmsingle and vmcluster. A
more sophisticated was originally added to the partition index PR
(#8134) and was aimed to test backup/restore when switching back and
forth between legacy and partition index. During the code review it was
decided that it would be good to have a separate test as well since
legacy code will be removed in future and so will the test.
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
This speeds up building the Go builder image significantly (from hours to a few minutes),
since the build speed was limited by the download speed from https://musl.cc , and this speed
was extremely slow (e.g. 10kb/s and slower).
This also improves build security, since the local mirror of musl.cc is under our control.
This PR addresses two issues:
When tenant labels (e.g. vm_account_id, vm_project_id) are passed via
extra_filters, they were not included in the rollupCache key. This could
cause cache entries to be reused across different tenants, resulting in
incorrect query results.
If a tenant is specified only via extra_filters, and that tenant does
not exist in TenantsCached, it gets silently filtered out by
GetTenantTokensFromFilters, causing the query to fall back to a global
(non-tenant) query — which is likely unexpected and potentially unsafe.
This fix ensures correct tenant scoping and avoids unintended data
exposure or cache pollution.
Related issue
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/9001
---------
Co-authored-by: Zakhar Bessarab <me@zekker.dev>
Co-authored-by: Max Kotliar <kotlyar.maksim@gmail.com>
### Describe Your Changes
Typo? It's called "fields" pipe, not "field".
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
Remove the community-provided dashboard as it remains without updates
for a few years already. Recommending it may hurt user's experience.
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Dual stack mode is disabled by default in order to avoid accidentally
exposing components via IPv6 networks.
vmctl does not expose any endpoints and does not allow using default Go
flags as it is using `urfave/cli` lib.
This commit enables IPv6 support by default since there is no security
risks related to network configuration and this make vmctl easier to use
with default configuration.
See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/9116
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
### Describe Your Changes
This change adds logging of the number of skipped bytes when a log line
exceeds the configured `insert.maxLineSizeBytes`.
it helps diagnose and tune systems dealing with oversized log records by
showing how much to increase the parameter for the log to fit in
storage.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/#pull-request-checklist).
---------
Signed-off-by: Dmitry Ponomaryov <iamhalje@gmail.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
### Describe Your Changes
Added a log limit if the 200 logs per second limit is reached and a
notification for the user asking them to add a filter to the query
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/#pull-request-checklist).
### Describe Your Changes
This PR adds the explore section to the docs. It emphasizes on
explaining and linking assets for VMUI and
MetricsQL
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/#pull-request-checklist).
### Describe Your Changes
Properly check precense of `/`, previously it was
ignoring a case where "/" would be at the beginning of the string.
This is a follow-up for 00712b18
---------
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
## Problem
In vlcluster evel setups, components like vlselect can still accept and
forward /insert requests. The lack of strict endpoint control increases
the risk of human error and undermines deployment security boundaries.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/9061
## Fix
Add flags to disable the vlinsert and vlselect endpoints. The
`-insert.disable` flag also disables the internalinsert endpoint.
Similarly for vlselect.
---------
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
Previously, if metric_usage_tracker file was corrupted. It prevented
VictoriaMetrics from start and required manual action. Corruption may
happen in various reasons, such as unclean shutdown of the process.
This commit changes panic into error message, in the same way as other
caches do.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/9074
Previously, any case when cache returned items was skipping lookup of
tenants at vmstorage nodes. This leaded to inconsistent results for
cases when cache contained items to cover only some part of requested
time range.
Fix this by forcing a cache item to cover full requested time range.
This forces cache hits to always be "full hits".
See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/9042
Target branch for this PR is another PR related to the same issue -
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/9048, this is in
order to avoid additional rebasing/merge as this PR will conflict with
cluster branch after initial PR merge. GH will change target for this pr
to cluster brance once #9048 will be merged.
---------
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
This commits adds additional vmselect routes.
Such as `/static`, `/api/v1/status/metric_names_stats` and others.
In addition it properly redirects `/vmui` and `/vmalert` access endpoints requests. Such endpoints require to preserve trailing `/`. Previously it was omitted and redirect requests failed.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/9003
### What this PR does
log error returned by `fastcache.LoadFromFile` before falling back to
creating a new cache instance. this improves observability and helps
detect problems like file corruption or permission issues early.
this replaces `fastcache.LoadFromFileOrNew` with a custom function
`loadFromFileOrNewWithLog` that explicitly logs errors encountered
during cache restoration.
---
### Related Issue
Closes#8934
---
### Test Plan
- manually tested by simulating a missing file scenario
- ensured expected log output on cache load failure
- verified normal cache creation fallback path
---
### Changelog
log error when cache fails to restore from file during workingsetcache
initialization (#8934)
---
### Checklist
- [x] Signed commits
- [x] Follows coding and commit message conventions
- [x] Tested manually
- [x] Scope limited to relevant change
- [x] Changelog entry added
Co-authored-by: Robin Hayer <rshayer95@gmail.com>
Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>
Add the step for running this command after publishing Docker images during the release process.
See docs/victoriametrics/Release-Guide.md
This commit resolves the issue with the missing `latest` and `stable` tags after the https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7336
This also resolve issues with accidental publishing of incorrect Docker images under the `latest` and `stable` tags.
It is very easy to fix incorrectly published `latest` and `stable` tags by re-running the `TAG=v1.x.y make publish-latest` command,
which updates the `latest` and `stable` tags, so they point to the given TAG=v1.x.y.
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
---------
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Max Kotliar <kotlyar.maksim@gmail.com>
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
### Describe Your Changes
Fixed grammatical and phrasing issues in first half of FAQ docs.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
---------
Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>
This should simplify working with big number of log fields in LogsQL queries.
Examples:
- `... | keep foo*` leaves only fields starting with `foo` prefix
- `... | rm foo*` removes all the fields starting with `foo` prefix
- `... | mv foo* bar*` replaces `foo` prefix with `bar` prefix in log fields
- `... | sum(foo*)` sums all the log fields starting with `foo` prefix
The DataBlock.GetTimestamps() was returning a slice of strings, which belong to the DataBlock.
These strings are changed whenever the DataBlock is re-used for the next block.
So these strings couldn't be assigned to logRow.timestamp and to tailProcessor.lastTimestamps,
which outlive the DataBlock. The commit aa8c18fc9f5d44091d7ca92be6935eeaf3b85d7f broke this assumption,
which triggered the following bugs:
1. The bug, which could return incorrectly sorted results from /select/logsql/query when the 'limit' query arg is passed to it.
The endpoint must return the last 'limit' log entries on the selected time range in this case, and these log entries
must be sorted by _time.
2. The bug, which could return incorrect results from /select/logsql/tail (e.g. it could incorrectly skip some matching logs,
it could return the same logs multiple times and it could return out-of-order logs without proper sorting by _time).
The solution is to return parsed timestamps from the DataBlock.GetTimestamps() function, so they could be safely
used by the caller without worries that they could be changed while in use.
Fix https://github.com/VictoriaMetrics/VictoriaMetrics/issues/9016.
Data will carry `vm_account_id` and `vm_project_id` labels when
exporting with native export API in cluster.
These labels could be treated as normal labels and be imported to
victoriametrics cluster, hence inconsistent with the source metrics
data.
e.g.:
1. source data: `{__name__="metrics_test"}`.
2. exported data: `{__name__="metrics_test", vm_account_id="0",
vm_project_id="0"}`.
3. re-imported data: `{__name__="metrics_test", vm_account_id="0",
vm_project_id="0", vm_account_id="0", vm_project_id="0"}`.
4. query result for MetricsQL `metrics_test{}`:
`{__name__="metrics_test", vm_account_id="0", vm_project_id="0"}`.
5. expect query result: `{__name__="metrics_test"}`
In VictoriaMetrics cluster, `vm_account_id` and `vm_project_id` label
are only useful when doing multi-tenant export/import. So they should be
remove if the export URL is not for multi-tenant.
This pull request:
- properly remove tenant info when exporting data in native format.
Note:
- Commit 67514c37ef23c22b91638e80e30504be23fa8dc1 is for apptest and
need to be cherry pick to master branch cc @rtm0 .
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
---------
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
Co-authored-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
### Describe Your Changes
`tests-full` (plural) target doesn't exist, but test (singular) does
discovered while working through unrelated PR
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
Add an option to configure metadata of objects when uploading backups.
For AWS S3 also support using object tagging.
Using metadata of objects is useful in order to get extended reports
about bucket content and billing details. It is also useful when
performing queries to bucket content based on metadata.
See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8010
---------
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
Previously, address was always parsed as "host:port" and added port if
it was missing. This leaded to hard to understand errors in case address
was provided in "http://host:port" format.
Improve error validation in order to provide more precise error message
in case of invalid address format.
See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/9029
Previous error message: `cannot dial storageNode
"http://localhost:8488": dial tcp4: address http://localhost:8488: too
many colons in address` and vminsert continue running.
Current error message: `cannot normalize
-storageNode="http://localhost:8480": invalid address
"http://localhost:8480"; expected format: host:port` and vminsert exists
with error status code.
---------
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Previously, if the value of rentetionFilter was changed within the same
retention, storage didn't start background merge for historical data.
This commits changes this behaviour by writing applied
filters into metadata.json. For backward-compatibility it reads content
of appliedRetention.txt file. It should prevent from triggering
background merge on storage update. If needed, manually remove appliedRetention.txt file from
storage/data/PART folder and remove storage.
Also, it properly applies retentionFilter for data back-filling.
Previously, it was ignored and data outside of retention could be
ingested.
In addition, it changes scheduling of historical merges.
Instead 2 separate background processes, storage launches a single
thread. It reduces CPU resource and disk IO resources usage.
Related issues:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8885https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4592
### Describe Your Changes
Previously, invalid label name or value could cause a panic of vmselect
or vmsingle as it was using MustNewLabelsFromString which was added for
usage in tests only.
Fix this by properly handling and propagating error to user interface if
there is any.
See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8661
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Previously, restore mark could be create to a backup which does not exist or incomplete. This would lead to a crash when attempting to perform restore later on.
This commit adds verification of backup availability and completion to prevent such issues from happening. It also adds a verification bypass mechanism for cases when user wants to create a restore mark which is not currently available.
related issues:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5361https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8771
- dynamically adjusts the concurrent dial limit between 8 and 64 based
on the `-search.maxConcurrentRequests`.
- goroutines now have the chance to access available connections while
awaiting the dial limit token.
Related PR:
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8922
Skipping downsampling rules with filters based on the timestamp and offset leads to unexpected behaviour in case both rules with and without filters are present.
For example, with the following configuration: `-downsampling.period='{__name__="foo"}:60d:2m,7d:4m'`
The user would expect `foo` metrics to be downsampled only after 60d to 2m intervals. But actually pre-filter would skip scoped rule and use global rule after 7d with 4m interval.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8969
### Describe Your Changes
Fix an issue where queries were not triggered when relative time was
selected and the chart was hidden.
Related issue: #8983
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
### Describe Your Changes
* UI now respects the `sort by` pipe in queries — if it's present, the
order returned by the server is preserved. Related issue: #8660.
* If no `sort by` pipe is used, logs are reversed on the client to show
the newest entries first (since VictoriaLogs returns them in ascending
time order — [see this in the
code](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vlselect/logsql/logsql.go#L1047)).
* Removed redundant client-side time-based sorting logic.
Additionally:
* Log record fields are now sorted alphabetically in UI selectors such
as **Group by field**, **Display fields**, and **Customize columns**.
Related issue: #8438.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
Move change 53a6bbfdf8 to the actual
release. Before, it was mistakenly merged to prev release.
Re-classify change from BUGFIX to FEATURE due to following reasons:
* the risk of facing this issue is low, as it reveals itself only for short staleness intervals
* it slightly changes increase_pure logic in a good way. But it is still a change, not bugfix.
Signed-off-by: hagen1778 <roman@victoriametrics.com>
This change has effect only if one of the flags below are set:
`-search.maxLookback`, `-search.setLookbackToStep` or
`-search.maxStalenessInterval`
These flags instruct query engine to ignore data points outside of the
look-behind window if these data points are beyond the staleness
interval.
This logic is used for `removeCounterResets` function, and in functions
`increase`, `increase_pure` or `delta`. The bug described in
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8935 hit the
corner case when `removeCounterResets` detected the stale series and
`increase` did not.
The reason why staleness detection failed for `increase` is that
`removeCounterResets` calculates interval between real data points. And
`realPrevValue` (that is used by those functions) calculates the
difference between look-behind window start and previous data point.
Which, at smaller gaps or smaller staleness intervals, could affect
staleness detection and make it different to `removeCounterResets`.
This change makes `realPrevValue` to acocunt for staleness between first
data point in captured look-behind window and previous data point.
-------
While there, also updated `increase_pure` logic. It was changed in
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1381 without
good explanation. Turns out, that `increase_pure` always compared last
value on the interval with value before the interval. While other
increase or delta functions did compare it with first data point on
interval, and only if it is missing - with the realPrevValue.
This change makes `increase_pure` logic consistent with other similar
function. The reason why it is not a separate PR is because tests
started to fail once `realPrevValue` callculation logic changed and
there were no good solution to isolate this change.
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
---------
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Max Kotliar <kotlyar.maksim@gmail.com>
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
### Describe Your Changes
When downloading archives for benchmarks, an error appears saying that
the archive was placed in a new path.
The error could have been prevented by providing the `-L (--location)`
flag that would tell curl to follow the redirect, so in addition to
updating the paths, this flag was added.
### Checklist
The following checks are **mandatory**:
- [X] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
### Describe Your Changes
Fix extra newline escape characters in panel descriptions.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
1. Add `!lex.isEnd()` to prevent an infinite loop. Although the current
code doesn't trigger this bug, it's a latent issue that could occur if
someone modifies the callers or adds new code paths without proper stop
tokens.
This PR improves integrations docs in 2 areas:
- Background set to white, avoiding issues when changing to dark mode.
- Height set to avoid blank spaces
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
### Describe Your Changes
When multiple users run tests on the same instance, the first user
creating a folder will own the testStoragePath, which can lead to issues
accessing this folder for other users. This change will allow us to
create unique folders per user.
```
% ls -ld /usr/tmp/vmalert-unittest/
drwxr-xr-x 2 some_user users 4096 May 12 17:22 /usr/tmp/vmalert-unittest/
...
2025-05-20T13:56:16.488Z panic lib/fs/fs.go:132 FATAL: cannot create directory: mkdir /usr/tmp/vmalert-unittest/1747749376488491648: permission denied
panic: FATAL: cannot create directory: mkdir /usr/tmp/vmalert-unittest/1747749376488491648: permission denied
```
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
---------
Co-authored-by: Hui Wang <haley@victoriametrics.com>
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8985
When using `AddTimeFilter`, it creates a string representation with the
exact same timestamps but doesn't transform the internal end value. This
is different from the `parseFilterTime` function, which makes the
behavior of these two paths different.
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
---------
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
### Describe Your Changes
compose.yml does not exist, only compose-base.yml
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
This commit re-fines the relabeling cookbok and moves all
relabeling related docs to the same page.
It also removes duplicated information from vmagent readme.
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
The link to enterprise release guide now points to the doc in
enterprise-single-node branch instead of enterprise master. This is
because we don't use enterprise master.
Additionally the `Public Announcement` section has been removed because
we don't make public announcements for releases.
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
### Describe Your Changes
As planned, we are adding a more structured way of explaining
deployments and tiers.
In this PR the following changes are added:
- The previous tiering section is moved under the deployments
placeholder
- Explanations for users to pick single or cluster
- Explanations for different parameters
- Re-styling of the docs to look more appealing
- Reorder some hanging docs so the sections are more clearly presented
to the users
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
---------
Co-authored-by: Alexander Marshalov <_@marshalov.org>
Apparently, when the doc was created for vmctl the anchor conflicts
weren't accounted for or weren't a thing yet. Now, various migration modes
have conflicting anchors. This should be addressed in follow-up commits.
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Improve wording and instructions. It has been a while since the last
time we updated it (more than 4 years!).
Signed-off-by: hagen1778 <roman@victoriametrics.com>
### Describe Your Changes
added command to run vmui locally backed by vm playground
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
### Describe Your Changes
Related issue: #8925
Properly escaped special characters in field values shown in
autocomplete suggestions.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
The following test produces duplicate per-day index records on a system
with 1 CPU even when data inserted sequentially:
```
GOEXPERIMENT=synctest taskset -c 0 go test ./lib/storage -run=TestStorageAddRowsForVariousDataPatternsConcurrently/perDayIndexes/serial/sameBatchMetrics/sameRowMetrics/sameBatchDates/diffRowDates
```
See: #8654
Make this test pass by relaxing got and want data equality requirement
if the number of CPUs is 1. This is temporary until one insertion corner
case is fixed:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8948
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
### Describe Your Changes
Related issue: #7046
Changes:
- add JSX automatic runtime import of React, you don't need to import
React anymore in JSX files.
- add unused imports eslint rule
- add headers to ProcessLiveTailRequest to enable client-side connection
setup
- refactor ExploreLogs: divided the component into several separate
components for better readability and code maintenance
- add live tailing tab to VictoriaLogs
short demo:
https://github.com/user-attachments/assets/3e5f57ee-8e72-4835-9fc6-35c6f38bc9ef
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
---------
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
### Describe Your Changes
Capitalize "Enterprise" in VictoriaMetrics Enterprise phrase
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
### Describe Your Changes
Improve relabel bug fix changelog message
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
Move anchors that were kept only for backward-compatibility reasons
to the bottom of the document. So they don't take extra space in main doc.
Demote anchor level to `h6`, so docs engine will stop rendering in the
right navigation column.
In this way, we still keep the backward compatibility: old links will
continue working. And free space occupied by these link in the main doc.
Thanks to @makasim for the idea.
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Commit 3b84f45e0a introduce a typo at `relabelConfigs.IsSet` function. It incorrectly returned value if relabeling configuration is set or not.
As a result, vmagent was not able to properly perform relabel configuration reload.
And incorrectly exposed metrics for reload configuration.
Related issue:
https://github.com/VictoriaMetrics/helm-charts/issues/2119
Previously, headers hash calculation had a typo, instead of `hash.Write` - `hash.Sum` method was used.
It discards any previous writes and as a result digest always had the same value. It prevented from proper config reload.
This commit fixes this typo and properly calculates headers digest.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8931
Previously, OTL attributes fields with KeyValue type were ingested as a single json formated field. It complicates requests and requires extra effort at query time.
This commit adds support for handling nested fields to match the behavior of
other handlers, such as `/jsonline`. KeyValue attribute will be converted into separate field.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8862
### Describe Your Changes
Added npm command `npm run start:logs:playground` to run VictoriaLogs UI
with playground env.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
### Describe Your Changes
Version with the latest changes from prometheus/common have been
released as v0.303.1 so builds are no longer failing.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
It solves the following build error:
```
duplicate menu entry with identifier "Release process guidance" in menu "docs"
```
Signed-off-by: hagen1778 <roman@victoriametrics.com>
### Describe Your Changes
Update /anomaly-detection/ docs according to release `v1.22.1`
P.s. The links in guides and quickstarts were not intentionally set to
v1.22.1 until we fully restore parallelization in forthcoming releases
and mark it as a go-to release everywhere
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
There are three possible cases for blockResultColumn.getValuesEncoded():
- The blockResultColumn.valuesEncoded is already set. This is the case for manually constructed blockResultColumn,
or if it is cloned via blockResult.clone().
- The blockResultColumn.chSrc is non-nil. In this case the valuesEncoded must be read from the corresponding br.bs,
by applying br.bm filter.
- The blockResultColumn.cSrc is non-nil. In this case the valuesEncoded must be read from the corresponding br.brSrc,
by applying br.bm filter.
It is better from maintainability and debuggability PoV to write this logic in a single getValuesEncoded() function
instead of indirecting it via valuesEncodedCreator.
VictoriaLogs stores min and max column values per every data block.
These values were incorrectly used by min() and max() stats functions
inside updateStatsForAllRows() function. It was assumed that this function
could use min / max values stored in the block, since all the rows in the blockResult
must be processed. But the blockResult contains _filtered_ rows,
e.g. it may have less rows than the number of rows in the original block.
In this case it is unsafe assuming that the min / max values from the original block
exist in the filtered rows inside blockResult.
Add blockResult.isFull() function, which returns true if the blockResult contains all rows
from the original block (e.g. they aren't filtered). Use this function in fast path,
while fall back to slow path, which triggers reading the column values and iterating over them.
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [X] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
* update sorting to reflect more important changes first
* use issue numbers instead of `this issue` wording
Signed-off-by: hagen1778 <roman@victoriametrics.com>
### Describe Your Changes
Add a link to UI on the welcome page.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
* actualize information
* explain alert state in more details
* re-order content by its importance
Signed-off-by: hagen1778 <roman@victoriametrics.com>
### Describe Your Changes
Fix logs not updating after query change
Related issue: #8912
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
### Describe Your Changes
Fixes UI freeze in logs timeline after repeated zooming and panning.
The issue was caused by excessive query requests triggered during fast
timeline interactions. Added debounce to limit the frequency of
requests, similar to the approach used in vmui for metrics.
Related issue: #8655
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* tidy up wording
* mention available integrations in quickstart
* cross-reference integrations and data ingestion
* move these docs closer to each other
Signed-off-by: hagen1778 <roman@victoriametrics.com>
### Describe Your Changes
Limit number of points per series and total response size (30 MiB) on
the Raw Query page to improve UI stability.
Related issue: #7895
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
* move example alerts out of /rules folder, since this folder should
contain only useful rules
* expose ports for vlogs and vmetrics for local debug
* add some comments explamining vmalert config
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
Signed-off-by: hagen1778 <roman@victoriametrics.com>
The new templating option could contain following options: `prometheus`,
`vlogs` or `graphite`.
It represents the datasource type this rule is evaluated against. The
new templating option
can be used in rule's annotations or for routing to a specific
destination (e.g. Grafana datasource).
---------
Signed-off-by: hagen1778 <roman@victoriametrics.com>
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
This functionality was broken in
0e313e5355
Was caught by integration tests:
```
--- FAIL: TestSingleVMAuthRouterWithInternalAddr (5.00s)
vmauth_routing_test.go:148: Could not start vmauth: could not extract some or all regexps from stderr: ["pprof handlers are exposed at http://(.*:\\d{1,5})/debug/pprof/"]
```
Signed-off-by: hagen1778 <roman@victoriametrics.com>
VictoriaLogs happily accepts logs without _msg field, and there are legitimate cases where the _msg field isn't needed.
Remove the warning in order to reduce the level of confusion for VictoriaLogs users.
This feature is useful when ingesting logs, which may contain timestamps across different fields.
Then the first non-empty field from the _time_field list is used as a timestamp field.
This is needed in order to be able to find these playgrounds by `demo` word.
While at it, replace misleading `playground` word with less confusing wording across different places of the docs.
Previously `*_last_reload_successful` metrics are set to 1 even when respective
configuration is not defined, besides stream aggregation and scraping.
This commit initializes config reload related metrics only when respective
configuration is set.
Related issue:
https://github.com/VictoriaMetrics/helm-charts/issues/2119
This commit adds the following changes to the enterprise version:
- add make target for testing in FIPS mode
- disallow using OVH in FIPS mode. OVH is using SHA1 for authentication via headers and SHA1 is not allowed to be used in FIPS mode. There is no option to switch to another hashing algorithm in OVH API, so disabling it completely.
- build fips binaries together with regular ones. This will allow to make sure that FIPS builds are always up to date and compatible with regular ones.
- disable CGO in FIPS builds for vmagent, since vmagent imports Kafka library which uses CGO imports. This might lead to using OpenSSL version which is not certified for FIPS mode. Using pure Go implementation allows to avoid this and keep all validations on Go build process side.
- vm_retention_filtering_partitions_scheduled is a guage metric that shows now
many partitions the retention filtering is currently being applied to.
- vm_retention_filtering_partitions_scheduled_size_bytes is a guage metric that
shows the total size (in bytes) of partitions the retention filtering is
currently being applied to.
These metrics are similar to vm_downsampling_partitions_scheduled and
vm_downsampling_partitions_scheduled_size_bytes and can be used for observing
how much time it took for a given vmstorage instance to perform retention
filtering across this many partitions whose total size was this many bytes.
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
Signed-off-by: hagen1778 <roman@victoriametrics.com>
### Describe Your Changes
fixed typos in docs and code
fixed collision in cloud docs
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
* {lib/backup,app/}: gracefully cancel currently running operation during graceful shutdown
Make backup/restore process interruptable by passing global context from the operation caller.
This is needed in order to reduce shutdown delays in case backup/restore cancellation is requested.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8554
This PR fixes two related bugs in the `replace_regexp` pipe:
1. **Infinite loop on empty matches when `limit` is not set**
[#8625](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8625)
When a regex pattern like `\d*`, `()`, or `\b` was used, the
implementation could repeatedly match the same zero-width position
without advancing the string, causing unbounded memory usage and
eventual OOM. This is now fixed by collecting all matches up front,
respecting the `limit`, and applying replacements in a single pass.
2. **Incorrect handling of anchors (`^` and `$`)**
The previous implementation applied regex matching to progressively
sliced substrings (`s = s[end:]`), which unintentionally caused anchor
patterns like `^` (start-of-string) to match at every new substring's
start. As a result, patterns that should have matched only once (e.g.,
`^|$`) ended up matching multiple times.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8625
Hopefully this should reduce the amounts of questions whether vlinsert replicates or shards the incoming logs.
The answer - it spreads the incoming log _evenly_ among the configured vlstorage nodes (e.g. it shards logs).
### Changes
Updated `lib/httpserver/httpserver.go` to include a flag that can toggle
CORS (defaults to true to keep the current behavior).
This PR relates to
[this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8680#issue-2983786438)
feature request
### Checklist
The following checks are **mandatory**:
- [x] My change does not break backwards compatibility (i.e., preserves
CORS being enabled unless specified otherwise via the
`-http.cors.disabled=true` flag & value)
---------
Co-authored-by: Jai Mehra <jai.mehra@nav-timing.safrangroup.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
### Describe Your Changes
#8842
As a quick fix, it seems like it works. checked on my failing tests
(they are not emitting this error anymore)
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>
These were added while working on paritition index (#8134). Submitting
the separately in order to:
1. Make sure partition index will not break anything and
2. Be able to compare performance before and after swiching to parition
index.
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
fixed not successful rebase in PR, moved svg icons to a sprite and added
VM logo to UI
<img width="1290" alt="image"
src="https://github.com/user-attachments/assets/34cfdc61-6349-4320-b133-38965cb6c30f"
/>
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
### Describe Your Changes
use the built-in max/min to simplify the code
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
Signed-off-by: pkucode <cssjtu@163.com>
The new rule should notify user when there is a job with 0 configured or
discovered targets, which is usually a sign of misconfiguration.
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
Signed-off-by: hagen1778 <roman@victoriametrics.com>
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
Signed-off-by: hagen1778 <roman@victoriametrics.com>
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8466
This pull request:
1. Add `GraphiteWrite`, `CSVImport`, `OpenTSDBHTTPImport` data import
methods for vmsingle and vminsert.
2. Add TCP `Write` method to `apptest.Client` to make it capable of
writing data in TCP-based protocol.
3. Add test cases:
1. for new import methods.
2. for corner cases placed under `app/victoria-metrics/main_test`.
4. Removed all test cases under `app/victoria-metrics/main_test`.
Signed-off-by: f41gh7 <nik@victoriametrics.com>
Co-authored-by: f41gh7 <nik@victoriametrics.com>
### Describe Your Changes
Updated "> note" sections in /anomaly-detection/ doc path for new
formatting and improved clarity
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
This is needed because Go interprets backticked string as a measurement unit for the given command-line flag:
The listed type, here int, can be changed by placing a back-quoted name in the flag's usage string;
the first such item in the message is taken to be a parameter name to show in the message and the back quotes
are stripped from the message when displayed
See https://pkg.go.dev/flag#PrintDefaults
Signed-off-by: hagen1778 <roman@victoriametrics.com>
### Describe Your Changes
1. Fixed log entry sorting in the "Group" view: newest entries are now
displayed at the top again. In v1.18.0, log entries were incorrectly
sorted (oldest first) due to changes related to nanosecond precision
support.
Related issue: #8726
2. Added alphabetical sorting of fields by key in the "Group" view for
easier navigation.
Related issue: #8438
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
### Describe Your Changes
Related issue: #8697
Renamed `retentionFilters` to `retentionFilter`.
🛑 before merge this pull request, need to merge API changes in
enterprise version.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
* set Y-min to 0. This helps to see real spikes in resource usage;
* add text panel explaining how to use query hash;
* support filtering by tenant;
* use `\tvm_slow_query_stats` prefix filter to filter out logs
that mention failed queries with `vm_slow_query_stats` word;
* add panel to show the minimum queried timestamp - see https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8828
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Based on the code, an active time series is one that has received at
least one sample during the last hour. It does not includes querying.
The first time the docs were updated with this was in
d06aae9454. But even at that time, the
code shows that querying a metric did not make it active.
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
* put important changes on top;
* replace `this issue` with corresponding issue number to make it more sense;
* consistently format components names.
Signed-off-by: hagen1778 <roman@victoriametrics.com>
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Co-authored-by: Jose Gómez-Sellés <14234281+jgomezselles@users.noreply.github.com>
* display max values instead of cumulative values
to outline anomalies;
* remove unnecessary overrides on column width;
* add corresponding formatting units to displayed values.
Signed-off-by: hagen1778 <roman@victoriametrics.com>
datadb.rb contains logRows shards, which weren't freed up after the data ingestion
for the given per-day datadb is stopped. This leads to slow memory leak when VictoriaLogs runs
for multiple days without restarts. Avoid this memory leak by freeing up the logRows shards
after converting them to in-memory parts. Re-use the freed up logRows shards via a pool in order
to reduce the pressure on GC.
This commit moved all the docs/*.md files to docs/victoriametrics/ folder. This broke direct links to these files
such as https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/docs/vmagent.md .
Add the missing `/victoriametrics/` part here, so the link becomes https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/docs/victoriametrics/vmagent.md .
The move of all the docs/*.md files into docs/victoriametrics/ folder is dubious because of the following reasons:
- It breaks direct links to *.md files like mentioned above. It is impossible to fix such links all over the Internet,
so they will remain broken :(
The best thing we can do is to fix them on the resources we control such as VictoriaMetrics repository.
- It breaks Google indexing of VictoriaMetrics docs. Google index contains old links such as https://docs.victoriametrics.com/vmagent/#features .
Now such links are automatically redirected to https://docs.victoriametrics.com/victoriametrics/vmagent/#features by a javascript on the https://docs.victoriametrics.com/vmagent/ page.
Google doesn't like redirects at javascript, since they are frequently used by black hat SEO purposes. This leads to pessimization of VictoriaMetrics docs
in Google search result :( We cannot update the old links all over the Internet in order to avoid the redirect by javascript :(
The best thing we can do is to add <meta rel="canonical"> header with the new location of the page at the old url,
and hope Google won't remove VictoriaMetrics docs from its search results. @AndrewChubatiuk , please do this.
- It breaks backwards navigation. When you click the https://docs.victoriametrics.com/vmagent/ link on some page and then press `back` button
in the web browser, it won't return back you to the original page because of the intermediate redirect :( The broken navigation cannot be fixed
for old links located all over the Internet.
- It increases chances of breaking old links left on the Internet in the future, which will lead to 404 Not Found pages
and angry users :(
The sad thing is that we hit the same wall with harmful redirects again :( In the beginning the VictoriaMetrics docs links had .html suffix,
and they were case-sensitive. For example, http://docs.victoriametrics.com/MetricsQL.html#rate . It has been decided for some unknown reason
that it is a good idea to remove the .html suffix and to make all the links lowercase. So now such links are automatically redirected by javascript
to https://docs.victoriametrics.com/metricsql/#rate and then redirected again by another javasript to https://docs.victoriametrics.com/victoriametrics/metricsql/#rate :(
There are many old links all over the Internet (for example, at Reddit, StackOverflow, some internal Wiki pages, etc.). We cannot fix all of them,
so we need to pray these links won't break in the future.
@hagen1778, @tenmozes, @makasim , please make sure we won't hit the same wall in a third time.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8595
Use multiple independent logRows shards for storing the pending log entries before converting them to searchable parts.
Every shard is protected by its own mutex, so multiple CPU cores may add multiple log rows into datadb at the same time.
This increases the performance of BenchmarkStorageMustAddRows/rowsPerInsert-1, which ingests log rows own-by-one
from concurrently running goroutines, by 2x.
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
Here a description of important terms related to Organizations is
provided. The intention is to leverage on a good UI design rather than
explaining all steps needed to perfomr certain actions.
However, an effort on explaining the meaning of internal states and
terms has been done.
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
This commit modifies the logging behavior for client network errors
(e.g., EOFs, timeouts) during the handshake process. They are now logged
as warnings instead of errors, as they are not actionable from the
server’s perspective. Here's some examples of such errors.
Timeouts during the initial read phase:
2025-04-09T07:08:59.323Z error
VictoriaMetrics/lib/vmselectapi/server.go:204 cannot perform
vmselect handshake with client "<REDACTED>": cannot read hello: cannot
read message with size 11: read tcp4 <REDACTED>-><REDACTED>: i/o
timeout; read only 0 bytes
EOFs occurring later in the handshake process:
2025-04-08T18:01:30.783Z error
VictoriaMetrics/lib/vmselectapi/server.go:204 cannot perform
vmselect handshake with client "<REDACTED>": cannot read isCompressed
flag: cannot read message with size 1: EOF; read only 0 bytes
By logging these as warnings, we reduce noise in error logs while
preserving valuble information for debug.
Previously, if `cpu.max` file has only `max` resource defined without
`period`, it was parsed incorrectly and silently drop error. While this
syntax is valid and actually used by some container runtimes. If period
is not defined, default value for it 100_000 must be used.
This commit fixes parsing function by using default value for period.
In addition, it adds zero value check, which fixes possible panic if
period has 0 value.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8808
This reverts commit fa6a32a39d.
Reason for revert: the broken tests were fixed on GOARCH=386 by skipping the check for the state size
after improting the state of stats function, since the state size depends on the hardware architecture.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8710
### Describe Your Changes
Two new columns were added to the **Cardinality Explorer** table:
`Requests count`
- Displays how many times a metric was queried since stats collection
began
- Highlighted **red** when the value is `0` or missing
`Last request`
- Shows when the metric was last queried
- Highlighted:
- **Red** if older than **30 days**
- **Yellow** if between **7 and 30 days**
- Shows exact timestamp on hover in `YYYY-MM-DD HH:mm:ss` format
Related issue: #6145
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
Co-authored-by: Hui Wang <haley@victoriametrics.com>
* downsampling: allow using `-downsampling.period=filter:0s:0s` to skip downsampling for time series that match the specified `filter`
* doc minor fix
* lib/storage/downsampling: fix a typo in error message
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
---------
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
### Describe Your Changes
Fixes incorrect /flags request in vmui when using URL prefixes.
Related issue: #8641
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
### Describe Your Changes
Hide identical series in the legend on the Raw Query page when
deduplication is disabled.
Related issue: #8688
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
Before filling a bug report it would be great to [upgrade](https://docs.victoriametrics.com/#how-to-upgrade)
Before filling a bug report it would be great to [upgrade](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#how-to-upgrade)
to [the latest available release](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest)
and verify whether the bug is reproducible there.
It's also recommended to read the [troubleshooting docs](https://docs.victoriametrics.com/troubleshooting/) first.
It's also recommended to read the [troubleshooting docs](https://docs.victoriametrics.com/victoriametrics/troubleshooting/) first.
- type:textarea
id:describe-the-bug
attributes:
@@ -64,8 +64,8 @@ body:
*[Grafana dashboard for VictoriaMetrics cluster](https://grafana.com/grafana/dashboards/11176)
See how to setup monitoring here:
*[monitoring for single-node VictoriaMetrics](https://docs.victoriametrics.com/#monitoring)
*[monitoring for VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#monitoring)
*[monitoring for single-node VictoriaMetrics](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#monitoring)
*[monitoring for VictoriaMetrics cluster](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#monitoring)
@@ -6,4 +6,4 @@ Please provide a brief description of the changes you made. Be as specific as po
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/).
- [ ] My change adheres to [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/#pull-request-checklist).
VictoriaMetrics is a fast, cost-saving, and scalable solution for monitoring and managing time series data. It delivers high performance and reliability, making it an ideal choice for businesses of all sizes.
@@ -21,10 +21,10 @@ VictoriaMetrics is a fast, cost-saving, and scalable solution for monitoring and
Here are some resources and information about VictoriaMetrics:
- Deployment types: [Single-node version](https://docs.victoriametrics.com/), [Cluster version](https://docs.victoriametrics.com/cluster-victoriametrics/), and [Enterprise version](https://docs.victoriametrics.com/enterprise/)
- Changelog: [CHANGELOG](https://docs.victoriametrics.com/changelog/), and [How to upgrade](https://docs.victoriametrics.com/#how-to-upgrade-victoriametrics)
- Deployment types: [Single-node version](https://docs.victoriametrics.com/), [Cluster version](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/), and [Enterprise version](https://docs.victoriametrics.com/victoriametrics/enterprise/)
- Changelog: [CHANGELOG](https://docs.victoriametrics.com/victoriametrics/changelog/), and [How to upgrade](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#how-to-upgrade-victoriametrics)
Yes, we open-source both the single-node VictoriaMetrics and the cluster version.
@@ -35,22 +35,22 @@ VictoriaMetrics is optimized for timeseries data, even when old time series are
* **Long-term storage for Prometheus** or as a drop-in replacement for Prometheus and Graphite in Grafana.
* **Powerful stream aggregation**: Can be used as a StatsD alternative.
* **Ideal for big data**: Works well with large amounts of time series data from APM, Kubernetes, IoT sensors, connected cars, industrial telemetry, financial data and various [Enterprise workloads](https://docs.victoriametrics.com/enterprise/).
* **Ideal for big data**: Works well with large amounts of time series data from APM, Kubernetes, IoT sensors, connected cars, industrial telemetry, financial data and various [Enterprise workloads](https://docs.victoriametrics.com/victoriametrics/enterprise/).
* **Query language**: Supports both PromQL and the more performant MetricsQL.
* **Easy to setup**: No dependencies, single [small binary](https://medium.com/@valyala/stripping-dependency-bloat-in-victoriametrics-docker-image-983fb5912b0d), configuration through command-line flags, but the default is also fine-tuned; backup and restore with [instant snapshots](https://medium.com/@valyala/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282).
* **Global query view**: Multiple Prometheus instances or any other data sources may ingest data into VictoriaMetrics and queried via a single query.
* **Various Protocols**: Support metric scraping, ingestion and backfilling in various protocol.
* [InfluxDB line protocol](https://docs.victoriametrics.com/#how-to-send-data-from-influxdb-compatible-agents-such-as-telegraf) over HTTP, TCP and UDP.
* [Graphite plaintext protocol](https://docs.victoriametrics.com/#how-to-send-data-from-graphite-compatible-agents-such-as-statsd) with [tags](https://graphite.readthedocs.io/en/latest/tags.html#carbon).
* [OpenTSDB put message](https://docs.victoriametrics.com/#sending-data-via-telnet-put-protocol).
- **Multiple retentions**: Reducing storage costs by specifying different retentions for different datasets.
- **Downsampling**: Reducing storage costs and increasing performance for queries over historical data.
- **Stable releases** with long-term support lines ([LTS](https://docs.victoriametrics.com/lts-releases/)).
- **Stable releases** with long-term support lines ([LTS](https://docs.victoriametrics.com/victoriametrics/lts-releases/)).
- **Comprehensive support**: First-class consulting, feature requests and technical support provided by the core VictoriaMetrics dev team.
- Many other features, which you can read about on [the Enterprise page](https://docs.victoriametrics.com/enterprise/).
- Many other features, which you can read about on [the Enterprise page](https://docs.victoriametrics.com/victoriametrics/enterprise/).
[Contact us](mailto:info@victoriametrics.com) if you need enterprise support for VictoriaMetrics. Or you can request a free trial license [here](https://victoriametrics.com/products/enterprise/trial/), downloaded Enterprise binaries are available at [Github Releases](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest).
@@ -77,7 +77,7 @@ Some good benchmarks VictoriaMetrics achieved:
* **Minimal memory footprint**: handling millions of unique timeseries with [10x less RAM](https://medium.com/@valyala/insert-benchmarks-with-inch-influxdb-vs-victoriametrics-e31a41ae2893) than InfluxDB, up to [7x less RAM](https://valyala.medium.com/prometheus-vs-victoriametrics-benchmark-on-node-exporter-metrics-4ca29c75590f) than Prometheus, Thanos or Cortex.
* **Highly scalable and performance** for [data ingestion](https://medium.com/@valyala/high-cardinality-tsdb-benchmarks-victoriametrics-vs-timescaledb-vs-influxdb-13e6ee64dd6b) and [querying](https://medium.com/@valyala/when-size-matters-benchmarking-victoriametrics-vs-timescale-and-influxdb-6035811952d4), [20x outperforms](https://medium.com/@valyala/insert-benchmarks-with-inch-influxdb-vs-victoriametrics-e31a41ae2893) InfluxDB and TimescaleDB.
* **High data compression**: [70x more data points](https://medium.com/@valyala/when-size-matters-benchmarking-victoriametrics-vs-timescale-and-influxdb-6035811952d4) may be stored into limited storage than TimescaleDB, [7x less storage](https://valyala.medium.com/prometheus-vs-victoriametrics-benchmark-on-node-exporter-metrics-4ca29c75590f) space is required than Prometheus, Thanos or Cortex.
* **Reducing storage costs**: [10x more effective](https://docs.victoriametrics.com/casestudies/#grammarly) than Graphite according to the Grammarly case study.
* **Reducing storage costs**: [10x more effective](https://docs.victoriametrics.com/victoriametrics/casestudies/#grammarly) than Graphite according to the Grammarly case study.
* **A single-node VictoriaMetrics** can replace medium-sized clusters built with competing solutions such as Thanos, M3DB, Cortex, InfluxDB or TimescaleDB. See [VictoriaMetrics vs Thanos](https://medium.com/@valyala/comparing-thanos-to-victoriametrics-cluster-b193bea1683), [Measuring vertical scalability](https://medium.com/@valyala/measuring-vertical-scalability-for-time-series-databases-in-google-cloud-92550d78d8ae), [Remote write storage wars - PromCon 2019](https://promcon.io/2019-munich/talks/remote-write-storage-wars/).
* **Optimized for storage**: [Works well with high-latency IO](https://medium.com/@valyala/high-cardinality-tsdb-benchmarks-victoriametrics-vs-timescaledb-vs-influxdb-13e6ee64dd6b) and low IOPS (HDD and network storage in AWS, Google Cloud, Microsoft Azure, etc.).
"See https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt . "+
"With enabled proxy protocol http server cannot serve regular /metrics endpoint. Use -pushmetrics.url for metrics pushing")
minScrapeInterval=flag.Duration("dedup.minScrapeInterval",0,"Leave only the last sample in every time series per each discrete interval "+
"equal to -dedup.minScrapeInterval > 0. See also -streamAggr.dedupInterval and https://docs.victoriametrics.com/#deduplication")
"equal to -dedup.minScrapeInterval > 0. See also -streamAggr.dedupInterval and https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#deduplication")
dryRun=flag.Bool("dryRun",false,"Whether to check config files without running VictoriaMetrics. The following config files are checked: "+
"-promscrape.config, -relabelConfig and -streamAggr.config. Unknown config entries aren't allowed in -promscrape.config by default. "+
"This can be changed with -promscrape.config.strictParse=false command-line flag")
@@ -45,7 +45,7 @@ var (
finalDedupScheduleInterval=flag.Duration("storage.finalDedupScheduleCheckInterval",time.Hour,"The interval for checking when final deduplication process should be started."+
"Storage unconditionally adds 25% jitter to the interval value on each check evaluation."+
" Changing the interval to the bigger values may delay downsampling, deduplication for historical data."+
" See also https://docs.victoriametrics.com/#deduplication")
" See also https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#deduplication")
t.Fatalf("not expected number of csv lines want=%d\ngot=%d test=%s.%s\n\response=%q",len(data),test.ExpectedResultLinesCount,q,test.Issue,strings.Join(data,"\n"))
disableInsert=flag.Bool("internalinsert.disable",false,"Whether to disable /internal/insert HTTP endpoint")
maxRequestSize=flagutil.NewBytes("internalinsert.maxRequestSize",64*1024*1024,"The maximum size in bytes of a single request, which can be accepted at /internal/insert HTTP endpoint")
journaldStreamFields=flagutil.NewArrayString("journald.streamFields","Comma-separated list of fields to use as log stream fields for logs ingested over journald protocol. "+
@@ -36,9 +37,7 @@ var (
"See https://docs.victoriametrics.com/victorialogs/data-ingestion/journald/#time-field")
journaldTenantID=flag.String("journald.tenantID","0:0","TenantID for logs ingested via the Journald endpoint. "+
"See https://docs.victoriametrics.com/victorialogs/data-ingestion/journald/#multitenancy")
journaldIncludeEntryMetadata=flag.Bool("journald.includeEntryMetadata",false,"Include journal entry fields, which with double underscores.")
maxRequestSize=flagutil.NewBytes("journald.maxRequestSize",64*1024*1024,"The maximum size in bytes of a single journald request")
journaldIncludeEntryMetadata=flag.Bool("journald.includeEntryMetadata",false,"Include Journald fields with double underscore prefixes")
decolorizeFieldsTCP=flagutil.NewArrayString("syslog.decolorizeFields.tcp","Fields to remove ANSI color codes across logs ingested via the corresponding -syslog.listenAddr.tcp. "+
decolorizeFieldsUDP=flagutil.NewArrayString("syslog.decolorizeFields.udp","Fields to remove ANSI color codes across logs ingested via the corresponding -syslog.listenAddr.udp. "+
{"priority":"123","facility":"15","severity":"3","format":"rfc5424","hostname":"mymachine.example.com","app_name":"appname","proc_id":"12345","msg_id":"ID47","exampleSDID@32473.iut":"3","exampleSDID@32473.eventSource":"Application 123 = ] 56","exampleSDID@32473.eventID":"11211","_msg":"This is a test message with structured data."}`
{"priority":"123","facility_keyword":"solaris-cron","level":"error","facility":"15","severity":"3","format":"rfc5424","hostname":"mymachine.example.com","app_name":"appname","proc_id":"12345","msg_id":"ID47","exampleSDID@32473.iut":"3","exampleSDID@32473.eventSource":"Application 123 = ] 56","exampleSDID@32473.eventID":"11211","_msg":"This is a test message with structured data."}`
addr=flag.String("addr","stdout","HTTP address to push the generated logs to; if it is set to stdout, then logs are generated to stdout")
workers=flag.Int("workers",1,"The number of workers to use to push logs to -addr")
start=newTimeFlag("start","-1d","Generated logs start from this time; see https://docs.victoriametrics.com/#timestamp-formats")
end=newTimeFlag("end","0s","Generated logs end at this time; see https://docs.victoriametrics.com/#timestamp-formats")
start=newTimeFlag("start","-1d","Generated logs start from this time; see https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#timestamp-formats")
end=newTimeFlag("end","0s","Generated logs end at this time; see https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#timestamp-formats")
activeStreams=flag.Int("activeStreams",100,"The number of active log streams to generate; see https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields")
totalStreams=flag.Int("totalStreams",0,"The number of total log streams; if -totalStreams > -activeStreams, then some active streams are substituted with new streams "+
maxQueueDuration=flag.Duration("search.maxQueueDuration",10*time.Second,"The maximum time the search request waits for execution when -search.maxConcurrentRequests "+
"limit is reached; see also -search.maxQueryDuration")
maxQueryDuration=flag.Duration("search.maxQueryDuration",time.Second*30,"The maximum duration for query execution. It can be overridden to a smaller value on a per-query basis via 'timeout' query arg")
disableSelect=flag.Bool("select.disable",false,"Whether to disable /select/* HTTP endpoints")
disableInternal=flag.Bool("internalselect.disable",false,"Whether to disable /internal/select/* HTTP endpoints")
)
funcgetDefaultMaxConcurrentRequests()int{
@@ -71,13 +74,31 @@ var vmuiFileServer = http.FileServer(http.FS(vmuiFiles))
// RequestHandler handles select requests for VictoriaLogs
minFreeDiskSpaceBytes=flagutil.NewBytes("storage.minFreeDiskSpaceBytes",10e6,"The minimum free disk space at -storageDataPath after which "+
"the storage stops accepting new data")
forceMergeAuthKey=flagutil.NewPassword("forceMergeAuthKey","authKey, which must be passed in query string to /internal/force_merge pages. It overrides -httpAuth.*")
forceMergeAuthKey=flagutil.NewPassword("forceMergeAuthKey","authKey, which must be passed in query string to /internal/force_merge . It overrides -httpAuth.* . "+
"See https://docs.victoriametrics.com/victorialogs/#forced-merge")
forceFlushAuthKey=flagutil.NewPassword("forceFlushAuthKey","authKey, which must be passed in query string to /internal/force_flush . It overrides -httpAuth.* . "+
"See https://docs.victoriametrics.com/victorialogs/#forced-flush")
storageNodeAddrs=flagutil.NewArrayString("storageNode","Comma-separated list of TCP addresses for storage nodes to route the ingested logs to and to send select queries to. "+
"If the list is empty, then the ingested logs are stored and queried locally from -storageDataPath")
fmt.Fprintf(w,"See docs at <a href='https://docs.victoriametrics.com/vmagent/'>https://docs.victoriametrics.com/vmagent/</a></br>")
fmt.Fprintf(w,"See docs at <a href='https://docs.victoriametrics.com/victoriametrics/vmagent/'>https://docs.victoriametrics.com/victoriametrics/vmagent/</a></br>")
fmt.Fprintf(w,"Useful endpoints:</br>")
httpserver.WriteAPIHelp(w,[][2]string{
{"targets","status for discovered active targets"},
@@ -752,7 +754,7 @@ func usage() {
consts=`
vmagent collects metrics data via popular data ingestion protocols and routes it to VictoriaMetrics.
See the docs at https://docs.victoriametrics.com/vmagent/ .
See the docs at https://docs.victoriametrics.com/victoriametrics/vmagent/ .
forcePromProto=flagutil.NewArrayBool("remoteWrite.forcePromProto","Whether to force Prometheus remote write protocol for sending data "+
"to the corresponding -remoteWrite.url . See https://docs.victoriametrics.com/vmagent/#victoriametrics-remote-write-protocol")
"to the corresponding -remoteWrite.url . See https://docs.victoriametrics.com/victoriametrics/vmagent/#victoriametrics-remote-write-protocol")
forceVMProto=flagutil.NewArrayBool("remoteWrite.forceVMProto","Whether to force VictoriaMetrics remote write protocol for sending data "+
"to the corresponding -remoteWrite.url . See https://docs.victoriametrics.com/vmagent/#victoriametrics-remote-write-protocol")
"to the corresponding -remoteWrite.url . See https://docs.victoriametrics.com/victoriametrics/vmagent/#victoriametrics-remote-write-protocol")
rateLimit=flagutil.NewArrayInt("remoteWrite.rateLimit",0,"Optional rate limit in bytes per second for data sent to the corresponding -remoteWrite.url. "+
"By default, the rate limit is disabled. It can be useful for limiting load on remote storage when big amounts of buffered data "+
@@ -458,13 +458,13 @@ again:
}elseifstatusCode==415||statusCode==400{
ifc.canDowngradeVMProto.Swap(false){
logger.Infof("received unsupported media type or bad request from remote storage at %q. Downgrading protocol from VictoriaMetrics to Prometheus remote write for all future requests. "+
"See https://docs.victoriametrics.com/vmagent/#victoriametrics-remote-write-protocol",c.sanitizedURL)
"See https://docs.victoriametrics.com/victoriametrics/vmagent/#victoriametrics-remote-write-protocol",c.sanitizedURL)
c.useVMProto.Store(false)
}
ifencoding.IsZstd(block){
logger.Infof("received unsupported media type or bad request from remote storage at %q. Re-packing the block to Prometheus remote write and retrying."+
"See https://docs.victoriametrics.com/vmagent/#victoriametrics-remote-write-protocol",c.sanitizedURL)
"See https://docs.victoriametrics.com/victoriametrics/vmagent/#victoriametrics-remote-write-protocol",c.sanitizedURL)
maxRowsPerBlock=flag.Int("remoteWrite.maxRowsPerBlock",10000,"The maximum number of samples to send in each block to remote storage. Higher number may improve performance at the cost of the increased memory usage. See also -remoteWrite.maxBlockSize")
vmProtoCompressLevel=flag.Int("remoteWrite.vmProtoCompressLevel",0,"The compression level for VictoriaMetrics remote write protocol. "+
"Higher values reduce network traffic at the cost of higher CPU usage. Negative values reduce CPU usage at the cost of increased network traffic. "+
"See https://docs.victoriametrics.com/vmagent/#victoriametrics-remote-write-protocol")
"See https://docs.victoriametrics.com/victoriametrics/vmagent/#victoriametrics-remote-write-protocol")
relabelConfigPathGlobal=flag.String("remoteWrite.relabelConfig","","Optional path to file with relabeling configs, which are applied "+
"to all the metrics before sending them to -remoteWrite.url. See also -remoteWrite.urlRelabelConfig. "+
"The path can point either to local file or to http url. "+
"See https://docs.victoriametrics.com/vmagent/#relabeling")
"See https://docs.victoriametrics.com/victoriametrics/relabeling/")
relabelConfigPaths=flagutil.NewArrayString("remoteWrite.urlRelabelConfig","Optional path to relabel configs for the corresponding -remoteWrite.url. "+
"See also -remoteWrite.relabelConfig. The path can point either to local file or to http url. "+
"See https://docs.victoriametrics.com/vmagent/#relabeling")
"See https://docs.victoriametrics.com/victoriametrics/relabeling/")
usePromCompatibleNaming=flag.Bool("usePromCompatibleNaming",false,"Whether to replace characters unsupported by Prometheus with underscores "+
"in the ingested metric names and label names. For example, foo.bar{a.b='c'} is transformed into foo_bar{a_b='c'} during data ingestion if this flag is set. "+
"Pass multiple -remoteWrite.url options in order to replicate the collected data to multiple remote storage systems. "+
"The data can be sharded among the configured remote storage systems if -remoteWrite.shardByURL flag is set")
enableMultitenantHandlers=flag.Bool("enableMultitenantHandlers",false,"Whether to process incoming data via multitenant insert handlers according to "+
"https://docs.victoriametrics.com/cluster-victoriametrics/#url-format . By default incoming data is processed via single-node insert handlers "+
"according to https://docs.victoriametrics.com/#how-to-import-time-series-data ."+
"See https://docs.victoriametrics.com/vmagent/#multitenancy for details")
"https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#url-format . By default incoming data is processed via single-node insert handlers "+
"according to https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#how-to-import-time-series-data ."+
"See https://docs.victoriametrics.com/victoriametrics/vmagent/#multitenancy for details")
shardByURL=flag.Bool("remoteWrite.shardByURL",false,"Whether to shard outgoing series across all the remote storage systems enumerated via -remoteWrite.url. "+
"By default the data is replicated across all the -remoteWrite.url . See https://docs.victoriametrics.com/vmagent/#sharding-among-remote-storages . "+
shardByURL=flag.Bool("remoteWrite.shardByURL",false,"Whether to shard outgoing series across all the remote storage systems enumerated via -remoteWrite.url. "+
"By default the data is replicated across all the -remoteWrite.url . See https://docs.victoriametrics.com/victoriametrics/vmagent/#sharding-among-remote-storages . "+
"See also -remoteWrite.shardByURLReplicas")
shardByURLReplicas=flag.Int("remoteWrite.shardByURLReplicas",1,"How many copies of data to make among remote storage systems enumerated via -remoteWrite.url "+
"when -remoteWrite.shardByURL is set. See https://docs.victoriametrics.com/vmagent/#sharding-among-remote-storages")
"when -remoteWrite.shardByURL is set. See https://docs.victoriametrics.com/victoriametrics/vmagent/#sharding-among-remote-storages")
shardByURLLabels=flagutil.NewArrayString("remoteWrite.shardByURL.labels","Optional list of labels, which must be used for sharding outgoing samples "+
"among remote storage systems if -remoteWrite.shardByURL command-line flag is set. By default all the labels are used for sharding in order to gain "+
"even distribution of series over the specified -remoteWrite.url systems. See also -remoteWrite.shardByURL.ignoreLabels")
shardByURLIgnoreLabels=flagutil.NewArrayString("remoteWrite.shardByURL.ignoreLabels","Optional list of labels, which must be ignored when sharding outgoing samples "+
"among remote storage systems if -remoteWrite.shardByURL command-line flag is set. By default all the labels are used for sharding in order to gain "+
"even distribution of series over the specified -remoteWrite.url systems. See also -remoteWrite.shardByURL.labels")
tmpDataPath=flag.String("remoteWrite.tmpDataPath","vmagent-remotewrite-data","Path to directory for storing pending data, which isn't sent to the configured -remoteWrite.url . "+
"See also -remoteWrite.maxDiskUsagePerURL and -remoteWrite.disableOnDiskQueue")
keepDanglingQueues=flag.Bool("remoteWrite.keepDanglingQueues",false,"Keep persistent queues contents at -remoteWrite.tmpDataPath in case there are no matching -remoteWrite.url. "+
@@ -81,28 +80,30 @@ var (
`For example, if m{k1="v1",k2="v2"} may be sent as m{k2="v2",k1="v1"}`+
`Enabled sorting for labels can slow down ingestion performance a bit`)
maxHourlySeries=flag.Int("remoteWrite.maxHourlySeries",0,"The maximum number of unique series vmagent can send to remote storage systems during the last hour. "+
"Excess series are logged and dropped. This can be useful for limiting series cardinality. See https://docs.victoriametrics.com/vmagent/#cardinality-limiter")
"Excess series are logged and dropped. This can be useful for limiting series cardinality. See https://docs.victoriametrics.com/victoriametrics/vmagent/#cardinality-limiter")
maxDailySeries=flag.Int("remoteWrite.maxDailySeries",0,"The maximum number of unique series vmagent can send to remote storage systems during the last 24 hours. "+
"Excess series are logged and dropped. This can be useful for limiting series churn rate. See https://docs.victoriametrics.com/vmagent/#cardinality-limiter")
"Excess series are logged and dropped. This can be useful for limiting series churn rate. See https://docs.victoriametrics.com/victoriametrics/vmagent/#cardinality-limiter")
maxIngestionRate=flag.Int("maxIngestionRate",0,"The maximum number of samples vmagent can receive per second. Data ingestion is paused when the limit is exceeded. "+
"By default there are no limits on samples ingestion rate. See also -remoteWrite.rateLimit")
disableOnDiskQueue=flagutil.NewArrayBool("remoteWrite.disableOnDiskQueue","Whether to disable storing pending data to -remoteWrite.tmpDataPath "+
"when the remote storage system at the corresponding -remoteWrite.url cannot keep up with the data ingestion rate. "+
"See https://docs.victoriametrics.com/vmagent#disabling-on-disk-persistence . See also -remoteWrite.dropSamplesOnOverload")
"See https://docs.victoriametrics.com/victoriametrics/vmagent/#disabling-on-disk-persistence . See also -remoteWrite.dropSamplesOnOverload")
dropSamplesOnOverload=flag.Bool("remoteWrite.dropSamplesOnOverload",false,"Whether to drop samples when -remoteWrite.disableOnDiskQueue is set and if the samples "+
"cannot be pushed into the configured -remoteWrite.url systems in a timely manner. See https://docs.victoriametrics.com/vmagent#disabling-on-disk-persistence")
"cannot be pushed into the configured -remoteWrite.url systems in a timely manner. See https://docs.victoriametrics.com/victoriametrics/vmagent/#disabling-on-disk-persistence")
)
var(
// rwctxsGlobal contains statically populated entries when -remoteWrite.url is specified.
t.Fatalf("average active time series before: %d, after: %d, changed: %.2f. threshold: %.2f",avgActiveTimeSeries1,avgActiveTimeSeries2,changed,threshold)
streamAggrGlobalConfig=flag.String("streamAggr.config","","Optional path to file with stream aggregation config. "+
"See https://docs.victoriametrics.com/stream-aggregation/ . "+
"See https://docs.victoriametrics.com/victoriametrics/stream-aggregation/ . "+
"See also -streamAggr.keepInput, -streamAggr.dropInput and -streamAggr.dedupInterval")
streamAggrGlobalKeepInput=flag.Bool("streamAggr.keepInput",false,"Whether to keep all the input samples after the aggregation "+
"with -streamAggr.config. By default, only aggregates samples are dropped, while the remaining samples "+
"are written to remote storages write. See also -streamAggr.dropInput and https://docs.victoriametrics.com/stream-aggregation/")
"are written to remote storages write. See also -streamAggr.dropInput and https://docs.victoriametrics.com/victoriametrics/stream-aggregation/")
streamAggrGlobalDropInput=flag.Bool("streamAggr.dropInput",false,"Whether to drop all the input samples after the aggregation "+
"with -remoteWrite.streamAggr.config. By default, only aggregates samples are dropped, while the remaining samples "+
"are written to remote storages write. See also -streamAggr.keepInput and https://docs.victoriametrics.com/stream-aggregation/")
"are written to remote storages write. See also -streamAggr.keepInput and https://docs.victoriametrics.com/victoriametrics/stream-aggregation/")
streamAggrGlobalDedupInterval=flag.Duration("streamAggr.dedupInterval",0,"Input samples are de-duplicated with this interval on "+
"aggregator before optional aggregation with -streamAggr.config . "+
"See also -dedup.minScrapeInterval and https://docs.victoriametrics.com/stream-aggregation/#deduplication")
"See also -dedup.minScrapeInterval and https://docs.victoriametrics.com/victoriametrics/stream-aggregation/#deduplication")
streamAggrGlobalIgnoreOldSamples=flag.Bool("streamAggr.ignoreOldSamples",false,"Whether to ignore input samples with old timestamps outside the "+
"current aggregation interval for aggregator. "+
"See https://docs.victoriametrics.com/stream-aggregation/#ignoring-old-samples")
"See https://docs.victoriametrics.com/victoriametrics/stream-aggregation/#ignoring-old-samples")
streamAggrGlobalIgnoreFirstIntervals=flag.Int("streamAggr.ignoreFirstIntervals",0,"Number of aggregation intervals to skip after the start for "+
"aggregator. Increase this value if you observe incorrect aggregation results after vmagent restarts. It could be caused by receiving unordered delayed data from "+
"clients pushing data into the vmagent. See https://docs.victoriametrics.com/stream-aggregation/#ignore-aggregation-intervals-on-start")
"clients pushing data into the vmagent. See https://docs.victoriametrics.com/victoriametrics/stream-aggregation/#ignore-aggregation-intervals-on-start")
streamAggrGlobalDropInputLabels=flagutil.NewArrayString("streamAggr.dropInputLabels","An optional list of labels to drop from samples for aggregator "+
"before stream de-duplication and aggregation . See https://docs.victoriametrics.com/stream-aggregation/#dropping-unneeded-labels")
"before stream de-duplication and aggregation . See https://docs.victoriametrics.com/victoriametrics/stream-aggregation/#dropping-unneeded-labels")
streamAggrGlobalEnableWindows=flag.Bool("streamAggr.enableWindows",false,"Enables aggregation within fixed windows for all global aggregators. "+
"This allows to get more precise results, but impacts resource usage as it requires twice more memory to store two states. "+
"See https://docs.victoriametrics.com/stream-aggregation/#aggregation-windows.")
"See https://docs.victoriametrics.com/victoriametrics/stream-aggregation/#aggregation-windows.")
// Per URL config
streamAggrConfig=flagutil.NewArrayString("remoteWrite.streamAggr.config","Optional path to file with stream aggregation config for the corresponding -remoteWrite.url. "+
"See https://docs.victoriametrics.com/stream-aggregation/ . "+
"See https://docs.victoriametrics.com/victoriametrics/stream-aggregation/ . "+
"See also -remoteWrite.streamAggr.keepInput, -remoteWrite.streamAggr.dropInput and -remoteWrite.streamAggr.dedupInterval")
streamAggrDropInput=flagutil.NewArrayBool("remoteWrite.streamAggr.dropInput","Whether to drop all the input samples after the aggregation "+
"with -remoteWrite.streamAggr.config at the corresponding -remoteWrite.url. By default, only aggregates samples are dropped, while the remaining samples "+
"are written to the corresponding -remoteWrite.url . See also -remoteWrite.streamAggr.keepInput and https://docs.victoriametrics.com/stream-aggregation/")
"are written to the corresponding -remoteWrite.url . See also -remoteWrite.streamAggr.keepInput and https://docs.victoriametrics.com/victoriametrics/stream-aggregation/")
streamAggrKeepInput=flagutil.NewArrayBool("remoteWrite.streamAggr.keepInput","Whether to keep all the input samples after the aggregation "+
"with -remoteWrite.streamAggr.config at the corresponding -remoteWrite.url. By default, only aggregates samples are dropped, while the remaining samples "+
"are written to the corresponding -remoteWrite.url . See also -remoteWrite.streamAggr.dropInput and https://docs.victoriametrics.com/stream-aggregation/")
"are written to the corresponding -remoteWrite.url . See also -remoteWrite.streamAggr.dropInput and https://docs.victoriametrics.com/victoriametrics/stream-aggregation/")
streamAggrDedupInterval=flagutil.NewArrayDuration("remoteWrite.streamAggr.dedupInterval",0,"Input samples are de-duplicated with this interval before optional aggregation "+
"with -remoteWrite.streamAggr.config at the corresponding -remoteWrite.url. See also -dedup.minScrapeInterval and https://docs.victoriametrics.com/stream-aggregation/#deduplication")
"with -remoteWrite.streamAggr.config at the corresponding -remoteWrite.url. See also -dedup.minScrapeInterval and https://docs.victoriametrics.com/victoriametrics/stream-aggregation/#deduplication")
streamAggrIgnoreOldSamples=flagutil.NewArrayBool("remoteWrite.streamAggr.ignoreOldSamples","Whether to ignore input samples with old timestamps outside the current "+
"aggregation interval for the corresponding -remoteWrite.streamAggr.config at the corresponding -remoteWrite.url. "+
"See https://docs.victoriametrics.com/stream-aggregation/#ignoring-old-samples")
"See https://docs.victoriametrics.com/victoriametrics/stream-aggregation/#ignoring-old-samples")
streamAggrIgnoreFirstIntervals=flagutil.NewArrayInt("remoteWrite.streamAggr.ignoreFirstIntervals",0,"Number of aggregation intervals to skip after the start "+
"for the corresponding -remoteWrite.streamAggr.config at the corresponding -remoteWrite.url. Increase this value if "+
"you observe incorrect aggregation results after vmagent restarts. It could be caused by receiving buffered delayed data from clients pushing data into the vmagent. "+
"See https://docs.victoriametrics.com/stream-aggregation/#ignore-aggregation-intervals-on-start")
"See https://docs.victoriametrics.com/victoriametrics/stream-aggregation/#ignore-aggregation-intervals-on-start")
streamAggrDropInputLabels=flagutil.NewArrayString("remoteWrite.streamAggr.dropInputLabels","An optional list of labels to drop from samples "+
"before stream de-duplication and aggregation with -remoteWrite.streamAggr.config and -remoteWrite.streamAggr.dedupInterval at the corresponding -remoteWrite.url. "+
"Multiple labels per remoteWrite.url must be delimited by '^^': -remoteWrite.streamAggr.dropInputLabels='replica^^az,replica'. "+
"See https://docs.victoriametrics.com/stream-aggregation/#dropping-unneeded-labels")
"See https://docs.victoriametrics.com/victoriametrics/stream-aggregation/#dropping-unneeded-labels")
streamAggrEnableWindows=flagutil.NewArrayBool("remoteWrite.streamAggr.enableWindows","Enables aggregation within fixed windows for all remote write's aggregators. "+
"This allows to get more precise results, but impacts resource usage as it requires twice more memory to store two states. "+
"See https://docs.victoriametrics.com/stream-aggregation/#aggregation-windows.")
"See https://docs.victoriametrics.com/victoriametrics/stream-aggregation/#aggregation-windows.")
)
// CheckStreamAggrConfigs checks -remoteWrite.streamAggr.config and -streamAggr.config.
See vmalert-tool docs [here](https://docs.victoriametrics.com/vmalert-tool.html).
See vmalert-tool docs [here](https://docs.victoriametrics.com/victoriametrics/vmalert-tool/).
vmalert-tool docs can be edited at [docs/vmalert-tool.md](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/docs/vmalert-tool.md).
vmalert-tool docs can be edited at [docs/vmalert-tool.md](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/docs/victoriametrics/vmalert-tool.md).
defaultRuleType=flag.String("rule.defaultRuleType","prometheus",`Default type for rule expressions, can be overridden via "type" parameter on the group level, see https://docs.victoriametrics.com/vmalert/#groups. Supported values: "graphite", "prometheus" and "vlogs".`)
defaultRuleType=flag.String("rule.defaultRuleType","prometheus",`Default type for rule expressions, can be overridden via "type" parameter on the group level, see https://docs.victoriametrics.com/victoriametrics/vmalert/#groups. Supported values: "graphite", "prometheus" and "vlogs".`)
@@ -44,7 +44,7 @@ Enterprise version of vmalert supports S3 and GCS paths to rules.
For example: gs://bucket/path/to/rules, s3://bucket/path/to/rules
S3 and GCS paths support only matching by prefix, e.g. s3://bucket/dir/rule_ matches
all files with prefix rule_ in folder dir.
See https://docs.victoriametrics.com/vmalert/#reading-rules-from-object-storage
See https://docs.victoriametrics.com/victoriametrics/vmalert/#reading-rules-from-object-storage
`)
ruleTemplatesPath=flagutil.NewArrayString("rule.templates",`Path or glob pattern to location with go template definitions `+
@@ -71,7 +71,7 @@ absolute path to all .tpl files in root.
externalURL=flag.String("external.url","","External URL is used as alert's source for sent alerts to the notifier. By default, hostname is used as address.")
externalAlertSource=flag.String("external.alert.source","",`External Alert Source allows to override the Source link for alerts sent to AlertManager `+
`for cases where you want to build a custom link to Grafana, Prometheus or any other service. `+
`Supports templating - see https://docs.victoriametrics.com/vmalert/#templating . `+
`Supports templating - see https://docs.victoriametrics.com/victoriametrics/vmalert/#templating . `+
`For example, link to Grafana: -external.alert.source='explore?orgId=1&left={"datasource":"VictoriaMetrics","queries":[{"expr":{{.Expr|jsonEscape|queryEscape}},"refId":"A"}],"range":{"from":"now-1h","to":"now"}}'. `+
`Link to VMUI: -external.alert.source='vmui/#/?g0.expr={{.Expr|queryEscape}}'. `+
`If empty 'vmalert/alert?group_id={{.GroupID}}&alert_id={{.AlertID}}' is used.`)
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
