Add build tag `disable_grpc_modules` for vmbackup, vmrestore and
vmbackupmanager. Binary size increases only for 3MB with it. It's
acceptable trade-off for security and feature updates.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8008
- Drop the code needed for asynchronous removal of the directory on NFS shares.
This code was needed when VictoriaMetrics could keep open files after their deletion
or renaming. This is no longer the case after the commit 43b24164ef .
Now files are deleted only after all the readers close them.
This updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/61
- Unify MustRemoveAll() and MustRemoveDirAtomic() into MustRemoveDir() and MustRemovePath()
functions:
- The MustRemoveDir() deletes the given directory with all its contents, in an "atomic" way:
it creates a special `.delete-this-dir` file in the directory, then removes all its contents
except of this file, and later removes the `.delete-this-dir` file together with the directory
itself. This makes possible easily determining whether the given directory needs to be deleted
after unclean shutdown - if it contains the `.delete-this-dir` file or if it is empty, it must be deleted.
Add IsPartiallyRemovedDir() function, which can be used for detecting whether the given directory must be removed
at starup.
Previously the MustRemoveDirAtomic() was using a "trick" for atomic directory removal: it was "atomically" renaming
the directory to a temporary directory with '.must-remove.' marker in the directory name, and after that it
was removing the renamed directory. On startup all the directories with the `.must-remove.` marker were deleted
if they are left after unclean shutdown. This "trick" doesn't work for NFS and object storage such as S3,
since these storage systems do not support atomic renaming of directories with multiple entries inside.
The new MustRemoveDir() function doesn't use this "trick", so it can be safely used in NFS and S3-like storage systems.
This is based on the pull request from @func25 - https://github.com/VictoriaMetrics/VictoriaMetrics/pull/9486/files .
- The MustRemovePath() deletes the given file or an empty directory.
- Delete the existing parts and partitions at startup if they were partially deleted.
- Consistently use fs.MustRemoveDir() and fs.MustRemovePath() instead of os.RemoveAll() across the codebase.
This reduces the amounts of bolierplate code related to error handling.
- Consistently use fs.MustWriteSync() instead of os.WriteFile() across the codebase.
### What this PR does
log error returned by `fastcache.LoadFromFile` before falling back to
creating a new cache instance. this improves observability and helps
detect problems like file corruption or permission issues early.
this replaces `fastcache.LoadFromFileOrNew` with a custom function
`loadFromFileOrNewWithLog` that explicitly logs errors encountered
during cache restoration.
---
### Related Issue
Closes#8934
---
### Test Plan
- manually tested by simulating a missing file scenario
- ensured expected log output on cache load failure
- verified normal cache creation fallback path
---
### Changelog
log error when cache fails to restore from file during workingsetcache
initialization (#8934)
---
### Checklist
- [x] Signed commits
- [x] Follows coding and commit message conventions
- [x] Tested manually
- [x] Scope limited to relevant change
- [x] Changelog entry added
Co-authored-by: Robin Hayer <rshayer95@gmail.com>
Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>
### Describe Your Changes
Version with the latest changes from prometheus/common have been
released as v0.303.1 so builds are no longer failing.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Support of the latest prometheus/common is not released yet so pin to previous version.
Related commit at prometheus/prometheus: 95f49dd84b
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
The new version of github.com/VictoriaMetrics/metricsql handles $__interval and $__rate_interval
inside rollup functions in more correct way - it drops square brakets, so VictoriaMetrics
could automatically detect the needed lookbehind window depending on the time distance between real samples.
For example, rate(m[$__rate_interval]) is parsed and processed as rate(m) now.
metricsql: bump to v0.83.0
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7703
The update also returns an error if metric name is specified twice in
metrics selector.
For example, `foo{__name__="bar"}` is not allowed anymore. It would
successfully parse before
this change, but it won't satisfy the search filter any way. So it had
no sense in supporting this. This is why some test cases were removed.
Signed-off-by: hagen1778 <roman@victoriametrics.com>
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
Signed-off-by: hagen1778 <roman@victoriametrics.com>
This introduces the ability to copy-n-paste queries with $__interval and $__rate_interval
placeholders into VictoriaMetrics - these placeholders are automatically replaced with 1i,
which equals to the `step` query arg value passed to /api/v1/query_range and /api/v1/query.
Add exclude google.golang.org/grpc/stats/opentelemetry v0.0.0-20240907200651-3ffb98b2c93a to go.mod
according to https://github.com/googleapis/google-cloud-go/issues/11283#issuecomment-2558515586 .
This fixes the following strange issue on `make vendor-update`:
cloud.google.com/go/storage imports
google.golang.org/grpc/stats/opentelemetry: ambiguous import: found package google.golang.org/grpc/stats/opentelemetry in multiple modules:
google.golang.org/grpc v1.69.0 (/go/pkg/mod/google.golang.org/grpc@v1.69.0/stats/opentelemetry)
google.golang.org/grpc/stats/opentelemetry v0.0.0-20240907200651-3ffb98b2c93a (/go/pkg/mod/google.golang.org/grpc/stats/opentelemetry@v0.0.0-20240907200651-3ffb98b2c93a)
This fixes possible `index out of range` panic when -search.logImplicitConversion
or -search.disableImplicitConversion command-line flags are passed to vmselect
and it tries executing incorrect query with too small number of arguments passed
to rollup function.
Previously, for `^` aka pow function calls, VictoriaMetrics returned `1`
if left arg was Nan. For example, given query=`(hour()==2)^1` returns 1
for NaN produced by hour() == 2 function. It added additional non-exist
datapoints to the timeseries.
This commit port bugfix from `metricql` package and adds test for it.
Now, VictoriaMetrics
correctly returns `NaN` for such cases.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7359
Signed-off-by: f41gh7 <nik@victoriametrics.com>
### Describe Your Changes
This PR continues the implementation of integration tests (#7199). It
adds the support for vm-single:
- A vmsingle app wrapper has been added
- Sample vmsingle tests that test the VM documentation related to
querying data (#7435)
- The tests use the go-cmp/{cmp,/cmpopts} packages, therefore they have
been added to ./vendor
- Minor refactoring: data objects have been moved to model.go
Advice on porting things to cluster branch:
- The build rule must include tests that start with TestVmsingle
(similarly to how TestCluster tests are skipped in master branch)
- The build rule must depend on `vmstorage vminsert vmselect` instead of
`victoria-metrics`
- The query_test.go can actually be implemented for cluster as well. To
do this the tests need to be renamed to start with TestCluster and the
tests must instantiace vm{storage,insert,select} instead of vmsingle.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
Fixes panic if incorrect metricsql expression passed to the prettifier API.
Prettify function had misleading panic for duration expression formatting. It expected all WITH templates to be already parsed.
But WITH expression expand was removed.
Bug was introduced at e712a49898
and present at v1.98.0+ releases
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6736
Signed-off-by: f41gh7 <nik@victoriametrics.com>
