rms/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2026-07-24 01:31:20 +03:00
Code Issues Packages Projects Releases Wiki Activity
12,604 Commits 218 Branches 1,017 Tags
RequestErrorsToAPI-update
Commit Graph

18 Commits

Author SHA1 Message Date
Max Kotliar
710c920d60 app/vmrestore: disallow restoring parts outside the configured -storageDataPath directory (#1051) 
A specifically crafted backup source (compromised S3/GCS/Azure bucket) could use `..`
components in object names to write files outside the `-storageDataPath`
directory.

Fix by validating all source parts against the destination directory
before restore begins (restore.go), and adding a defense-in-depth panic
guard in `NewDirectWriteCloser` (fslocal.go).

PR https://github.com/VictoriaMetrics/VictoriaMetrics-enterprise/pull/1051

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
 2026-06-18 13:21:12 +03:00
Aliaksandr Valialkin
acd06e37a8 all: use new canonical urls to vmbackup docs: https://docs.victoriametrics.com/victoriametrics/vmbackup/ 
This avoids a redirect from the old link https://docs.victoriametrics.com/vmbackup/ to https://docs.victoriametrics.com/victoriametrics/vmbackup/ ,
and fixes `backwards` navigation for these links across VictoriaMetrics docs.

This is a follow-up for f152021521
See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8595#issuecomment-2831598274
 2025-04-30 16:51:30 +02:00
Aliaksandr Valialkin
6e6bae3e8d all: replace old https://docs.victoriametrics.com/vmbackup.html url with the new one - https://docs.victoriametrics.com/vmbackup/ 2024-04-18 01:57:04 +02:00
Aliaksandr Valialkin
7e1dd8ab9d lib: consistently use atomic.* types instead of atomic.* functions 
See ea9e2b19a5
 2024-02-24 02:07:53 +02:00
Aliaksandr Valialkin
f5c4fcc250 lib/backup: consistently use path.Join() when constructing paths for s3, gs and azblob 
E.g. replace `fs.Dir + filePath` with `path.Join(fs.Dir, filePath)`

The fs.Dir is guaranteed to end with slash - see Init() functions.
The filePath may start with slash. If it starts with slash, then `fs.Dir + filePath` constructs
an incorrect path with double slashes.
path.Join() properly substitutes duplicate slashes with a single slash in this case.

While at it, also substitute incorrect usage of filepath.Join() with path.Join()
for constructing paths to object storage systems, which expect forward slashes in paths.
filepath.Join() substittues forward slashes with backslashes on Windows, so this may break
creating or managing backups from Windows.

This is a follow-up for 0399367be602b577baf6a872ca81bf0f99ba401b
Updates https://github.com/VictoriaMetrics/VictoriaMetrics-enterprise/pull/719
 2023-12-04 10:34:39 +02:00
Zakhar Bessarab
2fc7e9f47e lib/backup: add -deleteAllObjectVersions command-line flag (#5147) 
New flag enforces removal of all versions of the object in remote object storage.

See:
- https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5121
- https://docs.victoriametrics.com/vmbackup.html#permanent-deletion-of-objects-in-s3-compatible-storages
 2023-10-10 14:13:23 +02:00
Zakhar Bessarab
760cdcec68 lib/backup: fix issue with inconsistent copying of appliedRetention.txt (#5027) 
* lib/backup: fix issue with inconsistent copying of appliedRetention.txt

appliedRetention.txt can be modified in place, so it should be always copied just the same as parts.json

Updates: https://github.com/victoriaMetrics/victoriaMetrics/issues/5005
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* docs: add changelog entry for appliedRetention.txt copying fix

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
 2023-09-21 11:25:19 +02:00
Aliaksandr Valialkin
4d01bc6d52 lib/backup: properly copy parts.json files inside indexdb directory additional to data directory 
This is a follow-up for 264ffe3fa1

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5005
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5006
 2023-09-18 16:16:50 +02:00
Aliaksandr Valialkin
f93a7b8457 lib/backup/common: consistently use canonical path with / directory separators at Part.Path 
Previously Part.Path could contain `\` directory separators on Windows OS,
which could result in incorrect filepaths generation when making backups at object storage.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4704

This is a follow-up for f2df8ad480
 2023-09-18 16:15:34 +02:00
Zakhar Bessarab
264ffe3fa1 lib/backup: force copying of parts.json (#5006) 
* lib/backup: force copying of parts.json

Copying of parts.json is required because `part.key()` comparison can create same key value for files with different contents. This will result in inconsistent backup being created or restored.

See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5005
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* lib/backup: ensure parts.json is only copied once

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Nikolay <nik@victoriametrics.com>
 2023-09-15 19:04:38 +02:00
Aliaksandr Valialkin
a0f695f5de app/vmbackup: add ability to make server-side copying of existing backups 2023-08-13 17:24:24 -07:00
Nikolay
4786f036de lib/backup: fixes path generation for windows (#4133) 
replaces custom fsync function with standard Fsync methods for files.
fixes pattern matching for parts and properly generate backup path for local fs.
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/70
 2023-05-03 10:48:53 +02:00
Nikolay
06eff5a72c removes FileSize from backup part key (#1872) 
* removes FileSize from backup part key
it should fix download restoration for backups

* Update lib/backup/common/part.go

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2021-12-01 11:01:28 +02:00
Aliaksandr Valialkin
bdd0a1cdb2 lib/backup: increase backup chunk size from 128MB to 1GB 
This should reduce costs for object storage API calls by 8x. See https://cloud.google.com/storage/pricing#operations-pricing
 2021-01-13 12:16:35 +02:00
Aliaksandr Valialkin
b449607181 lib/backup: add MustStop() method for all remote filesystems 2020-10-09 15:32:19 +03:00
Aliaksandr Valialkin
f358fb72d1 app/{vmbackup,vmrestore}: add backup complete file to backup when it is complete and check for this file before restoring from backup 
This should prevent from restoring from incomplete backups.

Add `-skipBackupCompleteCheck` command-line flag to `vmrestore` in order to be able restoring from old backups without `backup complete` file.
 2020-01-09 15:35:38 +02:00
Oleg Kovalov
b4f44befa3 fix misspelled words (#229) 2019-11-12 00:16:42 +02:00
Aliaksandr Valialkin
1e46961d68 app/{vmbackup,vmrestore}: add vmbackup and vmrestore tools for creating backups on s3 or gcs from instant snapshots 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/203
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/38
 2019-11-08 21:21:07 +02:00