Commit Graph

13 Commits

Author SHA1 Message Date
Zakhar Bessarab
2efc21bfcb lib/backup: add support of object metadata configuration
Add an option to configure metadata of objects when uploading backups.
For AWS S3 also support using object tagging.

Using metadata of objects is useful in order to get extended reports
about bucket content and billing details. It is also useful when
performing queries to bucket content based on metadata.

See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8010

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2025-06-02 18:52:30 +04:00
f41gh7
3a2ca7601d {lib/backup,app/}: gracefully cancel currently running operation during graceful shutdown
* {lib/backup,app/}: gracefully cancel currently running operation during graceful shutdown

Make backup/restore process interruptable by passing global context from the operation caller.

This is needed in order to reduce shutdown delays in case backup/restore cancellation is requested.

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8554
2025-05-06 12:01:12 +02:00
Guillem Jover
1d8b7faf71 spelling and grammar fixes via codespell (#8497)
### Describe Your Changes

Fix many spelling errors and some grammar, including misspellings in
filenames.

The change also fixes a typo in metric `vm_mmaped_files` to `vm_mmapped_files`.
While this is a breaking change, this metric isn't used in alerts or dashboards.
So it seems to have low impact on users.

The change also deprecates `cspell` as it is much heavier and less usable.
---------

Co-authored-by: Andrii Chubatiuk <achubatiuk@victoriametrics.com>
Co-authored-by: Andrii Chubatiuk <andrew.chubatiuk@gmail.com>

(cherry picked from commit 76d205feae)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-03-17 16:38:11 +01:00
Aliaksandr Valialkin
2f4dc2aff1 lib/backup: consistently use path.Join() when constructing paths for s3, gs and azblob
E.g. replace `fs.Dir + filePath` with `path.Join(fs.Dir, filePath)`

The fs.Dir is guaranteed to end with slash - see Init() functions.
The filePath may start with slash. If it starts with slash, then `fs.Dir + filePath` constructs
an incorrect path with double slashes.
path.Join() properly substitutes duplicate slashes with a single slash in this case.

While at it, also substitute incorrect usage of filepath.Join() with path.Join()
for constructing paths to object storage systems, which expect forward slashes in paths.
filepath.Join() substittues forward slashes with backslashes on Windows, so this may break
creating or managing backups from Windows.

This is a follow-up for 0399367be602b577baf6a872ca81bf0f99ba401b
Updates https://github.com/VictoriaMetrics/VictoriaMetrics-enterprise/pull/719
2023-12-04 17:25:41 +02:00
Zakhar Bessarab
8b42a1733c lib/backup: add -deleteAllObjectVersions command-line flag (#5147)
New flag enforces removal of all versions of the object in remote object storage.

See:
- https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5121
- https://docs.victoriametrics.com/vmbackup.html#permanent-deletion-of-objects-in-s3-compatible-storages

(cherry picked from commit 2fc7e9f47e)
2023-10-10 14:14:21 +02:00
Aliaksandr Valialkin
1361239393 app/vmbackup: add ability to make server-side copying of existing backups 2023-08-13 17:26:26 -07:00
Aliaksandr Valialkin
e95e401e4d app/vmbackupmanager: sync with enterprise-single-node branch after 41a54c775891c87e3d5ed59ff0769c869dd2fe71 2023-04-13 19:38:28 -07:00
Zakhar Bessarab
6c65ee18d9 vmbackup: configure retries for GCS remote FS (#3156) 2022-09-26 16:32:53 +03:00
Aliaksandr Valialkin
de1c07b937 lib/backup: add MustStop() method for all remote filesystems 2020-10-09 15:32:13 +03:00
Aliaksandr Valialkin
d962568e93 all: use %w instead of %s for wrapping errors in fmt.Errorf
This will simplify examining the returned errors such as httpserver.ErrorWithStatusCode .
See https://blog.golang.org/go1.13-errors for details.
2020-06-30 23:33:46 +03:00
Aliaksandr Valialkin
705af61587 app/{vmbackup,vmrestore}: add backup complete file to backup when it is complete and check for this file before restoring from backup
This should prevent from restoring from incomplete backups.

Add `-skipBackupCompleteCheck` command-line flag to `vmrestore` in order to be able restoring from old backups without `backup complete` file.
2020-01-09 15:35:45 +02:00
Aliaksandr Valialkin
17eca31989 lib/backup: retrieve only the required metadata when reading GCS objects 2019-11-19 21:30:51 +02:00
Aliaksandr Valialkin
7d7fbf890e app/{vmbackup,vmrestore}: add vmbackup and vmrestore tools for creating backups on s3 or gcs from instant snapshots
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/203
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/38
2019-11-07 21:26:43 +02:00