wip

The commit automates several steps so they can be copy-pasted and
executed without any manual adjustments.

On success, it prints a message starting with “SUCCESS:” to make it
easier to distinguish from failures.

It also introduces the CURR_TAG and NEXT_TAG environment variables,
which are used by the automated commands.

docs/victoriametrics/Release-Guide.md

@@ -65,11 +65,19 @@ release candidate images are built and pushed, a draft GitHub release is created
 and the candidate is deployed to the sandbox environment.
 
 1. Lock merges for 24h with `/mutex merge` in Slack.
+1. Set NEXT_TAG, CURR_TAG (consult [changelog](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/docs/victoriametrics/changelog/CHANGELOG.md#tip) for the current version):
+   ```sh
+   export CURR_TAG=v1.xx.y && export NEXT_TAG=v1.xx.y
+   ```
+   
 1. Make sure you get all changes fetched.
 
    ```sh
-   git fetch opensource
-   git fetch enterprise
+   git checkout master && git pull opensource master && \
+   git checkout cluster && git pull opensource cluster && \
+   git checkout enterprise-single-node && git pull enterprise enterprise-single-node && \
+   git checkout enterprise-cluster && git pull enterprise enterprise-cluster && \
+   echo "SUCCESS: all branches are up to date"
    ```
 
 1. Make sure all the changes are synced between `master`, `cluster`, `enterprise-single-node` and `enterprise-cluster` branches.
@@ -77,51 +85,66 @@ and the candidate is deployed to the sandbox environment.
    You can use the following commands to gather the changes and then compare them using your favorite diff tool.
 
    ```sh
-   git checkout master && git log v1.xx.y..HEAD --oneline > master_changelog.txt
-   git checkout cluster && git log v1.xx.y-cluster..HEAD --oneline > cluster_changelog.txt
-   git checkout enterprise-single-node && git log v1.xx.y-enterprise..HEAD --oneline > enterprise_changelog.txt
-   git checkout enterprise-cluster && git log v1.xx.y-enterprise-cluster..HEAD --oneline > enterprise_cluster_changelog.txt
+   git checkout master && git log "${CURR_TAG}..HEAD" --oneline > master_changelog.txt && \
+   git checkout cluster && git log "${CURR_TAG}-cluster..HEAD" --oneline > cluster_changelog.txt && \
+   git checkout enterprise-single-node && git log "${CURR_TAG}-enterprise..HEAD" --oneline > enterprise_changelog.txt && \
+   git checkout enterprise-cluster && git log "${CURR_TAG}-enterprise-cluster..HEAD" --oneline > enterprise_cluster_changelog.txt && \
+   echo "SUCCESS: changelogs are gathered"
    ```
 
-1. Review bugfixes in the changelog to determine if they need to be backported to LTS versions.
+1. Review bugfixes in the [changelog](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/docs/victoriametrics/changelog/CHANGELOG.md#tip) to determine if they need to be backported to LTS versions.
    Cherry-pick bug fixes relevant for [LTS releases](https://docs.victoriametrics.com/victoriametrics/lts-releases/).
    This serves as a double-check. The initial assessment should already have been done by the person who merged a PR.
 
 1. Make sure tests pass on branches `master`, `cluster`, `enterprise-single-node` and `enterprise-cluster`.
 
    ```sh
-   make test-full
-   make check-all
+   git checkout master && make test check-all integration-test && \
+   git checkout cluster && make test check-all integration-test && \
+   git checkout enterprise-single-node && make test check-all integration-test && \
+   git checkout enterprise-cluster && make test check-all integration-test && \
+   echo "SUCCESS: all tests passed"
    ```
 
 1. Verify no CVEs in Go code or base images according to the [CVE handling policy](https://docs.victoriametrics.com/victoriametrics/#cve-handling-policy).
    It’s sufficient to run `govulncheck` on the `master` branch since other branches are checked in CI on regular bases. 
    For image scanning, build and check Alpine base image.
+   Alpine update [commit](https://github.com/VictoriaMetrics/VictoriaMetrics/commit/8f3e96fa38757cd2f55abe700abb540c90e4c2e9) example.
+   GOlang update [commit](https://github.com/VictoriaMetrics/VictoriaMetrics/commit/7dbe569fe71ac3a5e0068b1cd85adfff41e5a365) example.
 
    ```sh
    make govulncheck
    
    make package-base
-   grype  --only-fixed [base-image-tag]
+   grype db update && grype  --only-fixed [base-image-tag]
    ```
 
 1. Re-build `vmui` static files. Static assets needs to be rebuilt separately for oss and enterprise branches (changes should not be cherry-picked between these branches). See [commit example](https://github.com/VictoriaMetrics/VictoriaMetrics/commit/9dde5b8ee3fdc9d4cd495c8118e04ff4ee32e650).
 
    ```sh
-   make vmui-update
+   make vmui-update && \
+   echo "SUCCESS: vmui static files are rebuilt"
+   ```
+
+1. Update "available from" tooltip in docs and commit. See example in this [commit](https://github.com/VictoriaMetrics/VictoriaMetrics/commit/70afdd02854990cbf44ac2272b191c5d14d6e118).
+
+   ```sh
+   PKG_TAG=${NEXT_TAG} make docs-update-version && \
+   echo "SUCCESS: docs are updated"
    ```
 
-1. Make sure that the release branches have no security issues.
-1. Update release versions if needed in [SECURITY.md](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/SECURITY.md).
-1. Run `PKG_TAG=v1.xx.y make docs-update-version` command to update version help tooltips.
 1. Cut new version in [CHANGELOG.md](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/docs/victoriametrics/changelog/CHANGELOG.md) and commit it. See example in this [commit](https://github.com/VictoriaMetrics/VictoriaMetrics/commit/b771152039d23b5ccd637a23ea748bc44a9511a7).
 1. Create the following release tags:
-   * `git tag -s v1.xx.y` in `master` branch
-   * `git tag -s v1.xx.y-cluster` in `cluster` branch
-   * `git tag -s v1.xx.y-enterprise` in `enterprise-single-node` branch
-   * `git tag -s v1.xx.y-enterprise-cluster` in `enterprise-cluster` branch
-1. Run `TAG=v1.xx.y EXTRA_DOCKER_TAG_SUFFIX=-rcY make publish-release`. This command performs the following tasks:
 
+   ```sh
+   git checkout master && git tag -s "${NEXT_TAG}" -m "Release ${NEXT_TAG}" && \
+   git checkout cluster && git tag -s "${NEXT_TAG}-cluster" -m "Release ${NEXT_TAG}-cluster" && \
+   git checkout enterprise-single-node && git tag -s "${NEXT_TAG}-enterprise" -m "Release ${NEXT_TAG}-enterprise" && \
+   git checkout enterprise-cluster && git tag -s "${NEXT_TAG}-enterprise-cluster" -m "Release ${NEXT_TAG}-enterprise-cluster" && \
+   echo "SUCCESS: all tags are created"
+   ```
+
+1. Publish release candidate images with the command. This command performs the following tasks:
    * a) Build and package binaries in `*.tar.gz` release archives with the corresponding `_checksums.txt` files inside `bin` directory.
       This step can be run manually with the command `make release` from the needed git tag.
    * b)  Build and publish [multi-platform Docker images](https://docs.docker.com/build/buildx/multiplatform-images/)
@@ -135,8 +158,12 @@ and the candidate is deployed to the sandbox environment.
       * linux/386
       This step can be run manually with the command `make publish` from the needed git tag.
 
-1. Run `TAG=v1.xx.y make github-create-release github-upload-assets`. This command performs the following tasks:
+   ```sh
+   TAG="$NEXT_TAG" EXTRA_DOCKER_TAG_SUFFIX=-rc0 make publish-release && \
+   echo "SUCCESS: release candidate images are published"
+   ```
 
+1. Create a draft release on Github. This command performs the following tasks:
    * a) Create draft GitHub release with the name `TAG`. This step can be run manually
       with the command `TAG=v1.xx.y make github-create-release`.
       The release id is stored at `/tmp/vm-github-release` file.
@@ -151,6 +178,11 @@ and the candidate is deployed to the sandbox environment.
       * To run the command `TAG=v1.xx.y make github-create-release github-upload-assets`, so new release is created
         and all the needed assets are re-uploaded to it.
 
+   ```sh
+   TAG=${NEXT_TAG} make github-create-release github-upload-assets && \
+   echo "SUCCESS: draft release is created and assets are uploaded"
+   ```
+
 1. Go to <https://github.com/VictoriaMetrics/VictoriaMetrics/releases> and verify that draft release with the name `TAG` has been created
    and this release contains all the needed binaries and checksums.
 1. Update the release description with the content of [CHANGELOG](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/docs/victoriametrics/changelog/CHANGELOG.md) for this release. **Use "Save Draft" button, do not publish the release yet!**.
@@ -169,7 +201,7 @@ Issues included in the release are closed, with the comment.
 1. Run `TAG=v1.xx.y EXTRA_DOCKER_TAG_SUFFIX=-rc1 make publish-final-images`. This command publishes the final release images from release candidate image for given `EXTRA_DOCKER_TAG_SUFFIX` and updates  `latest` Docker image tag for the given `TAG`.
    This command must be run only for the latest officially published release. It must be skipped when publishing other releases such as
    [LTS releases](https://docs.victoriametrics.com/victoriametrics/lts-releases/) or some test releases.
-1. Deploy the final images to the sandbox environment and perform a quick smoke test to verify basic functionality works.
+1. Deploy the final images to the sandbox environment and perform a quick smoke test to verify basic functionality works. See example [commit](https://github.com/VictoriaMetrics/ops/pull/2546/commits/2fc794c74e768558434318955341444a2ffb4d6b).
 1. Push the tags `v1.xx.y` and `v1.xx.y-cluster` created at previous steps to public GitHub repository at [https://github.com/VictoriaMetrics/VictoriaMetrics](https://github.com/VictoriaMetrics/VictoriaMetrics):
 
    ```shell
@@ -200,6 +232,10 @@ Issues included in the release are closed, with the comment.
 1. Bump VictoriaMetrics version mentioned in [docs](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7388).
 1. Follow the instructions in [release follow-up](https://github.com/VictoriaMetrics/VictoriaMetrics-enterprise/blob/enterprise-single-node/Release-Guide.md).
 
+#### Playground
+
+1. Upgrade playground in ops repository. See example [commit](https://github.com/VictoriaMetrics/ops/pull/2479/commits/8768b0f5fef683bd7c436060a384e40bc95744d2).
+
 #### Operator
 
 The operator repository [https://github.com/VictoriaMetrics/operator/](https://github.com/VictoriaMetrics/operator/)

release/.gitignore

@@ -0,0 +1,3 @@
+/VictoriaMetrics
+/tmp
+/*_changelog.txt

release/Makefile

@@ -0,0 +1,126 @@
+
+CURR_TAG = v1.128.0
+NEXT_TAG = v1.129.0
+DIFF_CMD ?= diff
+
+.PHONY: init
+init:
+	mkdir -p tmp && rm -rf tmp/*
+
+	git clone git@github.com:VictoriaMetrics/VictoriaMetrics.git tmp/VictoriaMetrics
+
+	(cd tmp/VictoriaMetrics && git remote rename origin opensource)
+	(cd tmp/VictoriaMetrics && git remote add enterprise git@github.com:VictoriaMetrics/VictoriaMetrics-enterprise.git && git fetch enterprise)
+
+	(cd tmp/VictoriaMetrics && git checkout master && git pull opensource master)
+	(cd tmp/VictoriaMetrics && git checkout -b cluster opensource/cluster && git pull opensource cluster)
+	(cd tmp/VictoriaMetrics && git checkout -b enterprise-single-node enterprise/enterprise-single-node && git pull enterprise enterprise-single-node)
+	(cd tmp/VictoriaMetrics && git checkout -b enterprise-cluster enterprise/enterprise-cluster && git pull enterprise enterprise-cluster)
+	@echo "SUCCESS: Release from tag $(CURR_TAG) to $(NEXT_TAG) initialized"
+
+.PHONY: compare-git-history
+compare-git-history:
+	@echo "Collecting git logs from all branches..."
+	(cd tmp/VictoriaMetrics && git checkout master && git log --format=%s "${CURR_TAG}..HEAD" > ../master_changelog.txt)
+	(cd tmp/VictoriaMetrics && git checkout cluster && git log --format=%s "$(CURR_TAG)-cluster..HEAD" > ../cluster_changelog.txt)
+	(cd tmp/VictoriaMetrics && git checkout enterprise-single-node && git log --format=%s "$(CURR_TAG)-enterprise..HEAD" > ../enterprise_changelog.txt)
+	(cd tmp/VictoriaMetrics && git checkout enterprise-cluster && git log --format=%s "$(CURR_TAG)-enterprise-cluster..HEAD" > ../enterprise_cluster_changelog.txt)
+
+	@echo "=== Comparing master vs cluster ==="
+	@$(DIFF_CMD) tmp/master_changelog.txt tmp/cluster_changelog.txt || true
+	@echo ""
+	@read -p "Are master and cluster branches equal? (y/n): " answer; \
+	if [ "$$answer" != "y" ]; then \
+		echo "ERROR: Branches are not equal. Aborting."; \
+		exit 1; \
+	fi
+
+	@echo ""
+	@echo "=== Comparing master vs enterprise-single-node ==="
+	@$(DIFF_CMD) tmp/master_changelog.txt tmp/enterprise_changelog.txt || true
+	@echo ""
+	@read -p "Are master and enterprise-single-node branches equal? (y/n): " answer; \
+	if [ "$$answer" != "y" ]; then \
+		echo "ERROR: Branches are not equal. Aborting."; \
+		exit 1; \
+	fi
+
+	@echo ""
+	@echo "=== Comparing cluster vs enterprise-cluster ==="
+	@$(DIFF_CMD) tmp/cluster_changelog.txt tmp/enterprise_cluster_changelog.txt || true
+	@echo ""
+	@read -p "Are cluster and enterprise-cluster branches equal? (y/n): " answer; \
+	if [ "$$answer" != "y" ]; then \
+		echo "ERROR: Branches are not equal. Aborting."; \
+		exit 1; \
+	fi
+
+	@echo ""
+	@echo "SUCCESS: All branches confirmed equal"
+
+
+.PHONY: tests
+tests:
+	(cd tmp/VictoriaMetrics && git checkout master && make test check-all integration-test)
+	(cd tmp/VictoriaMetrics && git checkout cluster && make test check-all integration-test)
+	(cd tmp/VictoriaMetrics && git checkout enterprise-single-node && make test check-all integration-test)
+	(cd tmp/VictoriaMetrics && git checkout enterprise-cluster && make test check-all integration-test)
+	@echo "SUCCESS: all tests passed"
+
+.PHONY: security
+security:
+	(cd tmp/VictoriaMetrics && git checkout master)
+	(cd tmp/VictoriaMetrics && make govulncheck)
+
+	@echo "Checking image tag: docker.io/victoriametrics/victoria-metrics:$(NEXT_TAG)-security-checks"
+	(cd tmp/VictoriaMetrics && BUILDINFO_TAG=$(NEXT_TAG)-security-checks make package-victoria-metrics)
+	grype db update && grype --only-fixed docker.io/victoriametrics/victoria-metrics:$(NEXT_TAG)-security-checks
+	docker scout cves docker.io/victoriametrics/victoria-metrics:$(NEXT_TAG)-security-checks
+	@echo "SUCCESS: all security checks passed"
+
+.PHONY: vmui-opensource-update
+vmui-opensource-update:
+	@echo "=== Updating VMUI on master branch ==="
+	#(cd tmp/VictoriaMetrics && git checkout master)
+	(cd tmp/VictoriaMetrics && make vmui-update)
+	@echo ""
+	@echo "=== Changes in master branch ==="
+	(cd tmp/VictoriaMetrics && git diff --name-only)
+	@echo ""
+	@read -p "Commit changes to master? (y/n): " answer; \
+	if [ "$$answer" != "y" ]; then \
+		echo "Aborting. Rolling back changes..."; \
+		(cd tmp/VictoriaMetrics && git checkout .); \
+		exit 1; \
+	fi
+	(cd tmp/VictoriaMetrics && git add -A && git commit -m "app/vmui: update vmui build for release $(NEXT_TAG)")
+	@echo "Changes committed to master"
+	@echo ""
+	@echo "=== Cherry-picking to cluster branch ==="
+	(cd tmp/VictoriaMetrics && git checkout cluster && git cherry-pick master)
+	@echo "Cherry-picked to cluster"
+	@echo "SUCCESS: VMUI updated on master and cluster branches"
+
+.PHONY: vmui-enterprise-update
+vmui-enterprise-update:
+	@echo "=== Updating VMUI on enterprise-single-node branch ==="
+	(cd tmp/VictoriaMetrics && git checkout enterprise-single-node)
+	(cd tmp/VictoriaMetrics && make vmui-update)
+	@echo ""
+	@echo "=== Changes in enterprise-single-node branch ==="
+	(cd tmp/VictoriaMetrics && git diff --name-only)
+	@echo ""
+	@read -p "Commit changes to enterprise-single-node? (y/n): " answer; \
+	if [ "$$answer" != "y" ]; then \
+		echo "Aborting. Rolling back changes..."; \
+		(cd tmp/VictoriaMetrics && git checkout .); \
+		exit 1; \
+	fi
+	(cd tmp/VictoriaMetrics && git add -A && git commit -m "app/vmui: update vmui build for release $(NEXT_TAG)")
+	@echo "Changes committed to enterprise-single-node"
+	@echo ""
+	@echo "=== Cherry-picking to enterprise-cluster branch ==="
+	(cd tmp/VictoriaMetrics && git checkout enterprise-cluster && git cherry-pick enterprise-single-node)
+	@echo "Cherry-picked to enterprise-cluster"
+	@echo ""
+	@echo "SUCCESS: VMUI updated on enterprise-single-node and enterprise-cluster branches"

release/README.md

@@ -0,0 +1,260 @@
+# Release Guide
+
+## PreRequisites
+
+1. Make sure you have these tools installed `golang`, `docker`, `make`, `git`, `gcc`, `ca-certificates`, `curl`, `zip`:
+1. Make sure you have `enterprise` remote configured
+
+   ```sh
+   git remote add enterprise <url>
+   ```
+
+1. Replace `origin` remote with `opensource` to avoid accidentally pushing enterprise stuff to public repository.
+
+   ```sh
+   git remote remove origin
+   git remote add opensource git@github.com:VictoriaMetrics/VictoriaMetrics.git
+   ```
+
+1. Make sure you have singing key configured
+1. Make sure you have github token with at least `read:org, repo, write:packages` permissions exported under `GITHUB_TOKEN` env variable.
+   You can create a token in your [profile settings](https://github.com/settings/tokens)
+1. Make sure you're [authorized](https://hub.docker.com/orgs/victoriametrics/settings/enforce-sign-in/windows) for pushing docker images to [docker.io](https://hub.docker.com/u/victoriametrics) and [quay.io](https://quay.io/organization/victoriametrics).
+
+### For MacOS users
+
+Make sure you have GNU version of utilities `zip`, `tar`, `sha256sum`. To install them run the following commands:
+
+```sh
+brew install coreutils
+brew install gnu-tar
+export PATH="/usr/local/opt/coreutils/libexec/gnubin:$PATH"
+```
+
+Docker may need additional configuration changes:
+
+```sh
+docker buildx create --use --name=qemu
+docker buildx inspect --bootstrap
+```
+
+By default, docker on MacOS has limited amount of resources (CPU, mem) to use.
+Bumping the limits may significantly improve build speed.
+
+## Release
+
+The release process is divided into two steps.
+
+### Step 1
+
+In the first step, which usually takes place on Friday, the binaries are compiled,
+release candidate images are built and pushed, a draft GitHub release is created,
+and the candidate is deployed to the sandbox environment.
+
+1. Lock merges for 24h with `/mutex merge` in Slack.
+1. Update NEXT_TAG, CURR_TAG in `Makefile`:
+
+1. Init release. It git checkout VictoriaMetrics, set up git remotes, and fetch latest `master`, `cluster`, `enterprise-single-node` and `enterprise-cluster` branches.
+
+   ```sh
+   make init
+   ```
+
+1. Make sure all the changes are synced between `master`, `cluster`, `enterprise-single-node` and `enterprise-cluster` branches.
+   This serves as a double-check. The synchronization should already have been done by the person who merged a PR.
+   You can use the following commands to gather the changes and then compare them using your favorite diff tool.
+
+   ```sh
+   make compare-git-history
+   ```
+
+1. Review bugfixes in the [changelog](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/docs/victoriametrics/changelog/CHANGELOG.md#tip) to determine if they need to be backported to LTS versions.
+   Cherry-pick bug fixes relevant for [LTS releases](https://docs.victoriametrics.com/victoriametrics/lts-releases/).
+   This serves as a double-check. The initial assessment should already have been done by the person who merged a PR.
+
+1. Make sure tests pass on branches `master`, `cluster`, `enterprise-single-node` and `enterprise-cluster`.
+
+   ```sh
+   make tests
+   ```
+
+1. Verify no CVEs in Go code or base images according to the [CVE handling policy](https://docs.victoriametrics.com/victoriametrics/#cve-handling-policy).
+   It’s sufficient to run `govulncheck` on the `master` branch since other branches are checked in CI on regular bases.
+   For image scanning, build and check Alpine base image.
+   Alpine update [commit](https://github.com/VictoriaMetrics/VictoriaMetrics/commit/8f3e96fa38757cd2f55abe700abb540c90e4c2e9) example.
+   GOlang update [commit](https://github.com/VictoriaMetrics/VictoriaMetrics/commit/7dbe569fe71ac3a5e0068b1cd85adfff41e5a365) example.
+
+   ```sh
+   make security
+   ```
+
+1. Re-build `vmui` static files. Static assets needs to be rebuilt separately for oss and enterprise branches (changes should not be cherry-picked between these branches). See [commit example](https://github.com/VictoriaMetrics/VictoriaMetrics/commit/9dde5b8ee3fdc9d4cd495c8118e04ff4ee32e650).
+
+   ```sh
+   make vmui-opensource-update
+   make vmui-enterprise-update
+   ```
+
+----------
+
+1. Update "available from" tooltip in docs and commit. See example in this [commit](https://github.com/VictoriaMetrics/VictoriaMetrics/commit/70afdd02854990cbf44ac2272b191c5d14d6e118).
+
+   ```sh
+   PKG_TAG=${NEXT_TAG} make docs-update-version && \
+   echo "SUCCESS: docs are updated"
+   ```
+
+1. Cut new version in [CHANGELOG.md](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/docs/victoriametrics/changelog/CHANGELOG.md) and commit it. See example in this [commit](https://github.com/VictoriaMetrics/VictoriaMetrics/commit/b771152039d23b5ccd637a23ea748bc44a9511a7).
+1. Create the following release tags:
+
+   ```sh
+   git checkout master && git tag -s "${NEXT_TAG}" -m "Release ${NEXT_TAG}" && \
+   git checkout cluster && git tag -s "${NEXT_TAG}-cluster" -m "Release ${NEXT_TAG}-cluster" && \
+   git checkout enterprise-single-node && git tag -s "${NEXT_TAG}-enterprise" -m "Release ${NEXT_TAG}-enterprise" && \
+   git checkout enterprise-cluster && git tag -s "${NEXT_TAG}-enterprise-cluster" -m "Release ${NEXT_TAG}-enterprise-cluster" && \
+   echo "SUCCESS: all tags are created"
+   ```
+
+1. Publish release candidate images with the command. This command performs the following tasks:
+    * a) Build and package binaries in `*.tar.gz` release archives with the corresponding `_checksums.txt` files inside `bin` directory.
+      This step can be run manually with the command `make release` from the needed git tag.
+    * b)  Build and publish [multi-platform Docker images](https://docs.docker.com/build/buildx/multiplatform-images/)
+      for the given `TAG`, `TAG-cluster`, `TAG-enterprise` and `TAG-enterprise-cluster`.
+      The resulting docker images will have special release candidate suffix for the given `EXTRA_DOCKER_TAG_SUFFIX`.
+      The multi-platform Docker image is built for the following platforms:
+        * linux/amd64
+        * linux/arm64
+        * linux/arm
+        * linux/ppc64le
+        * linux/386
+          This step can be run manually with the command `make publish` from the needed git tag.
+
+   ```sh
+   TAG="$NEXT_TAG" EXTRA_DOCKER_TAG_SUFFIX=-rc0 make publish-release && \
+   echo "SUCCESS: release candidate images are published"
+   ```
+
+1. Create a draft release on Github. This command performs the following tasks:
+    * a) Create draft GitHub release with the name `TAG`. This step can be run manually
+      with the command `TAG=v1.xx.y make github-create-release`.
+      The release id is stored at `/tmp/vm-github-release` file.
+    * b) Upload all the binaries and checksums created at step `11a` to that release.
+      This step can be run manually with the command `make github-upload-assets`.
+      It is expected that the needed release id is stored at `/tmp/vm-github-release` file,
+      which must be created at the step `a`.
+      If the upload process is interrupted by any reason, then the following recovery steps must be performed:
+        * To delete the created draft release by running the command `make github-delete-release`.
+          This command expects that the id of the release to delete is located at `/tmp/vm-github-release`
+          file created at the step `a`.
+        * To run the command `TAG=v1.xx.y make github-create-release github-upload-assets`, so new release is created
+          and all the needed assets are re-uploaded to it.
+
+   ```sh
+   TAG=${NEXT_TAG} make github-create-release github-upload-assets && \
+   echo "SUCCESS: draft release is created and assets are uploaded"
+   ```
+
+1. Go to <https://github.com/VictoriaMetrics/VictoriaMetrics/releases> and verify that draft release with the name `TAG` has been created
+   and this release contains all the needed binaries and checksums.
+1. Update the release description with the content of [CHANGELOG](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/docs/victoriametrics/changelog/CHANGELOG.md) for this release. **Use "Save Draft" button, do not publish the release yet!**.
+1. Follow the instructions in [LTS release](https://github.com/VictoriaMetrics/VictoriaMetrics-enterprise/blob/enterprise-single-node/Release-Guide.md#lts-release).
+1. Deploy created images to sandbox environment. Make sure it is deployed and don't introduce any regressions. Follow instructions at [Testing Releases](https://github.com/VictoriaMetrics/VictoriaMetrics-enterprise/blob/enterprise-single-node/Release-Guide.md#testing-releases).
+
+### Step 2
+
+In the second step, typically on Monday, the release candidate's performance in the sandbox is reviewed.
+If no issues are found, the release candidate is promoted and officially published and announced to the community.
+Version numbers are updated in the codebase, documentation, and related projects.
+Issues included in the release are closed, with the comment.
+
+1. Review the performance of the release candidate in the sandbox environment.
+   If any issues are found, they must be addressed, and the release process restarted from [Step 1](#step-1) with an incremented release candidate version.
+1. Run `TAG=v1.xx.y EXTRA_DOCKER_TAG_SUFFIX=-rc1 make publish-final-images`. This command publishes the final release images from release candidate image for given `EXTRA_DOCKER_TAG_SUFFIX` and updates  `latest` Docker image tag for the given `TAG`.
+   This command must be run only for the latest officially published release. It must be skipped when publishing other releases such as
+   [LTS releases](https://docs.victoriametrics.com/victoriametrics/lts-releases/) or some test releases.
+1. Deploy the final images to the sandbox environment and perform a quick smoke test to verify basic functionality works. See example [commit](https://github.com/VictoriaMetrics/ops/pull/2546/commits/2fc794c74e768558434318955341444a2ffb4d6b).
+1. Push the tags `v1.xx.y` and `v1.xx.y-cluster` created at previous steps to public GitHub repository at [https://github.com/VictoriaMetrics/VictoriaMetrics](https://github.com/VictoriaMetrics/VictoriaMetrics):
+
+   ```shell
+   git push opensource v1.xx.y
+   git push opensource v1.xx.y-cluster
+   ```
+
+   Push the tags `v1.xx.y`, `v1.xx.y-cluster`, `v1.xx.y-enterprise` and `v1.xx.y-enterprise-cluster` to the corresponding
+   branches in private repository:
+
+   ```shell
+   git push enterprise v1.xx.y
+   git push enterprise v1.xx.y-cluster
+   git push enterprise v1.xx.y-enterprise
+   git push enterprise v1.xx.y-enterprise-cluster
+   ```
+
+   **Important note:** do not push enterprise tags to public GitHub repository - they must be pushed only to private repository.
+
+1. Publish release by pressing "Publish release" green button on GitHub [releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases).
+1. Update GitHub issues related to the new release. Usually, such issues have a label [waiting for release](https://github.com/VictoriaMetrics/VictoriaMetrics/issues?q=is%3Aopen+is%3Aissue+label%3A%22waiting+for+release%22). Close such issues by mentioning which release they were included into, and remove the label. See an example of a [closed issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6637#issuecomment-2390729511).
+1. Bump VictoriaMetrics version at `deployment/docker/*.yml`. For example:
+
+   ```shell
+   for f in $(grep "v1\.116\.0" -R deployment/docker/ -l); do sed -i 's/v1.116.0/v1.117.0/g' $f; done
+   ```
+
+1. Bump VictoriaMetrics version mentioned in [docs](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7388).
+1. Follow the instructions in [release follow-up](https://github.com/VictoriaMetrics/VictoriaMetrics-enterprise/blob/enterprise-single-node/Release-Guide.md).
+
+#### Playground
+
+1. Upgrade playground in ops repository. See example [commit](https://github.com/VictoriaMetrics/ops/pull/2479/commits/8768b0f5fef683bd7c436060a384e40bc95744d2).
+
+#### Operator
+
+The operator repository [https://github.com/VictoriaMetrics/operator/](https://github.com/VictoriaMetrics/operator/)
+
+1. Bump the VictoriaMetrics version in [file `internal/config/config.go`](https://github.com/VictoriaMetrics/operator/blob/master/internal/config/config.go) with new release version for:
+    * `VM_METRICS_VERSION` key in `defaultEnvs` map.
+    * `BaseOperatorConf.MetricsVersion` default value.
+1. Run `make docs`.
+1. Add the dependency to the new release to the tip section in `docs/CHANGELOG.md` ([example](https://github.com/VictoriaMetrics/operator/pull/1355/commits/1d7f4439c359b371b05a06e93f615dbcfb266cf5)).
+1. Commit and send a PR for review.
+
+#### Helm Charts
+
+The helm chart repository [https://github.com/VictoriaMetrics/helm-charts/](https://github.com/VictoriaMetrics/helm-charts/)
+
+Bump the version of images
+
+> Note that helm charts versioning uses its own versioning scheme. The version of the charts not tied to the version of VictoriaMetrics components.
+
+Bump `appVersion` field in `Chart.yaml` with new release version.
+Add new line to "Next release" section in `CHANGELOG.md` about version update (the line must always start with "`-`"). Do **NOT** change headers in `CHANGELOG.md`.
+Bump `version` field in `Chart.yaml` with incremental semver version (based on the `CHANGELOG.md` analysis).
+
+Do these updates to the following charts:
+
+1. Update `vmagent` chart `version` and `appVersion` in [`Chart.yaml`](https://github.com/VictoriaMetrics/helm-charts/blob/master/charts/victoria-metrics-agent/Chart.yaml)
+1. Update `vmalert` chart `version` and `appVersion` in [`Chart.yaml`](https://github.com/VictoriaMetrics/helm-charts/blob/master/charts/victoria-metrics-alert/Chart.yaml)
+1. Update `vmauth` chart `version` and `appVersion` in [`Chart.yaml`](https://github.com/VictoriaMetrics/helm-charts/blob/master/charts/victoria-metrics-auth/Chart.yaml)
+1. Update `cluster` chart `version` and `appVersion` in  [`Chart.yaml`](https://github.com/VictoriaMetrics/helm-charts/blob/master/charts/victoria-metrics-cluster/Chart.yaml)
+1. Update `distributed` chart `version` and `appVersion` in  [`Chart.yaml`](https://github.com/VictoriaMetrics/helm-charts/blob/master/charts/victoria-metrics-distributed/Chart.yaml)
+1. Update `gateway` chart `version` and `appVersion` in  [`Chart.yaml`](https://github.com/VictoriaMetrics/helm-charts/blob/master/charts/victoria-metrics-gateway/Chart.yaml)
+1. Update `k8s-stack` chart `version` and `appVersion` in [`Chart.yaml`](https://github.com/VictoriaMetrics/helm-charts/blob/master/charts/victoria-metrics-k8s-stack/Chart.yaml)
+1. Update `single` chart `version` and `appVersion` in [`Chart.yaml`](https://github.com/VictoriaMetrics/helm-charts/blob/master/charts/victoria-metrics-single/Chart.yaml)
+
+See commit example of a [helm chart](https://github.com/VictoriaMetrics/helm-charts/commit/0ec3ab81795cb098d4741451b66886cc6d9be36c).
+
+Once updated, run the following commands:
+
+1. Commit and push changes to `master`.
+1. Run "Release" action on Github:
+   ![release helm charts](Release-Guide_helm-release.webp)
+1. Merge new PRs *"Automatic update CHANGELOGs and READMEs"* and *"Synchronize docs"* after pipelines are complete.
+
+#### Ansible Roles
+
+> Note that ansible playbooks versioning uses its own versioning scheme. The version of the playbooks is not tied to the version of VictoriaMetrics components.
+
+1. Update the version of VictoriaMetrics components at [https://github.com/VictoriaMetrics/ansible-playbooks](https://github.com/VictoriaMetrics/ansible-playbooks).
+1. Commit changes.
+1. Create a new tag with `git tag -sm <TAG> <TAG>`.
+1. Push the changes with the new tag. This automatically publishes the new versions to galaxy.ansible.com.