vendor: make vendor-update

This syncs kubernetes SD with Prometheus 2.25
See 617c56f55a

.dockerignore

@@ -4,4 +4,3 @@ gocache-for-docker
 victoria-metrics-data
 vmstorage-data
 vmselect-cache
-.vscode

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,44 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: ''
+labels: ''
+assignees: ''
+
+---
+
+**Describe the bug**
+A clear and concise description of what the bug is.
+It would be great [upgrading](https://victoriametrics.github.io/#how-to-upgrade) to [the latest avaialble release](https://github.com/VictoriaMetrics/VictoriaMetrics/releases)
+and verifying whether the bug is reproducible there.
+It is also recommended reading [troubleshooting docs](https://victoriametrics.github.io/#troubleshooting).
+
+**To Reproduce**
+Steps to reproduce the behavior.
+
+**Expected behavior**
+A clear and concise description of what you expected to happen.
+
+**Screenshots**
+If applicable, add screenshots to help explain your problem.
+
+**Version**
+The line returned when passing `--version` command line flag to binary. For example:
+```
+$ ./victoria-metrics-prod --version
+victoria-metrics-20190730-121249-heads-single-node-0-g671d9e55
+```
+
+**Used command-line flags**
+Command-line flags are listed as `flag{name="httpListenAddr", value=":443"} 1` lines at `/metrics` page.
+See the following docs for details:
+
+* [monitoring for single-node VictoriaMetrics](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/README.md#monitoring)
+* [montioring for VictoriaMetrics cluster](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/cluster/README.md#monitoring)
+
+**Additional context**
+Add any other context about the problem here such as error logs from VictoriaMetrics and Prometheus,
+`/metrics` output, screenshots from the official Grafana dashboards for VictoriaMetrics:
+
+* [Grafana dashboard for single-node VictoriaMetrics](https://grafana.com/dashboards/10229)
+* [Grafana dashboard for VictoriaMetrics cluster](https://grafana.com/grafana/dashboards/11176)

.github/ISSUE_TEMPLATE/bug_report.yml

@@ -1,86 +0,0 @@
-name: Bug report
-description: Create a report to help us improve
-labels: [bug]
-body:
-  - type: markdown
-    attributes:
-      value: |
-        Before filling a bug report it would be great to [upgrade](https://docs.victoriametrics.com/#how-to-upgrade) 
-        to [the latest available release](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest)
-        and verify whether the bug is reproducible there.
-        It's also recommended to read the [troubleshooting docs](https://docs.victoriametrics.com/troubleshooting/) first.
-  - type: textarea
-    id: describe-the-bug
-    attributes:
-      label: Describe the bug
-      description: |
-        A clear and concise description of what the bug is.
-      placeholder: |
-        When I do `A` VictoriaMetrics does `B`. I expect it to do `C`.
-    validations:
-      required: true
-  - type: textarea
-    id: to-reproduce
-    attributes:
-      label: To Reproduce
-      description: |
-        Steps to reproduce the behavior.
-        If reproducing an issue requires some specific configuration file, please paste it here.
-      placeholder: |
-        Steps to reproduce the behavior.
-    validations:
-      required: true
-  - type: textarea
-    id: version
-    attributes:
-      label: Version
-      description: |
-        The line returned when passing `--version` command line flag to the binary. For example:
-        ```
-        $ ./victoria-metrics-prod --version
-        victoria-metrics-20190730-121249-heads-single-node-0-g671d9e55
-        ```
-    validations:
-      required: true
-  - type: textarea
-    id: logs
-    attributes:
-      label: Logs
-      description: |
-        Check if any warnings or errors were logged by VictoriaMetrics components
-        or components in communication with VictoriaMetrics (e.g. Prometheus, Grafana).
-    validations:
-      required: false
-  - type: textarea
-    id: screenshots
-    attributes:
-      label: Screenshots
-      description: |
-        If applicable, add screenshots to help explain your problem.
-  
-        For VictoriaMetrics health-state issues please provide full-length screenshots
-        of Grafana dashboards if possible:
-          * [Grafana dashboard for single-node VictoriaMetrics](https://grafana.com/grafana/dashboards/10229/)
-          * [Grafana dashboard for VictoriaMetrics cluster](https://grafana.com/grafana/dashboards/11176/)
-        
-        See how to setup monitoring here:
-          * [monitoring for single-node VictoriaMetrics](https://docs.victoriametrics.com/#monitoring)
-          * [monitoring for VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#monitoring)
-    validations:
-      required: false
-  - type: textarea
-    id: flags
-    attributes:
-      label: Used command-line flags
-      description: |
-        Please provide the command-line flags used for running VictoriaMetrics and its components.
-    validations:
-      required: false
-  - type: textarea
-    id: additional-info
-    attributes:
-      label: Additional information
-      placeholder: |
-        Additional information that doesn't fit elsewhere
-    validations:
-      required: false

.github/ISSUE_TEMPLATE/configuration.yml

@@ -1,5 +0,0 @@
-blank_issues_enabled: true
-contact_links:
-  - name: Ask on Slack
-    url: https://slack.victoriametrics.com/
-    about: You can ask for help here!

.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,20 @@
+---
+name: Feature request
+about: Suggest an idea for this project
+title: ''
+labels: ''
+assignees: ''
+
+---
+
+**Is your feature request related to a problem? Please describe.**
+A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+
+**Describe the solution you'd like**
+A clear and concise description of what you want to happen.
+
+**Describe alternatives you've considered**
+A clear and concise description of any alternative solutions or features you've considered.
+
+**Additional context**
+Add any other context or screenshots about the feature request here.

.github/ISSUE_TEMPLATE/feature_request.yml

@@ -1,43 +0,0 @@
-name: Feature request
-description: Suggest an idea for this project
-labels: [enhancement]
-body:
-  - type: textarea
-    id: describe-the-problem
-    attributes:
-      label: Is your feature request related to a problem? Please describe
-      description: |
-        A clear and concise description of what the problem is.
-      placeholder: |
-        Ex. I'm always frustrated when [...]
-    validations:
-      required: false
-  - type: textarea
-    id: describe-the-solution
-    attributes:
-      label: Describe the solution you'd like
-      description: |
-        A clear and concise description of what you want to happen.
-    validations:
-      required: true
-  - type: textarea
-    id: alternative-solutions
-    attributes:
-      label: Describe alternatives you've considered
-      description: |
-        A clear and concise description of any alternative solutions or features you've considered.
-      placeholder: |
-        I have tried to do `A`, but that doesn't solve a problem completely.
-        I have tried to do `A` and `B`, but implementing this would be better.
-    validations:
-      required: false
-  - type: textarea
-    id: feature-additional-info
-    attributes:
-      label: Additional information
-      description: |
-        Additional information which you consider helpful for implementing this feature.
-      placeholder: |
-        Add any other context or screenshots about the feature request here.
-    validations:
-      required: false

.github/ISSUE_TEMPLATE/question.yml

@@ -1,32 +0,0 @@
-name: Question
-description: Ask a question regarding VictoriaMetrics or its components
-labels: [question]
-body:
-  - type: textarea
-    id: describe-the-component
-    attributes:
-      label: Is your question request related to a specific component?
-      placeholder: |
-        VictoriaMetrics, vmagent, vmalert, vmui, etc...
-    validations:
-      required: false
-  - type: textarea
-    id: describe-the-question
-    attributes:
-      label: Describe the question in detail
-      description: |
-        A clear and concise description of the issue and the question.
-    validations:
-      required: true
-  - type: checkboxes
-    id: troubleshooting
-    attributes:
-      label: Troubleshooting docs
-      description: I am familiar with the following troubleshooting docs
-      options:
-        - label: General - https://docs.victoriametrics.com/troubleshooting/
-          required: false
-        - label: vmagent - https://docs.victoriametrics.com/vmagent/#troubleshooting
-          required: false
-        - label: vmalert - https://docs.victoriametrics.com/vmalert/#troubleshooting
-          required: false

.github/dependabot.yml

@@ -1,30 +0,0 @@
-version: 2
-updates:
-  - package-ecosystem: "github-actions"
-    directory: "/"
-    schedule:
-      interval: "daily"
-  - package-ecosystem: "gomod"
-    directory: "/"
-    schedule:
-      interval: "weekly"
-    open-pull-requests-limit: 0
-  - package-ecosystem: "bundler"
-    directory: "/docs"
-    schedule:
-      interval: "weekly"
-    open-pull-requests-limit: 0
-  - package-ecosystem: "gomod"
-    directory: "/app/vmui/packages/vmui/web"
-    schedule:
-      interval: "weekly"
-    open-pull-requests-limit: 0
-  - package-ecosystem: "docker"
-    directory: "/"
-    schedule:
-      interval: "daily"
-  - package-ecosystem: "npm"
-    directory: "/app/vmui/packages/vmui"
-    schedule:
-      interval: "weekly"
-    open-pull-requests-limit: 0

.github/pull_request_template.md

@@ -1,9 +0,0 @@
-### Describe Your Changes
-
-Please provide a brief description of the changes you made. Be as specific as possible to help others understand the purpose and impact of your modifications.
-
-### Checklist
-
-The following checks are **mandatory**:
-
-- [ ] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/).

.github/workflows/build.yml

@@ -1,54 +0,0 @@
-name: build
-
-on:
-  push:
-    branches:
-      - cluster
-      - master
-    paths:
-      - '**.go'
-      - '**/Dockerfile*' # The trailing * is for app/vmui/Dockerfile-*.
-      - '**/Makefile'
-  pull_request:
-    branches:
-      - cluster
-      - master
-    paths:
-      - '**.go'
-      - '**/Dockerfile*' # The trailing * is for app/vmui/Dockerfile-*.
-      - '**/Makefile'
-
-permissions:
-  contents: read
-
-concurrency:
-  cancel-in-progress: true
-  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
-
-jobs:
-  build:
-    name: Build
-    runs-on: ubuntu-latest
-    steps:
-      - name: Code checkout
-        uses: actions/checkout@v4
-
-      - name: Setup Go
-        id: go
-        uses: actions/setup-go@v5
-        with:
-          go-version: stable
-          cache: false
-
-      - name: Cache Go artifacts
-        uses: actions/cache@v4
-        with:
-          path: |
-            ~/.cache/go-build
-            ~/go/bin
-            ~/go/pkg/mod
-          key: go-artifacts-${{ runner.os }}-crossbuild-${{ steps.go.outputs.go-version }}-${{ hashFiles('go.sum', 'Makefile', 'app/**/Makefile') }}
-          restore-keys: go-artifacts-${{ runner.os }}-crossbuild-
-
-      - name: Run crossbuild
-        run: make crossbuild

.github/workflows/check-licenses.yml

@@ -1,38 +0,0 @@
-name: license-check
-on:
-  push:
-    paths:
-      - 'vendor'
-  pull_request:
-    paths:
-      - 'vendor'
-permissions:
-  contents: read
-
-jobs:
-  build:
-    name: Build
-    runs-on: ubuntu-latest
-    steps:
-      - name: Code checkout
-        uses: actions/checkout@master
-
-      - name: Setup Go
-        id: go
-        uses: actions/setup-go@v5
-        with:
-          go-version: stable
-          cache: false
-
-      - name: Cache Go artifacts
-        uses: actions/cache@v4
-        with:
-          path: |
-            ~/.cache/go-build
-            ~/go/pkg/mod
-            ~/go/bin
-          key: go-artifacts-${{ runner.os }}-check-licenses-${{ steps.go.outputs.go-version }}-${{ hashFiles('go.sum', 'Makefile', 'app/**/Makefile') }}
-          restore-keys: go-artifacts-${{ runner.os }}-check-licenses-
-
-      - name: Check License
-        run: make check-licenses

.github/workflows/codeql-analysis-go.yml

@@ -1,62 +0,0 @@
-name: 'CodeQL Go'
-
-on:
-  push:
-    branches:
-      - cluster
-      - master
-    paths:
-      - '**.go'
-  pull_request:
-    branches:
-      - cluster
-      - master
-    paths:
-      - '**.go'
-
-concurrency:
-  cancel-in-progress: true
-  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
-
-jobs:
-  analyze:
-    name: Analyze
-    runs-on: ubuntu-latest
-    permissions:
-      actions: read
-      contents: read
-      security-events: write
-
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@v4
-
-      - name: Set up Go
-        id: go
-        uses: actions/setup-go@v5
-        with:
-          cache: false
-          go-version: stable
-
-      - name: Cache Go artifacts
-        uses: actions/cache@v4
-        with:
-          path: |
-            ~/.cache/go-build
-            ~/go/bin
-            ~/go/pkg/mod
-          key: go-artifacts-${{ runner.os }}-codeql-analyze-${{ steps.go.outputs.go-version }}-${{ hashFiles('go.sum', 'Makefile', 'app/**/Makefile') }}
-          restore-keys: go-artifacts-${{ runner.os }}-codeql-analyze-
-
-      - name: Initialize CodeQL
-        uses: github/codeql-action/init@v3
-        with:
-          languages: go
-
-      - name: Autobuild
-        uses: github/codeql-action/autobuild@v3
-
-      - name: Perform CodeQL Analysis
-        uses: github/codeql-action/analyze@v3
-        with:
-          category: 'language:go'

.github/workflows/codeql-analysis-js-ts.yml

@@ -1,46 +0,0 @@
-name: 'CodeQL JS/TS'
-
-on:
-  push:
-    branches:
-      - cluster
-      - master
-    paths:
-      - '**.js'
-      - '**.ts'
-      - '**.tsx'
-  pull_request:
-    branches:
-      - cluster
-      - master
-    paths:
-      - '**.js'
-      - '**.ts'
-      - '**.tsx'
-
-concurrency:
-  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
-  cancel-in-progress: true
-
-jobs:
-  analyze:
-    name: Analyze
-    runs-on: ubuntu-latest
-    permissions:
-      actions: read
-      contents: read
-      security-events: write
-
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@v4
-
-      - name: Initialize CodeQL
-        uses: github/codeql-action/init@v3
-        with:
-          languages: javascript-typescript
-
-      - name: Perform CodeQL Analysis
-        uses: github/codeql-action/analyze@v3
-        with:
-          category: 'language:js/ts'

.github/workflows/github-pages.yml

@@ -0,0 +1,30 @@
+name: github-pages
+on:
+  push:
+    paths:
+      - 'docs/*'
+      - 'README.md'
+    branches:
+      - master
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@master
+      - name: publish
+        shell: bash
+        env:
+          TOKEN: ${{secrets.CI_TOKEN}}
+        run: |
+          git clone https://vika:${TOKEN}@github.com/VictoriaMetrics/VictoriaMetrics.github.io.git gpages
+          cp docs/* gpages
+          cp README.md gpages
+          cd gpages
+          git config --local user.email "info@victoriametrics.com"
+          git config --local user.name "Vika"
+          git add .
+          git commit -m "update github pages"
+          remote_repo="https://vika:${TOKEN}@github.com/VictoriaMetrics/VictoriaMetrics.github.io.git"
+          git push "${remote_repo}"
+          cd ..
+          rm -rf gpages

.github/workflows/main.yml

@@ -1,93 +1,67 @@
 name: main
-
 on:
   push:
-    branches:
-      - cluster
-      - master
-    paths:
-      - '**.go'
+    paths-ignore:
+      - 'docs/**'
+      - '**.md'
   pull_request:
-    branches:
-      - cluster
-      - master
-    paths:
-      - '**.go'
-
-permissions:
-  contents: read
-
-concurrency:
-  cancel-in-progress: true
-  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
-
+    paths-ignore:
+      - 'docs/**'
+      - '**.md'
 jobs:
-  lint:
-    name: lint
+  build:
+    name: Build
     runs-on: ubuntu-latest
     steps:
-      - name: Code checkout
-        uses: actions/checkout@v4
-
       - name: Setup Go
+        uses: actions/setup-go@main
+        with:
+          go-version: 1.16
         id: go
-        uses: actions/setup-go@v5
-        with:
-          cache: false
-          go-version: stable
-
-      - name: Cache Go artifacts
-        uses: actions/cache@v4
-        with:
-          path: |
-            ~/.cache/go-build
-            ~/go/bin
-            ~/go/pkg/mod
-          key: go-artifacts-${{ runner.os }}-check-all-${{ steps.go.outputs.go-version }}-${{ hashFiles('go.sum', 'Makefile', 'app/**/Makefile') }}
-          restore-keys: go-artifacts-${{ runner.os }}-check-all-
-
-      - name: Run check-all
+      - name: Dependencies
         run: |
+          go get -u golang.org/x/lint/golint
+          go get -u github.com/kisielk/errcheck
+          curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s -- -b $(go env GOPATH)/bin v1.29.0
+      - name: Code checkout
+        uses: actions/checkout@master
+      - name: Build
+        env:
+          GO111MODULE: on
+        run: |
+          export PATH=$PATH:$(go env GOPATH)/bin # temporary fix. See https://github.com/actions/setup-go/issues/14
           make check-all
           git diff --exit-code
-
-  test:
-    name: test
-    needs: lint
-    runs-on: ubuntu-latest
-
-    strategy:
-      matrix:
-        scenario:
-          - 'test-full'
-          - 'test-full-386'
-          - 'test-pure'
-
-    steps:
-      - name: Code checkout
-        uses: actions/checkout@v4
-
-      - name: Setup Go
-        id: go
-        uses: actions/setup-go@v5
-        with:
-          cache: false
-          go-version: stable
-
-      - name: Cache Go artifacts
-        uses: actions/cache@v4
-        with:
-          path: |
-            ~/.cache/go-build
-            ~/go/bin
-            ~/go/pkg/mod
-          key: go-artifacts-${{ runner.os }}-${{ matrix.scenario }}-${{ steps.go.outputs.go-version }}-${{ hashFiles('go.sum', 'Makefile', 'app/**/Makefile') }}
-          restore-keys: go-artifacts-${{ runner.os }}-${{ matrix.scenario }}-
-
-      - name: Run tests
-        run: make ${{ matrix.scenario}}
-
+          make test-full
+          make test-pure
+          make test-full-386
+          make victoria-metrics
+          make victoria-metrics-pure
+          make victoria-metrics-arm
+          make victoria-metrics-arm64
+          make vmutils
+          GOOS=freebsd go build -mod=vendor ./app/victoria-metrics
+          GOOS=freebsd go build -mod=vendor ./app/vmagent
+          GOOS=freebsd go build -mod=vendor ./app/vmalert
+          GOOS=freebsd go build -mod=vendor ./app/vmbackup
+          GOOS=freebsd go build -mod=vendor ./app/vmrestore
+          GOOS=freebsd go build -mod=vendor ./app/vmctl
+          GOOS=openbsd go build -mod=vendor ./app/victoria-metrics
+          GOOS=openbsd go build -mod=vendor ./app/vmagent
+          GOOS=openbsd go build -mod=vendor ./app/vmalert
+          GOOS=openbsd go build -mod=vendor ./app/vmbackup
+          GOOS=openbsd go build -mod=vendor ./app/vmrestore
+          GOOS=openbsd go build -mod=vendor ./app/vmctl
+          GOOS=darwin go build -mod=vendor ./app/victoria-metrics
+          GOOS=darwin go build -mod=vendor ./app/vmagent
+          GOOS=darwin go build -mod=vendor ./app/vmalert
+          GOOS=darwin go build -mod=vendor ./app/vmbackup
+          GOOS=darwin go build -mod=vendor ./app/vmrestore
+          GOOS=darwin go build -mod=vendor ./app/vmctl
+          CGO_ENABLED=0 GOOS=windows go build -mod=vendor ./app/vmagent
       - name: Publish coverage
-        uses: codecov/codecov-action@v4
+        uses: codecov/codecov-action@v1.0.6
         with:
+          token: ${{secrets.CODECOV_TOKEN}}
           file: ./coverage.txt
+

.github/workflows/sync-docs.yml

@@ -1,51 +0,0 @@
-name: publish-docs
-on:
-  push:
-    branches:
-      - 'master'
-    paths:
-      - 'docs/**'
-  workflow_dispatch: {}
-permissions:
-  contents: read  # This is required for actions/checkout and to commit back image update
-  deployments: write
-jobs:
-  build:
-    name: Build
-    runs-on: ubuntu-latest
-    steps:
-      - name: Code checkout
-        uses: actions/checkout@v4
-        with:
-          path: main
-      - name: Checkout private code
-        uses: actions/checkout@v4
-        with:
-          repository: VictoriaMetrics/vmdocs
-          token: ${{ secrets.VM_BOT_GH_TOKEN }}
-          path: docs
-      - name: Import GPG key
-        uses: crazy-max/ghaction-import-gpg@v6
-        with:
-          gpg_private_key: ${{ secrets.VM_BOT_GPG_PRIVATE_KEY }}
-          passphrase: ${{ secrets.VM_BOT_PASSPHRASE }}
-          git_user_signingkey: true
-          git_commit_gpgsign: true
-          workdir: docs
-      - name: Set short git commit SHA
-        id: vars
-        run: |
-          calculatedSha=$(git rev-parse --short ${{ github.sha }})
-          echo "short_sha=$calculatedSha" >> $GITHUB_OUTPUT
-        working-directory: main
-      - name: update code and commit
-        run: |
-          rm -rf content
-          cp -r ../main/docs content
-          make clean-after-copy
-          git config --global user.name "${{ steps.import-gpg.outputs.email }}"
-          git config --global user.email "${{ steps.import-gpg.outputs.email }}"
-          git add .
-          git commit -S -m "sync docs with VictoriaMetrics/VictoriaMetrics commit: ${{ steps.vars.outputs.short_sha }}"
-          git push
-        working-directory: docs

.github/workflows/wiki.yml

@@ -0,0 +1,28 @@
+name: wiki
+on:
+  push:
+    paths:
+      - 'docs/*'
+    branches:
+      - master
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@master
+      - name: publish
+        shell: bash
+        env:
+          TOKEN: ${{secrets.CI_TOKEN}}
+        run: |
+          git clone https://vika:${TOKEN}@github.com/VictoriaMetrics/VictoriaMetrics.wiki.git wiki
+          cp docs/* wiki
+          cd wiki
+          git config --local user.email "info@victoriametrics.com"
+          git config --local user.name "Vika"
+          git add .
+          git commit -m "update wiki pages"
+          remote_repo="https://vika:${TOKEN}@github.com/VictoriaMetrics/VictoriaMetrics.wiki.git"
+          git push "${remote_repo}"
+          cd ..
+          rm -rf wiki

.gitignore

@@ -4,12 +4,9 @@
 *.pprof
 /bin
 .idea
-.vscode
 *.test
 *.swp
-/vmdocs
 /gocache-for-docker
-/victoria-logs-data
 /victoria-metrics-data
 /vmagent-remotewrite-data
 /vmstorage-data
@@ -18,11 +15,3 @@
 /package/temp-rpm-*
 /package/*.deb
 /package/*.rpm
-.DS_store
-Gemfile.lock
-/_site
-_site
-*.tmp
-/docs/.jekyll-metadata
-coverage.txt
-cspell.json

.golangci.yml

@@ -1,22 +0,0 @@
-run:
-  timeout: 2m
-
-linters:
-  enable:
-  - revive
-
-issues:
-  exclude-rules:
-  - linters:
-    - staticcheck
-    text: "SA(4003|1019|5011):"
-  include:
-  - EXC0012
-  - EXC0014
-
-linters-settings:
-  errcheck:
-    exclude-functions:
-      - "fmt.Fprintf"
-      - "fmt.Fprint"
-      - "(net/http.ResponseWriter).Write"

.wwhrd.yml

@@ -1,6 +0,0 @@
-allowlist:
-  - Apache-2.0
-  - MIT
-  - BSD-3-Clause
-  - BSD-2-Clause
-  - ISC

CODE_OF_CONDUCT.md

@@ -7,7 +7,7 @@ contributors and maintainers pledge to making participation in our project and
 our community a harassment-free experience for everyone, regardless of age, body
 size, disability, ethnicity, sex characteristics, gender identity and expression,
 level of experience, education, socio-economic status, nationality, personal
-appearance, race, religion or sexual identity and orientation.
+appearance, race, religion, or sexual identity and orientation.
 
 ## Our Standards
 
@@ -24,9 +24,9 @@ Examples of unacceptable behavior by participants include:
 
 * The use of sexualized language or imagery and unwelcome sexual attention or
  advances
-* Trolling, insulting/derogatory comments and personal or political attacks
+* Trolling, insulting/derogatory comments, and personal or political attacks
 * Public or private harassment
-* Publishing others' private information, such as physical or electronic
+* Publishing others' private information, such as a physical or electronic
  address, without explicit permission
 * Other conduct which could reasonably be considered inappropriate in a
  professional setting
@@ -38,26 +38,26 @@ behavior and are expected to take appropriate and fair corrective action in
 response to any instances of unacceptable behavior.
 
 Project maintainers have the right and responsibility to remove, edit, or
-reject comments, commits, code, wiki edits, issues and other contributions
-that are not aligned to this Code of Conduct or to ban temporarily or
+reject comments, commits, code, wiki edits, issues, and other contributions
+that are not aligned to this Code of Conduct, or to ban temporarily or
 permanently any contributor for other behaviors that they deem inappropriate,
-threatening, offensive or harmful.
+threatening, offensive, or harmful.
 
 ## Scope
 
 This Code of Conduct applies both within project spaces and in public spaces
 when an individual is representing the project or its community. Examples of
 representing a project or community include using an official project e-mail
-address, posting via an official social media account or acting as an appointed
+address, posting via an official social media account, or acting as an appointed
 representative at an online or offline event. Representation of a project may be
 further defined and clarified by project maintainers.
 
 ## Enforcement
 
-Instances of abusive, harassing or otherwise unacceptable behavior may be
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
 reported by contacting the project team at info@victoriametrics.com. All
 complaints will be reviewed and investigated and will result in a response that
-is deemed necessary and appropriate for the circumstances. The project team is
+is deemed necessary and appropriate to the circumstances. The project team is
 obligated to maintain confidentiality with regard to the reporter of an incident.
 Further details of specific enforcement policies may be posted separately.
 
@@ -68,9 +68,9 @@ members of the project's leadership.
 ## Attribution
 
 This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
-available at <https://www.contributor-covenant.org/version/1/4/code-of-conduct.html>
+available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html
 
 [homepage]: https://www.contributor-covenant.org
 
 For answers to common questions about this code of conduct, see
-<https://www.contributor-covenant.org/faq>
+https://www.contributor-covenant.org/faq

CODE_OF_CONDUCT_RU.md

@@ -0,0 +1,120 @@
+
+# Кодекс Поведения участника
+
+## Наши обязательства
+
+Мы, как участники, авторы и лидеры обязуемся сделать участие в сообществе
+свободным от притеснений для всех, независимо от возраста, телосложения,
+видимых или невидимых ограничений способности, этнической принадлежности,
+половых признаков, гендерной идентичности и выражения, уровня опыта,
+образования, социо-экономического статуса, национальности, внешности,
+расы, религии, или сексуальной идентичности и ориентации.
+
+Мы обещаем действовать и взаимодействовать таким образом, чтобы вносить вклад в открытое,
+дружелюбное, многообразное, инклюзивное и здоровое сообщество.
+
+## Наши стандарты
+
+Примеры поведения, создающие условия для благоприятных взаимоотношений включают в себя:
+
+* Проявление доброты и эмпатии к другим участникам проекта
+* Уважение к чужой точке зрения и опыту
+* Конструктивная критика и принятие конструктивной критики
+* Принятие ответственности, принесение извинений тем, кто пострадал от наших ошибок
+  и извлечение уроков из опыта
+* Ориентирование на то, что лучше подходит для сообщества, а не только для нас лично
+
+Примеры неприемлемого поведения участников включают в себя:
+
+* Использование выражений или изображений сексуального характера и нежелательное сексуальное внимание или домогательство в любой форме
+* Троллинг, оскорбительные или уничижительные комментарии, переход на личности или затрагивание политических убеждений
+* Публичное или приватное домогательство
+* Публикация личной информации других лиц, например, физического или электронного адреса, без явного разрешения
+* Иное поведение, которое обоснованно считать неуместным в профессиональной обстановке
+
+## Обязанности
+
+Лидеры сообщества отвечают за разъяснение и применение наших стандартов приемлемого
+поведения и будут предпринимать соответствующие и честные меры по исправлению положения
+в ответ на любое поведение, которое они сочтут неприемлемым, угрожающим, оскорбительным или вредным.
+
+Лидеры сообщества обладают правом и обязанностью удалять, редактировать или отклонять
+комментарии, коммиты, код, изменения в вики, вопросы и другой вклад, который не совпадает
+с Кодексом Поведения, и предоставят причины принятого решения, когда сочтут нужным.
+
+## Область применения
+
+Данный Кодекс Поведения применим во всех во всех публичных физических и цифровых пространства сообщества,
+а также когда человек официально представляет сообщество в публичных местах.
+Примеры представления проекта или сообщества включают использование официальной электронной почты,
+публикации в официальном аккаунте в социальных сетях,
+или упоминания как представителя в онлайн или оффлайн мероприятии.
+
+## Приведение в исполнение
+
+О случаях домогательства, а так же оскорбительного или иного другого неприемлемого
+поведения можно сообщить ответственным лидерам сообщества с помощью письма на info@victoriametrics.com
+Все жалобы будут рассмотрены и расследованы оперативно и беспристрастно.
+
+Все лидеры сообщества обязаны уважать неприкосновенность частной жизни и личную
+неприкосновенность автора сообщения.
+
+## Руководство по исполнению
+
+Лидеры сообщества будут следовать следующим Принципам Воздействия в Сообществе,
+чтобы определить последствия для тех, кого они считают виновными в нарушении данного Кодекса Поведения:
+
+### 1. Исправление
+
+**Общественное влияние**: Использование недопустимой лексики или другое поведение,
+считающиеся непрофессиональным или нежелательным в сообществе.
+
+**Последствия**: Личное, письменное предупреждение от лидеров сообщества,
+объясняющее суть нарушения и почему такое поведение
+было неуместно. Лидеры сообщества могут попросить принести публичное извинение.
+
+### 2. Предупреждение
+
+**Общественное влияние**: Нарушение в результате одного инцидента или серии действий.
+
+**Последствия**: Предупреждение о последствиях в случае продолжающегося неуместного поведения.
+На определенное время не допускается взаимодействие с людьми, вовлеченными в инцидент,
+включая незапрошенное взаимодействие 
+с теми, кто обеспечивает соблюдение Кодекса. Это включает в себя избегание взаимодействия
+в публичных пространствах, а так же во внешних каналах,
+таких как социальные сети. Нарушение этих правил влечет за собой временный или вечный бан.
+
+### 3. Временный бан
+
+**Общественное влияние**: Серьёзное нарушение стандартов сообщества,
+включая продолжительное неуместное поведение.
+
+**Последствия**: Временный запрет (бан) на любое взаимодействие 
+или публичное общение с сообществом на определенный период времени.
+На этот период не допускается публичное или личное взаимодействие с людьми,
+вовлеченными в инцидент, включая незапрошенное взаимодействие 
+с теми, кто обеспечивает соблюдение Кодекса.
+Нарушение этих правил влечет за собой вечный бан.
+
+### 4. Вечный бан
+
+**Общественное влияние**: Демонстрация систематических нарушений стандартов сообщества,
+включая продолжающееся неуместное поведение, домогательство до отдельных лиц,
+или проявление агрессии либо пренебрежительного отношения к категориям лиц.
+
+**Последствия**: Вечный запрет на любое публичное взаимодействие с сообществом.
+
+## Атрибуция
+
+Данный Кодекс Поведения основан на [Кодекс Поведения участника][homepage],
+версии 2.0, доступной по адресу
+https://www.contributor-covenant.org/version/2/0/code_of_conduct.html.
+
+Принципы Воздействия в Сообществе были вдохновлены [Mozilla's code of conduct
+enforcement ladder](https://github.com/mozilla/diversity).
+
+[homepage]: https://www.contributor-covenant.org
+
+Ответы на общие вопросы о данном кодексе поведения ищите на странице FAQ:
+https://www.contributor-covenant.org/faq. Переводы доступны по адресу
+https://www.contributor-covenant.org/translations.

CONTRIBUTING.md

@@ -1 +1,16 @@
-The document has been moved [here](https://docs.victoriametrics.com/contributing/).
+If you like VictoriaMetrics and want to contribute, then we need the following:
+
+- Filing issues and feature requests [here](https://github.com/VictoriaMetrics/VictoriaMetrics/issues).
+- Spreading a word about VictoriaMetrics: conference talks, articles, comments, experience sharing with colleagues.
+- Updating documentation.
+
+We are open to third-party pull requests provided they follow [KISS design principle](https://en.wikipedia.org/wiki/KISS_principle):
+
+- Prefer simple code and architecture.
+- Avoid complex abstractions.
+- Avoid magic code and fancy algorithms.
+- Avoid [big external dependencies](https://medium.com/@valyala/stripping-dependency-bloat-in-victoriametrics-docker-image-983fb5912b0d).
+- Minimize the number of moving parts in the distributed system.
+- Avoid automated decisions, which may hurt cluster availability, consistency or performance.
+
+Adhering `KISS` principle simplifies the resulting code and architecture, so it can be reviewed, understood and verified by many people.

LICENSE

@@ -175,7 +175,7 @@
 
    END OF TERMS AND CONDITIONS
 
-   Copyright 2019-2024 VictoriaMetrics, Inc.
+   Copyright 2019-2021 VictoriaMetrics, Inc.
 
    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.

Makefile

@@ -1,40 +1,29 @@
 PKG_PREFIX := github.com/VictoriaMetrics/VictoriaMetrics
 
-MAKE_CONCURRENCY ?= $(shell getconf _NPROCESSORS_ONLN)
-MAKE_PARALLEL := $(MAKE) -j $(MAKE_CONCURRENCY)
-DATEINFO_TAG ?= $(shell date -u +'%Y%m%d-%H%M%S')
 BUILDINFO_TAG ?= $(shell echo $$(git describe --long --all | tr '/' '-')$$( \
-	      git diff-index --quiet HEAD -- || echo '-dirty-'$$(git diff-index -u HEAD | openssl sha1 | cut -d' ' -f2 | cut -c 1-8)))
-LATEST_TAG ?= latest
+	      git diff-index --quiet HEAD -- || echo '-dirty-'$$(git diff-index -u HEAD | openssl sha1 | cut -c 10-17)))
 
 PKG_TAG ?= $(shell git tag -l --points-at HEAD)
 ifeq ($(PKG_TAG),)
 PKG_TAG := $(BUILDINFO_TAG)
 endif
 
-GO_BUILDINFO = -X '$(PKG_PREFIX)/lib/buildinfo.Version=$(APP_NAME)-$(DATEINFO_TAG)-$(BUILDINFO_TAG)'
-TAR_OWNERSHIP ?= --owner=1000 --group=1000
+GO_BUILDINFO = -X '$(PKG_PREFIX)/lib/buildinfo.Version=$(APP_NAME)-$(shell date -u +'%Y%m%d-%H%M%S')-$(BUILDINFO_TAG)'
 
 .PHONY: $(MAKECMDGOALS)
 
-include app/*/Makefile
-include cspell/Makefile
-include docs/Makefile
-include deployment/*/Makefile
-include dashboards/Makefile
-include package/release/Makefile
-
 all: \
 	victoria-metrics-prod \
-	victoria-logs-prod \
 	vmagent-prod \
 	vmalert-prod \
-	vmalert-tool-prod \
 	vmauth-prod \
 	vmbackup-prod \
 	vmrestore-prod \
 	vmctl-prod
 
+include app/*/Makefile
+include deployment/*/Makefile
+
 clean:
 	rm -rf bin/*
 
@@ -42,7 +31,6 @@ publish: \
 	publish-victoria-metrics \
 	publish-vmagent \
 	publish-vmalert \
-	publish-vmalert-tool \
 	publish-vmauth \
 	publish-vmbackup \
 	publish-vmrestore \
@@ -50,10 +38,8 @@ publish: \
 
 package: \
 	package-victoria-metrics \
-	package-victoria-logs \
 	package-vmagent \
 	package-vmalert \
-	package-vmalert-tool \
 	package-vmauth \
 	package-vmbackup \
 	package-vmrestore \
@@ -62,455 +48,170 @@ package: \
 vmutils: \
 	vmagent \
 	vmalert \
-	vmalert-tool \
 	vmauth \
 	vmbackup \
 	vmrestore \
 	vmctl
 
-vmutils-pure: \
-	vmagent-pure \
-	vmalert-pure \
-	vmalert-tool-pure \
-	vmauth-pure \
-	vmbackup-pure \
-	vmrestore-pure \
-	vmctl-pure
+vmutils-arm64: \
+	vmagent-arm64 \
+	vmalert-arm64 \
+	vmauth-arm64 \
+	vmbackup-arm64 \
+	vmrestore-arm64 \
+	vmctl-arm64
 
-vmutils-linux-amd64: \
-	vmagent-linux-amd64 \
-	vmalert-linux-amd64 \
-	vmalert-tool-linux-amd64 \
-	vmauth-linux-amd64 \
-	vmbackup-linux-amd64 \
-	vmrestore-linux-amd64 \
-	vmctl-linux-amd64
+release-snap:
+	snapcraft
+	snapcraft upload "victoriametrics_$(PKG_TAG)_multi.snap" --release beta,edge,candidate
 
-vmutils-linux-arm64: \
-	vmagent-linux-arm64 \
-	vmalert-linux-arm64 \
-	vmalert-tool-linux-arm64 \
-	vmauth-linux-arm64 \
-	vmbackup-linux-arm64 \
-	vmrestore-linux-arm64 \
-	vmctl-linux-arm64
-
-vmutils-linux-arm: \
-	vmagent-linux-arm \
-	vmalert-linux-arm \
-	vmalert-tool-linux-arm \
-	vmauth-linux-arm \
-	vmbackup-linux-arm \
-	vmrestore-linux-arm \
-	vmctl-linux-arm
-
-vmutils-linux-386: \
-	vmagent-linux-386 \
-	vmalert-linux-386 \
-	vmalert-tool-linux-386 \
-	vmauth-linux-386 \
-	vmbackup-linux-386 \
-	vmrestore-linux-386 \
-	vmctl-linux-386
-
-vmutils-linux-ppc64le: \
-	vmagent-linux-ppc64le \
-	vmalert-linux-ppc64le \
-	vmalert-tool-linux-ppc64le \
-	vmauth-linux-ppc64le \
-	vmbackup-linux-ppc64le \
-	vmrestore-linux-ppc64le \
-	vmctl-linux-ppc64le
-
-vmutils-darwin-amd64: \
-	vmagent-darwin-amd64 \
-	vmalert-darwin-amd64 \
-	vmalert-tool-darwin-amd64 \
-	vmauth-darwin-amd64 \
-	vmbackup-darwin-amd64 \
-	vmrestore-darwin-amd64 \
-	vmctl-darwin-amd64
-
-vmutils-darwin-arm64: \
-	vmagent-darwin-arm64 \
-	vmalert-darwin-arm64 \
-	vmalert-tool-darwin-arm64 \
-	vmauth-darwin-arm64 \
-	vmbackup-darwin-arm64 \
-	vmrestore-darwin-arm64 \
-	vmctl-darwin-arm64
-
-vmutils-freebsd-amd64: \
-	vmagent-freebsd-amd64 \
-	vmalert-freebsd-amd64 \
-	vmalert-tool-freebsd-amd64 \
-	vmauth-freebsd-amd64 \
-	vmbackup-freebsd-amd64 \
-	vmrestore-freebsd-amd64 \
-	vmctl-freebsd-amd64
-
-vmutils-openbsd-amd64: \
-	vmagent-openbsd-amd64 \
-	vmalert-openbsd-amd64 \
-	vmalert-tool-openbsd-amd64 \
-	vmauth-openbsd-amd64 \
-	vmbackup-openbsd-amd64 \
-	vmrestore-openbsd-amd64 \
-	vmctl-openbsd-amd64
-
-vmutils-windows-amd64: \
-	vmagent-windows-amd64 \
-	vmalert-windows-amd64 \
-	vmalert-tool-windows-amd64 \
-	vmauth-windows-amd64 \
-	vmbackup-windows-amd64 \
-	vmrestore-windows-amd64 \
-	vmctl-windows-amd64
-
-crossbuild:
-	$(MAKE_PARALLEL) victoria-metrics-crossbuild vmutils-crossbuild
-
-victoria-metrics-crossbuild: \
-	victoria-metrics-linux-386 \
-	victoria-metrics-linux-amd64 \
-	victoria-metrics-linux-arm64 \
-	victoria-metrics-linux-arm \
-	victoria-metrics-linux-ppc64le \
-	victoria-metrics-darwin-amd64 \
-	victoria-metrics-darwin-arm64 \
-	victoria-metrics-freebsd-amd64 \
-	victoria-metrics-openbsd-amd64 \
-	victoria-metrics-windows-amd64
-
-vmutils-crossbuild: \
-	vmutils-linux-386 \
-	vmutils-linux-amd64 \
-	vmutils-linux-arm64 \
-	vmutils-linux-arm \
-	vmutils-linux-ppc64le \
-	vmutils-darwin-amd64 \
-	vmutils-darwin-arm64 \
-	vmutils-freebsd-amd64 \
-	vmutils-openbsd-amd64 \
-	vmutils-windows-amd64
-
-publish-release:
-	rm -rf bin/*
-	git checkout $(TAG) && $(MAKE) release && LATEST_TAG=stable $(MAKE) publish && \
-		git checkout $(TAG)-cluster && $(MAKE) release && LATEST_TAG=cluster-stable $(MAKE) publish && \
-		git checkout $(TAG)-enterprise && $(MAKE) release && LATEST_TAG=enterprise-stable $(MAKE) publish && \
-		git checkout $(TAG)-enterprise-cluster && $(MAKE) release && LATEST_TAG=enterprise-cluster-stable $(MAKE) publish
-
-release:
-	$(MAKE_PARALLEL) \
-		release-victoria-metrics \
-		release-vmutils
+release: \
+	release-victoria-metrics \
+	release-vmutils
 
 release-victoria-metrics: \
-	release-victoria-metrics-linux-386 \
-	release-victoria-metrics-linux-amd64 \
-	release-victoria-metrics-linux-arm \
-	release-victoria-metrics-linux-arm64 \
-	release-victoria-metrics-darwin-amd64 \
-	release-victoria-metrics-darwin-arm64 \
-	release-victoria-metrics-freebsd-amd64 \
-	release-victoria-metrics-openbsd-amd64 \
-	release-victoria-metrics-windows-amd64
+	release-victoria-metrics-amd64 \
+	release-victoria-metrics-arm64
 
-release-victoria-metrics-linux-386:
-	GOOS=linux GOARCH=386 $(MAKE) release-victoria-metrics-goos-goarch
+release-victoria-metrics-amd64:
+	GOARCH=amd64 $(MAKE) release-victoria-metrics-generic
 
-release-victoria-metrics-linux-amd64:
-	GOOS=linux GOARCH=amd64 $(MAKE) release-victoria-metrics-goos-goarch
+release-victoria-metrics-arm64:
+	GOARCH=arm64 $(MAKE) release-victoria-metrics-generic
 
-release-victoria-metrics-linux-arm:
-	GOOS=linux GOARCH=arm $(MAKE) release-victoria-metrics-goos-goarch
-
-release-victoria-metrics-linux-arm64:
-	GOOS=linux GOARCH=arm64 $(MAKE) release-victoria-metrics-goos-goarch
-
-release-victoria-metrics-darwin-amd64:
-	GOOS=darwin GOARCH=amd64 $(MAKE) release-victoria-metrics-goos-goarch
-
-release-victoria-metrics-darwin-arm64:
-	GOOS=darwin GOARCH=arm64 $(MAKE) release-victoria-metrics-goos-goarch
-
-release-victoria-metrics-freebsd-amd64:
-	GOOS=freebsd GOARCH=amd64 $(MAKE) release-victoria-metrics-goos-goarch
-
-release-victoria-metrics-openbsd-amd64:
-	GOOS=openbsd GOARCH=amd64 $(MAKE) release-victoria-metrics-goos-goarch
-
-release-victoria-metrics-windows-amd64:
-	GOARCH=amd64 $(MAKE) release-victoria-metrics-windows-goarch
-
-release-victoria-metrics-goos-goarch: victoria-metrics-$(GOOS)-$(GOARCH)-prod
+release-victoria-metrics-generic: victoria-metrics-$(GOARCH)-prod
 	cd bin && \
-		tar $(TAR_OWNERSHIP) --transform="flags=r;s|-$(GOOS)-$(GOARCH)||" -czf victoria-metrics-$(GOOS)-$(GOARCH)-$(PKG_TAG).tar.gz \
-			victoria-metrics-$(GOOS)-$(GOARCH)-prod \
-		&& sha256sum victoria-metrics-$(GOOS)-$(GOARCH)-$(PKG_TAG).tar.gz \
-			victoria-metrics-$(GOOS)-$(GOARCH)-prod \
-			| sed s/-$(GOOS)-$(GOARCH)-prod/-prod/ > victoria-metrics-$(GOOS)-$(GOARCH)-$(PKG_TAG)_checksums.txt
-	cd bin && rm -rf victoria-metrics-$(GOOS)-$(GOARCH)-prod
-
-release-victoria-metrics-windows-goarch: victoria-metrics-windows-$(GOARCH)-prod
-	cd bin && \
-		zip victoria-metrics-windows-$(GOARCH)-$(PKG_TAG).zip \
-			victoria-metrics-windows-$(GOARCH)-prod.exe \
-		&& sha256sum victoria-metrics-windows-$(GOARCH)-$(PKG_TAG).zip \
-			victoria-metrics-windows-$(GOARCH)-prod.exe \
-			> victoria-metrics-windows-$(GOARCH)-$(PKG_TAG)_checksums.txt
-	cd bin && rm -rf \
-		victoria-metrics-windows-$(GOARCH)-prod.exe
-
-release-victoria-logs:
-	$(MAKE_PARALLEL) release-victoria-logs-linux-386 \
-		release-victoria-logs-linux-amd64 \
-		release-victoria-logs-linux-arm \
-		release-victoria-logs-linux-arm64 \
-		release-victoria-logs-darwin-amd64 \
-		release-victoria-logs-darwin-arm64 \
-		release-victoria-logs-freebsd-amd64 \
-		release-victoria-logs-openbsd-amd64 \
-		release-victoria-logs-windows-amd64
-
-release-victoria-logs-linux-386:
-	GOOS=linux GOARCH=386 $(MAKE) release-victoria-logs-goos-goarch
-
-release-victoria-logs-linux-amd64:
-	GOOS=linux GOARCH=amd64 $(MAKE) release-victoria-logs-goos-goarch
-
-release-victoria-logs-linux-arm:
-	GOOS=linux GOARCH=arm $(MAKE) release-victoria-logs-goos-goarch
-
-release-victoria-logs-linux-arm64:
-	GOOS=linux GOARCH=arm64 $(MAKE) release-victoria-logs-goos-goarch
-
-release-victoria-logs-darwin-amd64:
-	GOOS=darwin GOARCH=amd64 $(MAKE) release-victoria-logs-goos-goarch
-
-release-victoria-logs-darwin-arm64:
-	GOOS=darwin GOARCH=arm64 $(MAKE) release-victoria-logs-goos-goarch
-
-release-victoria-logs-freebsd-amd64:
-	GOOS=freebsd GOARCH=amd64 $(MAKE) release-victoria-logs-goos-goarch
-
-release-victoria-logs-openbsd-amd64:
-	GOOS=openbsd GOARCH=amd64 $(MAKE) release-victoria-logs-goos-goarch
-
-release-victoria-logs-windows-amd64:
-	GOARCH=amd64 $(MAKE) release-victoria-logs-windows-goarch
-
-release-victoria-logs-goos-goarch: victoria-logs-$(GOOS)-$(GOARCH)-prod
-	cd bin && \
-		tar $(TAR_OWNERSHIP) --transform="flags=r;s|-$(GOOS)-$(GOARCH)||" -czf victoria-logs-$(GOOS)-$(GOARCH)-$(PKG_TAG).tar.gz \
-			victoria-logs-$(GOOS)-$(GOARCH)-prod \
-		&& sha256sum victoria-logs-$(GOOS)-$(GOARCH)-$(PKG_TAG).tar.gz \
-			victoria-logs-$(GOOS)-$(GOARCH)-prod \
-			| sed s/-$(GOOS)-$(GOARCH)-prod/-prod/ > victoria-logs-$(GOOS)-$(GOARCH)-$(PKG_TAG)_checksums.txt
-	cd bin && rm -rf victoria-logs-$(GOOS)-$(GOARCH)-prod
-
-release-victoria-logs-windows-goarch: victoria-logs-windows-$(GOARCH)-prod
-	cd bin && \
-		zip victoria-logs-windows-$(GOARCH)-$(PKG_TAG).zip \
-			victoria-logs-windows-$(GOARCH)-prod.exe \
-		&& sha256sum victoria-logs-windows-$(GOARCH)-$(PKG_TAG).zip \
-			victoria-logs-windows-$(GOARCH)-prod.exe \
-			> victoria-logs-windows-$(GOARCH)-$(PKG_TAG)_checksums.txt
-	cd bin && rm -rf \
-		victoria-logs-windows-$(GOARCH)-prod.exe
+		tar --transform="flags=r;s|-$(GOARCH)||" -czf victoria-metrics-$(GOARCH)-$(PKG_TAG).tar.gz \
+			victoria-metrics-$(GOARCH)-prod \
+		&& sha256sum victoria-metrics-$(GOARCH)-$(PKG_TAG).tar.gz \
+			victoria-metrics-$(GOARCH)-prod \
+			| sed s/-$(GOARCH)// > victoria-metrics-$(GOARCH)-$(PKG_TAG)_checksums.txt
 
 release-vmutils: \
-	release-vmutils-linux-386 \
-	release-vmutils-linux-amd64 \
-	release-vmutils-linux-arm64 \
-	release-vmutils-linux-arm \
-	release-vmutils-darwin-amd64 \
-	release-vmutils-darwin-arm64 \
-	release-vmutils-freebsd-amd64 \
-	release-vmutils-openbsd-amd64 \
-	release-vmutils-windows-amd64
+	release-vmutils-amd64 \
+	release-vmutils-arm64
 
-release-vmutils-linux-386:
-	GOOS=linux GOARCH=386 $(MAKE) release-vmutils-goos-goarch
-	
-release-vmutils-linux-amd64:
-	GOOS=linux GOARCH=amd64 $(MAKE) release-vmutils-goos-goarch
+release-vmutils-amd64:
+	GOARCH=amd64 $(MAKE) release-vmutils-generic
 
-release-vmutils-linux-arm64:
-	GOOS=linux GOARCH=arm64 $(MAKE) release-vmutils-goos-goarch
+release-vmutils-arm64:
+	GOARCH=arm64 $(MAKE) release-vmutils-generic
 
-release-vmutils-linux-arm:
-	GOOS=linux GOARCH=arm $(MAKE) release-vmutils-goos-goarch
-
-release-vmutils-darwin-amd64:
-	GOOS=darwin GOARCH=amd64 $(MAKE) release-vmutils-goos-goarch
-
-release-vmutils-darwin-arm64:
-	GOOS=darwin GOARCH=arm64 $(MAKE) release-vmutils-goos-goarch
-
-release-vmutils-freebsd-amd64:
-	GOOS=freebsd GOARCH=amd64 $(MAKE) release-vmutils-goos-goarch
-
-release-vmutils-openbsd-amd64:
-	GOOS=openbsd GOARCH=amd64 $(MAKE) release-vmutils-goos-goarch
-
-release-vmutils-windows-amd64:
-	GOARCH=amd64 $(MAKE) release-vmutils-windows-goarch
-
-release-vmutils-goos-goarch: \
-	vmagent-$(GOOS)-$(GOARCH)-prod \
-	vmalert-$(GOOS)-$(GOARCH)-prod \
-	vmalert-tool-$(GOOS)-$(GOARCH)-prod \
-	vmauth-$(GOOS)-$(GOARCH)-prod \
-	vmbackup-$(GOOS)-$(GOARCH)-prod \
-	vmrestore-$(GOOS)-$(GOARCH)-prod \
-	vmctl-$(GOOS)-$(GOARCH)-prod
+release-vmutils-generic: \
+	vmagent-$(GOARCH)-prod \
+	vmalert-$(GOARCH)-prod \
+	vmauth-$(GOARCH)-prod \
+	vmbackup-$(GOARCH)-prod \
+	vmrestore-$(GOARCH)-prod \
+	vmctl-$(GOARCH)-prod
 	cd bin && \
-		tar $(TAR_OWNERSHIP) --transform="flags=r;s|-$(GOOS)-$(GOARCH)||" -czf vmutils-$(GOOS)-$(GOARCH)-$(PKG_TAG).tar.gz \
-			vmagent-$(GOOS)-$(GOARCH)-prod \
-			vmalert-$(GOOS)-$(GOARCH)-prod \
-			vmalert-tool-$(GOOS)-$(GOARCH)-prod \
-			vmauth-$(GOOS)-$(GOARCH)-prod \
-			vmbackup-$(GOOS)-$(GOARCH)-prod \
-			vmrestore-$(GOOS)-$(GOARCH)-prod \
-			vmctl-$(GOOS)-$(GOARCH)-prod \
-		&& sha256sum vmutils-$(GOOS)-$(GOARCH)-$(PKG_TAG).tar.gz \
-			vmagent-$(GOOS)-$(GOARCH)-prod \
-			vmalert-$(GOOS)-$(GOARCH)-prod \
-			vmalert-tool-$(GOOS)-$(GOARCH)-prod \
-			vmauth-$(GOOS)-$(GOARCH)-prod \
-			vmbackup-$(GOOS)-$(GOARCH)-prod \
-			vmrestore-$(GOOS)-$(GOARCH)-prod \
-			vmctl-$(GOOS)-$(GOARCH)-prod \
-			| sed s/-$(GOOS)-$(GOARCH)-prod/-prod/ > vmutils-$(GOOS)-$(GOARCH)-$(PKG_TAG)_checksums.txt
-	cd bin && rm -rf \
-		vmagent-$(GOOS)-$(GOARCH)-prod \
-		vmalert-$(GOOS)-$(GOARCH)-prod \
-		vmalert-tool-$(GOOS)-$(GOARCH)-prod \
-		vmauth-$(GOOS)-$(GOARCH)-prod \
-		vmbackup-$(GOOS)-$(GOARCH)-prod \
-		vmrestore-$(GOOS)-$(GOARCH)-prod \
-		vmctl-$(GOOS)-$(GOARCH)-prod
-
-release-vmutils-windows-goarch: \
-	vmagent-windows-$(GOARCH)-prod \
-	vmalert-windows-$(GOARCH)-prod \
-	vmalert-tool-windows-$(GOARCH)-prod \
-	vmauth-windows-$(GOARCH)-prod \
-	vmbackup-windows-$(GOARCH)-prod \
-	vmrestore-windows-$(GOARCH)-prod \
-	vmctl-windows-$(GOARCH)-prod
-	cd bin && \
-		zip vmutils-windows-$(GOARCH)-$(PKG_TAG).zip \
-			vmagent-windows-$(GOARCH)-prod.exe \
-			vmalert-windows-$(GOARCH)-prod.exe \
-			vmalert-tool-windows-$(GOARCH)-prod.exe \
-			vmauth-windows-$(GOARCH)-prod.exe \
-			vmbackup-windows-$(GOARCH)-prod.exe \
-			vmrestore-windows-$(GOARCH)-prod.exe \
-			vmctl-windows-$(GOARCH)-prod.exe \
-		&& sha256sum vmutils-windows-$(GOARCH)-$(PKG_TAG).zip \
-			vmagent-windows-$(GOARCH)-prod.exe \
-			vmalert-windows-$(GOARCH)-prod.exe \
-			vmalert-tool-windows-$(GOARCH)-prod.exe \
-			vmauth-windows-$(GOARCH)-prod.exe \
-			vmbackup-windows-$(GOARCH)-prod.exe \
-			vmrestore-windows-$(GOARCH)-prod.exe \
-			vmctl-windows-$(GOARCH)-prod.exe \
-			> vmutils-windows-$(GOARCH)-$(PKG_TAG)_checksums.txt
-	cd bin && rm -rf \
-		vmagent-windows-$(GOARCH)-prod.exe \
-		vmalert-windows-$(GOARCH)-prod.exe \
-		vmalert-tool-windows-$(GOARCH)-prod.exe \
-		vmauth-windows-$(GOARCH)-prod.exe \
-		vmbackup-windows-$(GOARCH)-prod.exe \
-		vmrestore-windows-$(GOARCH)-prod.exe \
-		vmctl-windows-$(GOARCH)-prod.exe
+		tar --transform="flags=r;s|-$(GOARCH)||" -czf vmutils-$(GOARCH)-$(PKG_TAG).tar.gz \
+			vmagent-$(GOARCH)-prod \
+			vmalert-$(GOARCH)-prod \
+			vmauth-$(GOARCH)-prod \
+			vmbackup-$(GOARCH)-prod \
+			vmrestore-$(GOARCH)-prod \
+			vmctl-$(GOARCH)-prod \
+		&& sha256sum vmutils-$(GOARCH)-$(PKG_TAG).tar.gz \
+			vmagent-$(GOARCH)-prod \
+			vmalert-$(GOARCH)-prod \
+			vmauth-$(GOARCH)-prod \
+			vmbackup-$(GOARCH)-prod \
+			vmrestore-$(GOARCH)-prod \
+			vmctl-$(GOARCH)-prod \
+			| sed s/-$(GOARCH)// > vmutils-$(GOARCH)-$(PKG_TAG)_checksums.txt
 
 pprof-cpu:
 	go tool pprof -trim_path=github.com/VictoriaMetrics/VictoriaMetrics@ $(PPROF_FILE)
 
 fmt:
-	gofmt -l -w -s ./lib
-	gofmt -l -w -s ./app
+	GO111MODULE=on gofmt -l -w -s ./lib
+	GO111MODULE=on gofmt -l -w -s ./app
 
 vet:
-	go vet ./lib/...
-	go vet ./app/...
+	GO111MODULE=on go vet -mod=vendor ./lib/...
+	GO111MODULE=on go vet -mod=vendor ./app/...
 
-check-all: fmt vet golangci-lint govulncheck
+lint: install-golint
+	golint lib/...
+	golint app/...
 
-clean-checkers: remove-golangci-lint remove-govulncheck
+install-golint:
+	which golint || go install golang.org/x/lint/golint
+
+errcheck: install-errcheck
+	errcheck -exclude=errcheck_excludes.txt ./lib/...
+	errcheck -exclude=errcheck_excludes.txt ./app/vminsert/...
+	errcheck -exclude=errcheck_excludes.txt ./app/vmselect/...
+	errcheck -exclude=errcheck_excludes.txt ./app/vmstorage/...
+	errcheck -exclude=errcheck_excludes.txt ./app/vmagent/...
+	errcheck -exclude=errcheck_excludes.txt ./app/vmalert/...
+	errcheck -exclude=errcheck_excludes.txt ./app/vmauth/...
+	errcheck -exclude=errcheck_excludes.txt ./app/vmbackup/...
+	errcheck -exclude=errcheck_excludes.txt ./app/vmrestore/...
+	errcheck -exclude=errcheck_excludes.txt ./app/vmctl/...
+
+install-errcheck:
+	which errcheck || go install github.com/kisielk/errcheck
+
+check-all: fmt vet lint errcheck golangci-lint
 
 test:
-	go test ./lib/... ./app/...
+	GO111MODULE=on go test -mod=vendor ./lib/... ./app/...
 
 test-race:
-	go test -race ./lib/... ./app/...
+	GO111MODULE=on go test -mod=vendor -race ./lib/... ./app/...
 
 test-pure:
-	CGO_ENABLED=0 go test ./lib/... ./app/...
+	GO111MODULE=on CGO_ENABLED=0 go test -mod=vendor ./lib/... ./app/...
 
 test-full:
-	go test -coverprofile=coverage.txt -covermode=atomic ./lib/... ./app/...
+	GO111MODULE=on go test -mod=vendor -coverprofile=coverage.txt -covermode=atomic ./lib/... ./app/...
 
 test-full-386:
-	GOARCH=386 go test -coverprofile=coverage.txt -covermode=atomic ./lib/... ./app/...
+	GO111MODULE=on GOARCH=386 go test -mod=vendor -coverprofile=coverage.txt -covermode=atomic ./lib/... ./app/...
 
 benchmark:
-	go test -bench=. ./lib/...
-	go test -bench=. ./app/...
+	GO111MODULE=on go test -mod=vendor -bench=. ./lib/...
+	GO111MODULE=on go test -mod=vendor -bench=. ./app/...
 
 benchmark-pure:
-	CGO_ENABLED=0 go test -bench=. ./lib/...
-	CGO_ENABLED=0 go test -bench=. ./app/...
+	GO111MODULE=on CGO_ENABLED=0 go test -mod=vendor -bench=. ./lib/...
+	GO111MODULE=on CGO_ENABLED=0 go test -mod=vendor -bench=. ./app/...
 
 vendor-update:
-	go get -u -d ./lib/...
-	go get -u -d ./app/...
-	go mod tidy -compat=1.22
-	go mod vendor
+	GO111MODULE=on go get -u -d ./lib/...
+	GO111MODULE=on go get -u -d ./app/...
+	GO111MODULE=on go mod tidy
+	GO111MODULE=on go mod vendor
 
 app-local:
-	CGO_ENABLED=1 go build $(RACE) -ldflags "$(GO_BUILDINFO)" -o bin/$(APP_NAME)$(RACE) $(PKG_PREFIX)/app/$(APP_NAME)
+	CGO_ENABLED=1 GO111MODULE=on go build $(RACE) -mod=vendor -ldflags "$(GO_BUILDINFO)" -o bin/$(APP_NAME)$(RACE) $(PKG_PREFIX)/app/$(APP_NAME)
 
 app-local-pure:
-	CGO_ENABLED=0 go build $(RACE) -ldflags "$(GO_BUILDINFO)" -o bin/$(APP_NAME)-pure$(RACE) $(PKG_PREFIX)/app/$(APP_NAME)
+	CGO_ENABLED=0 GO111MODULE=on go build $(RACE) -mod=vendor -ldflags "$(GO_BUILDINFO)" -o bin/$(APP_NAME)-pure$(RACE) $(PKG_PREFIX)/app/$(APP_NAME)
 
-app-local-goos-goarch:
-	CGO_ENABLED=$(CGO_ENABLED) GOOS=$(GOOS) GOARCH=$(GOARCH) go build $(RACE) -ldflags "$(GO_BUILDINFO)" -o bin/$(APP_NAME)-$(GOOS)-$(GOARCH)$(RACE) $(PKG_PREFIX)/app/$(APP_NAME)
-
-app-local-windows-goarch:
-	CGO_ENABLED=0 GOOS=windows GOARCH=$(GOARCH) go build $(RACE) -ldflags "$(GO_BUILDINFO)" -o bin/$(APP_NAME)-windows-$(GOARCH)$(RACE).exe $(PKG_PREFIX)/app/$(APP_NAME)
+app-local-with-goarch:
+	GO111MODULE=on go build $(RACE) -mod=vendor -ldflags "$(GO_BUILDINFO)" -o bin/$(APP_NAME)-$(GOARCH)$(RACE) $(PKG_PREFIX)/app/$(APP_NAME)
 
 quicktemplate-gen: install-qtc
 	qtc
 
 install-qtc:
-	which qtc || go install github.com/valyala/quicktemplate/qtc@latest
+	which qtc || go install github.com/valyala/quicktemplate/qtc
 
 
 golangci-lint: install-golangci-lint
-	golangci-lint run
+	golangci-lint run --exclude '(SA4003|SA1019|SA5011):' -D errcheck -D structcheck --timeout 2m
 
 install-golangci-lint:
-	which golangci-lint || curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s -- -b $(shell go env GOPATH)/bin v1.60.1
+	which golangci-lint || curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s -- -b $(shell go env GOPATH)/bin v1.29.0
 
-remove-golangci-lint:
-	rm -rf `which golangci-lint`
-
-govulncheck: install-govulncheck
-	govulncheck ./...
-
-install-govulncheck:
-	which govulncheck || go install golang.org/x/vuln/cmd/govulncheck@latest
-
-remove-govulncheck:
-	rm -rf `which govulncheck`
-
-install-wwhrd:
-	which wwhrd || go install github.com/frapposelli/wwhrd@latest
-
-check-licenses: install-wwhrd
-	wwhrd check -f .wwhrd.yml
+docs-sync:
+	cp app/vmagent/README.md docs/vmagent.md
+	cp app/vmalert/README.md docs/vmalert.md
+	cp app/vmauth/README.md docs/vmauth.md
+	cp app/vmbackup/README.md docs/vmbackup.md
+	cp app/vmrestore/README.md docs/vmrestore.md
+	cp app/vmctl/README.md docs/vmctl.md
+	cp README.md docs/Single-server-VictoriaMetrics.md

SECURITY.md

@@ -1,18 +0,0 @@
-# Security Policy
-
-## Supported Versions
-
-The following versions of VictoriaMetrics receive regular security fixes:
-
-| Version | Supported          |
-|---------|--------------------|
-| [latest release](https://docs.victoriametrics.com/changelog/) | :white_check_mark: |
-| v1.102.x [LTS line](https://docs.victoriametrics.com/lts-releases/) | :white_check_mark: |
-| v1.97.x [LTS line](https://docs.victoriametrics.com/lts-releases/) | :white_check_mark: |
-| other releases  | :x:                |
-
-See [this page](https://victoriametrics.com/security/) for more details.
-
-## Reporting a Vulnerability
-
-Please report any security issues to <security@victoriametrics.com>

app/victoria-logs/Makefile

@@ -1,113 +0,0 @@
-# All these commands must run from repository root.
-
-victoria-logs:
-	APP_NAME=victoria-logs $(MAKE) app-local
-
-victoria-logs-race:
-	APP_NAME=victoria-logs RACE=-race $(MAKE) app-local
-
-victoria-logs-prod:
-	APP_NAME=victoria-logs $(MAKE) app-via-docker
-
-victoria-logs-pure-prod:
-	APP_NAME=victoria-logs $(MAKE) app-via-docker-pure
-
-victoria-logs-linux-amd64-prod:
-	APP_NAME=victoria-logs $(MAKE) app-via-docker-linux-amd64
-
-victoria-logs-linux-arm-prod:
-	APP_NAME=victoria-logs $(MAKE) app-via-docker-linux-arm
-
-victoria-logs-linux-arm64-prod:
-	APP_NAME=victoria-logs $(MAKE) app-via-docker-linux-arm64
-
-victoria-logs-linux-ppc64le-prod:
-	APP_NAME=victoria-logs $(MAKE) app-via-docker-linux-ppc64le
-
-victoria-logs-linux-386-prod:
-	APP_NAME=victoria-logs $(MAKE) app-via-docker-linux-386
-
-victoria-logs-darwin-amd64-prod:
-	APP_NAME=victoria-logs $(MAKE) app-via-docker-darwin-amd64
-
-victoria-logs-darwin-arm64-prod:
-	APP_NAME=victoria-logs $(MAKE) app-via-docker-darwin-arm64
-
-victoria-logs-freebsd-amd64-prod:
-	APP_NAME=victoria-logs $(MAKE) app-via-docker-freebsd-amd64
-
-victoria-logs-openbsd-amd64-prod:
-	APP_NAME=victoria-logs $(MAKE) app-via-docker-openbsd-amd64
-
-victoria-logs-windows-amd64-prod:
-	APP_NAME=victoria-logs $(MAKE) app-via-docker-windows-amd64
-
-package-victoria-logs:
-	APP_NAME=victoria-logs $(MAKE) package-via-docker
-
-package-victoria-logs-pure:
-	APP_NAME=victoria-logs $(MAKE) package-via-docker-pure
-
-package-victoria-logs-amd64:
-	APP_NAME=victoria-logs $(MAKE) package-via-docker-amd64
-
-package-victoria-logs-arm:
-	APP_NAME=victoria-logs $(MAKE) package-via-docker-arm
-
-package-victoria-logs-arm64:
-	APP_NAME=victoria-logs $(MAKE) package-via-docker-arm64
-
-package-victoria-logs-ppc64le:
-	APP_NAME=victoria-logs $(MAKE) package-via-docker-ppc64le
-
-package-victoria-logs-386:
-	APP_NAME=victoria-logs $(MAKE) package-via-docker-386
-
-publish-victoria-logs:
-	APP_NAME=victoria-logs $(MAKE) publish-via-docker
-
-victoria-logs-linux-amd64:
-	APP_NAME=victoria-logs CGO_ENABLED=1 GOOS=linux GOARCH=amd64 $(MAKE) app-local-goos-goarch
-
-victoria-logs-linux-arm:
-	APP_NAME=victoria-logs CGO_ENABLED=0 GOOS=linux GOARCH=arm $(MAKE) app-local-goos-goarch
-
-victoria-logs-linux-arm64:
-	APP_NAME=victoria-logs CGO_ENABLED=0 GOOS=linux GOARCH=arm64 $(MAKE) app-local-goos-goarch
-
-victoria-logs-linux-ppc64le:
-	APP_NAME=victoria-logs CGO_ENABLED=0 GOOS=linux GOARCH=ppc64le $(MAKE) app-local-goos-goarch
-
-victoria-logs-linux-s390x:
-	APP_NAME=victoria-logs CGO_ENABLED=0 GOOS=linux GOARCH=s390x $(MAKE) app-local-goos-goarch
-
-victoria-logs-linux-loong64:
-	APP_NAME=victoria-logs CGO_ENABLED=0 GOOS=linux GOARCH=loong64 $(MAKE) app-local-goos-goarch
-
-victoria-logs-linux-386:
-	APP_NAME=victoria-logs CGO_ENABLED=0 GOOS=linux GOARCH=386 $(MAKE) app-local-goos-goarch
-
-victoria-logs-darwin-amd64:
-	APP_NAME=victoria-logs CGO_ENABLED=0 GOOS=darwin GOARCH=amd64 $(MAKE) app-local-goos-goarch
-
-victoria-logs-darwin-arm64:
-	APP_NAME=victoria-logs CGO_ENABLED=0 GOOS=darwin GOARCH=arm64 $(MAKE) app-local-goos-goarch
-
-victoria-logs-freebsd-amd64:
-	APP_NAME=victoria-logs CGO_ENABLED=0 GOOS=freebsd GOARCH=amd64 $(MAKE) app-local-goos-goarch
-
-victoria-logs-openbsd-amd64:
-	APP_NAME=victoria-logs CGO_ENABLED=0 GOOS=openbsd GOARCH=amd64 $(MAKE) app-local-goos-goarch
-
-victoria-logs-windows-amd64:
-	GOARCH=amd64 APP_NAME=victoria-logs $(MAKE) app-local-windows-goarch
-
-victoria-logs-pure:
-	APP_NAME=victoria-logs $(MAKE) app-local-pure
-
-run-victoria-logs:
-	mkdir -p victoria-logs-data
-	DOCKER_OPTS='-v $(shell pwd)/victoria-logs-data:/victoria-logs-data' \
-	APP_NAME=victoria-logs \
-	ARGS='' \
-	$(MAKE) run-via-docker

app/victoria-logs/deployment/Dockerfile

@@ -1,8 +0,0 @@
-ARG base_image
-FROM $base_image
-
-EXPOSE 9428
-
-ENTRYPOINT ["/victoria-logs-prod"]
-ARG src_binary
-COPY $src_binary ./victoria-logs-prod

app/victoria-logs/main.go

@@ -1,105 +0,0 @@
-package main
-
-import (
-	"flag"
-	"fmt"
-	"net/http"
-	"os"
-	"time"
-
-	"github.com/VictoriaMetrics/VictoriaMetrics/app/vlinsert"
-	"github.com/VictoriaMetrics/VictoriaMetrics/app/vlselect"
-	"github.com/VictoriaMetrics/VictoriaMetrics/app/vlstorage"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/buildinfo"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/envflag"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/flagutil"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/fs"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/procutil"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/pushmetrics"
-)
-
-var (
-	httpListenAddrs  = flagutil.NewArrayString("httpListenAddr", "TCP address to listen for incoming http requests. See also -httpListenAddr.useProxyProtocol")
-	useProxyProtocol = flagutil.NewArrayBool("httpListenAddr.useProxyProtocol", "Whether to use proxy protocol for connections accepted at the given -httpListenAddr . "+
-		"See https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt . "+
-		"With enabled proxy protocol http server cannot serve regular /metrics endpoint. Use -pushmetrics.url for metrics pushing")
-)
-
-func main() {
-	// Write flags and help message to stdout, since it is easier to grep or pipe.
-	flag.CommandLine.SetOutput(os.Stdout)
-	flag.Usage = usage
-	envflag.Parse()
-	buildinfo.Init()
-	logger.Init()
-
-	listenAddrs := *httpListenAddrs
-	if len(listenAddrs) == 0 {
-		listenAddrs = []string{":9428"}
-	}
-	logger.Infof("starting VictoriaLogs at %q...", listenAddrs)
-	startTime := time.Now()
-
-	vlstorage.Init()
-	vlselect.Init()
-	vlinsert.Init()
-
-	go httpserver.Serve(listenAddrs, useProxyProtocol, requestHandler)
-	logger.Infof("started VictoriaLogs in %.3f seconds; see https://docs.victoriametrics.com/victorialogs/", time.Since(startTime).Seconds())
-
-	pushmetrics.Init()
-	sig := procutil.WaitForSigterm()
-	logger.Infof("received signal %s", sig)
-	pushmetrics.Stop()
-
-	logger.Infof("gracefully shutting down webservice at %q", listenAddrs)
-	startTime = time.Now()
-	if err := httpserver.Stop(listenAddrs); err != nil {
-		logger.Fatalf("cannot stop the webservice: %s", err)
-	}
-	logger.Infof("successfully shut down the webservice in %.3f seconds", time.Since(startTime).Seconds())
-
-	vlinsert.Stop()
-	vlselect.Stop()
-	vlstorage.Stop()
-
-	fs.MustStopDirRemover()
-
-	logger.Infof("the VictoriaLogs has been stopped in %.3f seconds", time.Since(startTime).Seconds())
-}
-
-func requestHandler(w http.ResponseWriter, r *http.Request) bool {
-	if r.URL.Path == "/" {
-		if r.Method != http.MethodGet {
-			return false
-		}
-		w.Header().Add("Content-Type", "text/html; charset=utf-8")
-		fmt.Fprintf(w, "<h2>Single-node VictoriaLogs</h2></br>")
-		fmt.Fprintf(w, "See docs at <a href='https://docs.victoriametrics.com/victorialogs/'>https://docs.victoriametrics.com/victorialogs/</a></br>")
-		fmt.Fprintf(w, "Useful endpoints:</br>")
-		httpserver.WriteAPIHelp(w, [][2]string{
-			{"select/vmui", "Web UI for VictoriaLogs"},
-			{"metrics", "available service metrics"},
-			{"flags", "command-line flags"},
-		})
-		return true
-	}
-	if vlinsert.RequestHandler(w, r) {
-		return true
-	}
-	if vlselect.RequestHandler(w, r) {
-		return true
-	}
-	return false
-}
-
-func usage() {
-	const s = `
-victoria-logs is a log management and analytics service.
-
-See the docs at https://docs.victoriametrics.com/victorialogs/
-`
-	flagutil.Usage(s)
-}

app/victoria-logs/multiarch/Dockerfile

@@ -1,12 +0,0 @@
-# See https://medium.com/on-docker/use-multi-stage-builds-to-inject-ca-certs-ad1e8f01de1b
-ARG certs_image
-ARG root_image
-FROM $certs_image AS certs
-RUN apk update && apk upgrade && apk --update --no-cache add ca-certificates
-
-FROM $root_image
-COPY --from=certs /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
-EXPOSE 9428
-ENTRYPOINT ["/victoria-logs-prod"]
-ARG TARGETARCH
-COPY victoria-logs-linux-${TARGETARCH}-prod ./victoria-logs-prod

app/victoria-metrics/Makefile

@@ -12,35 +12,20 @@ victoria-metrics-prod:
 victoria-metrics-pure-prod:
 	APP_NAME=victoria-metrics $(MAKE) app-via-docker-pure
 
-victoria-metrics-linux-amd64-prod:
-	APP_NAME=victoria-metrics $(MAKE) app-via-docker-linux-amd64
+victoria-metrics-amd64-prod:
+	APP_NAME=victoria-metrics $(MAKE) app-via-docker-amd64
 
-victoria-metrics-linux-arm-prod:
-	APP_NAME=victoria-metrics $(MAKE) app-via-docker-linux-arm
+victoria-metrics-arm-prod:
+	APP_NAME=victoria-metrics $(MAKE) app-via-docker-arm
 
-victoria-metrics-linux-arm64-prod:
-	APP_NAME=victoria-metrics $(MAKE) app-via-docker-linux-arm64
+victoria-metrics-arm64-prod:
+	APP_NAME=victoria-metrics $(MAKE) app-via-docker-arm64
 
-victoria-metrics-linux-ppc64le-prod:
-	APP_NAME=victoria-metrics $(MAKE) app-via-docker-linux-ppc64le
+victoria-metrics-ppc64le-prod:
+	APP_NAME=victoria-metrics $(MAKE) app-via-docker-ppc64le
 
-victoria-metrics-linux-386-prod:
-	APP_NAME=victoria-metrics $(MAKE) app-via-docker-linux-386
-
-victoria-metrics-darwin-amd64-prod:
-	APP_NAME=victoria-metrics $(MAKE) app-via-docker-darwin-amd64
-
-victoria-metrics-darwin-arm64-prod:
-	APP_NAME=victoria-metrics $(MAKE) app-via-docker-darwin-arm64
-
-victoria-metrics-freebsd-amd64-prod:
-	APP_NAME=victoria-metrics $(MAKE) app-via-docker-freebsd-amd64
-
-victoria-metrics-openbsd-amd64-prod:
-	APP_NAME=victoria-metrics $(MAKE) app-via-docker-openbsd-amd64
-
-victoria-metrics-windows-amd64-prod:
-	APP_NAME=victoria-metrics $(MAKE) app-via-docker-windows-amd64
+victoria-metrics-386-prod:
+	APP_NAME=victoria-metrics $(MAKE) app-via-docker-386
 
 package-victoria-metrics:
 	APP_NAME=victoria-metrics $(MAKE) package-via-docker
@@ -73,77 +58,60 @@ run-victoria-metrics:
 	ARGS='-graphiteListenAddr=:2003 -opentsdbListenAddr=:4242 -retentionPeriod=12 -search.maxUniqueTimeseries=1000000 -search.maxQueryDuration=10m' \
 	$(MAKE) run-via-docker
 
-victoria-metrics-linux-amd64:
-	APP_NAME=victoria-metrics CGO_ENABLED=1 GOOS=linux GOARCH=amd64 $(MAKE) app-local-goos-goarch
+victoria-metrics-amd64:
+	CGO_ENABLED=1 GOOS=linux GOARCH=amd64 GO111MODULE=on go build -mod=vendor -ldflags "$(GO_BUILDINFO)" -o bin/victoria-metrics-amd64 ./app/victoria-metrics
 
-victoria-metrics-linux-arm:
-	APP_NAME=victoria-metrics CGO_ENABLED=0 GOOS=linux GOARCH=arm $(MAKE) app-local-goos-goarch
+victoria-metrics-arm:
+	CGO_ENABLED=0 GOOS=linux GOARCH=arm GO111MODULE=on go build -mod=vendor -ldflags "$(GO_BUILDINFO)" -o bin/victoria-metrics-arm ./app/victoria-metrics
 
-victoria-metrics-linux-arm64:
-	APP_NAME=victoria-metrics CGO_ENABLED=0 GOOS=linux GOARCH=arm64 $(MAKE) app-local-goos-goarch
+victoria-metrics-arm64:
+	CGO_ENABLED=0 GOOS=linux GOARCH=arm64 GO111MODULE=on go build -mod=vendor -ldflags "$(GO_BUILDINFO)" -o bin/victoria-metrics-arm64 ./app/victoria-metrics
 
-victoria-metrics-linux-ppc64le:
-	APP_NAME=victoria-metrics CGO_ENABLED=0 GOOS=linux GOARCH=ppc64le $(MAKE) app-local-goos-goarch
+victoria-metrics-ppc64le:
+	CGO_ENABLED=0 GOOS=linux GOARCH=ppc64le GO111MODULE=on go build -mod=vendor -ldflags "$(GO_BUILDINFO)" -o bin/victoria-metrics-ppc64le ./app/victoria-metrics
 
-victoria-metrics-linux-s390x:
-	APP_NAME=victoria-metrics CGO_ENABLED=0 GOOS=linux GOARCH=s390x $(MAKE) app-local-goos-goarch
-
-victoria-metrics-linux-loong64:
-	APP_NAME=victoria-metrics CGO_ENABLED=0 GOOS=linux GOARCH=loong64 $(MAKE) app-local-goos-goarch
-
-victoria-metrics-linux-386:
-	APP_NAME=victoria-metrics CGO_ENABLED=0 GOOS=linux GOARCH=386 $(MAKE) app-local-goos-goarch
-
-victoria-metrics-darwin-amd64:
-	APP_NAME=victoria-metrics CGO_ENABLED=0 GOOS=darwin GOARCH=amd64 $(MAKE) app-local-goos-goarch
-
-victoria-metrics-darwin-arm64:
-	APP_NAME=victoria-metrics CGO_ENABLED=0 GOOS=darwin GOARCH=arm64 $(MAKE) app-local-goos-goarch
-
-victoria-metrics-freebsd-amd64:
-	APP_NAME=victoria-metrics CGO_ENABLED=0 GOOS=freebsd GOARCH=amd64 $(MAKE) app-local-goos-goarch
-
-victoria-metrics-openbsd-amd64:
-	APP_NAME=victoria-metrics CGO_ENABLED=0 GOOS=openbsd GOARCH=amd64 $(MAKE) app-local-goos-goarch
-
-victoria-metrics-windows-amd64:
-	GOARCH=amd64 APP_NAME=victoria-metrics $(MAKE) app-local-windows-goarch
+victoria-metrics-386:
+	CGO_ENABLED=0 GOOS=linux GOARCH=386 GO111MODULE=on go build -mod=vendor -ldflags "$(GO_BUILDINFO)" -o bin/victoria-metrics-386 ./app/victoria-metrics
 
 victoria-metrics-pure:
 	APP_NAME=victoria-metrics $(MAKE) app-local-pure
 
 ### Packaging as DEB - amd64
-victoria-metrics-package-deb-amd64: victoria-metrics-linux-amd64-prod
+victoria-metrics-package-deb: victoria-metrics-prod
 	./package/package_deb.sh amd64
 
 ### Packaging as DEB - arm64
-victoria-metrics-package-deb-arm: victoria-metrics-linux-arm-prod
-	./package/package_deb.sh arm
-
-### Packaging as DEB - arm64
-victoria-metrics-package-deb-arm64: victoria-metrics-linux-arm64-prod
+victoria-metrics-package-deb-arm64: victoria-metrics-arm64-prod
 	./package/package_deb.sh arm64
 
 ### Packaging as DEB - all
-victoria-metrics-package-deb: \
-        victoria-metrics-package-deb-amd64 \
-        victoria-metrics-package-deb-arm \
+victoria-metrics-package-deb-all: \
+        victoria-metrics-package-deb \
         victoria-metrics-package-deb-arm64
 
 ### Packaging as RPM - amd64
-victoria-metrics-package-rpm-amd64: victoria-metrics-linux-amd64-prod
+victoria-metrics-package-rpm: victoria-metrics-prod
 	./package/package_rpm.sh amd64
 
 ### Packaging as RPM - arm64
-victoria-metrics-package-rpm-arm64: victoria-metrics-linux-arm64-prod
+victoria-metrics-package-rpm-arm64: victoria-metrics-arm64-prod
 	./package/package_rpm.sh arm64
 
 ### Packaging as RPM - all
-victoria-metrics-package-rpm: \
-        victoria-metrics-package-rpm-amd64 \
+victoria-metrics-package-rpm-all: \
+        victoria-metrics-package-rpm \
         victoria-metrics-package-rpm-arm64
 
 ### Packaging as both DEB and RPM - all
-victoria-metrics-package-deb-rpm: \
+victoria-metrics-package-deb-rpm-all: \
         victoria-metrics-package-deb \
-        victoria-metrics-package-rpm
+        victoria-metrics-package-deb-arm64 \
+        victoria-metrics-package-rpm \
+        victoria-metrics-package-rpm-arm64
+
+### Packaging as snap
+victoria-metrics-package-snap:
+	which snapcraft || snap install snapcraft
+	which multipass || snap install multipass
+	snapcraft
+

app/victoria-metrics/main.go

@@ -3,13 +3,13 @@ package main
 import (
 	"flag"
 	"fmt"
+	"io"
 	"net/http"
 	"os"
+	"path"
 	"time"
 
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vminsert"
-	vminsertcommon "github.com/VictoriaMetrics/VictoriaMetrics/app/vminsert/common"
-	vminsertrelabel "github.com/VictoriaMetrics/VictoriaMetrics/app/vminsert/relabel"
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect"
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect/promql"
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmstorage"
@@ -21,26 +21,24 @@ import (
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/procutil"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/promscrape"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/pushmetrics"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/storage"
 )
 
 var (
-	httpListenAddrs  = flagutil.NewArrayString("httpListenAddr", "TCP addresses to listen for incoming http requests. See also -tls and -httpListenAddr.useProxyProtocol")
-	useProxyProtocol = flagutil.NewArrayBool("httpListenAddr.useProxyProtocol", "Whether to use proxy protocol for connections accepted at the corresponding -httpListenAddr . "+
-		"See https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt . "+
-		"With enabled proxy protocol http server cannot serve regular /metrics endpoint. Use -pushmetrics.url for metrics pushing")
-	minScrapeInterval = flag.Duration("dedup.minScrapeInterval", 0, "Leave only the last sample in every time series per each discrete interval "+
-		"equal to -dedup.minScrapeInterval > 0. See also -streamAggr.dedupInterval and https://docs.victoriametrics.com/#deduplication")
-	dryRun = flag.Bool("dryRun", false, "Whether to check config files without running VictoriaMetrics. The following config files are checked: "+
-		"-promscrape.config, -relabelConfig and -streamAggr.config. Unknown config entries aren't allowed in -promscrape.config by default. "+
-		"This can be changed with -promscrape.config.strictParse=false command-line flag")
-	inmemoryDataFlushInterval = flag.Duration("inmemoryDataFlushInterval", 5*time.Second, "The interval for guaranteed saving of in-memory data to disk. "+
-		"The saved data survives unclean shutdowns such as OOM crash, hardware reset, SIGKILL, etc. "+
-		"Bigger intervals may help increase the lifetime of flash storage with limited write cycles (e.g. Raspberry PI). "+
-		"Smaller intervals increase disk IO load. Minimum supported value is 1s")
+	httpListenAddr    = flag.String("httpListenAddr", ":8428", "TCP address to listen for http connections")
+	minScrapeInterval = flag.Duration("dedup.minScrapeInterval", 0, "Remove superflouos samples from time series if they are located closer to each other than this duration. "+
+		"This may be useful for reducing overhead when multiple identically configured Prometheus instances write data to the same VictoriaMetrics. "+
+		"Deduplication is disabled if the -dedup.minScrapeInterval is 0")
+	dryRun = flag.Bool("dryRun", false, "Whether to check only -promscrape.config and then exit. "+
+		"Unknown config entries are allowed in -promscrape.config by default. This can be changed with -promscrape.config.strictParse")
 )
 
+// custom api help links [["/api","doc"]] without http.pathPrefix.
+var customAPIPathList = [][]string{
+	{"/graph/explore", "explore metrics grafana page"},
+	{"/graph/d/prometheus-advanced/advanced-data-exploration", "PMM grafana dashboard"},
+}
+
 func main() {
 	// Write flags and help message to stdout, since it is easier to grep or pipe.
 	flag.CommandLine.SetOutput(os.Stdout)
@@ -56,47 +54,33 @@ func main() {
 		if err := promscrape.CheckConfig(); err != nil {
 			logger.Fatalf("error when checking -promscrape.config: %s", err)
 		}
-		if err := vminsertrelabel.CheckRelabelConfig(); err != nil {
-			logger.Fatalf("error when checking -relabelConfig: %s", err)
-		}
-		if err := vminsertcommon.CheckStreamAggrConfig(); err != nil {
-			logger.Fatalf("error when checking -streamAggr.config: %s", err)
-		}
-		logger.Infof("-promscrape.config is ok; exiting with 0 status code")
+		logger.Infof("-promscrape.config is ok; exitting with 0 status code")
 		return
 	}
 
-	listenAddrs := *httpListenAddrs
-	if len(listenAddrs) == 0 {
-		listenAddrs = []string{":8428"}
-	}
-	logger.Infof("starting VictoriaMetrics at %q...", listenAddrs)
+	logger.Infof("starting VictoriaMetrics at %q...", *httpListenAddr)
 	startTime := time.Now()
-	storage.SetDedupInterval(*minScrapeInterval)
-	storage.SetDataFlushInterval(*inmemoryDataFlushInterval)
+	storage.SetMinScrapeIntervalForDeduplication(*minScrapeInterval)
 	vmstorage.Init(promql.ResetRollupResultCacheIfNeeded)
 	vmselect.Init()
 	vminsert.Init()
-
 	startSelfScraper()
 
-	go httpserver.Serve(listenAddrs, useProxyProtocol, requestHandler)
+	go httpserver.Serve(*httpListenAddr, requestHandler)
 	logger.Infof("started VictoriaMetrics in %.3f seconds", time.Since(startTime).Seconds())
 
-	pushmetrics.Init()
 	sig := procutil.WaitForSigterm()
 	logger.Infof("received signal %s", sig)
-	pushmetrics.Stop()
 
 	stopSelfScraper()
 
-	logger.Infof("gracefully shutting down webservice at %q", listenAddrs)
+	logger.Infof("gracefully shutting down webservice at %q", *httpListenAddr)
 	startTime = time.Now()
-	if err := httpserver.Stop(listenAddrs); err != nil {
+	if err := httpserver.Stop(*httpListenAddr); err != nil {
 		logger.Fatalf("cannot stop the webservice: %s", err)
 	}
-	logger.Infof("successfully shut down the webservice in %.3f seconds", time.Since(startTime).Seconds())
 	vminsert.Stop()
+	logger.Infof("successfully shut down the webservice in %.3f seconds", time.Since(startTime).Seconds())
 
 	vmstorage.Stop()
 	vmselect.Stop()
@@ -108,27 +92,14 @@ func main() {
 
 func requestHandler(w http.ResponseWriter, r *http.Request) bool {
 	if r.URL.Path == "/" {
-		if r.Method != http.MethodGet {
-			return false
-		}
-		w.Header().Add("Content-Type", "text/html; charset=utf-8")
-		fmt.Fprintf(w, "<h2>Single-node VictoriaMetrics</h2></br>")
-		fmt.Fprintf(w, "See docs at <a href='https://docs.victoriametrics.com/'>https://docs.victoriametrics.com/</a></br>")
-		fmt.Fprintf(w, "Useful endpoints:</br>")
-		httpserver.WriteAPIHelp(w, [][2]string{
-			{"vmui", "Web UI"},
-			{"targets", "status for discovered active targets"},
-			{"service-discovery", "labels before and after relabeling for discovered targets"},
-			{"metric-relabel-debug", "debug metric relabeling"},
-			{"expand-with-exprs", "WITH expressions' tutorial"},
-			{"api/v1/targets", "advanced information about discovered targets in JSON format"},
-			{"config", "-promscrape.config contents"},
-			{"metrics", "available service metrics"},
-			{"flags", "command-line flags"},
-			{"api/v1/status/tsdb", "tsdb status page"},
-			{"api/v1/status/top_queries", "top queries"},
-			{"api/v1/status/active_queries", "active queries"},
-			{"-/reload", "reload configuration"},
+		fmt.Fprintf(w, "<h2>Single-node VictoriaMetrics.</h2></br>")
+		fmt.Fprintf(w, "See docs at <a href='https://victoriametrics.github.io/'>https://victoriametrics.github.io/</a></br>")
+		fmt.Fprintf(w, "Useful endpoints: </br>")
+		writeAPIHelp(w, [][]string{
+			{"/targets", "discovered targets list"},
+			{"/api/v1/targets", "advanced information about discovered targets in JSON format"},
+			{"/metrics", "available service metrics"},
+			{"/api/v1/status/tsdb", "tsdb status page"},
 		})
 		return true
 	}
@@ -144,11 +115,24 @@ func requestHandler(w http.ResponseWriter, r *http.Request) bool {
 	return false
 }
 
+func writeAPIHelp(w io.Writer, pathList [][]string) {
+	pathPrefix := httpserver.GetPathPrefix()
+	for _, p := range pathList {
+		p, doc := p[0], p[1]
+		p = path.Join(pathPrefix, p)
+		fmt.Fprintf(w, "<a href='%s'>%q</a> - %s<br/>", p, p, doc)
+	}
+	for _, p := range customAPIPathList {
+		p, doc := p[0], p[1]
+		fmt.Fprintf(w, "<a href='%s'>%q</a> - %s<br/>", p, p, doc)
+	}
+}
+
 func usage() {
 	const s = `
 victoria-metrics is a time series database and monitoring solution.
 
-See the docs at https://docs.victoriametrics.com/
+See the docs at https://victoriametrics.github.io/
 `
 	flagutil.Usage(s)
 }

app/victoria-metrics/main_test.go

@@ -3,18 +3,16 @@ package main
 import (
 	"bytes"
 	"encoding/json"
-	"errors"
 	"flag"
 	"fmt"
 	"io"
+	"io/ioutil"
 	"log"
-	"math/rand"
 	"net"
 	"net/http"
 	"os"
 	"path/filepath"
 	"reflect"
-	"strconv"
 	"strings"
 	"testing"
 	"time"
@@ -24,6 +22,7 @@ import (
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect"
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect/promql"
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmstorage"
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/envflag"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/fs"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
@@ -40,13 +39,11 @@ const (
 )
 
 const (
-	testReadHTTPPath           = "http://127.0.0.1" + testHTTPListenAddr
-	testWriteHTTPPath          = "http://127.0.0.1" + testHTTPListenAddr + "/write"
-	testOpenTSDBWriteHTTPPath  = "http://127.0.0.1" + testOpenTSDBHTTPListenAddr + "/api/put"
-	testPromWriteHTTPPath      = "http://127.0.0.1" + testHTTPListenAddr + "/api/v1/write"
-	testImportCSVWriteHTTPPath = "http://127.0.0.1" + testHTTPListenAddr + "/api/v1/import/csv"
-
-	testHealthHTTPPath = "http://127.0.0.1" + testHTTPListenAddr + "/health"
+	testReadHTTPPath          = "http://127.0.0.1" + testHTTPListenAddr
+	testWriteHTTPPath         = "http://127.0.0.1" + testHTTPListenAddr + "/write"
+	testOpenTSDBWriteHTTPPath = "http://127.0.0.1" + testOpenTSDBHTTPListenAddr + "/api/put"
+	testPromWriteHTTPPath     = "http://127.0.0.1" + testHTTPListenAddr + "/api/v1/write"
+	testHealthHTTPPath        = "http://127.0.0.1" + testHTTPListenAddr + "/health"
 )
 
 const (
@@ -59,15 +56,15 @@ var (
 )
 
 type test struct {
-	Name                     string   `json:"name"`
-	Data                     []string `json:"data"`
-	InsertQuery              string   `json:"insert_query"`
-	Query                    []string `json:"query"`
-	ResultMetrics            []Metric `json:"result_metrics"`
-	ResultSeries             Series   `json:"result_series"`
-	ResultQuery              Query    `json:"result_query"`
-	Issue                    string   `json:"issue"`
-	ExpectedResultLinesCount int      `json:"expected_result_lines_count"`
+	Name             string     `json:"name"`
+	Data             []string   `json:"data"`
+	InsertQuery      string     `json:"insert_query"`
+	Query            []string   `json:"query"`
+	ResultMetrics    []Metric   `json:"result_metrics"`
+	ResultSeries     Series     `json:"result_series"`
+	ResultQuery      Query      `json:"result_query"`
+	ResultQueryRange QueryRange `json:"result_query_range"`
+	Issue            string     `json:"issue"`
 }
 
 type Metric struct {
@@ -85,90 +82,42 @@ type Series struct {
 	Status string              `json:"status"`
 	Data   []map[string]string `json:"data"`
 }
-
 type Query struct {
-	Status string `json:"status"`
-	Data   struct {
-		ResultType string          `json:"resultType"`
-		Result     json.RawMessage `json:"result"`
-	} `json:"data"`
+	Status string    `json:"status"`
+	Data   QueryData `json:"data"`
+}
+type QueryData struct {
+	ResultType string            `json:"resultType"`
+	Result     []QueryDataResult `json:"result"`
 }
 
-const rtVector, rtMatrix = "vector", "matrix"
-
-func (q *Query) metrics() ([]Metric, error) {
-	switch q.Data.ResultType {
-	case rtVector:
-		var r QueryInstant
-		if err := json.Unmarshal(q.Data.Result, &r.Result); err != nil {
-			return nil, err
-		}
-		return r.metrics()
-	case rtMatrix:
-		var r QueryRange
-		if err := json.Unmarshal(q.Data.Result, &r.Result); err != nil {
-			return nil, err
-		}
-		return r.metrics()
-	default:
-		return nil, fmt.Errorf("unknown result type %q", q.Data.ResultType)
-	}
+type QueryDataResult struct {
+	Metric map[string]string `json:"metric"`
+	Value  []interface{}     `json:"value"`
 }
 
-type QueryInstant struct {
-	Result []struct {
-		Labels map[string]string `json:"metric"`
-		TV     [2]any            `json:"value"`
-	} `json:"result"`
-}
-
-func (q QueryInstant) metrics() ([]Metric, error) {
-	result := make([]Metric, len(q.Result))
-	for i, res := range q.Result {
-		f, err := strconv.ParseFloat(res.TV[1].(string), 64)
-		if err != nil {
-			return nil, fmt.Errorf("metric %v, unable to parse float64 from %s: %w", res, res.TV[1], err)
-		}
-		var m Metric
-		m.Metric = res.Labels
-		m.Timestamps = append(m.Timestamps, int64(res.TV[0].(float64)))
-		m.Values = append(m.Values, f)
-		result[i] = m
-	}
-	return result, nil
+func (r *QueryDataResult) UnmarshalJSON(b []byte) error {
+	type plain QueryDataResult
+	return json.Unmarshal(testutil.PopulateTimeTpl(b, insertionTime), (*plain)(r))
 }
 
 type QueryRange struct {
-	Result []struct {
-		Metric map[string]string `json:"metric"`
-		Values [][]any           `json:"values"`
-	} `json:"result"`
+	Status string         `json:"status"`
+	Data   QueryRangeData `json:"data"`
+}
+type QueryRangeData struct {
+	ResultType string                 `json:"resultType"`
+	Result     []QueryRangeDataResult `json:"result"`
 }
 
-func (q QueryRange) metrics() ([]Metric, error) {
-	var result []Metric
-	for i, res := range q.Result {
-		var m Metric
-		for _, tv := range res.Values {
-			f, err := strconv.ParseFloat(tv[1].(string), 64)
-			if err != nil {
-				return nil, fmt.Errorf("metric %v, unable to parse float64 from %s: %w", res, tv[1], err)
-			}
-			m.Values = append(m.Values, f)
-			m.Timestamps = append(m.Timestamps, int64(tv[0].(float64)))
-		}
-		if len(m.Values) < 1 || len(m.Timestamps) < 1 {
-			return nil, fmt.Errorf("metric %v contains no values", res)
-		}
-		m.Metric = q.Result[i].Metric
-		result = append(result, m)
-	}
-	return result, nil
+type QueryRangeDataResult struct {
+	Metric map[string]string `json:"metric"`
+	Values [][]interface{}   `json:"values"`
 }
 
-func (q *Query) UnmarshalJSON(b []byte) error {
-	type plain Query
-	return json.Unmarshal(testutil.PopulateTimeTpl(b, insertionTime), (*plain)(q))
+func (r *QueryRangeDataResult) UnmarshalJSON(b []byte) error {
+	type plain QueryRangeDataResult
+	return json.Unmarshal(testutil.PopulateTimeTpl(b, insertionTime), (*plain)(r))
 }
 
 func TestMain(m *testing.M) {
@@ -182,16 +131,16 @@ func setUp() {
 	storagePath = filepath.Join(os.TempDir(), testStorageSuffix)
 	processFlags()
 	logger.Init()
-	vmstorage.Init(promql.ResetRollupResultCacheIfNeeded)
+	vmstorage.InitWithoutMetrics(promql.ResetRollupResultCacheIfNeeded)
 	vmselect.Init()
 	vminsert.Init()
-	go httpserver.Serve(*httpListenAddrs, useProxyProtocol, requestHandler)
+	go httpserver.Serve(*httpListenAddr, requestHandler)
 	readyStorageCheckFunc := func() bool {
 		resp, err := http.Get(testHealthHTTPPath)
 		if err != nil {
 			return false
 		}
-		_ = resp.Body.Close()
+		resp.Body.Close()
 		return resp.StatusCode == 200
 	}
 	if err := waitFor(testStorageInitTimeout, readyStorageCheckFunc); err != nil {
@@ -200,7 +149,7 @@ func setUp() {
 }
 
 func processFlags() {
-	flag.Parse()
+	envflag.Parse()
 	for _, fv := range []struct {
 		flag  string
 		value string
@@ -231,7 +180,7 @@ func waitFor(timeout time.Duration, f func() bool) error {
 }
 
 func tearDown() {
-	if err := httpserver.Stop(*httpListenAddrs); err != nil {
+	if err := httpserver.Stop(*httpListenAddr); err != nil {
 		log.Printf("cannot stop the webservice: %s", err)
 	}
 	vminsert.Stop()
@@ -242,41 +191,31 @@ func tearDown() {
 
 func TestWriteRead(t *testing.T) {
 	t.Run("write", testWrite)
-	time.Sleep(500 * time.Millisecond)
-	vmstorage.Storage.DebugFlush()
-	time.Sleep(1500 * time.Millisecond)
+	time.Sleep(1 * time.Second)
+	vmstorage.Stop()
+	// open storage after stop in write
+	vmstorage.InitWithoutMetrics(promql.ResetRollupResultCacheIfNeeded)
 	t.Run("read", testRead)
 }
 
 func testWrite(t *testing.T) {
 	t.Run("prometheus", func(t *testing.T) {
-		for _, test := range readIn("prometheus", insertionTime) {
-			if test.Data == nil {
-				continue
-			}
+		for _, test := range readIn("prometheus", t, insertionTime) {
+			s := newSuite(t)
 			r := testutil.WriteRequest{}
-			testData := strings.Join(test.Data, "\n")
-			if err := json.Unmarshal([]byte(testData), &r.Timeseries); err != nil {
-				panic(fmt.Errorf("BUG: cannot unmarshal TimeSeries: %s\ntest data\n%s", err, testData))
+			s.noError(json.Unmarshal([]byte(strings.Join(test.Data, "\n")), &r.Timeseries))
+			data, err := testutil.Compress(r)
+			s.greaterThan(len(r.Timeseries), 0)
+			if err != nil {
+				t.Errorf("error compressing %v %s", r, err)
+				t.Fail()
 			}
-			if n := len(r.Timeseries); n <= 0 {
-				panic(fmt.Errorf("BUG: expecting non-empty Timeseries in test data:\n%s", testData))
-			}
-			data := testutil.Compress(r)
 			httpWrite(t, testPromWriteHTTPPath, test.InsertQuery, bytes.NewBuffer(data))
 		}
 	})
-	t.Run("csv", func(t *testing.T) {
-		for _, test := range readIn("csv", insertionTime) {
-			if test.Data == nil {
-				continue
-			}
-			httpWrite(t, testImportCSVWriteHTTPPath, test.InsertQuery, bytes.NewBuffer([]byte(strings.Join(test.Data, "\n"))))
-		}
-	})
 
 	t.Run("influxdb", func(t *testing.T) {
-		for _, x := range readIn("influxdb", insertionTime) {
+		for _, x := range readIn("influxdb", t, insertionTime) {
 			test := x
 			t.Run(test.Name, func(t *testing.T) {
 				t.Parallel()
@@ -285,7 +224,7 @@ func testWrite(t *testing.T) {
 		}
 	})
 	t.Run("graphite", func(t *testing.T) {
-		for _, x := range readIn("graphite", insertionTime) {
+		for _, x := range readIn("graphite", t, insertionTime) {
 			test := x
 			t.Run(test.Name, func(t *testing.T) {
 				t.Parallel()
@@ -294,7 +233,7 @@ func testWrite(t *testing.T) {
 		}
 	})
 	t.Run("opentsdb", func(t *testing.T) {
-		for _, x := range readIn("opentsdb", insertionTime) {
+		for _, x := range readIn("opentsdb", t, insertionTime) {
 			test := x
 			t.Run(test.Name, func(t *testing.T) {
 				t.Parallel()
@@ -303,7 +242,7 @@ func testWrite(t *testing.T) {
 		}
 	})
 	t.Run("opentsdbhttp", func(t *testing.T) {
-		for _, x := range readIn("opentsdbhttp", insertionTime) {
+		for _, x := range readIn("opentsdbhttp", t, insertionTime) {
 			test := x
 			t.Run(test.Name, func(t *testing.T) {
 				t.Parallel()
@@ -315,23 +254,18 @@ func testWrite(t *testing.T) {
 }
 
 func testRead(t *testing.T) {
-	for _, engine := range []string{"csv", "prometheus", "graphite", "opentsdb", "influxdb", "opentsdbhttp"} {
+	for _, engine := range []string{"prometheus", "graphite", "opentsdb", "influxdb", "opentsdbhttp"} {
 		t.Run(engine, func(t *testing.T) {
-			for _, x := range readIn(engine, insertionTime) {
+			for _, x := range readIn(engine, t, insertionTime) {
 				test := x
 				t.Run(test.Name, func(t *testing.T) {
 					t.Parallel()
 					for _, q := range test.Query {
 						q = testutil.PopulateTimeTplString(q, insertionTime)
 						if test.Issue != "" {
-							test.Issue = "\nRegression in " + test.Issue
+							test.Issue = "Regression in " + test.Issue
 						}
-						switch {
-						case strings.HasPrefix(q, "/api/v1/export/csv"):
-							data := strings.Split(string(httpReadData(t, testReadHTTPPath, q)), "\n")
-							if len(data) == test.ExpectedResultLinesCount {
-								t.Fatalf("not expected number of csv lines want=%d\ngot=%d test=%s.%s\n\response=%q", len(data), test.ExpectedResultLinesCount, q, test.Issue, strings.Join(data, "\n"))
-							}
+						switch true {
 						case strings.HasPrefix(q, "/api/v1/export"):
 							if err := checkMetricsResult(httpReadMetrics(t, testReadHTTPPath, q), test.ResultMetrics); err != nil {
 								t.Fatalf("Export. %s fails with error %s.%s", q, err, test.Issue)
@@ -342,19 +276,17 @@ func testRead(t *testing.T) {
 							if err := checkSeriesResult(s, test.ResultSeries); err != nil {
 								t.Fatalf("Series. %s fails with error %s.%s", q, err, test.Issue)
 							}
+						case strings.HasPrefix(q, "/api/v1/query_range"):
+							queryResult := QueryRange{}
+							httpReadStruct(t, testReadHTTPPath, q, &queryResult)
+							if err := checkQueryRangeResult(queryResult, test.ResultQueryRange); err != nil {
+								t.Fatalf("Query Range. %s fails with error %s.%s", q, err, test.Issue)
+							}
 						case strings.HasPrefix(q, "/api/v1/query"):
 							queryResult := Query{}
 							httpReadStruct(t, testReadHTTPPath, q, &queryResult)
-							gotMetrics, err := queryResult.metrics()
-							if err != nil {
-								t.Fatalf("failed to parse query response: %s", err)
-							}
-							expMetrics, err := test.ResultQuery.metrics()
-							if err != nil {
-								t.Fatalf("failed to parse expected response: %s", err)
-							}
-							if err := checkMetricsResult(gotMetrics, expMetrics); err != nil {
-								t.Fatalf("%q fails with error %s.%s", q, err, test.Issue)
+							if err := checkQueryResult(queryResult, test.ResultQuery); err != nil {
+								t.Fatalf("Query. %s fails with error %s.%s", q, err, test.Issue)
 							}
 						default:
 							t.Fatalf("unsupported read query %s", q)
@@ -366,141 +298,76 @@ func testRead(t *testing.T) {
 	}
 }
 
-func readIn(readFor string, insertTime time.Time) []test {
-	testDir := filepath.Join(testFixturesDir, readFor)
+func readIn(readFor string, t *testing.T, insertTime time.Time) []test {
+	t.Helper()
+	s := newSuite(t)
 	var tt []test
-	err := filepath.Walk(testDir, func(path string, _ os.FileInfo, err error) error {
+	s.noError(filepath.Walk(filepath.Join(testFixturesDir, readFor), func(path string, info os.FileInfo, err error) error {
 		if err != nil {
 			return err
 		}
 		if filepath.Ext(path) != ".json" {
 			return nil
 		}
-		b, err := os.ReadFile(path)
-		if err != nil {
-			panic(fmt.Errorf("BUG: cannot read %s: %s", path, err))
-		}
+		b, err := ioutil.ReadFile(path)
+		s.noError(err)
 		item := test{}
-		if err := json.Unmarshal(b, &item); err != nil {
-			panic(fmt.Errorf("cannot parse %T from %s: %s; data:\n%s", &item, path, err, b))
-		}
+		s.noError(json.Unmarshal(b, &item))
 		for i := range item.Data {
 			item.Data[i] = testutil.PopulateTimeTplString(item.Data[i], insertTime)
 		}
 		tt = append(tt, item)
 		return nil
-	})
-	if err != nil {
-		panic(fmt.Errorf("BUG: cannot read test data at %s: %w", testDir, err))
-	}
+	}))
 	if len(tt) == 0 {
-		panic(fmt.Errorf("BUG: no tests found in %s", testDir))
+		t.Fatalf("no test found in %s", filepath.Join(testFixturesDir, readFor))
 	}
 	return tt
 }
 
 func httpWrite(t *testing.T, address, query string, r io.Reader) {
 	t.Helper()
-
-	requestURL := address + query
-	resp, err := http.Post(requestURL, "", r)
-	if err != nil {
-		t.Fatalf("cannot send request to %s: %s", requestURL, err)
-	}
-	_ = resp.Body.Close()
-	if resp.StatusCode != 204 {
-		t.Fatalf("unexpected status code received from %s; got %d; want 204", requestURL, resp.StatusCode)
-	}
+	s := newSuite(t)
+	resp, err := http.Post(address+query, "", r)
+	s.noError(err)
+	s.noError(resp.Body.Close())
+	s.equalInt(resp.StatusCode, 204)
 }
 
-func tcpWrite(t *testing.T, address, data string) {
+func tcpWrite(t *testing.T, address string, data string) {
 	t.Helper()
-
+	s := newSuite(t)
 	conn, err := net.Dial("tcp", address)
-	if err != nil {
-		t.Fatalf("cannot dial %s: %s", address, err)
-	}
-	defer func() {
-		_ = conn.Close()
-	}()
+	s.noError(err)
+	defer conn.Close()
 	n, err := conn.Write([]byte(data))
-	if err != nil {
-		t.Fatalf("cannot write %d bytes to %s: %s", len(data), address, err)
-	}
-	if n != len(data) {
-		panic(fmt.Errorf("BUG: conn.Write() returned unexpected number of written bytes to %s; got %d; want %d", address, n, len(data)))
-	}
+	s.noError(err)
+	s.equalInt(n, len(data))
 }
 
 func httpReadMetrics(t *testing.T, address, query string) []Metric {
 	t.Helper()
-
-	requestURL := address + query
-	resp, err := http.Get(requestURL)
-	if err != nil {
-		t.Fatalf("cannot send request to %s: %s", requestURL, err)
-	}
-	defer func() {
-		_ = resp.Body.Close()
-	}()
-	if resp.StatusCode != 200 {
-		t.Fatalf("unexpected status code received from %s; got %d; want 200", requestURL, resp.StatusCode)
-	}
-
+	s := newSuite(t)
+	resp, err := http.Get(address + query)
+	s.noError(err)
+	defer resp.Body.Close()
+	s.equalInt(resp.StatusCode, 200)
 	var rows []Metric
-	dec := json.NewDecoder(resp.Body)
-	for {
+	for dec := json.NewDecoder(resp.Body); dec.More(); {
 		var row Metric
-		err := dec.Decode(&row)
-		if err != nil {
-			if errors.Is(err, io.EOF) {
-				return rows
-			}
-			t.Fatalf("cannot decode %T from response received from %s: %s", &row, requestURL, err)
-		}
+		s.noError(dec.Decode(&row))
 		rows = append(rows, row)
 	}
+	return rows
 }
-
-func httpReadStruct(t *testing.T, address, query string, dst any) {
+func httpReadStruct(t *testing.T, address, query string, dst interface{}) {
 	t.Helper()
-
-	requestURL := address + query
-	resp, err := http.Get(requestURL)
-	if err != nil {
-		t.Fatalf("cannot send request to %s: %s", requestURL, err)
-	}
-	defer func() {
-		_ = resp.Body.Close()
-	}()
-	if resp.StatusCode != 200 {
-		t.Fatalf("unexpected status code received from %s; got %d; want 200", requestURL, resp.StatusCode)
-	}
-	err = json.NewDecoder(resp.Body).Decode(dst)
-	if err != nil {
-		t.Fatalf("cannot decode %T from response received from %s: %s", dst, requestURL, err)
-	}
-}
-
-func httpReadData(t *testing.T, address, query string) []byte {
-	t.Helper()
-
-	requestURL := address + query
-	resp, err := http.Get(requestURL)
-	if err != nil {
-		t.Fatalf("cannot send request to %s: %s", requestURL, err)
-	}
-	defer func() {
-		_ = resp.Body.Close()
-	}()
-	if resp.StatusCode != 200 {
-		t.Fatalf("unexpected status code received from %s; got %d; want 200", requestURL, resp.StatusCode)
-	}
-	data, err := io.ReadAll(resp.Body)
-	if err != nil {
-		t.Fatalf("cannot read response from %s: %s", requestURL, err)
-	}
-	return data
+	s := newSuite(t)
+	resp, err := http.Get(address + query)
+	s.noError(err)
+	defer resp.Body.Close()
+	s.equalInt(resp.StatusCode, 200)
+	s.noError(json.NewDecoder(resp.Body).Decode(dst))
 }
 
 func checkMetricsResult(got, want []Metric) error {
@@ -548,72 +415,84 @@ func removeIfFoundSeries(r map[string]string, contains []map[string]string) []ma
 	return contains
 }
 
-func TestImportJSONLines(t *testing.T) {
-	f := func(labelsCount, labelLen int) {
-		t.Helper()
-
-		reqURL := fmt.Sprintf("http://localhost%s/api/v1/import", testHTTPListenAddr)
-		line := generateJSONLine(labelsCount, labelLen)
-		req, err := http.NewRequest("POST", reqURL, bytes.NewBufferString(line))
-		if err != nil {
-			t.Fatalf("cannot create request: %s", err)
-		}
-		resp, err := http.DefaultClient.Do(req)
-		if err != nil {
-			t.Fatalf("cannot perform request for labelsCount=%d, labelLen=%d: %s", labelsCount, labelLen, err)
-		}
-		if resp.StatusCode != 204 {
-			t.Fatalf("unexpected statusCode for labelsCount=%d, labelLen=%d; got %d; want 204", labelsCount, labelLen, resp.StatusCode)
-		}
+func checkQueryResult(got, want Query) error {
+	if got.Status != want.Status {
+		return fmt.Errorf("status mismatch %q - %q", want.Status, got.Status)
 	}
-
-	// labels with various lengths
-	for i := 0; i < 500; i++ {
-		f(10, i*5)
+	if got.Data.ResultType != want.Data.ResultType {
+		return fmt.Errorf("result type mismatch %q - %q", want.Data.ResultType, got.Data.ResultType)
 	}
-
-	// Too many labels
-	f(1000, 100)
-
-	// Too long labels
-	f(1, 100_000)
-	f(10, 100_000)
-	f(10, 10_000)
+	wantData := append([]QueryDataResult(nil), want.Data.Result...)
+	for _, r := range got.Data.Result {
+		wantData = removeIfFoundQueryData(r, wantData)
+	}
+	if len(wantData) > 0 {
+		return fmt.Errorf("expected query result %+v not found in %+v", wantData, got.Data.Result)
+	}
+	return nil
 }
 
-func generateJSONLine(labelsCount, labelLen int) string {
-	m := make(map[string]string, labelsCount)
-	m["__name__"] = generateSizedRandomString(labelLen)
-	for j := 1; j < labelsCount; j++ {
-		labelName := generateSizedRandomString(labelLen)
-		labelValue := generateSizedRandomString(labelLen)
-		m[labelName] = labelValue
+func removeIfFoundQueryData(r QueryDataResult, contains []QueryDataResult) []QueryDataResult {
+	for i, item := range contains {
+		if reflect.DeepEqual(r.Metric, item.Metric) && reflect.DeepEqual(r.Value[0], item.Value[0]) && reflect.DeepEqual(r.Value[1], item.Value[1]) {
+			contains[i] = contains[len(contains)-1]
+			return contains[:len(contains)-1]
+		}
 	}
+	return contains
+}
 
-	type jsonLine struct {
-		Metric     map[string]string `json:"metric"`
-		Values     []float64         `json:"values"`
-		Timestamps []int64           `json:"timestamps"`
+func checkQueryRangeResult(got, want QueryRange) error {
+	if got.Status != want.Status {
+		return fmt.Errorf("status mismatch %q - %q", want.Status, got.Status)
 	}
-	line := &jsonLine{
-		Metric:     m,
-		Values:     []float64{1.34},
-		Timestamps: []int64{time.Now().UnixNano() / 1e6},
+	if got.Data.ResultType != want.Data.ResultType {
+		return fmt.Errorf("result type mismatch %q - %q", want.Data.ResultType, got.Data.ResultType)
 	}
-	data, err := json.Marshal(&line)
+	wantData := append([]QueryRangeDataResult(nil), want.Data.Result...)
+	for _, r := range got.Data.Result {
+		wantData = removeIfFoundQueryRangeData(r, wantData)
+	}
+	if len(wantData) > 0 {
+		return fmt.Errorf("expected query range result %+v not found in %+v", wantData, got.Data.Result)
+	}
+	return nil
+}
+
+func removeIfFoundQueryRangeData(r QueryRangeDataResult, contains []QueryRangeDataResult) []QueryRangeDataResult {
+	for i, item := range contains {
+		if reflect.DeepEqual(r.Metric, item.Metric) && reflect.DeepEqual(r.Values, item.Values) {
+			contains[i] = contains[len(contains)-1]
+			return contains[:len(contains)-1]
+		}
+	}
+	return contains
+}
+
+type suite struct{ t *testing.T }
+
+func newSuite(t *testing.T) *suite { return &suite{t: t} }
+
+func (s *suite) noError(err error) {
+	s.t.Helper()
 	if err != nil {
-		panic(fmt.Errorf("cannot marshal JSON: %w", err))
+		s.t.Errorf("unexpected error %v", err)
+		s.t.FailNow()
 	}
-	data = append(data, '\n')
-	return string(data)
 }
 
-const alphabetSample = `qwertyuiopasdfghjklzxcvbnm`
-
-func generateSizedRandomString(size int) string {
-	dst := make([]byte, size)
-	for i := range dst {
-		dst[i] = alphabetSample[rand.Intn(len(alphabetSample))]
+func (s *suite) equalInt(a, b int) {
+	s.t.Helper()
+	if a != b {
+		s.t.Errorf("%d not equal %d", a, b)
+		s.t.FailNow()
+	}
+}
+
+func (s *suite) greaterThan(a, b int) {
+	s.t.Helper()
+	if a <= b {
+		s.t.Errorf("%d less or equal then %d", a, b)
+		s.t.FailNow()
 	}
-	return string(dst)
 }

app/victoria-metrics/multiarch/Dockerfile

@@ -1,12 +1,12 @@
 # See https://medium.com/on-docker/use-multi-stage-builds-to-inject-ca-certs-ad1e8f01de1b
 ARG certs_image
 ARG root_image
-FROM $certs_image AS certs
-RUN apk update && apk upgrade && apk --update --no-cache add ca-certificates
+FROM $certs_image as certs
+RUN apk --update --no-cache add ca-certificates
 
 FROM $root_image
 COPY --from=certs /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
 EXPOSE 8428
 ENTRYPOINT ["/victoria-metrics-prod"]
 ARG TARGETARCH
-COPY victoria-metrics-linux-${TARGETARCH}-prod ./victoria-metrics-prod
+COPY victoria-metrics-${TARGETARCH}-prod ./victoria-metrics-prod

app/victoria-metrics/self_scraper.go

@@ -6,9 +6,8 @@ import (
 	"time"
 
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmstorage"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/appmetrics"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/bytesutil"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/decimal"
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/prompb"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/protoparser/prometheus"
@@ -50,10 +49,18 @@ func selfScraper(scrapeInterval time.Duration) {
 	var mrs []storage.MetricRow
 	var labels []prompb.Label
 	t := time.NewTicker(scrapeInterval)
-	f := func(currentTime time.Time, sendStaleMarkers bool) {
-		currentTimestamp := currentTime.UnixNano() / 1e6
+	var currentTimestamp int64
+	for {
+		select {
+		case <-selfScraperStopCh:
+			t.Stop()
+			logger.Infof("stopped self-scraping `/metrics` page")
+			return
+		case currentTime := <-t.C:
+			currentTimestamp = currentTime.UnixNano() / 1e6
+		}
 		bb.Reset()
-		appmetrics.WritePrometheusMetrics(&bb)
+		httpserver.WritePrometheusMetrics(&bb)
 		s := bytesutil.ToUnsafeString(bb.B)
 		rows.Reset()
 		rows.Unmarshal(s)
@@ -76,26 +83,9 @@ func selfScraper(scrapeInterval time.Duration) {
 			mr := &mrs[len(mrs)-1]
 			mr.MetricNameRaw = storage.MarshalMetricNameRaw(mr.MetricNameRaw[:0], labels)
 			mr.Timestamp = currentTimestamp
-			if sendStaleMarkers {
-				mr.Value = decimal.StaleNaN
-			} else {
-				mr.Value = r.Value
-			}
-		}
-		if err := vmstorage.AddRows(mrs); err != nil {
-			logger.Errorf("cannot store self-scraped metrics: %s", err)
-		}
-	}
-	for {
-		select {
-		case <-selfScraperStopCh:
-			f(time.Now(), true)
-			t.Stop()
-			logger.Infof("stopped self-scraping `/metrics` page")
-			return
-		case currentTime := <-t.C:
-			f(currentTime, false)
+			mr.Value = r.Value
 		}
+		vmstorage.AddRows(mrs)
 	}
 }
 
@@ -106,7 +96,7 @@ func addLabel(dst []prompb.Label, key, value string) []prompb.Label {
 		dst = append(dst, prompb.Label{})
 	}
 	lb := &dst[len(dst)-1]
-	lb.Name = key
-	lb.Value = value
+	lb.Name = bytesutil.ToUnsafeBytes(key)
+	lb.Value = bytesutil.ToUnsafeBytes(value)
 	return dst
 }

app/victoria-metrics/test/parser_test.go

@@ -12,7 +12,7 @@ func TestPopulateTimeTplString(t *testing.T) {
 	}
 	f := func(s, resultExpected string) {
 		t.Helper()
-		result := PopulateTimeTplString(s, now.UTC())
+		result := PopulateTimeTplString(s, now)
 		if result != resultExpected {
 			t.Fatalf("unexpected result; got %q; want %q", result, resultExpected)
 		}

app/victoria-metrics/test/prom_writer.go

@@ -1,16 +0,0 @@
-package test
-
-import (
-	"fmt"
-
-	"github.com/golang/snappy"
-)
-
-// Compress marshals and compresses wr.
-func Compress(wr WriteRequest) []byte {
-	data, err := wr.Marshal()
-	if err != nil {
-		panic(fmt.Errorf("BUG: cannot compress WriteRequest: %s", err))
-	}
-	return snappy.Encode(nil, data)
-}

app/victoria-metrics/test/prom_writter.go

@@ -0,0 +1,12 @@
+package test
+
+import "github.com/golang/snappy"
+
+// Compress marshals and compresses wr.
+func Compress(wr WriteRequest) ([]byte, error) {
+	data, err := wr.Marshal()
+	if err != nil {
+		return nil, err
+	}
+	return snappy.Encode(nil, data), nil
+}

app/victoria-metrics/testdata/csv/basic.json

@@ -1,14 +0,0 @@
-{
-  "name": "csv export",
-  "data": [
-    "rfc3339,4,{TIME_MS}",
-    "rfc3339milli,6,{TIME_MS}",
-    "ts,8,{TIME_MS}",
-    "tsms,10,{TIME_MS},"
-  ],
-  "insert_query": "?format=1:label:tfmt,2:metric:test_csv,3:time:unix_ms",
-  "query": [
-    "/api/v1/export/csv?format=__name__,tfmt,__value__,__timestamp__:rfc3339&match[]={__name__=\"test_csv\"}&step=30s&start={TIME_MS-180s}"
-  ],
-  "expected_result_lines_count": 4
-}

app/victoria-metrics/testdata/csv/with_extra_labels.json

@@ -1,14 +0,0 @@
-{
-  "name": "csv export with extra_labels",
-  "data": [
-    "location-1,4,{TIME_MS}",
-    "location-2,6,{TIME_MS}",
-    "location-3,8,{TIME_MS}",
-    "location-4,10,{TIME_MS},"
-  ],
-  "insert_query": "?format=1:label:location,2:metric:test_csv_labels,3:time:unix_ms&extra_label=location=location-1",
-  "query": [
-    "/api/v1/export/csv?format=__name__,location,__value__,__timestamp__:unix_ms&match[]={__name__=\"test_csv\"}&step=30s&start={TIME_MS-180s}"
-  ],
-  "expected_result_lines_count": 4
-}

app/victoria-metrics/testdata/graphite/comparison-not-inf-not-nan.json

@@ -7,10 +7,10 @@
     "not_nan_not_inf;item=y 3 {TIME_S-1m}",
     "not_nan_not_inf;item=y 1 {TIME_S-2m}"],
   "query": ["/api/v1/query_range?query=1/(not_nan_not_inf-1)!=inf!=nan&start={TIME_S-3m}&end={TIME_S}&step=60"],
-  "result_query": {
+  "result_query_range": {
     "status":"success",
     "data":{"resultType":"matrix",
       "result":[
-	      {"metric":{"item":"y"},"values":[["{TIME_S-1m}","0.5"], ["{TIME_S}","0.5"]]}
+	      {"metric":{"item":"y"},"values":[["{TIME_S-1m}","0.5"]]}
       ]}}
 }

app/victoria-metrics/testdata/graphite/empty-label-match.json

@@ -6,7 +6,7 @@
     "empty_label_match;foo=bar 2 {TIME_S-1m}",
     "empty_label_match;foo=baz 3 {TIME_S-1m}"],
   "query": ["/api/v1/query_range?query=empty_label_match{foo=~'bar|'}&start={TIME_S-1m}&end={TIME_S}&step=60"],
-  "result_query": {
+  "result_query_range": {
     "status":"success",
     "data":{"resultType":"matrix",
       "result":[

app/victoria-metrics/testdata/graphite/max_lookback_set.json

@@ -8,7 +8,7 @@
     "max_lookback_set 4 {TIME_S-150s}"
   ],
   "query": ["/api/v1/query_range?query=max_lookback_set&start={TIME_S-150s}&end={TIME_S}&step=10s&max_lookback=1s"],
-  "result_query": {
+  "result_query_range": {
     "status":"success",
     "data":{"resultType":"matrix",
       "result":[{"metric":{"__name__":"max_lookback_set"},"values":[

app/victoria-metrics/testdata/graphite/max_lookback_unset.json

@@ -8,7 +8,7 @@
     "max_lookback_unset 4 {TIME_S-150s}"
   ],
   "query": ["/api/v1/query_range?query=max_lookback_unset&start={TIME_S-150s}&end={TIME_S}&step=10s"],
-  "result_query": {
+  "result_query_range": {
     "status":"success",
     "data":{"resultType":"matrix",
       "result":[{"metric":{"__name__":"max_lookback_unset"},"values":[
@@ -19,7 +19,6 @@
 	      ["{TIME_S-110s}","3"],
 	      ["{TIME_S-100s}","3"],
 	      ["{TIME_S-90s}","3"],
-	      ["{TIME_S-80s}","3"],
 	      ["{TIME_S-60s}","2"],
 	      ["{TIME_S-50s}","2"],
 	      ["{TIME_S-40s}","2"],

app/victoria-metrics/testdata/graphite/not-nan-as-missing-data.json

@@ -8,11 +8,11 @@
     "not_nan_as_missing_data;item=y 3 {TIME_S-1m}"
   ],
   "query": ["/api/v1/query_range?query=not_nan_as_missing_data>1&start={TIME_S-2m}&end={TIME_S}&step=60"],
-  "result_query": {
+  "result_query_range": {
     "status":"success",
     "data":{"resultType":"matrix",
       "result":[
 	      {"metric":{"__name__":"not_nan_as_missing_data","item":"x"},"values":[["{TIME_S-2m}","2"]]},
-	      {"metric":{"__name__":"not_nan_as_missing_data","item":"y"},"values":[["{TIME_S-2m}","4"],["{TIME_S-1m}","3"],["{TIME_S}", "3"]]}
+	      {"metric":{"__name__":"not_nan_as_missing_data","item":"y"},"values":[["{TIME_S-2m}","4"],["{TIME_S-1m}","3"]]}
       ]}}
 }

app/victoria-metrics/testdata/graphite/subquery-aggregation.json

@@ -6,7 +6,7 @@
     "forms_daily_count;item=x 2 {TIME_S-2m}",
     "forms_daily_count;item=y 3 {TIME_S-1m}",
     "forms_daily_count;item=y 4 {TIME_S-2m}"],
-  "query": ["/api/v1/query?query=min%20by%20(item)%20(min_over_time(forms_daily_count[10m:1m]))&time={TIME_S-1m}&latency_offset=1ms"],
+  "query": ["/api/v1/query?query=min%20by%20(item)%20(min_over_time(forms_daily_count[10m:1m]))&time={TIME_S-1m}"],
   "result_query": {
     "status":"success",
     "data":{"resultType":"vector","result":[{"metric":{"item":"x"},"value":["{TIME_S-1m}","2"]},{"metric":{"item":"y"},"value":["{TIME_S-1m}","4"]}]}

app/victoria-metrics/testdata/prometheus/instant-matrix.json

@@ -1,12 +0,0 @@
-{
-  "name": "instant query with look-behind window",
-  "data": ["[{\"labels\":[{\"name\":\"__name__\",\"value\":\"foo\"}],\"samples\":[{\"value\":1,\"timestamp\":\"{TIME_MS-60s}\"}]}]"],
-  "query": ["/api/v1/query?query=foo[5m]"],
-  "result_query": {
-      "status": "success",
-      "data":{
-        "resultType":"matrix",
-        "result":[{"metric":{"__name__":"foo"},"values":[["{TIME_S-60s}", "1"]]}]
-      }
-    }
-}

app/victoria-metrics/testdata/prometheus/instant-scalar.json

@@ -1,11 +0,0 @@
-{
-  "name": "instant scalar query",
-  "query": ["/api/v1/query?query=42&time={TIME_S}"],
-  "result_query": {
-      "status": "success",
-      "data":{
-        "resultType":"vector",
-        "result":[{"metric":{},"value":["{TIME_S}", "42"]}]
-      }
-    }
-}

app/victoria-metrics/testdata/prometheus/issue-5553-too-big-lookback.json

@@ -1,13 +0,0 @@
-{
-  "name": "too big look-behind window",
-  "issue": "https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5553",
-  "data": ["[{\"labels\":[{\"name\":\"__name__\",\"value\":\"foo\"},{\"name\":\"issue\",\"value\":\"5553\"}],\"samples\":[{\"value\":1,\"timestamp\":\"{TIME_MS-60s}\"}]}]"],
-  "query": ["/api/v1/query?query=foo{issue=\"5553\"}[100y]"],
-  "result_query": {
-      "status": "success",
-      "data":{
-        "resultType":"matrix",
-        "result":[{"metric":{"__name__":"foo", "issue": "5553"},"values":[["{TIME_S-60s}", "1"]]}]
-      }
-    }
-}

app/victoria-metrics/testdata/prometheus/query-range.json

@@ -1,18 +0,0 @@
-{
-  "name": "query range",
-  "issue": "https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5553",
-  "data": ["[{\"labels\":[{\"name\":\"__name__\",\"value\":\"bar\"}],\"samples\":[{\"value\":1,\"timestamp\":\"{TIME_MS-60s}\"}, {\"value\":2,\"timestamp\":\"{TIME_MS-120s}\"}, {\"value\":1,\"timestamp\":\"{TIME_MS-180s}\"}]}]"],
-  "query": ["/api/v1/query_range?query=bar&step=30s&start={TIME_MS-180s}"],
-  "result_query": {
-      "status": "success",
-      "data":{
-        "resultType":"matrix",
-        "result":[
-          {
-            "metric":{"__name__":"bar"},
-            "values":[["{TIME_S-180s}", "1"],["{TIME_S-150s}", "1"],["{TIME_S-120s}", "2"],["{TIME_S-90s}", "2"], ["{TIME_S-60s}", "1"], ["{TIME_S-30s}", "1"], ["{TIME_S}", "1"]]
-          }
-        ]
-      }
-    }
-}

app/vlinsert/elasticsearch/bulk_response.qtpl

@@ -1,20 +0,0 @@
-{% stripspace %}
-
-{% func BulkResponse(n int, tookMs int64) %}
-{
-	"took":{%dl tookMs %},
-	"errors":false,
-	"items":[
-		{% for i := 0; i < n; i++ %}
-		{
-			"create":{
-				"status":201
-			}
-		}
-		{% if i+1 < n %},{% endif %}
-		{% endfor %}
-	]
-}
-{% endfunc %}
-
-{% endstripspace %}

app/vlinsert/elasticsearch/bulk_response.qtpl.go

@@ -1,69 +0,0 @@
-// Code generated by qtc from "bulk_response.qtpl". DO NOT EDIT.
-// See https://github.com/valyala/quicktemplate for details.
-
-//line app/vlinsert/elasticsearch/bulk_response.qtpl:3
-package elasticsearch
-
-//line app/vlinsert/elasticsearch/bulk_response.qtpl:3
-import (
-	qtio422016 "io"
-
-	qt422016 "github.com/valyala/quicktemplate"
-)
-
-//line app/vlinsert/elasticsearch/bulk_response.qtpl:3
-var (
-	_ = qtio422016.Copy
-	_ = qt422016.AcquireByteBuffer
-)
-
-//line app/vlinsert/elasticsearch/bulk_response.qtpl:3
-func StreamBulkResponse(qw422016 *qt422016.Writer, n int, tookMs int64) {
-//line app/vlinsert/elasticsearch/bulk_response.qtpl:3
-	qw422016.N().S(`{"took":`)
-//line app/vlinsert/elasticsearch/bulk_response.qtpl:5
-	qw422016.N().DL(tookMs)
-//line app/vlinsert/elasticsearch/bulk_response.qtpl:5
-	qw422016.N().S(`,"errors":false,"items":[`)
-//line app/vlinsert/elasticsearch/bulk_response.qtpl:8
-	for i := 0; i < n; i++ {
-//line app/vlinsert/elasticsearch/bulk_response.qtpl:8
-		qw422016.N().S(`{"create":{"status":201}}`)
-//line app/vlinsert/elasticsearch/bulk_response.qtpl:14
-		if i+1 < n {
-//line app/vlinsert/elasticsearch/bulk_response.qtpl:14
-			qw422016.N().S(`,`)
-//line app/vlinsert/elasticsearch/bulk_response.qtpl:14
-		}
-//line app/vlinsert/elasticsearch/bulk_response.qtpl:15
-	}
-//line app/vlinsert/elasticsearch/bulk_response.qtpl:15
-	qw422016.N().S(`]}`)
-//line app/vlinsert/elasticsearch/bulk_response.qtpl:18
-}
-
-//line app/vlinsert/elasticsearch/bulk_response.qtpl:18
-func WriteBulkResponse(qq422016 qtio422016.Writer, n int, tookMs int64) {
-//line app/vlinsert/elasticsearch/bulk_response.qtpl:18
-	qw422016 := qt422016.AcquireWriter(qq422016)
-//line app/vlinsert/elasticsearch/bulk_response.qtpl:18
-	StreamBulkResponse(qw422016, n, tookMs)
-//line app/vlinsert/elasticsearch/bulk_response.qtpl:18
-	qt422016.ReleaseWriter(qw422016)
-//line app/vlinsert/elasticsearch/bulk_response.qtpl:18
-}
-
-//line app/vlinsert/elasticsearch/bulk_response.qtpl:18
-func BulkResponse(n int, tookMs int64) string {
-//line app/vlinsert/elasticsearch/bulk_response.qtpl:18
-	qb422016 := qt422016.AcquireByteBuffer()
-//line app/vlinsert/elasticsearch/bulk_response.qtpl:18
-	WriteBulkResponse(qb422016, n, tookMs)
-//line app/vlinsert/elasticsearch/bulk_response.qtpl:18
-	qs422016 := string(qb422016.B)
-//line app/vlinsert/elasticsearch/bulk_response.qtpl:18
-	qt422016.ReleaseByteBuffer(qb422016)
-//line app/vlinsert/elasticsearch/bulk_response.qtpl:18
-	return qs422016
-//line app/vlinsert/elasticsearch/bulk_response.qtpl:18
-}

app/vlinsert/elasticsearch/elasticsearch.go

@@ -1,280 +0,0 @@
-package elasticsearch
-
-import (
-	"bufio"
-	"errors"
-	"flag"
-	"fmt"
-	"io"
-	"math"
-	"net/http"
-	"strconv"
-	"strings"
-	"time"
-
-	"github.com/VictoriaMetrics/metrics"
-
-	"github.com/VictoriaMetrics/VictoriaMetrics/app/vlinsert/insertutils"
-	"github.com/VictoriaMetrics/VictoriaMetrics/app/vlstorage"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/bufferedwriter"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/bytesutil"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logstorage"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/protoparser/common"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/writeconcurrencylimiter"
-)
-
-var (
-	elasticsearchVersion = flag.String("elasticsearch.version", "8.9.0", "Elasticsearch version to report to client")
-)
-
-// RequestHandler processes Elasticsearch insert requests
-func RequestHandler(path string, w http.ResponseWriter, r *http.Request) bool {
-	w.Header().Add("Content-Type", "application/json")
-	// This header is needed for Logstash
-	w.Header().Set("X-Elastic-Product", "Elasticsearch")
-
-	if strings.HasPrefix(path, "/_ilm/policy") {
-		// Return fake response for Elasticsearch ilm request.
-		fmt.Fprintf(w, `{}`)
-		return true
-	}
-	if strings.HasPrefix(path, "/_index_template") {
-		// Return fake response for Elasticsearch index template request.
-		fmt.Fprintf(w, `{}`)
-		return true
-	}
-	if strings.HasPrefix(path, "/_ingest") {
-		// Return fake response for Elasticsearch ingest pipeline request.
-		// See: https://www.elastic.co/guide/en/elasticsearch/reference/8.8/put-pipeline-api.html
-		fmt.Fprintf(w, `{}`)
-		return true
-	}
-	if strings.HasPrefix(path, "/_nodes") {
-		// Return fake response for Elasticsearch nodes discovery request.
-		// See: https://www.elastic.co/guide/en/elasticsearch/reference/8.8/cluster.html
-		fmt.Fprintf(w, `{}`)
-		return true
-	}
-	if strings.HasPrefix(path, "/logstash") || strings.HasPrefix(path, "/_logstash") {
-		// Return fake response for Logstash APIs requests.
-		// See: https://www.elastic.co/guide/en/elasticsearch/reference/8.8/logstash-apis.html
-		fmt.Fprintf(w, `{}`)
-		return true
-	}
-	switch path {
-	case "/":
-		switch r.Method {
-		case http.MethodGet:
-			// Return fake response for Elasticsearch ping request.
-			// See the latest available version for Elasticsearch at https://github.com/elastic/elasticsearch/releases
-			fmt.Fprintf(w, `{
-			"version": {
-				"number": %q
-			}
-		}`, *elasticsearchVersion)
-		case http.MethodHead:
-			// Return empty response for Logstash ping request.
-		}
-
-		return true
-	case "/_license":
-		// Return fake response for Elasticsearch license request.
-		fmt.Fprintf(w, `{
-			"license": {
-				"uid": "cbff45e7-c553-41f7-ae4f-9205eabd80xx",
-				"type": "oss",
-				"status": "active",
-				"expiry_date_in_millis" : 4000000000000
-			}
-		}`)
-		return true
-	case "/_bulk":
-		startTime := time.Now()
-		bulkRequestsTotal.Inc()
-
-		cp, err := insertutils.GetCommonParams(r)
-		if err != nil {
-			httpserver.Errorf(w, r, "%s", err)
-			return true
-		}
-		if err := vlstorage.CanWriteData(); err != nil {
-			httpserver.Errorf(w, r, "%s", err)
-			return true
-		}
-		lmp := cp.NewLogMessageProcessor()
-		isGzip := r.Header.Get("Content-Encoding") == "gzip"
-		n, err := readBulkRequest(r.Body, isGzip, cp.TimeField, cp.MsgField, lmp)
-		lmp.MustClose()
-		if err != nil {
-			logger.Warnf("cannot decode log message #%d in /_bulk request: %s, stream fields: %s", n, err, cp.StreamFields)
-			return true
-		}
-
-		tookMs := time.Since(startTime).Milliseconds()
-		bw := bufferedwriter.Get(w)
-		defer bufferedwriter.Put(bw)
-		WriteBulkResponse(bw, n, tookMs)
-		_ = bw.Flush()
-
-		// update bulkRequestDuration only for successfully parsed requests
-		// There is no need in updating bulkRequestDuration for request errors,
-		// since their timings are usually much smaller than the timing for successful request parsing.
-		bulkRequestDuration.UpdateDuration(startTime)
-
-		return true
-	default:
-		return false
-	}
-}
-
-var (
-	bulkRequestsTotal   = metrics.NewCounter(`vl_http_requests_total{path="/insert/elasticsearch/_bulk"}`)
-	rowsIngestedTotal   = metrics.NewCounter(`vl_rows_ingested_total{type="elasticsearch_bulk"}`)
-	bulkRequestDuration = metrics.NewHistogram(`vl_http_request_duration_seconds{path="/insert/elasticsearch/_bulk"}`)
-)
-
-func readBulkRequest(r io.Reader, isGzip bool, timeField, msgField string, lmp insertutils.LogMessageProcessor) (int, error) {
-	// See https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html
-
-	if isGzip {
-		zr, err := common.GetGzipReader(r)
-		if err != nil {
-			return 0, fmt.Errorf("cannot read gzipped _bulk request: %w", err)
-		}
-		defer common.PutGzipReader(zr)
-		r = zr
-	}
-
-	wcr := writeconcurrencylimiter.GetReader(r)
-	defer writeconcurrencylimiter.PutReader(wcr)
-
-	lb := lineBufferPool.Get()
-	defer lineBufferPool.Put(lb)
-
-	lb.B = bytesutil.ResizeNoCopyNoOverallocate(lb.B, insertutils.MaxLineSizeBytes.IntN())
-	sc := bufio.NewScanner(wcr)
-	sc.Buffer(lb.B, len(lb.B))
-
-	n := 0
-	nCheckpoint := 0
-	for {
-		ok, err := readBulkLine(sc, timeField, msgField, lmp)
-		wcr.DecConcurrency()
-		if err != nil || !ok {
-			rowsIngestedTotal.Add(n - nCheckpoint)
-			return n, err
-		}
-		n++
-		if batchSize := n - nCheckpoint; n >= 1000 {
-			rowsIngestedTotal.Add(batchSize)
-			nCheckpoint = n
-		}
-	}
-}
-
-var lineBufferPool bytesutil.ByteBufferPool
-
-func readBulkLine(sc *bufio.Scanner, timeField, msgField string, lmp insertutils.LogMessageProcessor) (bool, error) {
-	var line []byte
-
-	// Read the command, must be "create" or "index"
-	for len(line) == 0 {
-		if !sc.Scan() {
-			if err := sc.Err(); err != nil {
-				if errors.Is(err, bufio.ErrTooLong) {
-					return false, fmt.Errorf(`cannot read "create" or "index" command, since its size exceeds -insert.maxLineSizeBytes=%d`,
-						insertutils.MaxLineSizeBytes.IntN())
-				}
-				return false, err
-			}
-			return false, nil
-		}
-		line = sc.Bytes()
-	}
-	lineStr := bytesutil.ToUnsafeString(line)
-	if !strings.Contains(lineStr, `"create"`) && !strings.Contains(lineStr, `"index"`) {
-		return false, fmt.Errorf(`unexpected command %q; expecting "create" or "index"`, line)
-	}
-
-	// Decode log message
-	if !sc.Scan() {
-		if err := sc.Err(); err != nil {
-			if errors.Is(err, bufio.ErrTooLong) {
-				return false, fmt.Errorf("cannot read log message, since its size exceeds -insert.maxLineSizeBytes=%d", insertutils.MaxLineSizeBytes.IntN())
-			}
-			return false, err
-		}
-		return false, fmt.Errorf(`missing log message after the "create" or "index" command`)
-	}
-	line = sc.Bytes()
-	p := logstorage.GetJSONParser()
-	if err := p.ParseLogMessage(line); err != nil {
-		return false, fmt.Errorf("cannot parse json-encoded log entry: %w", err)
-	}
-
-	ts, err := extractTimestampFromFields(timeField, p.Fields)
-	if err != nil {
-		return false, fmt.Errorf("cannot parse timestamp: %w", err)
-	}
-	if ts == 0 {
-		ts = time.Now().UnixNano()
-	}
-	logstorage.RenameField(p.Fields, msgField, "_msg")
-	lmp.AddRow(ts, p.Fields)
-	logstorage.PutJSONParser(p)
-
-	return true, nil
-}
-
-func extractTimestampFromFields(timeField string, fields []logstorage.Field) (int64, error) {
-	for i := range fields {
-		f := &fields[i]
-		if f.Name != timeField {
-			continue
-		}
-		timestamp, err := parseElasticsearchTimestamp(f.Value)
-		if err != nil {
-			return 0, err
-		}
-		f.Value = ""
-		return timestamp, nil
-	}
-	return 0, nil
-}
-
-func parseElasticsearchTimestamp(s string) (int64, error) {
-	if s == "0" || s == "" {
-		// Special case - zero or empty timestamp must be substituted
-		// with the current time by the caller.
-		return 0, nil
-	}
-	if len(s) < len("YYYY-MM-DD") || s[len("YYYY")] != '-' {
-		// Try parsing timestamp in milliseconds
-		n, err := strconv.ParseInt(s, 10, 64)
-		if err != nil {
-			return 0, fmt.Errorf("cannot parse timestamp in milliseconds from %q: %w", s, err)
-		}
-		if n > int64(math.MaxInt64)/1e6 {
-			return 0, fmt.Errorf("too big timestamp in milliseconds: %d; mustn't exceed %d", n, int64(math.MaxInt64)/1e6)
-		}
-		if n < int64(math.MinInt64)/1e6 {
-			return 0, fmt.Errorf("too small timestamp in milliseconds: %d; must be bigger than %d", n, int64(math.MinInt64)/1e6)
-		}
-		n *= 1e6
-		return n, nil
-	}
-	if len(s) == len("YYYY-MM-DD") {
-		t, err := time.Parse("2006-01-02", s)
-		if err != nil {
-			return 0, fmt.Errorf("cannot parse date %q: %w", s, err)
-		}
-		return t.UnixNano(), nil
-	}
-	nsecs, ok := logstorage.TryParseTimestampRFC3339Nano(s)
-	if !ok {
-		return 0, fmt.Errorf("cannot parse timestamp %q", s)
-	}
-	return nsecs, nil
-}

app/vlinsert/elasticsearch/elasticsearch_test.go

@@ -1,103 +0,0 @@
-package elasticsearch
-
-import (
-	"bytes"
-	"compress/gzip"
-	"fmt"
-	"testing"
-
-	"github.com/VictoriaMetrics/VictoriaMetrics/app/vlinsert/insertutils"
-)
-
-func TestReadBulkRequest_Failure(t *testing.T) {
-	f := func(data string) {
-		t.Helper()
-
-		tlp := &insertutils.TestLogMessageProcessor{}
-		r := bytes.NewBufferString(data)
-		rows, err := readBulkRequest(r, false, "_time", "_msg", tlp)
-		if err == nil {
-			t.Fatalf("expecting non-empty error")
-		}
-		if rows != 0 {
-			t.Fatalf("unexpected non-zero rows=%d", rows)
-		}
-	}
-	f("foobar")
-	f(`{}`)
-	f(`{"create":{}}`)
-	f(`{"creat":{}}
-{}`)
-	f(`{"create":{}}
-foobar`)
-}
-
-func TestReadBulkRequest_Success(t *testing.T) {
-	f := func(data, timeField, msgField string, rowsExpected int, timestampsExpected []int64, resultExpected string) {
-		t.Helper()
-
-		tlp := &insertutils.TestLogMessageProcessor{}
-
-		// Read the request without compression
-		r := bytes.NewBufferString(data)
-		rows, err := readBulkRequest(r, false, timeField, msgField, tlp)
-		if err != nil {
-			t.Fatalf("unexpected error: %s", err)
-		}
-		if rows != rowsExpected {
-			t.Fatalf("unexpected rows read; got %d; want %d", rows, rowsExpected)
-		}
-		if err := tlp.Verify(rowsExpected, timestampsExpected, resultExpected); err != nil {
-			t.Fatal(err)
-		}
-
-		// Read the request with compression
-		tlp = &insertutils.TestLogMessageProcessor{}
-		compressedData := compressData(data)
-		r = bytes.NewBufferString(compressedData)
-		rows, err = readBulkRequest(r, true, timeField, msgField, tlp)
-		if err != nil {
-			t.Fatalf("unexpected error: %s", err)
-		}
-		if rows != rowsExpected {
-			t.Fatalf("unexpected rows read; got %d; want %d", rows, rowsExpected)
-		}
-		if err := tlp.Verify(rowsExpected, timestampsExpected, resultExpected); err != nil {
-			t.Fatalf("verification failure after compression: %s", err)
-		}
-	}
-
-	// Verify an empty data
-	f("", "_time", "_msg", 0, nil, "")
-	f("\n", "_time", "_msg", 0, nil, "")
-	f("\n\n", "_time", "_msg", 0, nil, "")
-
-	// Verify non-empty data
-	data := `{"create":{"_index":"filebeat-8.8.0"}}
-{"@timestamp":"2023-06-06T04:48:11.735Z","log":{"offset":71770,"file":{"path":"/var/log/auth.log"}},"message":"foobar"}
-{"create":{"_index":"filebeat-8.8.0"}}
-{"@timestamp":"2023-06-06T04:48:12.735Z","message":"baz"}
-{"index":{"_index":"filebeat-8.8.0"}}
-{"message":"xyz","@timestamp":"2023-06-06T04:48:13.735Z","x":"y"}
-`
-	timeField := "@timestamp"
-	msgField := "message"
-	rowsExpected := 3
-	timestampsExpected := []int64{1686026891735000000, 1686026892735000000, 1686026893735000000}
-	resultExpected := `{"@timestamp":"","log.offset":"71770","log.file.path":"/var/log/auth.log","_msg":"foobar"}
-{"@timestamp":"","_msg":"baz"}
-{"_msg":"xyz","@timestamp":"","x":"y"}`
-	f(data, timeField, msgField, rowsExpected, timestampsExpected, resultExpected)
-}
-
-func compressData(s string) string {
-	var bb bytes.Buffer
-	zw := gzip.NewWriter(&bb)
-	if _, err := zw.Write([]byte(s)); err != nil {
-		panic(fmt.Errorf("unexpected error when compressing data: %w", err))
-	}
-	if err := zw.Close(); err != nil {
-		panic(fmt.Errorf("unexpected error when closing gzip writer: %w", err))
-	}
-	return bb.String()
-}

app/vlinsert/elasticsearch/elasticsearch_timing_test.go

@@ -1,50 +0,0 @@
-package elasticsearch
-
-import (
-	"bytes"
-	"fmt"
-	"testing"
-
-	"github.com/VictoriaMetrics/VictoriaMetrics/app/vlinsert/insertutils"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/bytesutil"
-)
-
-func BenchmarkReadBulkRequest(b *testing.B) {
-	b.Run("gzip:off", func(b *testing.B) {
-		benchmarkReadBulkRequest(b, false)
-	})
-	b.Run("gzip:on", func(b *testing.B) {
-		benchmarkReadBulkRequest(b, true)
-	})
-}
-
-func benchmarkReadBulkRequest(b *testing.B, isGzip bool) {
-	data := `{"create":{"_index":"filebeat-8.8.0"}}
-{"@timestamp":"2023-06-06T04:48:11.735Z","log":{"offset":71770,"file":{"path":"/var/log/auth.log"}},"message":"foobar"}
-{"create":{"_index":"filebeat-8.8.0"}}
-{"@timestamp":"2023-06-06T04:48:12.735Z","message":"baz"}
-{"create":{"_index":"filebeat-8.8.0"}}
-{"message":"xyz","@timestamp":"2023-06-06T04:48:13.735Z","x":"y"}
-`
-	if isGzip {
-		data = compressData(data)
-	}
-	dataBytes := bytesutil.ToUnsafeBytes(data)
-
-	timeField := "@timestamp"
-	msgField := "message"
-	blp := &insertutils.BenchmarkLogMessageProcessor{}
-
-	b.ReportAllocs()
-	b.SetBytes(int64(len(data)))
-	b.RunParallel(func(pb *testing.PB) {
-		r := &bytes.Reader{}
-		for pb.Next() {
-			r.Reset(dataBytes)
-			_, err := readBulkRequest(r, isGzip, timeField, msgField, blp)
-			if err != nil {
-				panic(fmt.Errorf("unexpected error: %w", err))
-			}
-		}
-	})
-}

app/vlinsert/insertutils/common_params.go

@@ -1,200 +0,0 @@
-package insertutils
-
-import (
-	"net/http"
-	"sync"
-	"time"
-
-	"github.com/VictoriaMetrics/metrics"
-
-	"github.com/VictoriaMetrics/VictoriaMetrics/app/vlstorage"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/httputils"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logstorage"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/timeutil"
-)
-
-// CommonParams contains common HTTP parameters used by log ingestion APIs.
-//
-// See https://docs.victoriametrics.com/victorialogs/data-ingestion/#http-parameters
-type CommonParams struct {
-	TenantID     logstorage.TenantID
-	TimeField    string
-	MsgField     string
-	StreamFields []string
-	IgnoreFields []string
-
-	Debug           bool
-	DebugRequestURI string
-	DebugRemoteAddr string
-}
-
-// GetCommonParams returns CommonParams from r.
-func GetCommonParams(r *http.Request) (*CommonParams, error) {
-	// Extract tenantID
-	tenantID, err := logstorage.GetTenantIDFromRequest(r)
-	if err != nil {
-		return nil, err
-	}
-
-	// Extract time field name from _time_field query arg
-	var timeField = "_time"
-	if tf := r.FormValue("_time_field"); tf != "" {
-		timeField = tf
-	}
-
-	// Extract message field name from _msg_field query arg
-	var msgField = ""
-	if msgf := r.FormValue("_msg_field"); msgf != "" {
-		msgField = msgf
-	}
-
-	streamFields := httputils.GetArray(r, "_stream_fields")
-	ignoreFields := httputils.GetArray(r, "ignore_fields")
-
-	debug := httputils.GetBool(r, "debug")
-	debugRequestURI := ""
-	debugRemoteAddr := ""
-	if debug {
-		debugRequestURI = httpserver.GetRequestURI(r)
-		debugRemoteAddr = httpserver.GetQuotedRemoteAddr(r)
-	}
-
-	cp := &CommonParams{
-		TenantID:        tenantID,
-		TimeField:       timeField,
-		MsgField:        msgField,
-		StreamFields:    streamFields,
-		IgnoreFields:    ignoreFields,
-		Debug:           debug,
-		DebugRequestURI: debugRequestURI,
-		DebugRemoteAddr: debugRemoteAddr,
-	}
-	return cp, nil
-}
-
-// GetCommonParamsForSyslog returns common params needed for parsing syslog messages and storing them to the given tenantID.
-func GetCommonParamsForSyslog(tenantID logstorage.TenantID) *CommonParams {
-	// See https://docs.victoriametrics.com/victorialogs/logsql/#unpack_syslog-pipe
-	cp := &CommonParams{
-		TenantID:  tenantID,
-		TimeField: "timestamp",
-		MsgField:  "message",
-		StreamFields: []string{
-			"hostname",
-			"app_name",
-			"proc_id",
-		},
-	}
-
-	return cp
-}
-
-// LogMessageProcessor is an interface for log message processors.
-type LogMessageProcessor interface {
-	// AddRow must add row to the LogMessageProcessor with the given timestamp and the given fields.
-	//
-	// The LogMessageProcessor implementation cannot hold references to fields, since the caller can re-use them.
-	AddRow(timestamp int64, fields []logstorage.Field)
-
-	// MustClose() must flush all the remaining fields and free up resources occupied by LogMessageProcessor.
-	MustClose()
-}
-
-type logMessageProcessor struct {
-	mu            sync.Mutex
-	wg            sync.WaitGroup
-	stopCh        chan struct{}
-	lastFlushTime time.Time
-
-	cp *CommonParams
-	lr *logstorage.LogRows
-}
-
-func (lmp *logMessageProcessor) initPeriodicFlush() {
-	lmp.lastFlushTime = time.Now()
-
-	lmp.wg.Add(1)
-	go func() {
-		defer lmp.wg.Done()
-
-		d := timeutil.AddJitterToDuration(time.Second)
-		ticker := time.NewTicker(d)
-		defer ticker.Stop()
-
-		for {
-			select {
-			case <-lmp.stopCh:
-				return
-			case <-ticker.C:
-				lmp.mu.Lock()
-				if time.Since(lmp.lastFlushTime) >= d {
-					lmp.flushLocked()
-				}
-				lmp.mu.Unlock()
-			}
-		}
-	}()
-}
-
-// AddRow adds new log message to lmp with the given timestamp and fields.
-func (lmp *logMessageProcessor) AddRow(timestamp int64, fields []logstorage.Field) {
-	lmp.mu.Lock()
-	defer lmp.mu.Unlock()
-
-	if len(fields) > *MaxFieldsPerLine {
-		rf := logstorage.RowFormatter(fields)
-		logger.Warnf("dropping log line with %d fields; it exceeds -insert.maxFieldsPerLine=%d; %s", len(fields), *MaxFieldsPerLine, rf)
-		rowsDroppedTotalTooManyFields.Inc()
-		return
-	}
-
-	lmp.lr.MustAdd(lmp.cp.TenantID, timestamp, fields)
-	if lmp.cp.Debug {
-		s := lmp.lr.GetRowString(0)
-		lmp.lr.ResetKeepSettings()
-		logger.Infof("remoteAddr=%s; requestURI=%s; ignoring log entry because of `debug` query arg: %s", lmp.cp.DebugRemoteAddr, lmp.cp.DebugRequestURI, s)
-		rowsDroppedTotalDebug.Inc()
-		return
-	}
-	if lmp.lr.NeedFlush() {
-		lmp.flushLocked()
-	}
-}
-
-// flushLocked must be called under locked lmp.mu.
-func (lmp *logMessageProcessor) flushLocked() {
-	lmp.lastFlushTime = time.Now()
-	vlstorage.MustAddRows(lmp.lr)
-	lmp.lr.ResetKeepSettings()
-}
-
-// MustClose flushes the remaining data to the underlying storage and closes lmp.
-func (lmp *logMessageProcessor) MustClose() {
-	close(lmp.stopCh)
-	lmp.wg.Wait()
-
-	lmp.flushLocked()
-	logstorage.PutLogRows(lmp.lr)
-	lmp.lr = nil
-}
-
-// NewLogMessageProcessor returns new LogMessageProcessor for the given cp.
-//
-// MustClose() must be called on the returned LogMessageProcessor when it is no longer needed.
-func (cp *CommonParams) NewLogMessageProcessor() LogMessageProcessor {
-	lr := logstorage.GetLogRows(cp.StreamFields, cp.IgnoreFields)
-	lmp := &logMessageProcessor{
-		cp: cp,
-		lr: lr,
-
-		stopCh: make(chan struct{}),
-	}
-	lmp.initPeriodicFlush()
-
-	return lmp
-}
-
-var rowsDroppedTotalDebug = metrics.NewCounter(`vl_rows_dropped_total{reason="debug"}`)
-var rowsDroppedTotalTooManyFields = metrics.NewCounter(`vl_rows_dropped_total{reason="too_many_fields"}`)

app/vlinsert/insertutils/flags.go

@@ -1,15 +0,0 @@
-package insertutils
-
-import (
-	"flag"
-
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/flagutil"
-)
-
-var (
-	// MaxLineSizeBytes is the maximum length of a single line for /insert/* handlers
-	MaxLineSizeBytes = flagutil.NewBytes("insert.maxLineSizeBytes", 256*1024, "The maximum size of a single line, which can be read by /insert/* handlers")
-
-	// MaxFieldsPerLine is the maximum number of fields per line for /insert/* handlers
-	MaxFieldsPerLine = flag.Int("insert.maxFieldsPerLine", 1000, "The maximum number of log fields per line, which can be read by /insert/* handlers")
-)

app/vlinsert/insertutils/testutils.go

@@ -1,53 +0,0 @@
-package insertutils
-
-import (
-	"fmt"
-	"reflect"
-	"strings"
-
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logstorage"
-)
-
-// TestLogMessageProcessor implements LogMessageProcessor for testing.
-type TestLogMessageProcessor struct {
-	timestamps []int64
-	rows       []string
-}
-
-// AddRow adds row with the given timestamp and fields to tlp
-func (tlp *TestLogMessageProcessor) AddRow(timestamp int64, fields []logstorage.Field) {
-	tlp.timestamps = append(tlp.timestamps, timestamp)
-	tlp.rows = append(tlp.rows, string(logstorage.MarshalFieldsToJSON(nil, fields)))
-}
-
-// MustClose closes tlp.
-func (tlp *TestLogMessageProcessor) MustClose() {
-}
-
-// Verify verifies the number of rows, timestamps and results after AddRow calls.
-func (tlp *TestLogMessageProcessor) Verify(rowsExpected int, timestampsExpected []int64, resultExpected string) error {
-	result := strings.Join(tlp.rows, "\n")
-	if len(tlp.rows) != rowsExpected {
-		return fmt.Errorf("unexpected rows read; got %d; want %d;\nrows read:\n%s\nrows wanted\n%s", len(tlp.rows), rowsExpected, result, resultExpected)
-	}
-
-	if !reflect.DeepEqual(tlp.timestamps, timestampsExpected) {
-		return fmt.Errorf("unexpected timestamps;\ngot\n%d\nwant\n%d", tlp.timestamps, timestampsExpected)
-	}
-	if result != resultExpected {
-		return fmt.Errorf("unexpected result;\ngot\n%s\nwant\n%s", result, resultExpected)
-	}
-
-	return nil
-}
-
-// BenchmarkLogMessageProcessor implements LogMessageProcessor for benchmarks.
-type BenchmarkLogMessageProcessor struct{}
-
-// AddRow implements LogMessageProcessor interface.
-func (blp *BenchmarkLogMessageProcessor) AddRow(_ int64, _ []logstorage.Field) {
-}
-
-// MustClose implements LogMessageProcessor interface.
-func (blp *BenchmarkLogMessageProcessor) MustClose() {
-}

app/vlinsert/insertutils/timestamp.go

@@ -1,33 +0,0 @@
-package insertutils
-
-import (
-	"fmt"
-	"time"
-
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logstorage"
-)
-
-// ExtractTimestampRFC3339NanoFromFields extracts RFC3339 timestamp in nanoseconds from the field with the name timeField at fields.
-//
-// The value for the timeField is set to empty string after returning from the function,
-// so it could be ignored during data ingestion.
-//
-// The current timestamp is returned if fields do not contain a field with timeField name or if the timeField value is empty.
-func ExtractTimestampRFC3339NanoFromFields(timeField string, fields []logstorage.Field) (int64, error) {
-	for i := range fields {
-		f := &fields[i]
-		if f.Name != timeField {
-			continue
-		}
-		if f.Value == "" || f.Value == "0" {
-			return time.Now().UnixNano(), nil
-		}
-		nsecs, ok := logstorage.TryParseTimestampRFC3339Nano(f.Value)
-		if !ok {
-			return 0, fmt.Errorf("cannot unmarshal rfc3339 timestamp from %s=%q", timeField, f.Value)
-		}
-		f.Value = ""
-		return nsecs, nil
-	}
-	return time.Now().UnixNano(), nil
-}

app/vlinsert/insertutils/timestamp_test.go

@@ -1,75 +0,0 @@
-package insertutils
-
-import (
-	"testing"
-
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logstorage"
-)
-
-func TestExtractTimestampRFC3339NanoFromFields_Success(t *testing.T) {
-	f := func(timeField string, fields []logstorage.Field, nsecsExpected int64) {
-		t.Helper()
-
-		nsecs, err := ExtractTimestampRFC3339NanoFromFields(timeField, fields)
-		if err != nil {
-			t.Fatalf("unexpected error: %s", err)
-		}
-		if nsecs != nsecsExpected {
-			t.Fatalf("unexpected nsecs; got %d; want %d", nsecs, nsecsExpected)
-		}
-
-		for _, f := range fields {
-			if f.Name == timeField {
-				if f.Value != "" {
-					t.Fatalf("unexpected value for field %s; got %q; want %q", timeField, f.Value, "")
-				}
-			}
-		}
-	}
-
-	f("time", []logstorage.Field{
-		{Name: "foo", Value: "bar"},
-		{Name: "time", Value: "2024-06-18T23:37:20Z"},
-	}, 1718753840000000000)
-
-	f("time", []logstorage.Field{
-		{Name: "foo", Value: "bar"},
-		{Name: "time", Value: "2024-06-18T23:37:20+08:00"},
-	}, 1718725040000000000)
-
-	f("time", []logstorage.Field{
-		{Name: "foo", Value: "bar"},
-		{Name: "time", Value: "2024-06-18T23:37:20.123-05:30"},
-	}, 1718773640123000000)
-
-	f("time", []logstorage.Field{
-		{Name: "time", Value: "2024-06-18T23:37:20.123456789-05:30"},
-		{Name: "foo", Value: "bar"},
-	}, 1718773640123456789)
-}
-
-func TestExtractTimestampRFC3339NanoFromFields_Error(t *testing.T) {
-	f := func(s string) {
-		t.Helper()
-
-		fields := []logstorage.Field{
-			{Name: "time", Value: s},
-		}
-		nsecs, err := ExtractTimestampRFC3339NanoFromFields("time", fields)
-		if err == nil {
-			t.Fatalf("expecting non-nil error")
-		}
-		if nsecs != 0 {
-			t.Fatalf("unexpected nsecs; got %d; want %d", nsecs, 0)
-		}
-	}
-
-	f("foobar")
-
-	// no Z at the end
-	f("2024-06-18T23:37:20")
-
-	// incomplete time
-	f("2024-06-18")
-	f("2024-06-18T23:37")
-}

app/vlinsert/jsonline/jsonline.go

@@ -1,135 +0,0 @@
-package jsonline
-
-import (
-	"bufio"
-	"errors"
-	"fmt"
-	"io"
-	"net/http"
-	"time"
-
-	"github.com/VictoriaMetrics/VictoriaMetrics/app/vlinsert/insertutils"
-	"github.com/VictoriaMetrics/VictoriaMetrics/app/vlstorage"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/bytesutil"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logstorage"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/protoparser/common"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/writeconcurrencylimiter"
-	"github.com/VictoriaMetrics/metrics"
-)
-
-// RequestHandler processes jsonline insert requests
-func RequestHandler(w http.ResponseWriter, r *http.Request) {
-	startTime := time.Now()
-	w.Header().Add("Content-Type", "application/json")
-
-	if r.Method != "POST" {
-		w.WriteHeader(http.StatusMethodNotAllowed)
-		return
-	}
-
-	requestsTotal.Inc()
-
-	cp, err := insertutils.GetCommonParams(r)
-	if err != nil {
-		httpserver.Errorf(w, r, "%s", err)
-		return
-	}
-	if err := vlstorage.CanWriteData(); err != nil {
-		httpserver.Errorf(w, r, "%s", err)
-		return
-	}
-
-	reader := r.Body
-	if r.Header.Get("Content-Encoding") == "gzip" {
-		zr, err := common.GetGzipReader(reader)
-		if err != nil {
-			logger.Errorf("cannot read gzipped jsonline request: %s", err)
-			return
-		}
-		defer common.PutGzipReader(zr)
-		reader = zr
-	}
-
-	lmp := cp.NewLogMessageProcessor()
-	err = processStreamInternal(reader, cp.TimeField, cp.MsgField, lmp)
-	lmp.MustClose()
-
-	if err != nil {
-		logger.Errorf("jsonline: %s", err)
-	} else {
-		// update requestDuration only for successfully parsed requests.
-		// There is no need in updating requestDuration for request errors,
-		// since their timings are usually much smaller than the timing for successful request parsing.
-		requestDuration.UpdateDuration(startTime)
-	}
-}
-
-func processStreamInternal(r io.Reader, timeField, msgField string, lmp insertutils.LogMessageProcessor) error {
-	wcr := writeconcurrencylimiter.GetReader(r)
-	defer writeconcurrencylimiter.PutReader(wcr)
-
-	lb := lineBufferPool.Get()
-	defer lineBufferPool.Put(lb)
-
-	lb.B = bytesutil.ResizeNoCopyNoOverallocate(lb.B, insertutils.MaxLineSizeBytes.IntN())
-	sc := bufio.NewScanner(wcr)
-	sc.Buffer(lb.B, len(lb.B))
-
-	n := 0
-	for {
-		ok, err := readLine(sc, timeField, msgField, lmp)
-		wcr.DecConcurrency()
-		if err != nil {
-			errorsTotal.Inc()
-			return fmt.Errorf("cannot read line #%d in /jsonline request: %s", n, err)
-		}
-		if !ok {
-			return nil
-		}
-		n++
-		rowsIngestedTotal.Inc()
-	}
-}
-
-func readLine(sc *bufio.Scanner, timeField, msgField string, lmp insertutils.LogMessageProcessor) (bool, error) {
-	var line []byte
-	for len(line) == 0 {
-		if !sc.Scan() {
-			if err := sc.Err(); err != nil {
-				if errors.Is(err, bufio.ErrTooLong) {
-					return false, fmt.Errorf(`cannot read json line, since its size exceeds -insert.maxLineSizeBytes=%d`, insertutils.MaxLineSizeBytes.IntN())
-				}
-				return false, err
-			}
-			return false, nil
-		}
-		line = sc.Bytes()
-	}
-
-	p := logstorage.GetJSONParser()
-	if err := p.ParseLogMessage(line); err != nil {
-		return false, fmt.Errorf("cannot parse json-encoded log entry: %w", err)
-	}
-	ts, err := insertutils.ExtractTimestampRFC3339NanoFromFields(timeField, p.Fields)
-	if err != nil {
-		return false, fmt.Errorf("cannot get timestamp: %w", err)
-	}
-	logstorage.RenameField(p.Fields, msgField, "_msg")
-	lmp.AddRow(ts, p.Fields)
-	logstorage.PutJSONParser(p)
-
-	return true, nil
-}
-
-var lineBufferPool bytesutil.ByteBufferPool
-
-var (
-	rowsIngestedTotal = metrics.NewCounter(`vl_rows_ingested_total{type="jsonline"}`)
-
-	requestsTotal = metrics.NewCounter(`vl_http_requests_total{path="/insert/jsonline"}`)
-	errorsTotal   = metrics.NewCounter(`vl_http_errors_total{path="/insert/jsonline"}`)
-
-	requestDuration = metrics.NewHistogram(`vl_http_request_duration_seconds{path="/insert/jsonline"}`)
-)

app/vlinsert/jsonline/jsonline_test.go

@@ -1,55 +0,0 @@
-package jsonline
-
-import (
-	"bytes"
-	"testing"
-
-	"github.com/VictoriaMetrics/VictoriaMetrics/app/vlinsert/insertutils"
-)
-
-func TestProcessStreamInternal_Success(t *testing.T) {
-	f := func(data, timeField, msgField string, rowsExpected int, timestampsExpected []int64, resultExpected string) {
-		t.Helper()
-
-		tlp := &insertutils.TestLogMessageProcessor{}
-		r := bytes.NewBufferString(data)
-		if err := processStreamInternal(r, timeField, msgField, tlp); err != nil {
-			t.Fatalf("unexpected error: %s", err)
-		}
-
-		if err := tlp.Verify(rowsExpected, timestampsExpected, resultExpected); err != nil {
-			t.Fatal(err)
-		}
-	}
-
-	data := `{"@timestamp":"2023-06-06T04:48:11.735Z","log":{"offset":71770,"file":{"path":"/var/log/auth.log"}},"message":"foobar"}
-{"@timestamp":"2023-06-06T04:48:12.735Z","message":"baz"}
-{"message":"xyz","@timestamp":"2023-06-06T04:48:13.735Z","x":"y"}
-`
-	timeField := "@timestamp"
-	msgField := "message"
-	rowsExpected := 3
-	timestampsExpected := []int64{1686026891735000000, 1686026892735000000, 1686026893735000000}
-	resultExpected := `{"@timestamp":"","log.offset":"71770","log.file.path":"/var/log/auth.log","_msg":"foobar"}
-{"@timestamp":"","_msg":"baz"}
-{"_msg":"xyz","@timestamp":"","x":"y"}`
-	f(data, timeField, msgField, rowsExpected, timestampsExpected, resultExpected)
-}
-
-func TestProcessStreamInternal_Failure(t *testing.T) {
-	f := func(data string) {
-		t.Helper()
-
-		tlp := &insertutils.TestLogMessageProcessor{}
-		r := bytes.NewBufferString(data)
-		if err := processStreamInternal(r, "time", "", tlp); err == nil {
-			t.Fatalf("expecting non-nil error")
-		}
-	}
-
-	// invalid json
-	f("foobar")
-
-	// invalid timestamp field
-	f(`{"time":"foobar"}`)
-}

app/vlinsert/loki/loki.go

@@ -1,59 +0,0 @@
-package loki
-
-import (
-	"net/http"
-
-	"github.com/VictoriaMetrics/VictoriaMetrics/app/vlinsert/insertutils"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logstorage"
-)
-
-// RequestHandler processes Loki insert requests
-func RequestHandler(path string, w http.ResponseWriter, r *http.Request) bool {
-	switch path {
-	case "/api/v1/push":
-		handleInsert(r, w)
-		return true
-	case "/ready":
-		// See https://grafana.com/docs/loki/latest/api/#identify-ready-loki-instance
-		w.WriteHeader(http.StatusOK)
-		w.Write([]byte("ready"))
-		return true
-	default:
-		return false
-	}
-}
-
-// See https://grafana.com/docs/loki/latest/api/#push-log-entries-to-loki
-func handleInsert(r *http.Request, w http.ResponseWriter) {
-	contentType := r.Header.Get("Content-Type")
-	switch contentType {
-	case "application/json":
-		handleJSON(r, w)
-	default:
-		// Protobuf request body should be handled by default according to https://grafana.com/docs/loki/latest/api/#push-log-entries-to-loki
-		handleProtobuf(r, w)
-	}
-}
-
-func getCommonParams(r *http.Request) (*insertutils.CommonParams, error) {
-	cp, err := insertutils.GetCommonParams(r)
-	if err != nil {
-		return nil, err
-	}
-
-	// If parsed tenant is (0,0) it is likely to be default tenant
-	// Try parsing tenant from Loki headers
-	if cp.TenantID.AccountID == 0 && cp.TenantID.ProjectID == 0 {
-		org := r.Header.Get("X-Scope-OrgID")
-		if org != "" {
-			tenantID, err := logstorage.ParseTenantID(org)
-			if err != nil {
-				return nil, err
-			}
-			cp.TenantID = tenantID
-		}
-
-	}
-
-	return cp, nil
-}

app/vlinsert/loki/loki_json.go

@@ -1,202 +0,0 @@
-package loki
-
-import (
-	"fmt"
-	"io"
-	"math"
-	"net/http"
-	"strconv"
-	"time"
-
-	"github.com/VictoriaMetrics/VictoriaMetrics/app/vlinsert/insertutils"
-	"github.com/VictoriaMetrics/VictoriaMetrics/app/vlstorage"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/bytesutil"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logstorage"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/protoparser/common"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/writeconcurrencylimiter"
-	"github.com/VictoriaMetrics/metrics"
-	"github.com/valyala/fastjson"
-)
-
-var parserPool fastjson.ParserPool
-
-func handleJSON(r *http.Request, w http.ResponseWriter) {
-	startTime := time.Now()
-	requestsJSONTotal.Inc()
-	reader := r.Body
-	if r.Header.Get("Content-Encoding") == "gzip" {
-		zr, err := common.GetGzipReader(reader)
-		if err != nil {
-			httpserver.Errorf(w, r, "cannot initialize gzip reader: %s", err)
-			return
-		}
-		defer common.PutGzipReader(zr)
-		reader = zr
-	}
-
-	wcr := writeconcurrencylimiter.GetReader(reader)
-	data, err := io.ReadAll(wcr)
-	writeconcurrencylimiter.PutReader(wcr)
-	if err != nil {
-		httpserver.Errorf(w, r, "cannot read request body: %s", err)
-		return
-	}
-
-	cp, err := getCommonParams(r)
-	if err != nil {
-		httpserver.Errorf(w, r, "cannot parse common params from request: %s", err)
-		return
-	}
-	if err := vlstorage.CanWriteData(); err != nil {
-		httpserver.Errorf(w, r, "%s", err)
-		return
-	}
-	lmp := cp.NewLogMessageProcessor()
-	n, err := parseJSONRequest(data, lmp)
-	lmp.MustClose()
-	if err != nil {
-		httpserver.Errorf(w, r, "cannot parse Loki json request: %s", err)
-		return
-	}
-
-	rowsIngestedJSONTotal.Add(n)
-
-	// update requestJSONDuration only for successfully parsed requests
-	// There is no need in updating requestJSONDuration for request errors,
-	// since their timings are usually much smaller than the timing for successful request parsing.
-	requestJSONDuration.UpdateDuration(startTime)
-}
-
-var (
-	requestsJSONTotal     = metrics.NewCounter(`vl_http_requests_total{path="/insert/loki/api/v1/push",format="json"}`)
-	rowsIngestedJSONTotal = metrics.NewCounter(`vl_rows_ingested_total{type="loki",format="json"}`)
-	requestJSONDuration   = metrics.NewHistogram(`vl_http_request_duration_seconds{path="/insert/loki/api/v1/push",format="json"}`)
-)
-
-func parseJSONRequest(data []byte, lmp insertutils.LogMessageProcessor) (int, error) {
-	p := parserPool.Get()
-	defer parserPool.Put(p)
-	v, err := p.ParseBytes(data)
-	if err != nil {
-		return 0, fmt.Errorf("cannot parse JSON request body: %w", err)
-	}
-
-	streamsV := v.Get("streams")
-	if streamsV == nil {
-		return 0, fmt.Errorf("missing `streams` item in the parsed JSON: %q", v)
-	}
-	streams, err := streamsV.Array()
-	if err != nil {
-		return 0, fmt.Errorf("`streams` item in the parsed JSON must contain an array; got %q", streamsV)
-	}
-
-	currentTimestamp := time.Now().UnixNano()
-	var commonFields []logstorage.Field
-	rowsIngested := 0
-	for _, stream := range streams {
-		// populate common labels from `stream` dict
-		commonFields = commonFields[:0]
-		labelsV := stream.Get("stream")
-		var labels *fastjson.Object
-		if labelsV != nil {
-			o, err := labelsV.Object()
-			if err != nil {
-				return rowsIngested, fmt.Errorf("`stream` item in the parsed JSON must contain an object; got %q", labelsV)
-			}
-			labels = o
-		}
-		labels.Visit(func(k []byte, v *fastjson.Value) {
-			if err != nil {
-				return
-			}
-			vStr, errLocal := v.StringBytes()
-			if errLocal != nil {
-				err = fmt.Errorf("unexpected label value type for %q:%q; want string", k, v)
-				return
-			}
-			commonFields = append(commonFields, logstorage.Field{
-				Name:  bytesutil.ToUnsafeString(k),
-				Value: bytesutil.ToUnsafeString(vStr),
-			})
-		})
-		if err != nil {
-			return rowsIngested, fmt.Errorf("error when parsing `stream` object: %w", err)
-		}
-
-		// populate messages from `values` array
-		linesV := stream.Get("values")
-		if linesV == nil {
-			return rowsIngested, fmt.Errorf("missing `values` item in the parsed JSON %q", stream)
-		}
-		lines, err := linesV.Array()
-		if err != nil {
-			return rowsIngested, fmt.Errorf("`values` item in the parsed JSON must contain an array; got %q", linesV)
-		}
-
-		fields := commonFields
-		for _, line := range lines {
-			lineA, err := line.Array()
-			if err != nil {
-				return rowsIngested, fmt.Errorf("unexpected contents of `values` item; want array; got %q", line)
-			}
-			if len(lineA) != 2 {
-				return rowsIngested, fmt.Errorf("unexpected number of values in `values` item array %q; got %d want 2", line, len(lineA))
-			}
-
-			// parse timestamp
-			timestamp, err := lineA[0].StringBytes()
-			if err != nil {
-				return rowsIngested, fmt.Errorf("unexpected log timestamp type for %q; want string", lineA[0])
-			}
-			ts, err := parseLokiTimestamp(bytesutil.ToUnsafeString(timestamp))
-			if err != nil {
-				return rowsIngested, fmt.Errorf("cannot parse log timestamp %q: %w", timestamp, err)
-			}
-			if ts == 0 {
-				ts = currentTimestamp
-			}
-
-			// parse log message
-			msg, err := lineA[1].StringBytes()
-			if err != nil {
-				return rowsIngested, fmt.Errorf("unexpected log message type for %q; want string", lineA[1])
-			}
-
-			fields = append(fields[:len(commonFields)], logstorage.Field{
-				Name:  "_msg",
-				Value: bytesutil.ToUnsafeString(msg),
-			})
-			lmp.AddRow(ts, fields)
-		}
-		rowsIngested += len(lines)
-	}
-
-	return rowsIngested, nil
-}
-
-func parseLokiTimestamp(s string) (int64, error) {
-	if s == "" {
-		// Special case - an empty timestamp must be substituted with the current time by the caller.
-		return 0, nil
-	}
-	n, err := strconv.ParseInt(s, 10, 64)
-	if err != nil {
-		// Fall back to parsing floating-point value
-		f, err := strconv.ParseFloat(s, 64)
-		if err != nil {
-			return 0, err
-		}
-		if f > math.MaxInt64 {
-			return 0, fmt.Errorf("too big timestamp in nanoseconds: %v; mustn't exceed %v", f, int64(math.MaxInt64))
-		}
-		if f < math.MinInt64 {
-			return 0, fmt.Errorf("too small timestamp in nanoseconds: %v; must be bigger or equal to %v", f, int64(math.MinInt64))
-		}
-		n = int64(f)
-	}
-	if n < 0 {
-		return 0, fmt.Errorf("too small timestamp in nanoseconds: %d; must be bigger than 0", n)
-	}
-	return n, nil
-}

app/vlinsert/loki/loki_json_test.go

@@ -1,119 +0,0 @@
-package loki
-
-import (
-	"testing"
-
-	"github.com/VictoriaMetrics/VictoriaMetrics/app/vlinsert/insertutils"
-)
-
-func TestParseJSONRequest_Failure(t *testing.T) {
-	f := func(s string) {
-		t.Helper()
-
-		tlp := &insertutils.TestLogMessageProcessor{}
-		n, err := parseJSONRequest([]byte(s), tlp)
-		if err == nil {
-			t.Fatalf("expecting non-nil error")
-		}
-		if n != 0 {
-			t.Fatalf("unexpected number of parsed lines: %d; want 0", n)
-		}
-	}
-	f(``)
-
-	// Invalid json
-	f(`{}`)
-	f(`[]`)
-	f(`"foo"`)
-	f(`123`)
-
-	// invalid type for `streams` item
-	f(`{"streams":123}`)
-
-	// Missing `values` item
-	f(`{"streams":[{}]}`)
-
-	// Invalid type for `values` item
-	f(`{"streams":[{"values":"foobar"}]}`)
-
-	// Invalid type for `stream` item
-	f(`{"streams":[{"stream":[],"values":[]}]}`)
-
-	// Invalid type for `values` individual item
-	f(`{"streams":[{"values":[123]}]}`)
-
-	// Invalid length of `values` individual item
-	f(`{"streams":[{"values":[[]]}]}`)
-	f(`{"streams":[{"values":[["123"]]}]}`)
-	f(`{"streams":[{"values":[["123","456","789"]]}]}`)
-
-	// Invalid type for timestamp inside `values` individual item
-	f(`{"streams":[{"values":[[123,"456"]}]}`)
-
-	// Invalid type for log message
-	f(`{"streams":[{"values":[["123",1234]]}]}`)
-}
-
-func TestParseJSONRequest_Success(t *testing.T) {
-	f := func(s string, timestampsExpected []int64, resultExpected string) {
-		t.Helper()
-
-		tlp := &insertutils.TestLogMessageProcessor{}
-
-		n, err := parseJSONRequest([]byte(s), tlp)
-		if err != nil {
-			t.Fatalf("unexpected error: %s", err)
-		}
-		if err := tlp.Verify(n, timestampsExpected, resultExpected); err != nil {
-			t.Fatal(err)
-		}
-	}
-
-	// Empty streams
-	f(`{"streams":[]}`, nil, ``)
-	f(`{"streams":[{"values":[]}]}`, nil, ``)
-	f(`{"streams":[{"stream":{},"values":[]}]}`, nil, ``)
-	f(`{"streams":[{"stream":{"foo":"bar"},"values":[]}]}`, nil, ``)
-
-	// Empty stream labels
-	f(`{"streams":[{"values":[["1577836800000000001", "foo bar"]]}]}`, []int64{1577836800000000001}, `{"_msg":"foo bar"}`)
-	f(`{"streams":[{"stream":{},"values":[["1577836800000000001", "foo bar"]]}]}`, []int64{1577836800000000001}, `{"_msg":"foo bar"}`)
-
-	// Non-empty stream labels
-	f(`{"streams":[{"stream":{
-	"label1": "value1",
-	"label2": "value2"
-},"values":[
-	["1577836800000000001", "foo bar"],
-	["1477836900005000002", "abc"],
-	["147.78369e9", "foobar"]
-]}]}`, []int64{1577836800000000001, 1477836900005000002, 147783690000}, `{"label1":"value1","label2":"value2","_msg":"foo bar"}
-{"label1":"value1","label2":"value2","_msg":"abc"}
-{"label1":"value1","label2":"value2","_msg":"foobar"}`)
-
-	// Multiple streams
-	f(`{
-	"streams": [
-		{
-			"stream": {
-				"foo": "bar",
-				"a": "b"
-			},
-			"values": [
-				["1577836800000000001", "foo bar"],
-				["1577836900005000002", "abc"]
-			]
-		},
-		{
-			"stream": {
-				"x": "y"
-			},
-			"values": [
-				["1877836900005000002", "yx"]
-			]
-		}
-	]
-}`, []int64{1577836800000000001, 1577836900005000002, 1877836900005000002}, `{"foo":"bar","a":"b","_msg":"foo bar"}
-{"foo":"bar","a":"b","_msg":"abc"}
-{"x":"y","_msg":"yx"}`)
-}

app/vlinsert/loki/loki_json_timing_test.go

@@ -1,79 +0,0 @@
-package loki
-
-import (
-	"fmt"
-	"strconv"
-	"testing"
-	"time"
-
-	"github.com/VictoriaMetrics/VictoriaMetrics/app/vlinsert/insertutils"
-)
-
-func BenchmarkParseJSONRequest(b *testing.B) {
-	for _, streams := range []int{5, 10} {
-		for _, rows := range []int{100, 1000} {
-			for _, labels := range []int{10, 50} {
-				b.Run(fmt.Sprintf("streams_%d/rows_%d/labels_%d", streams, rows, labels), func(b *testing.B) {
-					benchmarkParseJSONRequest(b, streams, rows, labels)
-				})
-			}
-		}
-	}
-}
-
-func benchmarkParseJSONRequest(b *testing.B, streams, rows, labels int) {
-	blp := &insertutils.BenchmarkLogMessageProcessor{}
-	b.ReportAllocs()
-	b.SetBytes(int64(streams * rows))
-	b.RunParallel(func(pb *testing.PB) {
-		data := getJSONBody(streams, rows, labels)
-		for pb.Next() {
-			_, err := parseJSONRequest(data, blp)
-			if err != nil {
-				panic(fmt.Errorf("unexpected error: %w", err))
-			}
-		}
-	})
-}
-
-func getJSONBody(streams, rows, labels int) []byte {
-	body := append([]byte{}, `{"streams":[`...)
-	now := time.Now().UnixNano()
-	valuePrefix := fmt.Sprintf(`["%d","value_`, now)
-
-	for i := 0; i < streams; i++ {
-		body = append(body, `{"stream":{`...)
-
-		for j := 0; j < labels; j++ {
-			body = append(body, `"label_`...)
-			body = strconv.AppendInt(body, int64(j), 10)
-			body = append(body, `":"value_`...)
-			body = strconv.AppendInt(body, int64(j), 10)
-			body = append(body, '"')
-			if j < labels-1 {
-				body = append(body, ',')
-			}
-
-		}
-		body = append(body, `}, "values":[`...)
-
-		for j := 0; j < rows; j++ {
-			body = append(body, valuePrefix...)
-			body = strconv.AppendInt(body, int64(j), 10)
-			body = append(body, `"]`...)
-			if j < rows-1 {
-				body = append(body, ',')
-			}
-		}
-
-		body = append(body, `]}`...)
-		if i < streams-1 {
-			body = append(body, ',')
-		}
-
-	}
-
-	body = append(body, `]}`...)
-
-	return body
-}

app/vlinsert/loki/loki_protobuf.go

@@ -1,218 +0,0 @@
-package loki
-
-import (
-	"fmt"
-	"io"
-	"net/http"
-	"strconv"
-	"strings"
-	"sync"
-	"time"
-
-	"github.com/VictoriaMetrics/VictoriaMetrics/app/vlinsert/insertutils"
-	"github.com/VictoriaMetrics/VictoriaMetrics/app/vlstorage"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/bytesutil"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logstorage"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/writeconcurrencylimiter"
-	"github.com/VictoriaMetrics/metrics"
-	"github.com/golang/snappy"
-)
-
-var (
-	bytesBufPool bytesutil.ByteBufferPool
-	pushReqsPool sync.Pool
-)
-
-func handleProtobuf(r *http.Request, w http.ResponseWriter) {
-	startTime := time.Now()
-	requestsProtobufTotal.Inc()
-	wcr := writeconcurrencylimiter.GetReader(r.Body)
-	data, err := io.ReadAll(wcr)
-	writeconcurrencylimiter.PutReader(wcr)
-	if err != nil {
-		httpserver.Errorf(w, r, "cannot read request body: %s", err)
-		return
-	}
-
-	cp, err := getCommonParams(r)
-	if err != nil {
-		httpserver.Errorf(w, r, "cannot parse common params from request: %s", err)
-		return
-	}
-	if err := vlstorage.CanWriteData(); err != nil {
-		httpserver.Errorf(w, r, "%s", err)
-		return
-	}
-	lmp := cp.NewLogMessageProcessor()
-	n, err := parseProtobufRequest(data, lmp)
-	lmp.MustClose()
-	if err != nil {
-		httpserver.Errorf(w, r, "cannot parse Loki protobuf request: %s", err)
-		return
-	}
-
-	rowsIngestedProtobufTotal.Add(n)
-
-	// update requestProtobufDuration only for successfully parsed requests
-	// There is no need in updating requestProtobufDuration for request errors,
-	// since their timings are usually much smaller than the timing for successful request parsing.
-	requestProtobufDuration.UpdateDuration(startTime)
-}
-
-var (
-	requestsProtobufTotal     = metrics.NewCounter(`vl_http_requests_total{path="/insert/loki/api/v1/push",format="protobuf"}`)
-	rowsIngestedProtobufTotal = metrics.NewCounter(`vl_rows_ingested_total{type="loki",format="protobuf"}`)
-	requestProtobufDuration   = metrics.NewHistogram(`vl_http_request_duration_seconds{path="/insert/loki/api/v1/push",format="protobuf"}`)
-)
-
-func parseProtobufRequest(data []byte, lmp insertutils.LogMessageProcessor) (int, error) {
-	bb := bytesBufPool.Get()
-	defer bytesBufPool.Put(bb)
-
-	buf, err := snappy.Decode(bb.B[:cap(bb.B)], data)
-	if err != nil {
-		return 0, fmt.Errorf("cannot decode snappy-encoded request body: %w", err)
-	}
-	bb.B = buf
-
-	req := getPushRequest()
-	defer putPushRequest(req)
-
-	err = req.UnmarshalProtobuf(bb.B)
-	if err != nil {
-		return 0, fmt.Errorf("cannot parse request body: %w", err)
-	}
-
-	fields := getFields()
-	defer putFields(fields)
-
-	rowsIngested := 0
-	streams := req.Streams
-	currentTimestamp := time.Now().UnixNano()
-	for i := range streams {
-		stream := &streams[i]
-		// st.Labels contains labels for the stream.
-		// Labels are same for all entries in the stream.
-		fields.fields, err = parsePromLabels(fields.fields[:0], stream.Labels)
-		if err != nil {
-			return rowsIngested, fmt.Errorf("cannot parse stream labels %q: %w", stream.Labels, err)
-		}
-		commonFieldsLen := len(fields.fields)
-
-		entries := stream.Entries
-		for j := range entries {
-			e := &entries[j]
-			fields.fields = fields.fields[:commonFieldsLen]
-
-			for _, lp := range e.StructuredMetadata {
-				fields.fields = append(fields.fields, logstorage.Field{
-					Name:  lp.Name,
-					Value: lp.Value,
-				})
-			}
-
-			fields.fields = append(fields.fields, logstorage.Field{
-				Name:  "_msg",
-				Value: e.Line,
-			})
-
-			ts := e.Timestamp.UnixNano()
-			if ts == 0 {
-				ts = currentTimestamp
-			}
-
-			lmp.AddRow(ts, fields.fields)
-		}
-		rowsIngested += len(stream.Entries)
-	}
-	return rowsIngested, nil
-}
-
-func getFields() *fields {
-	v := fieldsPool.Get()
-	if v == nil {
-		return &fields{}
-	}
-	return v.(*fields)
-}
-
-func putFields(f *fields) {
-	f.fields = f.fields[:0]
-	fieldsPool.Put(f)
-}
-
-var fieldsPool sync.Pool
-
-type fields struct {
-	fields []logstorage.Field
-}
-
-// parsePromLabels parses log fields in Prometheus text exposition format from s, appends them to dst and returns the result.
-//
-// See test data of promtail for examples: https://github.com/grafana/loki/blob/a24ef7b206e0ca63ee74ca6ecb0a09b745cd2258/pkg/push/types_test.go
-func parsePromLabels(dst []logstorage.Field, s string) ([]logstorage.Field, error) {
-	// Make sure s is wrapped into `{...}`
-	s = strings.TrimSpace(s)
-	if len(s) < 2 {
-		return nil, fmt.Errorf("too short string to parse: %q", s)
-	}
-	if s[0] != '{' {
-		return nil, fmt.Errorf("missing `{` at the beginning of %q", s)
-	}
-	if s[len(s)-1] != '}' {
-		return nil, fmt.Errorf("missing `}` at the end of %q", s)
-	}
-	s = s[1 : len(s)-1]
-
-	for len(s) > 0 {
-		// Parse label name
-		n := strings.IndexByte(s, '=')
-		if n < 0 {
-			return nil, fmt.Errorf("cannot find `=` char for label value at %s", s)
-		}
-		name := s[:n]
-		s = s[n+1:]
-
-		// Parse label value
-		qs, err := strconv.QuotedPrefix(s)
-		if err != nil {
-			return nil, fmt.Errorf("cannot parse value for label %q at %s: %w", name, s, err)
-		}
-		s = s[len(qs):]
-		value, err := strconv.Unquote(qs)
-		if err != nil {
-			return nil, fmt.Errorf("cannot unquote value %q for label %q: %w", qs, name, err)
-		}
-
-		// Append the found field to dst.
-		dst = append(dst, logstorage.Field{
-			Name:  name,
-			Value: value,
-		})
-
-		// Check whether there are other labels remaining
-		if len(s) == 0 {
-			break
-		}
-		if !strings.HasPrefix(s, ",") {
-			return nil, fmt.Errorf("missing `,` char at %s", s)
-		}
-		s = s[1:]
-		s = strings.TrimPrefix(s, " ")
-	}
-	return dst, nil
-}
-
-func getPushRequest() *PushRequest {
-	v := pushReqsPool.Get()
-	if v == nil {
-		return &PushRequest{}
-	}
-	return v.(*PushRequest)
-}
-
-func putPushRequest(req *PushRequest) {
-	req.reset()
-	pushReqsPool.Put(req)
-}

app/vlinsert/loki/loki_protobuf_test.go

@@ -1,168 +0,0 @@
-package loki
-
-import (
-	"fmt"
-	"strings"
-	"testing"
-	"time"
-
-	"github.com/VictoriaMetrics/VictoriaMetrics/app/vlinsert/insertutils"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logstorage"
-	"github.com/golang/snappy"
-)
-
-type testLogMessageProcessor struct {
-	pr PushRequest
-}
-
-func (tlp *testLogMessageProcessor) AddRow(timestamp int64, fields []logstorage.Field) {
-	msg := ""
-	for _, f := range fields {
-		if f.Name == "_msg" {
-			msg = f.Value
-		}
-	}
-	var a []string
-	for _, f := range fields {
-		if f.Name == "_msg" {
-			continue
-		}
-		item := fmt.Sprintf("%s=%q", f.Name, f.Value)
-		a = append(a, item)
-	}
-	labels := "{" + strings.Join(a, ", ") + "}"
-	tlp.pr.Streams = append(tlp.pr.Streams, Stream{
-		Labels: labels,
-		Entries: []Entry{
-			{
-				Timestamp: time.Unix(0, timestamp),
-				Line:      strings.Clone(msg),
-			},
-		},
-	})
-}
-
-func (tlp *testLogMessageProcessor) MustClose() {
-}
-
-func TestParseProtobufRequest_Success(t *testing.T) {
-	f := func(s string, timestampsExpected []int64, resultExpected string) {
-		t.Helper()
-
-		tlp := &testLogMessageProcessor{}
-		n, err := parseJSONRequest([]byte(s), tlp)
-		if err != nil {
-			t.Fatalf("unexpected error: %s", err)
-		}
-		if n != len(tlp.pr.Streams) {
-			t.Fatalf("unexpected number of streams; got %d; want %d", len(tlp.pr.Streams), n)
-		}
-
-		data := tlp.pr.MarshalProtobuf(nil)
-		encodedData := snappy.Encode(nil, data)
-
-		tlp2 := &insertutils.TestLogMessageProcessor{}
-		n, err = parseProtobufRequest(encodedData, tlp2)
-		if err != nil {
-			t.Fatalf("unexpected error: %s", err)
-		}
-		if err := tlp2.Verify(n, timestampsExpected, resultExpected); err != nil {
-			t.Fatal(err)
-		}
-	}
-
-	// Empty streams
-	f(`{"streams":[]}`, nil, ``)
-	f(`{"streams":[{"values":[]}]}`, nil, ``)
-	f(`{"streams":[{"stream":{},"values":[]}]}`, nil, ``)
-	f(`{"streams":[{"stream":{"foo":"bar"},"values":[]}]}`, nil, ``)
-
-	// Empty stream labels
-	f(`{"streams":[{"values":[["1577836800000000001", "foo bar"]]}]}`, []int64{1577836800000000001}, `{"_msg":"foo bar"}`)
-	f(`{"streams":[{"stream":{},"values":[["1577836800000000001", "foo bar"]]}]}`, []int64{1577836800000000001}, `{"_msg":"foo bar"}`)
-
-	// Non-empty stream labels
-	f(`{"streams":[{"stream":{
-	"label1": "value1",
-	"label2": "value2"
-},"values":[
-	["1577836800000000001", "foo bar"],
-	["1477836900005000002", "abc"],
-	["147.78369e9", "foobar"]
-]}]}`, []int64{1577836800000000001, 1477836900005000002, 147783690000}, `{"label1":"value1","label2":"value2","_msg":"foo bar"}
-{"label1":"value1","label2":"value2","_msg":"abc"}
-{"label1":"value1","label2":"value2","_msg":"foobar"}`)
-
-	// Multiple streams
-	f(`{
-	"streams": [
-		{
-			"stream": {
-				"foo": "bar",
-				"a": "b"
-			},
-			"values": [
-				["1577836800000000001", "foo bar"],
-				["1577836900005000002", "abc"]
-			]
-		},
-		{
-			"stream": {
-				"x": "y"
-			},
-			"values": [
-				["1877836900005000002", "yx"]
-			]
-		}
-	]
-}`, []int64{1577836800000000001, 1577836900005000002, 1877836900005000002}, `{"foo":"bar","a":"b","_msg":"foo bar"}
-{"foo":"bar","a":"b","_msg":"abc"}
-{"x":"y","_msg":"yx"}`)
-}
-
-func TestParsePromLabels_Success(t *testing.T) {
-	f := func(s string) {
-		t.Helper()
-		fields, err := parsePromLabels(nil, s)
-		if err != nil {
-			t.Fatalf("unexpected error: %s", err)
-		}
-
-		var a []string
-		for _, f := range fields {
-			a = append(a, fmt.Sprintf("%s=%q", f.Name, f.Value))
-		}
-		result := "{" + strings.Join(a, ", ") + "}"
-		if result != s {
-			t.Fatalf("unexpected result;\ngot\n%s\nwant\n%s", result, s)
-		}
-	}
-
-	f("{}")
-	f(`{foo="bar"}`)
-	f(`{foo="bar", baz="x", y="z"}`)
-	f(`{foo="ba\"r\\z\n", a="", b="\"\\"}`)
-}
-
-func TestParsePromLabels_Failure(t *testing.T) {
-	f := func(s string) {
-		t.Helper()
-		fields, err := parsePromLabels(nil, s)
-		if err == nil {
-			t.Fatalf("expecting non-nil error")
-		}
-		if len(fields) > 0 {
-			t.Fatalf("unexpected non-empty fields: %s", fields)
-		}
-	}
-
-	f("")
-	f("{")
-	f(`{foo}`)
-	f(`{foo=bar}`)
-	f(`{foo="bar}`)
-	f(`{foo="ba\",r}`)
-	f(`{foo="bar" baz="aa"}`)
-	f(`foobar`)
-	f(`foo{bar="baz"}`)
-}

app/vlinsert/loki/loki_protobuf_timing_test.go

@@ -1,86 +0,0 @@
-package loki
-
-import (
-	"fmt"
-	"strconv"
-	"testing"
-	"time"
-
-	"github.com/golang/snappy"
-
-	"github.com/VictoriaMetrics/VictoriaMetrics/app/vlinsert/insertutils"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/bytesutil"
-)
-
-func BenchmarkParseProtobufRequest(b *testing.B) {
-	for _, streams := range []int{5, 10} {
-		for _, rows := range []int{100, 1000} {
-			for _, labels := range []int{10, 50} {
-				b.Run(fmt.Sprintf("streams_%d/rows_%d/labels_%d", streams, rows, labels), func(b *testing.B) {
-					benchmarkParseProtobufRequest(b, streams, rows, labels)
-				})
-			}
-		}
-	}
-}
-
-func benchmarkParseProtobufRequest(b *testing.B, streams, rows, labels int) {
-	blp := &insertutils.BenchmarkLogMessageProcessor{}
-	b.ReportAllocs()
-	b.SetBytes(int64(streams * rows))
-	b.RunParallel(func(pb *testing.PB) {
-		body := getProtobufBody(streams, rows, labels)
-		for pb.Next() {
-			_, err := parseProtobufRequest(body, blp)
-			if err != nil {
-				panic(fmt.Errorf("unexpected error: %w", err))
-			}
-		}
-	})
-}
-
-func getProtobufBody(streamsCount, rowsCount, labelsCount int) []byte {
-	var b []byte
-	var entries []Entry
-	streams := make([]Stream, streamsCount)
-	for i := range streams {
-		b = b[:0]
-		b = append(b, '{')
-		for j := 0; j < labelsCount; j++ {
-			b = append(b, "label_"...)
-			b = strconv.AppendInt(b, int64(j), 10)
-			b = append(b, `="value_`...)
-			b = strconv.AppendInt(b, int64(j), 10)
-			b = append(b, '"')
-			if j < labelsCount-1 {
-				b = append(b, ',')
-			}
-		}
-		b = append(b, '}')
-		labels := string(b)
-
-		var rowsBuf []byte
-		entriesLen := len(entries)
-		for j := 0; j < rowsCount; j++ {
-			rowsBufLen := len(rowsBuf)
-			rowsBuf = append(rowsBuf, "value_"...)
-			rowsBuf = strconv.AppendInt(rowsBuf, int64(j), 10)
-			entries = append(entries, Entry{
-				Timestamp: time.Now(),
-				Line:      bytesutil.ToUnsafeString(rowsBuf[rowsBufLen:]),
-			})
-		}
-
-		st := &streams[i]
-		st.Labels = labels
-		st.Entries = entries[entriesLen:]
-	}
-	pr := PushRequest{
-		Streams: streams,
-	}
-
-	body := pr.MarshalProtobuf(nil)
-	encodedBody := snappy.Encode(nil, body)
-
-	return encodedBody
-}

app/vlinsert/loki/pb.go

@@ -1,302 +0,0 @@
-// Code generated by protoc-gen-gogo. DO NOT EDIT.
-// source: push_request.proto
-// source: https://raw.githubusercontent.com/grafana/loki/main/pkg/push/push_request.proto
-// Licensed under the Apache License, Version 2.0 (the "License");
-// https://github.com/grafana/loki/blob/main/pkg/push/LICENSE
-
-package loki
-
-import (
-	"fmt"
-	"time"
-
-	"github.com/VictoriaMetrics/easyproto"
-)
-
-var mp easyproto.MarshalerPool
-
-// PushRequest represents Loki PushRequest
-//
-// See https://github.com/grafana/loki/blob/4220737a52da7ab6c9346b12d5a5d7bedbcd641d/pkg/push/push.proto#L14C1-L14C20
-type PushRequest struct {
-	Streams []Stream
-
-	entriesBuf   []Entry
-	labelPairBuf []LabelPair
-}
-
-func (pr *PushRequest) reset() {
-	pr.Streams = pr.Streams[:0]
-
-	pr.entriesBuf = pr.entriesBuf[:0]
-	pr.labelPairBuf = pr.labelPairBuf[:0]
-}
-
-// UnmarshalProtobuf unmarshals pr from protobuf message at src.
-//
-// pr remains valid until src is modified.
-func (pr *PushRequest) UnmarshalProtobuf(src []byte) error {
-	pr.reset()
-	var err error
-	pr.entriesBuf, pr.labelPairBuf, err = pr.unmarshalProtobuf(pr.entriesBuf, pr.labelPairBuf, src)
-	return err
-}
-
-// MarshalProtobuf marshals r to protobuf message, appends it to dst and returns the result.
-func (pr *PushRequest) MarshalProtobuf(dst []byte) []byte {
-	m := mp.Get()
-	pr.marshalProtobuf(m.MessageMarshaler())
-	dst = m.Marshal(dst)
-	mp.Put(m)
-	return dst
-}
-
-func (pr *PushRequest) marshalProtobuf(mm *easyproto.MessageMarshaler) {
-	for _, s := range pr.Streams {
-		s.marshalProtobuf(mm.AppendMessage(1))
-	}
-}
-
-func (pr *PushRequest) unmarshalProtobuf(entriesBuf []Entry, labelPairBuf []LabelPair, src []byte) ([]Entry, []LabelPair, error) {
-	// message PushRequest {
-	//   repeated Stream streams = 1;
-	// }
-	var err error
-	var fc easyproto.FieldContext
-	for len(src) > 0 {
-		src, err = fc.NextField(src)
-		if err != nil {
-			return entriesBuf, labelPairBuf, fmt.Errorf("cannot read next field in PushRequest: %w", err)
-		}
-		switch fc.FieldNum {
-		case 1:
-			data, ok := fc.MessageData()
-			if !ok {
-				return entriesBuf, labelPairBuf, fmt.Errorf("cannot read Stream data")
-			}
-			pr.Streams = append(pr.Streams, Stream{})
-			s := &pr.Streams[len(pr.Streams)-1]
-			entriesBuf, labelPairBuf, err = s.unmarshalProtobuf(entriesBuf, labelPairBuf, data)
-			if err != nil {
-				return entriesBuf, labelPairBuf, fmt.Errorf("cannot unmarshal Stream: %w", err)
-			}
-		}
-	}
-	return entriesBuf, labelPairBuf, nil
-}
-
-// Stream represents Loki stream.
-//
-// See https://github.com/grafana/loki/blob/4220737a52da7ab6c9346b12d5a5d7bedbcd641d/pkg/push/push.proto#L23
-type Stream struct {
-	Labels  string
-	Entries []Entry
-}
-
-func (s *Stream) marshalProtobuf(mm *easyproto.MessageMarshaler) {
-	mm.AppendString(1, s.Labels)
-	for _, e := range s.Entries {
-		e.marshalProtobuf(mm.AppendMessage(2))
-	}
-}
-
-func (s *Stream) unmarshalProtobuf(entriesBuf []Entry, labelPairBuf []LabelPair, src []byte) ([]Entry, []LabelPair, error) {
-	// message Stream {
-	//   string labels = 1;
-	//   repeated Entry entries = 2;
-	// }
-	var err error
-	var fc easyproto.FieldContext
-	entriesBufLen := len(entriesBuf)
-	for len(src) > 0 {
-		src, err = fc.NextField(src)
-		if err != nil {
-			return entriesBuf, labelPairBuf, fmt.Errorf("cannot read next field in Stream: %w", err)
-		}
-		switch fc.FieldNum {
-		case 1:
-			labels, ok := fc.String()
-			if !ok {
-				return entriesBuf, labelPairBuf, fmt.Errorf("cannot read labels")
-			}
-			s.Labels = labels
-		case 2:
-			data, ok := fc.MessageData()
-			if !ok {
-				return entriesBuf, labelPairBuf, fmt.Errorf("cannot read Entry data")
-			}
-			entriesBuf = append(entriesBuf, Entry{})
-			e := &entriesBuf[len(entriesBuf)-1]
-			labelPairBuf, err = e.unmarshalProtobuf(labelPairBuf, data)
-			if err != nil {
-				return entriesBuf, labelPairBuf, fmt.Errorf("cannot unmarshal Entry: %w", err)
-			}
-		}
-	}
-	s.Entries = entriesBuf[entriesBufLen:]
-	return entriesBuf, labelPairBuf, nil
-}
-
-// Entry represents Loki entry.
-//
-// See https://github.com/grafana/loki/blob/4220737a52da7ab6c9346b12d5a5d7bedbcd641d/pkg/push/push.proto#L38
-type Entry struct {
-	Timestamp          time.Time
-	Line               string
-	StructuredMetadata []LabelPair
-}
-
-func (e *Entry) marshalProtobuf(mm *easyproto.MessageMarshaler) {
-	marshalTime(mm, 1, e.Timestamp)
-	mm.AppendString(2, e.Line)
-	for _, lp := range e.StructuredMetadata {
-		lp.marshalProtobuf(mm.AppendMessage(3))
-	}
-}
-
-func (e *Entry) unmarshalProtobuf(labelPairBuf []LabelPair, src []byte) ([]LabelPair, error) {
-	// message Entry {
-	//   Timestamp timestamp = 1;
-	//   string line = 2;
-	//   repeated LabelPair structuredMetadata = 3;
-	// }
-	var err error
-	var fc easyproto.FieldContext
-	labelPairBufLen := len(labelPairBuf)
-	for len(src) > 0 {
-		src, err = fc.NextField(src)
-		if err != nil {
-			return labelPairBuf, fmt.Errorf("cannot read next field in Entry: %w", err)
-		}
-		switch fc.FieldNum {
-		case 1:
-			data, ok := fc.MessageData()
-			if !ok {
-				return labelPairBuf, fmt.Errorf("cannot read Timestamp data")
-			}
-			timestamp, err := unmarshalTime(data)
-			if err != nil {
-				return labelPairBuf, fmt.Errorf("cannot unmarshal Timestamp: %w", err)
-			}
-			e.Timestamp = timestamp
-		case 2:
-			line, ok := fc.String()
-			if !ok {
-				return labelPairBuf, fmt.Errorf("cannot read Line")
-			}
-			e.Line = line
-		case 3:
-			data, ok := fc.MessageData()
-			if !ok {
-				return labelPairBuf, fmt.Errorf("cannot read StructuredMetadata")
-			}
-			labelPairBuf = append(labelPairBuf, LabelPair{})
-			lp := &labelPairBuf[len(labelPairBuf)-1]
-			if err := lp.unmarshalProtobuf(data); err != nil {
-				return labelPairBuf, fmt.Errorf("cannot unmarshal StructuredMetadata: %w", err)
-			}
-		}
-	}
-	e.StructuredMetadata = labelPairBuf[labelPairBufLen:]
-	return labelPairBuf, nil
-}
-
-// LabelPair represents Loki label pair.
-//
-// See https://github.com/grafana/loki/blob/4220737a52da7ab6c9346b12d5a5d7bedbcd641d/pkg/push/push.proto#L33
-type LabelPair struct {
-	Name  string
-	Value string
-}
-
-func (lp *LabelPair) marshalProtobuf(mm *easyproto.MessageMarshaler) {
-	mm.AppendString(1, lp.Name)
-	mm.AppendString(2, lp.Value)
-}
-
-func (lp *LabelPair) unmarshalProtobuf(src []byte) (err error) {
-	// message LabelPair {
-	//   string name = 1;
-	//   string value = 2;
-	// }
-	var fc easyproto.FieldContext
-	for len(src) > 0 {
-		src, err = fc.NextField(src)
-		if err != nil {
-			return fmt.Errorf("cannot read next field in LabelPair: %w", err)
-		}
-		switch fc.FieldNum {
-		case 1:
-			name, ok := fc.String()
-			if !ok {
-				return fmt.Errorf("cannot read name")
-			}
-			lp.Name = name
-		case 2:
-			value, ok := fc.String()
-			if !ok {
-				return fmt.Errorf("cannot unmarshal value")
-			}
-			lp.Value = value
-		}
-	}
-	return nil
-}
-
-func marshalTime(mm *easyproto.MessageMarshaler, fieldNum uint32, timestamp time.Time) {
-	nsecs := timestamp.UnixNano()
-	ts := Timestamp{
-		Seconds: nsecs / 1e9,
-		Nanos:   int32(nsecs % 1e9),
-	}
-	ts.marshalProtobuf(mm.AppendMessage(fieldNum))
-}
-
-func unmarshalTime(src []byte) (time.Time, error) {
-	var ts Timestamp
-	if err := ts.unmarshalProtobuf(src); err != nil {
-		return time.Time{}, err
-	}
-	timestamp := time.Unix(ts.Seconds, int64(ts.Nanos)).UTC()
-	return timestamp, nil
-}
-
-// Timestamp is protobuf well-known timestamp type.
-type Timestamp struct {
-	Seconds int64
-	Nanos   int32
-}
-
-func (ts *Timestamp) marshalProtobuf(mm *easyproto.MessageMarshaler) {
-	mm.AppendInt64(1, ts.Seconds)
-	mm.AppendInt32(2, ts.Nanos)
-}
-
-func (ts *Timestamp) unmarshalProtobuf(src []byte) (err error) {
-	// message Timestamp {
-	//   int64 seconds = 1;
-	//   int32 nanos = 2;
-	// }
-	var fc easyproto.FieldContext
-	for len(src) > 0 {
-		src, err = fc.NextField(src)
-		if err != nil {
-			return fmt.Errorf("cannot read next field in Timestamp: %w", err)
-		}
-		switch fc.FieldNum {
-		case 1:
-			seconds, ok := fc.Int64()
-			if !ok {
-				return fmt.Errorf("cannot read Seconds")
-			}
-			ts.Seconds = seconds
-		case 2:
-			nanos, ok := fc.Int32()
-			if !ok {
-				return fmt.Errorf("cannot read Nanos")
-			}
-			ts.Nanos = nanos
-		}
-	}
-	return nil
-}

app/vlinsert/main.go

@@ -1,47 +0,0 @@
-package vlinsert
-
-import (
-	"net/http"
-	"strings"
-
-	"github.com/VictoriaMetrics/VictoriaMetrics/app/vlinsert/elasticsearch"
-	"github.com/VictoriaMetrics/VictoriaMetrics/app/vlinsert/jsonline"
-	"github.com/VictoriaMetrics/VictoriaMetrics/app/vlinsert/loki"
-	"github.com/VictoriaMetrics/VictoriaMetrics/app/vlinsert/syslog"
-)
-
-// Init initializes vlinsert
-func Init() {
-	syslog.MustInit()
-}
-
-// Stop stops vlinsert
-func Stop() {
-	syslog.MustStop()
-}
-
-// RequestHandler handles insert requests for VictoriaLogs
-func RequestHandler(w http.ResponseWriter, r *http.Request) bool {
-	path := r.URL.Path
-	if !strings.HasPrefix(path, "/insert/") {
-		// Skip requests, which do not start with /insert/, since these aren't our requests.
-		return false
-	}
-	path = strings.TrimPrefix(path, "/insert")
-	path = strings.ReplaceAll(path, "//", "/")
-
-	if path == "/jsonline" {
-		jsonline.RequestHandler(w, r)
-		return true
-	}
-	switch {
-	case strings.HasPrefix(path, "/elasticsearch/"):
-		path = strings.TrimPrefix(path, "/elasticsearch")
-		return elasticsearch.RequestHandler(path, w, r)
-	case strings.HasPrefix(path, "/loki/"):
-		path = strings.TrimPrefix(path, "/loki")
-		return loki.RequestHandler(path, w, r)
-	default:
-		return false
-	}
-}

app/vlinsert/syslog/syslog.go

@@ -1,531 +0,0 @@
-package syslog
-
-import (
-	"bufio"
-	"crypto/tls"
-	"errors"
-	"flag"
-	"fmt"
-	"io"
-	"net"
-	"strconv"
-	"strings"
-	"sync"
-	"sync/atomic"
-	"time"
-
-	"github.com/klauspost/compress/gzip"
-
-	"github.com/VictoriaMetrics/VictoriaMetrics/app/vlinsert/insertutils"
-	"github.com/VictoriaMetrics/VictoriaMetrics/app/vlstorage"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/bytesutil"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/cgroup"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/flagutil"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/ingestserver"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logstorage"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/netutil"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/protoparser/common"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/slicesutil"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/writeconcurrencylimiter"
-	"github.com/VictoriaMetrics/metrics"
-)
-
-var (
-	syslogTimezone = flag.String("syslog.timezone", "Local", "Timezone to use when parsing timestamps in RFC3164 syslog messages. Timezone must be a valid IANA Time Zone. "+
-		"For example: America/New_York, Europe/Berlin, Etc/GMT+3 . See https://docs.victoriametrics.com/victorialogs/data-ingestion/syslog/")
-
-	syslogTenantIDTCP = flagutil.NewArrayString("syslog.tenantID.tcp", "TenantID for logs ingested via the corresponding -syslog.listenAddr.tcp. "+
-		"See https://docs.victoriametrics.com/victorialogs/data-ingestion/syslog/")
-	syslogTenantIDUDP = flagutil.NewArrayString("syslog.tenantID.udp", "TenantID for logs ingested via the corresponding -syslog.listenAddr.udp. "+
-		"See https://docs.victoriametrics.com/victorialogs/data-ingestion/syslog/")
-
-	listenAddrTCP = flagutil.NewArrayString("syslog.listenAddr.tcp", "Comma-separated list of TCP addresses to listen to for Syslog messages. "+
-		"See https://docs.victoriametrics.com/victorialogs/data-ingestion/syslog/")
-	listenAddrUDP = flagutil.NewArrayString("syslog.listenAddr.udp", "Comma-separated list of UDP address to listen to for Syslog messages. "+
-		"See https://docs.victoriametrics.com/victorialogs/data-ingestion/syslog/")
-
-	tlsEnable = flagutil.NewArrayBool("syslog.tls", "Whether to enable TLS for receiving syslog messages at the corresponding -syslog.listenAddr.tcp. "+
-		"The corresponding -syslog.tlsCertFile and -syslog.tlsKeyFile must be set if -syslog.tls is set. See https://docs.victoriametrics.com/victorialogs/data-ingestion/syslog/#security")
-	tlsCertFile = flagutil.NewArrayString("syslog.tlsCertFile", "Path to file with TLS certificate for the corresponding -syslog.listenAddr.tcp if the corresponding -syslog.tls is set. "+
-		"Prefer ECDSA certs instead of RSA certs as RSA certs are slower. The provided certificate file is automatically re-read every second, so it can be dynamically updated. "+
-		"See https://docs.victoriametrics.com/victorialogs/data-ingestion/syslog/#security")
-	tlsKeyFile = flagutil.NewArrayString("syslog.tlsKeyFile", "Path to file with TLS key for the corresponding -syslog.listenAddr.tcp if the corresponding -syslog.tls is set. "+
-		"The provided key file is automatically re-read every second, so it can be dynamically updated. "+
-		"See https://docs.victoriametrics.com/victorialogs/data-ingestion/syslog/#security")
-	tlsCipherSuites = flagutil.NewArrayString("syslog.tlsCipherSuites", "Optional list of TLS cipher suites for -syslog.listenAddr.tcp if -syslog.tls is set. "+
-		"See the list of supported cipher suites at https://pkg.go.dev/crypto/tls#pkg-constants . "+
-		"See also https://docs.victoriametrics.com/victorialogs/data-ingestion/syslog/#security")
-	tlsMinVersion = flag.String("syslog.tlsMinVersion", "TLS13", "The minimum TLS version to use for -syslog.listenAddr.tcp if -syslog.tls is set. "+
-		"Supported values: TLS10, TLS11, TLS12, TLS13. "+
-		"See https://docs.victoriametrics.com/victorialogs/data-ingestion/syslog/#security")
-
-	compressMethodTCP = flagutil.NewArrayString("syslog.compressMethod.tcp", "Compression method for syslog messages received at the corresponding -syslog.listenAddr.tcp. "+
-		"Supported values: none, gzip, deflate. See https://docs.victoriametrics.com/victorialogs/data-ingestion/syslog/#compression")
-	compressMethodUDP = flagutil.NewArrayString("syslog.compressMethod.udp", "Compression method for syslog messages received at the corresponding -syslog.listenAddr.udp. "+
-		"Supported values: none, gzip, deflate. See https://docs.victoriametrics.com/victorialogs/data-ingestion/syslog/#compression")
-
-	useLocalTimestampTCP = flagutil.NewArrayBool("syslog.useLocalTimestamp.tcp", "Whether to use local timestamp instead of the original timestamp for the ingested syslog messages "+
-		"at the corresponding -syslog.listenAddr.tcp. See https://docs.victoriametrics.com/victorialogs/data-ingestion/syslog/#log-timestamps")
-	useLocalTimestampUDP = flagutil.NewArrayBool("syslog.useLocalTimestamp.udp", "Whether to use local timestamp instead of the original timestamp for the ingested syslog messages "+
-		"at the corresponding -syslog.listenAddr.udp. See https://docs.victoriametrics.com/victorialogs/data-ingestion/syslog/#log-timestamps")
-)
-
-// MustInit initializes syslog parser at the given -syslog.listenAddr.tcp and -syslog.listenAddr.udp ports
-//
-// This function must be called after flag.Parse().
-//
-// MustStop() must be called in order to free up resources occupied by the initialized syslog parser.
-func MustInit() {
-	if workersStopCh != nil {
-		logger.Panicf("BUG: MustInit() called twice without MustStop() call")
-	}
-	workersStopCh = make(chan struct{})
-
-	for argIdx, addr := range *listenAddrTCP {
-		workersWG.Add(1)
-		go func(addr string, argIdx int) {
-			runTCPListener(addr, argIdx)
-			workersWG.Done()
-		}(addr, argIdx)
-	}
-
-	for argIdx, addr := range *listenAddrUDP {
-		workersWG.Add(1)
-		go func(addr string, argIdx int) {
-			runUDPListener(addr, argIdx)
-			workersWG.Done()
-		}(addr, argIdx)
-	}
-
-	currentYear := time.Now().Year()
-	globalCurrentYear.Store(int64(currentYear))
-	workersWG.Add(1)
-	go func() {
-		ticker := time.NewTicker(time.Minute)
-		for {
-			select {
-			case <-workersStopCh:
-				ticker.Stop()
-				workersWG.Done()
-				return
-			case <-ticker.C:
-				currentYear := time.Now().Year()
-				globalCurrentYear.Store(int64(currentYear))
-			}
-		}
-	}()
-
-	if *syslogTimezone != "" {
-		tz, err := time.LoadLocation(*syslogTimezone)
-		if err != nil {
-			logger.Fatalf("cannot parse -syslog.timezone=%q: %s", *syslogTimezone, err)
-		}
-		globalTimezone = tz
-	} else {
-		globalTimezone = time.Local
-	}
-}
-
-var (
-	globalCurrentYear atomic.Int64
-	globalTimezone    *time.Location
-)
-
-var (
-	workersWG     sync.WaitGroup
-	workersStopCh chan struct{}
-)
-
-// MustStop stops syslog parser initialized via MustInit()
-func MustStop() {
-	close(workersStopCh)
-	workersWG.Wait()
-	workersStopCh = nil
-}
-
-func runUDPListener(addr string, argIdx int) {
-	ln, err := net.ListenPacket(netutil.GetUDPNetwork(), addr)
-	if err != nil {
-		logger.Fatalf("cannot start UDP syslog server at %q: %s", addr, err)
-	}
-
-	tenantIDStr := syslogTenantIDUDP.GetOptionalArg(argIdx)
-	tenantID, err := logstorage.ParseTenantID(tenantIDStr)
-	if err != nil {
-		logger.Fatalf("cannot parse -syslog.tenantID.udp=%q for -syslog.listenAddr.udp=%q: %s", tenantIDStr, addr, err)
-	}
-
-	compressMethod := compressMethodUDP.GetOptionalArg(argIdx)
-	checkCompressMethod(compressMethod, addr, "udp")
-
-	useLocalTimestamp := useLocalTimestampUDP.GetOptionalArg(argIdx)
-
-	doneCh := make(chan struct{})
-	go func() {
-		serveUDP(ln, tenantID, compressMethod, useLocalTimestamp)
-		close(doneCh)
-	}()
-
-	logger.Infof("started accepting syslog messages at -syslog.listenAddr.udp=%q", addr)
-	<-workersStopCh
-	if err := ln.Close(); err != nil {
-		logger.Fatalf("syslog: cannot close UDP listener at %s: %s", addr, err)
-	}
-	<-doneCh
-	logger.Infof("finished accepting syslog messages at -syslog.listenAddr.udp=%q", addr)
-}
-
-func runTCPListener(addr string, argIdx int) {
-	var tlsConfig *tls.Config
-	if tlsEnable.GetOptionalArg(argIdx) {
-		certFile := tlsCertFile.GetOptionalArg(argIdx)
-		keyFile := tlsKeyFile.GetOptionalArg(argIdx)
-		tc, err := netutil.GetServerTLSConfig(certFile, keyFile, *tlsMinVersion, *tlsCipherSuites)
-		if err != nil {
-			logger.Fatalf("cannot load TLS cert from -syslog.tlsCertFile=%q, -syslog.tlsKeyFile=%q, -syslog.tlsMinVersion=%q, -syslog.tlsCipherSuites=%q: %s",
-				certFile, keyFile, *tlsMinVersion, *tlsCipherSuites, err)
-		}
-		tlsConfig = tc
-	}
-	ln, err := netutil.NewTCPListener("syslog", addr, false, tlsConfig)
-	if err != nil {
-		logger.Fatalf("syslog: cannot start TCP listener at %s: %s", addr, err)
-	}
-
-	tenantIDStr := syslogTenantIDTCP.GetOptionalArg(argIdx)
-	tenantID, err := logstorage.ParseTenantID(tenantIDStr)
-	if err != nil {
-		logger.Fatalf("cannot parse -syslog.tenantID.tcp=%q for -syslog.listenAddr.tcp=%q: %s", tenantIDStr, addr, err)
-	}
-
-	compressMethod := compressMethodTCP.GetOptionalArg(argIdx)
-	checkCompressMethod(compressMethod, addr, "tcp")
-
-	useLocalTimestamp := useLocalTimestampTCP.GetOptionalArg(argIdx)
-
-	doneCh := make(chan struct{})
-	go func() {
-		serveTCP(ln, tenantID, compressMethod, useLocalTimestamp)
-		close(doneCh)
-	}()
-
-	logger.Infof("started accepting syslog messages at -syslog.listenAddr.tcp=%q", addr)
-	<-workersStopCh
-	if err := ln.Close(); err != nil {
-		logger.Fatalf("syslog: cannot close TCP listener at %s: %s", addr, err)
-	}
-	<-doneCh
-	logger.Infof("finished accepting syslog messages at -syslog.listenAddr.tcp=%q", addr)
-}
-
-func checkCompressMethod(compressMethod, addr, protocol string) {
-	switch compressMethod {
-	case "", "none", "gzip", "deflate":
-		return
-	default:
-		logger.Fatalf("unsupported -syslog.compressMethod.%s=%q for -syslog.listenAddr.%s=%q; supported values: 'none', 'gzip', 'deflate'", protocol, compressMethod, protocol, addr)
-	}
-}
-
-func serveUDP(ln net.PacketConn, tenantID logstorage.TenantID, compressMethod string, useLocalTimestamp bool) {
-	gomaxprocs := cgroup.AvailableCPUs()
-	var wg sync.WaitGroup
-	localAddr := ln.LocalAddr()
-	for i := 0; i < gomaxprocs; i++ {
-		wg.Add(1)
-		go func() {
-			defer wg.Done()
-			cp := insertutils.GetCommonParamsForSyslog(tenantID)
-			var bb bytesutil.ByteBuffer
-			bb.B = bytesutil.ResizeNoCopyNoOverallocate(bb.B, 64*1024)
-			for {
-				bb.Reset()
-				bb.B = bb.B[:cap(bb.B)]
-				n, remoteAddr, err := ln.ReadFrom(bb.B)
-				if err != nil {
-					udpErrorsTotal.Inc()
-					var ne net.Error
-					if errors.As(err, &ne) {
-						if ne.Temporary() {
-							logger.Errorf("syslog: temporary error when listening for UDP at %q: %s", localAddr, err)
-							time.Sleep(time.Second)
-							continue
-						}
-						if strings.Contains(err.Error(), "use of closed network connection") {
-							break
-						}
-					}
-					logger.Errorf("syslog: cannot read UDP data from %s at %s: %s", remoteAddr, localAddr, err)
-					continue
-				}
-				bb.B = bb.B[:n]
-				udpRequestsTotal.Inc()
-				if err := processStream(bb.NewReader(), compressMethod, useLocalTimestamp, cp); err != nil {
-					logger.Errorf("syslog: cannot process UDP data from %s at %s: %s", remoteAddr, localAddr, err)
-				}
-			}
-		}()
-	}
-	wg.Wait()
-}
-
-func serveTCP(ln net.Listener, tenantID logstorage.TenantID, compressMethod string, useLocalTimestamp bool) {
-	var cm ingestserver.ConnsMap
-	cm.Init("syslog")
-
-	var wg sync.WaitGroup
-	addr := ln.Addr()
-	for {
-		c, err := ln.Accept()
-		if err != nil {
-			var ne net.Error
-			if errors.As(err, &ne) {
-				if ne.Temporary() {
-					logger.Errorf("syslog: temporary error when listening for TCP addr %q: %s", addr, err)
-					time.Sleep(time.Second)
-					continue
-				}
-				if strings.Contains(err.Error(), "use of closed network connection") {
-					break
-				}
-				logger.Fatalf("syslog: unrecoverable error when accepting TCP connections at %q: %s", addr, err)
-			}
-			logger.Fatalf("syslog: unexpected error when accepting TCP connections at %q: %s", addr, err)
-		}
-		if !cm.Add(c) {
-			_ = c.Close()
-			break
-		}
-
-		wg.Add(1)
-		go func() {
-			cp := insertutils.GetCommonParamsForSyslog(tenantID)
-			if err := processStream(c, compressMethod, useLocalTimestamp, cp); err != nil {
-				logger.Errorf("syslog: cannot process TCP data at %q: %s", addr, err)
-			}
-
-			cm.Delete(c)
-			_ = c.Close()
-			wg.Done()
-		}()
-	}
-
-	cm.CloseAll(0)
-	wg.Wait()
-}
-
-// processStream parses a stream of syslog messages from r and ingests them into vlstorage.
-func processStream(r io.Reader, compressMethod string, useLocalTimestamp bool, cp *insertutils.CommonParams) error {
-	if err := vlstorage.CanWriteData(); err != nil {
-		return err
-	}
-
-	lmp := cp.NewLogMessageProcessor()
-	err := processStreamInternal(r, compressMethod, useLocalTimestamp, lmp)
-	lmp.MustClose()
-
-	return err
-}
-
-func processStreamInternal(r io.Reader, compressMethod string, useLocalTimestamp bool, lmp insertutils.LogMessageProcessor) error {
-	switch compressMethod {
-	case "", "none":
-	case "gzip":
-		zr, err := common.GetGzipReader(r)
-		if err != nil {
-			return fmt.Errorf("cannot read gzipped data: %w", err)
-		}
-		r = zr
-	case "deflate":
-		zr, err := common.GetZlibReader(r)
-		if err != nil {
-			return fmt.Errorf("cannot read deflated data: %w", err)
-		}
-		r = zr
-	default:
-		logger.Panicf("BUG: unsupported compressMethod=%q; supported values: none, gzip, deflate", compressMethod)
-	}
-
-	err := processUncompressedStream(r, useLocalTimestamp, lmp)
-
-	switch compressMethod {
-	case "gzip":
-		zr := r.(*gzip.Reader)
-		common.PutGzipReader(zr)
-	case "deflate":
-		zr := r.(io.ReadCloser)
-		common.PutZlibReader(zr)
-	}
-
-	return err
-}
-
-func processUncompressedStream(r io.Reader, useLocalTimestamp bool, lmp insertutils.LogMessageProcessor) error {
-	wcr := writeconcurrencylimiter.GetReader(r)
-	defer writeconcurrencylimiter.PutReader(wcr)
-
-	slr := getSyslogLineReader(wcr)
-	defer putSyslogLineReader(slr)
-
-	n := 0
-	for {
-		ok := slr.nextLine()
-		wcr.DecConcurrency()
-		if !ok {
-			break
-		}
-
-		currentYear := int(globalCurrentYear.Load())
-		err := processLine(slr.line, currentYear, globalTimezone, useLocalTimestamp, lmp)
-		if err != nil {
-			errorsTotal.Inc()
-			return fmt.Errorf("cannot read line #%d: %s", n, err)
-		}
-		n++
-		rowsIngestedTotal.Inc()
-	}
-	return slr.Error()
-}
-
-type syslogLineReader struct {
-	line []byte
-
-	br  *bufio.Reader
-	err error
-}
-
-func (slr *syslogLineReader) reset(r io.Reader) {
-	slr.line = slr.line[:0]
-	slr.br.Reset(r)
-	slr.err = nil
-}
-
-// Error returns the last error occurred in slr.
-func (slr *syslogLineReader) Error() error {
-	if slr.err == nil || slr.err == io.EOF {
-		return nil
-	}
-	return slr.err
-}
-
-// nextLine reads the next syslog line from slr and stores it at slr.line.
-//
-// false is returned if the next line cannot be read. Error() must be called in this case
-// in order to verify whether there is an error or just slr stream has been finished.
-func (slr *syslogLineReader) nextLine() bool {
-	if slr.err != nil {
-		return false
-	}
-
-again:
-	prefix, err := slr.br.ReadSlice(' ')
-	if err != nil {
-		if err != io.EOF {
-			slr.err = fmt.Errorf("cannot read message frame prefix: %w", err)
-			return false
-		}
-		if len(prefix) == 0 {
-			slr.err = err
-			return false
-		}
-	}
-	// skip empty lines
-	for len(prefix) > 0 && prefix[0] == '\n' {
-		prefix = prefix[1:]
-	}
-	if len(prefix) == 0 {
-		// An empty prefix or a prefix with empty lines - try reading yet another prefix.
-		goto again
-	}
-
-	if prefix[0] >= '0' && prefix[0] <= '9' {
-		// This is octet-counting method. See https://www.ietf.org/archive/id/draft-gerhards-syslog-plain-tcp-07.html#msgxfer
-		msgLenStr := bytesutil.ToUnsafeString(prefix[:len(prefix)-1])
-		msgLen, err := strconv.ParseUint(msgLenStr, 10, 64)
-		if err != nil {
-			slr.err = fmt.Errorf("cannot parse message length from %q: %w", msgLenStr, err)
-			return false
-		}
-		if maxMsgLen := insertutils.MaxLineSizeBytes.IntN(); msgLen > uint64(maxMsgLen) {
-			slr.err = fmt.Errorf("cannot read message longer than %d bytes; msgLen=%d", maxMsgLen, msgLen)
-			return false
-		}
-		slr.line = slicesutil.SetLength(slr.line, int(msgLen))
-		if _, err := io.ReadFull(slr.br, slr.line); err != nil {
-			slr.err = fmt.Errorf("cannot read message with size %d bytes: %w", msgLen, err)
-			return false
-		}
-		return true
-	}
-
-	// This is octet-stuffing method. See https://www.ietf.org/archive/id/draft-gerhards-syslog-plain-tcp-07.html#octet-stuffing-legacy
-	slr.line = append(slr.line[:0], prefix...)
-	for {
-		line, err := slr.br.ReadSlice('\n')
-		if err == nil {
-			slr.line = append(slr.line, line[:len(line)-1]...)
-			return true
-		}
-		if err == io.EOF {
-			slr.line = append(slr.line, line...)
-			return true
-		}
-		if err == bufio.ErrBufferFull {
-			slr.line = append(slr.line, line...)
-			continue
-		}
-		slr.err = fmt.Errorf("cannot read message in octet-stuffing method: %w", err)
-		return false
-	}
-}
-
-func getSyslogLineReader(r io.Reader) *syslogLineReader {
-	v := syslogLineReaderPool.Get()
-	if v == nil {
-		br := bufio.NewReaderSize(r, 64*1024)
-		return &syslogLineReader{
-			br: br,
-		}
-	}
-	slr := v.(*syslogLineReader)
-	slr.reset(r)
-	return slr
-}
-
-func putSyslogLineReader(slr *syslogLineReader) {
-	syslogLineReaderPool.Put(slr)
-}
-
-var syslogLineReaderPool sync.Pool
-
-func processLine(line []byte, currentYear int, timezone *time.Location, useLocalTimestamp bool, lmp insertutils.LogMessageProcessor) error {
-	p := logstorage.GetSyslogParser(currentYear, timezone)
-	lineStr := bytesutil.ToUnsafeString(line)
-	p.Parse(lineStr)
-
-	var ts int64
-	if useLocalTimestamp {
-		ts = time.Now().UnixNano()
-	} else {
-		nsecs, err := insertutils.ExtractTimestampRFC3339NanoFromFields("timestamp", p.Fields)
-		if err != nil {
-			return fmt.Errorf("cannot get timestamp from syslog line %q: %w", line, err)
-		}
-		ts = nsecs
-	}
-	logstorage.RenameField(p.Fields, "message", "_msg")
-	lmp.AddRow(ts, p.Fields)
-	logstorage.PutSyslogParser(p)
-
-	return nil
-}
-
-var (
-	rowsIngestedTotal = metrics.NewCounter(`vl_rows_ingested_total{type="syslog"}`)
-
-	errorsTotal = metrics.NewCounter(`vl_errors_total{type="syslog"}`)
-
-	udpRequestsTotal = metrics.NewCounter(`vl_udp_reqests_total{type="syslog"}`)
-	udpErrorsTotal   = metrics.NewCounter(`vl_udp_errors_total{type="syslog"}`)
-)

app/vlinsert/syslog/syslog_test.go

@@ -1,130 +0,0 @@
-package syslog
-
-import (
-	"bytes"
-	"reflect"
-	"testing"
-	"time"
-
-	"github.com/VictoriaMetrics/VictoriaMetrics/app/vlinsert/insertutils"
-)
-
-func TestSyslogLineReader_Success(t *testing.T) {
-	f := func(data string, linesExpected []string) {
-		t.Helper()
-
-		r := bytes.NewBufferString(data)
-		slr := getSyslogLineReader(r)
-		defer putSyslogLineReader(slr)
-
-		var lines []string
-		for slr.nextLine() {
-			lines = append(lines, string(slr.line))
-		}
-		if err := slr.Error(); err != nil {
-			t.Fatalf("unexpected error: %s", err)
-		}
-		if !reflect.DeepEqual(lines, linesExpected) {
-			t.Fatalf("unexpected lines read;\ngot\n%q\nwant\n%q", lines, linesExpected)
-		}
-	}
-
-	f("", nil)
-	f("\n", nil)
-	f("\n\n\n", nil)
-
-	f("foobar", []string{"foobar"})
-	f("foobar\n", []string{"foobar\n"})
-	f("\n\nfoo\n\nbar\n\n", []string{"foo\n\nbar\n\n"})
-
-	f(`Jun  3 12:08:33 abcd systemd: Starting Update the local ESM caches...`, []string{"Jun  3 12:08:33 abcd systemd: Starting Update the local ESM caches..."})
-
-	f(`Jun  3 12:08:33 abcd systemd: Starting Update the local ESM caches...
-
-48 <165>Jun  4 12:08:33 abcd systemd[345]: abc defg<123>1 2023-06-03T17:42:12.345Z mymachine.example.com appname 12345 ID47 [exampleSDID@32473 iut="3" eventSource="Application 123 = ] 56" eventID="11211"] This is a test message with structured data.
-
-`, []string{
-		"Jun  3 12:08:33 abcd systemd: Starting Update the local ESM caches...",
-		"<165>Jun  4 12:08:33 abcd systemd[345]: abc defg",
-		`<123>1 2023-06-03T17:42:12.345Z mymachine.example.com appname 12345 ID47 [exampleSDID@32473 iut="3" eventSource="Application 123 = ] 56" eventID="11211"] This is a test message with structured data.`,
-	})
-}
-
-func TestSyslogLineReader_Failure(t *testing.T) {
-	f := func(data string) {
-		t.Helper()
-
-		r := bytes.NewBufferString(data)
-		slr := getSyslogLineReader(r)
-		defer putSyslogLineReader(slr)
-
-		if slr.nextLine() {
-			t.Fatalf("expecting failure to read the first line")
-		}
-		if err := slr.Error(); err == nil {
-			t.Fatalf("expecting non-nil error")
-		}
-	}
-
-	// invalid format for message size
-	f("12foo bar")
-
-	// too big message size
-	f("123 aa")
-	f("1233423432 abc")
-}
-
-func TestProcessStreamInternal_Success(t *testing.T) {
-	f := func(data string, currentYear, rowsExpected int, timestampsExpected []int64, resultExpected string) {
-		t.Helper()
-
-		MustInit()
-		defer MustStop()
-
-		globalTimezone = time.UTC
-		globalCurrentYear.Store(int64(currentYear))
-
-		tlp := &insertutils.TestLogMessageProcessor{}
-		r := bytes.NewBufferString(data)
-		if err := processStreamInternal(r, "", false, tlp); err != nil {
-			t.Fatalf("unexpected error: %s", err)
-		}
-		if err := tlp.Verify(rowsExpected, timestampsExpected, resultExpected); err != nil {
-			t.Fatal(err)
-		}
-	}
-
-	data := `Jun  3 12:08:33 abcd systemd: Starting Update the local ESM caches...
-
-48 <165>Jun  4 12:08:33 abcd systemd[345]: abc defg<123>1 2023-06-03T17:42:12.345Z mymachine.example.com appname 12345 ID47 [exampleSDID@32473 iut="3" eventSource="Application 123 = ] 56" eventID="11211"] This is a test message with structured data.
-`
-	currentYear := 2023
-	rowsExpected := 3
-	timestampsExpected := []int64{1685794113000000000, 1685880513000000000, 1685814132345000000}
-	resultExpected := `{"format":"rfc3164","timestamp":"","hostname":"abcd","app_name":"systemd","_msg":"Starting Update the local ESM caches..."}
-{"priority":"165","facility":"20","severity":"5","format":"rfc3164","timestamp":"","hostname":"abcd","app_name":"systemd","proc_id":"345","_msg":"abc defg"}
-{"priority":"123","facility":"15","severity":"3","format":"rfc5424","timestamp":"","hostname":"mymachine.example.com","app_name":"appname","proc_id":"12345","msg_id":"ID47","exampleSDID@32473.iut":"3","exampleSDID@32473.eventSource":"Application 123 = ] 56","exampleSDID@32473.eventID":"11211","_msg":"This is a test message with structured data."}`
-	f(data, currentYear, rowsExpected, timestampsExpected, resultExpected)
-}
-
-func TestProcessStreamInternal_Failure(t *testing.T) {
-	f := func(data string) {
-		t.Helper()
-
-		MustInit()
-		defer MustStop()
-
-		tlp := &insertutils.TestLogMessageProcessor{}
-		r := bytes.NewBufferString(data)
-		if err := processStreamInternal(r, "", false, tlp); err == nil {
-			t.Fatalf("expecting non-nil error")
-		}
-	}
-
-	// invalid format for message size
-	f("12foo bar")
-
-	// too big message size
-	f("123 foo")
-	f("123456789 bar")
-}

app/vlogsgenerator/Makefile

@@ -1,7 +0,0 @@
-# All these commands must run from repository root.
-
-vlogsgenerator:
-	APP_NAME=vlogsgenerator $(MAKE) app-local
-
-vlogsgenerator-race:
-	APP_NAME=vlogsgenerator RACE=-race $(MAKE) app-local

app/vlogsgenerator/README.md

@@ -1,158 +0,0 @@
-# vlogsgenerator
-
-Logs generator for [VictoriaLogs](https://docs.victoriametrics.com/victorialogs/).
-
-## How to build vlogsgenerator?
-
-Run `make vlogsgenerator` from the repository root. This builds `bin/vlogsgenerator` binary.
-
-## How run vlogsgenerator?
-
-`vlogsgenerator` generates logs in [JSON line format](https://jsonlines.org/) suitable for the ingestion
-via [`/insert/jsonline` endpoint at VictoriaLogs](https://docs.victoriametrics.com/victorialogs/data-ingestion/#json-stream-api).
-
-By default it writes the generated logs into `stdout`. For example, the following command writes generated logs to `stdout`:
-
-```
-bin/vlogsgenerator
-```
-
-It is possible to redirect the generated logs to file. For example, the following command writes the generated logs to `logs.json` file:
-
-```
-bin/vlogsgenerator > logs.json
-```
-
-The generated logs at `logs.json` file can be inspected with the following command:
-
-```
-head logs.json | jq .
-```
-
-Below is an example output:
-
-```json
-{
-  "_time": "2024-05-08T14:34:00.854Z",
-  "_msg": "message for the stream 8 and worker 0; ip=185.69.136.129; uuid=b4fe8f1a-c93c-dea3-ba11-5b9f0509291e; u64=8996587920687045253",
-  "host": "host_8",
-  "worker_id": "0",
-  "run_id": "f9b3deee-e6b6-7f56-5deb-1586e4e81725",
-  "const_0": "some value 0 8",
-  "const_1": "some value 1 8",
-  "const_2": "some value 2 8",
-  "var_0": "some value 0 12752539384823438260",
-  "dict_0": "warn",
-  "dict_1": "info",
-  "u8_0": "6",
-  "u16_0": "35202",
-  "u32_0": "1964973739",
-  "u64_0": "4810489083243239145",
-  "float_0": "1.868",
-  "ip_0": "250.34.75.125",
-  "timestamp_0": "1799-03-16T01:34:18.311Z",
-  "json_0": "{\"foo\":\"bar_3\",\"baz\":{\"a\":[\"x\",\"y\"]},\"f3\":NaN,\"f4\":32}"
-}
-{
-  "_time": "2024-05-08T14:34:00.854Z",
-  "_msg": "message for the stream 9 and worker 0; ip=164.244.254.194; uuid=7e8373b1-ce0d-1ce7-8e96-4bcab8955598; u64=13949903463741076522",
-  "host": "host_9",
-  "worker_id": "0",
-  "run_id": "f9b3deee-e6b6-7f56-5deb-1586e4e81725",
-  "const_0": "some value 0 9",
-  "const_1": "some value 1 9",
-  "const_2": "some value 2 9",
-  "var_0": "some value 0 5371555382075206134",
-  "dict_0": "INFO",
-  "dict_1": "FATAL",
-  "u8_0": "219",
-  "u16_0": "31459",
-  "u32_0": "3918836777",
-  "u64_0": "6593354256620219850",
-  "float_0": "1.085",
-  "ip_0": "253.151.88.158",
-  "timestamp_0": "2042-10-05T16:42:57.082Z",
-  "json_0": "{\"foo\":\"bar_5\",\"baz\":{\"a\":[\"x\",\"y\"]},\"f3\":NaN,\"f4\":27}"
-}
-```
-
-The `run_id` field uniquely identifies every `vlogsgenerator` invocation.
-
-### How to write logs to VictoriaLogs?
-
-The generated logs can be written directly to VictoriaLogs by passing the address of [`/insert/jsonline` endpoint](https://docs.victoriametrics.com/victorialogs/data-ingestion/#json-stream-api)
-to `-addr` command-line flag. For example, the following command writes the generated logs to VictoriaLogs running at `localhost`:
-
-```
-bin/vlogsgenerator -addr=http://localhost:9428/insert/jsonline
-```
-
-### Configuration
-
-`vlogsgenerator` accepts various command-line flags, which can be used for configuring the number and the shape of the generated logs.
-These flags can be inspected by running `vlogsgenerator -help`. Below are the most interesting flags:
-
-* `-start` - starting timestamp for generating logs. Logs are evenly generated on the [`-start` ... `-end`] interval.
-* `-end` - ending timestamp for generating logs. Logs are evenly generated on the [`-start` ... `-end`] interval.
-* `-activeStreams` - the number of active [log streams](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields) to generate.
-* `-logsPerStream` - the number of log entries to generate per each log stream. Log entries are evenly distributed on the [`-start` ... `-end`] interval.
-
-The total number of generated logs can be calculated as `-activeStreams` * `-logsPerStream`.
-
-For example, the following command generates `1_000_000` log entries on the time range `[2024-01-01 - 2024-02-01]` across `100`
-[log streams](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields), where every logs stream contains `10_000` log entries,
-and writes them to `http://localhost:9428/insert/jsonline`:
-
-```
-bin/vlogsgenerator \
-  -start=2024-01-01 -end=2024-02-01 \
-  -activeStreams=100 \
-  -logsPerStream=10_000 \
-  -addr=http://localhost:9428/insert/jsonline
-```
-
-### Churn rate
-
-It is possible to generate churn rate for active [log streams](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields)
-by specifying `-totalStreams` command-line flag bigger than `-activeStreams`. For example, the following command generates
-logs for `1000` total streams, while the number of active streams equals to `100`. This means that at every time there are logs for `100` streams,
-but these streams change over the given [`-start` ... `-end`] time range, so the total number of streams on the given time range becomes `1000`:
-
-```
-bin/vlogsgenerator \
-  -start=2024-01-01 -end=2024-02-01 \
-  -activeStreams=100 \
-  -totalStreams=1_000 \
-  -logsPerStream=10_000 \
-  -addr=http://localhost:9428/insert/jsonline
-```
-
-In this case the total number of generated logs equals to `-totalStreams` * `-logsPerStream` = `10_000_000`.
-
-### Benchmark tuning
-
-By default `vlogsgenerator` generates and writes logs by a single worker. This may limit the maximum data ingestion rate during benchmarks.
-The number of workers can be changed via `-workers` command-line flag. For example, the following command generates and writes logs with `16` workers:
-
-```
-bin/vlogsgenerator \
-  -start=2024-01-01 -end=2024-02-01 \
-  -activeStreams=100 \
-  -logsPerStream=10_000 \
-  -addr=http://localhost:9428/insert/jsonline \
-  -workers=16
-```
-
-### Output statistics
-
-Every 10 seconds `vlogsgenerator` writes statistics about the generated logs into `stderr`. The frequency of the generated statistics can be adjusted via `-statInterval` command-line flag.
-For example, the following command writes statistics every 2 seconds:
-
-```
-bin/vlogsgenerator \
-  -start=2024-01-01 -end=2024-02-01 \
-  -activeStreams=100 \
-  -logsPerStream=10_000 \
-  -addr=http://localhost:9428/insert/jsonline \
-  -statInterval=2s
-```

app/vlogsgenerator/main.go

@@ -1,344 +0,0 @@
-package main
-
-import (
-	"bufio"
-	"flag"
-	"fmt"
-	"io"
-	"math"
-	"math/rand"
-	"net/http"
-	"net/url"
-	"os"
-	"strconv"
-	"sync"
-	"sync/atomic"
-	"time"
-
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/buildinfo"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/envflag"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/promutils"
-)
-
-var (
-	addr    = flag.String("addr", "stdout", "HTTP address to push the generated logs to; if it is set to stdout, then logs are generated to stdout")
-	workers = flag.Int("workers", 1, "The number of workers to use to push logs to -addr")
-
-	start         = newTimeFlag("start", "-1d", "Generated logs start from this time; see https://docs.victoriametrics.com/#timestamp-formats")
-	end           = newTimeFlag("end", "0s", "Generated logs end at this time; see https://docs.victoriametrics.com/#timestamp-formats")
-	activeStreams = flag.Int("activeStreams", 100, "The number of active log streams to generate; see https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields")
-	totalStreams  = flag.Int("totalStreams", 0, "The number of total log streams; if -totalStreams > -activeStreams, then some active streams are substituted with new streams "+
-		"during data generation")
-	logsPerStream     = flag.Int64("logsPerStream", 1_000, "The number of log entries to generate per each log stream. Log entries are evenly distributed between -start and -end")
-	constFieldsPerLog = flag.Int("constFieldsPerLog", 3, "The number of fields with constaint values to generate per each log entry; "+
-		"see https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model")
-	varFieldsPerLog = flag.Int("varFieldsPerLog", 1, "The number of fields with variable values to generate per each log entry; "+
-		"see https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model")
-	dictFieldsPerLog = flag.Int("dictFieldsPerLog", 2, "The number of fields with up to 8 different values to generate per each log entry; "+
-		"see https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model")
-	u8FieldsPerLog = flag.Int("u8FieldsPerLog", 1, "The number of fields with uint8 values to generate per each log entry; "+
-		"see https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model")
-	u16FieldsPerLog = flag.Int("u16FieldsPerLog", 1, "The number of fields with uint16 values to generate per each log entry; "+
-		"see https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model")
-	u32FieldsPerLog = flag.Int("u32FieldsPerLog", 1, "The number of fields with uint32 values to generate per each log entry; "+
-		"see https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model")
-	u64FieldsPerLog = flag.Int("u64FieldsPerLog", 1, "The number of fields with uint64 values to generate per each log entry; "+
-		"see https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model")
-	floatFieldsPerLog = flag.Int("floatFieldsPerLog", 1, "The number of fields with float64 values to generate per each log entry; "+
-		"see https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model")
-	ipFieldsPerLog = flag.Int("ipFieldsPerLog", 1, "The number of fields with IPv4 values to generate per each log entry; "+
-		"see https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model")
-	timestampFieldsPerLog = flag.Int("timestampFieldsPerLog", 1, "The number of fields with ISO8601 timestamps per each log entry; "+
-		"see https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model")
-	jsonFieldsPerLog = flag.Int("jsonFieldsPerLog", 1, "The number of JSON fields to generate per each log entry; "+
-		"see https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model")
-
-	statInterval = flag.Duration("statInterval", 10*time.Second, "The interval between publishing the stats")
-)
-
-func main() {
-	// Write flags and help message to stdout, since it is easier to grep or pipe.
-	flag.CommandLine.SetOutput(os.Stdout)
-	envflag.Parse()
-	buildinfo.Init()
-	logger.Init()
-
-	var remoteWriteURL *url.URL
-	if *addr != "stdout" {
-		urlParsed, err := url.Parse(*addr)
-		if err != nil {
-			logger.Fatalf("cannot parse -addr=%q: %s", *addr, err)
-		}
-		qs, err := url.ParseQuery(urlParsed.RawQuery)
-		if err != nil {
-			logger.Fatalf("cannot parse query string in -addr=%q: %w", *addr, err)
-		}
-		qs.Set("_stream_fields", "host,worker_id")
-		urlParsed.RawQuery = qs.Encode()
-		remoteWriteURL = urlParsed
-	}
-
-	if start.nsec >= end.nsec {
-		logger.Fatalf("-start=%s must be smaller than -end=%s", start, end)
-	}
-	if *activeStreams <= 0 {
-		logger.Fatalf("-activeStreams must be bigger than 0; got %d", *activeStreams)
-	}
-	if *logsPerStream <= 0 {
-		logger.Fatalf("-logsPerStream must be bigger than 0; got %d", *logsPerStream)
-	}
-	if *totalStreams < *activeStreams {
-		*totalStreams = *activeStreams
-	}
-
-	cfg := &workerConfig{
-		url:           remoteWriteURL,
-		activeStreams: *activeStreams,
-		totalStreams:  *totalStreams,
-	}
-
-	// divide total and active streams among workers
-	if *workers <= 0 {
-		logger.Fatalf("-workers must be bigger than 0; got %d", *workers)
-	}
-	if *workers > *activeStreams {
-		logger.Fatalf("-workers=%d cannot exceed -activeStreams=%d", *workers, *activeStreams)
-	}
-	cfg.activeStreams /= *workers
-	cfg.totalStreams /= *workers
-
-	logger.Infof("start -workers=%d workers for ingesting -logsPerStream=%d log entries per each -totalStreams=%d (-activeStreams=%d) on a time range -start=%s, -end=%s to -addr=%s",
-		*workers, *logsPerStream, *totalStreams, *activeStreams, toRFC3339(start.nsec), toRFC3339(end.nsec), *addr)
-
-	startTime := time.Now()
-	var wg sync.WaitGroup
-	for i := 0; i < *workers; i++ {
-		wg.Add(1)
-		go func(workerID int) {
-			defer wg.Done()
-			generateAndPushLogs(cfg, workerID)
-		}(i)
-	}
-
-	go func() {
-		prevEntries := uint64(0)
-		prevBytes := uint64(0)
-		ticker := time.NewTicker(*statInterval)
-		for range ticker.C {
-			currEntries := logEntriesCount.Load()
-			deltaEntries := currEntries - prevEntries
-			rateEntries := float64(deltaEntries) / statInterval.Seconds()
-
-			currBytes := bytesGenerated.Load()
-			deltaBytes := currBytes - prevBytes
-			rateBytes := float64(deltaBytes) / statInterval.Seconds()
-			logger.Infof("generated %dK log entries (%dK total) at %.0fK entries/sec, %dMB (%dMB total) at %.0fMB/sec",
-				deltaEntries/1e3, currEntries/1e3, rateEntries/1e3, deltaBytes/1e6, currBytes/1e6, rateBytes/1e6)
-
-			prevEntries = currEntries
-			prevBytes = currBytes
-		}
-	}()
-
-	wg.Wait()
-
-	dSecs := time.Since(startTime).Seconds()
-	currEntries := logEntriesCount.Load()
-	currBytes := bytesGenerated.Load()
-	rateEntries := float64(currEntries) / dSecs
-	rateBytes := float64(currBytes) / dSecs
-	logger.Infof("ingested %dK log entries (%dMB) in %.3f seconds; avg ingestion rate: %.0fK entries/sec, %.0fMB/sec", currEntries/1e3, currBytes/1e6, dSecs, rateEntries/1e3, rateBytes/1e6)
-}
-
-var logEntriesCount atomic.Uint64
-
-var bytesGenerated atomic.Uint64
-
-type workerConfig struct {
-	url           *url.URL
-	activeStreams int
-	totalStreams  int
-}
-
-type statWriter struct {
-	w io.Writer
-}
-
-func (sw *statWriter) Write(p []byte) (int, error) {
-	bytesGenerated.Add(uint64(len(p)))
-	return sw.w.Write(p)
-}
-
-func generateAndPushLogs(cfg *workerConfig, workerID int) {
-	pr, pw := io.Pipe()
-	sw := &statWriter{
-		w: pw,
-	}
-	bw := bufio.NewWriter(sw)
-	doneCh := make(chan struct{})
-	go func() {
-		generateLogs(bw, workerID, cfg.activeStreams, cfg.totalStreams)
-		_ = bw.Flush()
-		_ = pw.Close()
-		close(doneCh)
-	}()
-
-	if cfg.url == nil {
-		_, err := io.Copy(os.Stdout, pr)
-		if err != nil {
-			logger.Fatalf("unexpected error when writing logs to stdout: %s", err)
-		}
-		return
-	}
-
-	req, err := http.NewRequest("POST", cfg.url.String(), pr)
-	if err != nil {
-		logger.Fatalf("cannot create request to %q: %s", cfg.url, err)
-	}
-
-	resp, err := http.DefaultClient.Do(req)
-	if err != nil {
-		logger.Fatalf("cannot perform request to %q: %s", cfg.url, err)
-	}
-	if resp.StatusCode/100 != 2 {
-		logger.Fatalf("unexpected status code got from %q: %d; want 2xx", cfg.url, err)
-	}
-
-	// Wait until all the generateLogs goroutine is finished.
-	<-doneCh
-}
-
-func generateLogs(bw *bufio.Writer, workerID, activeStreams, totalStreams int) {
-	streamLifetime := int64(float64(end.nsec-start.nsec) * (float64(activeStreams) / float64(totalStreams)))
-	streamStep := int64(float64(end.nsec-start.nsec) / float64(totalStreams-activeStreams+1))
-	step := streamLifetime / (*logsPerStream - 1)
-
-	currNsec := start.nsec
-	for currNsec < end.nsec {
-		firstStreamID := int((currNsec - start.nsec) / streamStep)
-		generateLogsAtTimestamp(bw, workerID, currNsec, firstStreamID, activeStreams)
-		currNsec += step
-	}
-}
-
-var runID = toUUID(rand.Uint64(), rand.Uint64())
-
-func generateLogsAtTimestamp(bw *bufio.Writer, workerID int, ts int64, firstStreamID, activeStreams int) {
-	streamID := firstStreamID
-	timeStr := toRFC3339(ts)
-	for i := 0; i < activeStreams; i++ {
-		ip := toIPv4(rand.Uint32())
-		uuid := toUUID(rand.Uint64(), rand.Uint64())
-		fmt.Fprintf(bw, `{"_time":"%s","_msg":"message for the stream %d and worker %d; ip=%s; uuid=%s; u64=%d","host":"host_%d","worker_id":"%d"`,
-			timeStr, streamID, workerID, ip, uuid, rand.Uint64(), streamID, workerID)
-		fmt.Fprintf(bw, `,"run_id":"%s"`, runID)
-		for j := 0; j < *constFieldsPerLog; j++ {
-			fmt.Fprintf(bw, `,"const_%d":"some value %d %d"`, j, j, streamID)
-		}
-		for j := 0; j < *varFieldsPerLog; j++ {
-			fmt.Fprintf(bw, `,"var_%d":"some value %d %d"`, j, j, rand.Uint64())
-		}
-		for j := 0; j < *dictFieldsPerLog; j++ {
-			fmt.Fprintf(bw, `,"dict_%d":"%s"`, j, dictValues[rand.Intn(len(dictValues))])
-		}
-		for j := 0; j < *u8FieldsPerLog; j++ {
-			fmt.Fprintf(bw, `,"u8_%d":"%d"`, j, uint8(rand.Uint32()))
-		}
-		for j := 0; j < *u16FieldsPerLog; j++ {
-			fmt.Fprintf(bw, `,"u16_%d":"%d"`, j, uint16(rand.Uint32()))
-		}
-		for j := 0; j < *u32FieldsPerLog; j++ {
-			fmt.Fprintf(bw, `,"u32_%d":"%d"`, j, rand.Uint32())
-		}
-		for j := 0; j < *u64FieldsPerLog; j++ {
-			fmt.Fprintf(bw, `,"u64_%d":"%d"`, j, rand.Uint64())
-		}
-		for j := 0; j < *floatFieldsPerLog; j++ {
-			fmt.Fprintf(bw, `,"float_%d":"%v"`, j, math.Round(10_000*rand.Float64())/1000)
-		}
-		for j := 0; j < *ipFieldsPerLog; j++ {
-			ip := toIPv4(rand.Uint32())
-			fmt.Fprintf(bw, `,"ip_%d":"%s"`, j, ip)
-		}
-		for j := 0; j < *timestampFieldsPerLog; j++ {
-			timestamp := toISO8601(int64(rand.Uint64()))
-			fmt.Fprintf(bw, `,"timestamp_%d":"%s"`, j, timestamp)
-		}
-		for j := 0; j < *jsonFieldsPerLog; j++ {
-			fmt.Fprintf(bw, `,"json_%d":"{\"foo\":\"bar_%d\",\"baz\":{\"a\":[\"x\",\"y\"]},\"f3\":NaN,\"f4\":%d}"`, j, rand.Intn(10), rand.Intn(100))
-		}
-		fmt.Fprintf(bw, "}\n")
-
-		logEntriesCount.Add(1)
-		streamID++
-	}
-}
-
-var dictValues = []string{
-	"debug",
-	"info",
-	"warn",
-	"error",
-	"fatal",
-	"ERROR",
-	"FATAL",
-	"INFO",
-}
-
-func newTimeFlag(name, defaultValue, description string) *timeFlag {
-	var tf timeFlag
-	if err := tf.Set(defaultValue); err != nil {
-		logger.Panicf("invalid defaultValue=%q for flag %q: %w", defaultValue, name, err)
-	}
-	flag.Var(&tf, name, description)
-	return &tf
-}
-
-type timeFlag struct {
-	s    string
-	nsec int64
-}
-
-func (tf *timeFlag) Set(s string) error {
-	msec, err := promutils.ParseTimeMsec(s)
-	if err != nil {
-		return fmt.Errorf("cannot parse time from %q: %w", s, err)
-	}
-	tf.s = s
-	tf.nsec = msec * 1e6
-	return nil
-}
-
-func (tf *timeFlag) String() string {
-	return tf.s
-}
-
-func toRFC3339(nsec int64) string {
-	return time.Unix(0, nsec).UTC().Format(time.RFC3339Nano)
-}
-
-func toISO8601(nsec int64) string {
-	return time.Unix(0, nsec).UTC().Format("2006-01-02T15:04:05.000Z")
-}
-
-func toIPv4(n uint32) string {
-	dst := make([]byte, 0, len("255.255.255.255"))
-	dst = marshalUint64(dst, uint64(n>>24))
-	dst = append(dst, '.')
-	dst = marshalUint64(dst, uint64((n>>16)&0xff))
-	dst = append(dst, '.')
-	dst = marshalUint64(dst, uint64((n>>8)&0xff))
-	dst = append(dst, '.')
-	dst = marshalUint64(dst, uint64(n&0xff))
-	return string(dst)
-}
-
-func toUUID(a, b uint64) string {
-	return fmt.Sprintf("%08x-%04x-%04x-%04x-%012x", a&(1<<32-1), (a>>32)&(1<<16-1), (a >> 48), b&(1<<16-1), b>>16)
-}
-
-// marshalUint64 appends string representation of n to dst and returns the result.
-func marshalUint64(dst []byte, n uint64) []byte {
-	return strconv.AppendUint(dst, n, 10)
-}

app/vlselect/logsql/buffered_writer.go

@@ -1,47 +0,0 @@
-package logsql
-
-import (
-	"bufio"
-	"io"
-	"sync"
-)
-
-func getBufferedWriter(w io.Writer) *bufferedWriter {
-	v := bufferedWriterPool.Get()
-	if v == nil {
-		return &bufferedWriter{
-			bw: bufio.NewWriter(w),
-		}
-	}
-	bw := v.(*bufferedWriter)
-	bw.bw.Reset(w)
-	return bw
-}
-
-func putBufferedWriter(bw *bufferedWriter) {
-	bw.reset()
-	bufferedWriterPool.Put(bw)
-}
-
-var bufferedWriterPool sync.Pool
-
-type bufferedWriter struct {
-	mu sync.Mutex
-	bw *bufio.Writer
-}
-
-func (bw *bufferedWriter) reset() {
-	// nothing to do
-}
-
-func (bw *bufferedWriter) WriteIgnoreErrors(p []byte) {
-	bw.mu.Lock()
-	_, _ = bw.bw.Write(p)
-	bw.mu.Unlock()
-}
-
-func (bw *bufferedWriter) FlushIgnoreErrors() {
-	bw.mu.Lock()
-	_ = bw.bw.Flush()
-	bw.mu.Unlock()
-}

app/vlselect/logsql/hits_response.qtpl

@@ -1,70 +0,0 @@
-{% import (
-	"slices"
-
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logstorage"
-) %}
-
-{% stripspace %}
-
-// FieldsForHits formats labels for /select/logsql/hits response
-{% func FieldsForHits(columns []logstorage.BlockColumn, rowIdx int) %}
-{
-	{% if len(columns) > 0 %}
-		{%q= columns[0].Name %}:{%q= columns[0].Values[rowIdx] %}
-		{% for _, c := range columns[1:] %}
-			,{%q= c.Name %}:{%q= c.Values[rowIdx] %}
-		{% endfor %}
-	{% endif %}
-}
-{% endfunc %}
-
-{% func HitsSeries(m map[string]*hitsSeries) %}
-{
-	{% code
-		sortedKeys := make([]string, 0, len(m))
-		for k := range m {
-			sortedKeys = append(sortedKeys, k)
-		}
-		slices.Sort(sortedKeys)
-	%}
-	"hits":[
-		{% if len(sortedKeys) > 0 %}
-			{%= hitsSeriesLine(m, sortedKeys[0]) %}
-			{% for _, k := range sortedKeys[1:] %}
-				,{%= hitsSeriesLine(m, k) %}
-			{% endfor %}
-		{% endif %}
-	]
-}
-{% endfunc %}
-
-{% func hitsSeriesLine(m map[string]*hitsSeries, k string) %}
-{
-	{% code
-		hs := m[k]
-		hs.sort()
-		timestamps := hs.timestamps
-		hits := hs.hits
-	%}
-	"fields":{%s= k %},
-	"timestamps":[
-		{% if len(timestamps) > 0 %}
-			{%q= timestamps[0] %}
-			{% for _, ts := range timestamps[1:] %}
-				,{%q= ts %}
-			{% endfor %}
-		{% endif %}
-	],
-	"values":[
-		{% if len(hits) > 0 %}
-			{%dul= hits[0] %}
-			{% for _, v := range hits[1:] %}
-				,{%dul= v %}
-			{% endfor %}
-		{% endif %}
-	],
-	"total":{%dul= hs.hitsTotal %}
-}
-{% endfunc %}
-
-{% endstripspace %}

app/vlselect/logsql/hits_response.qtpl.go

@@ -1,223 +0,0 @@
-// Code generated by qtc from "hits_response.qtpl". DO NOT EDIT.
-// See https://github.com/valyala/quicktemplate for details.
-
-//line app/vlselect/logsql/hits_response.qtpl:1
-package logsql
-
-//line app/vlselect/logsql/hits_response.qtpl:1
-import (
-	"slices"
-
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logstorage"
-)
-
-// FieldsForHits formats labels for /select/logsql/hits response
-
-//line app/vlselect/logsql/hits_response.qtpl:10
-import (
-	qtio422016 "io"
-
-	qt422016 "github.com/valyala/quicktemplate"
-)
-
-//line app/vlselect/logsql/hits_response.qtpl:10
-var (
-	_ = qtio422016.Copy
-	_ = qt422016.AcquireByteBuffer
-)
-
-//line app/vlselect/logsql/hits_response.qtpl:10
-func StreamFieldsForHits(qw422016 *qt422016.Writer, columns []logstorage.BlockColumn, rowIdx int) {
-//line app/vlselect/logsql/hits_response.qtpl:10
-	qw422016.N().S(`{`)
-//line app/vlselect/logsql/hits_response.qtpl:12
-	if len(columns) > 0 {
-//line app/vlselect/logsql/hits_response.qtpl:13
-		qw422016.N().Q(columns[0].Name)
-//line app/vlselect/logsql/hits_response.qtpl:13
-		qw422016.N().S(`:`)
-//line app/vlselect/logsql/hits_response.qtpl:13
-		qw422016.N().Q(columns[0].Values[rowIdx])
-//line app/vlselect/logsql/hits_response.qtpl:14
-		for _, c := range columns[1:] {
-//line app/vlselect/logsql/hits_response.qtpl:14
-			qw422016.N().S(`,`)
-//line app/vlselect/logsql/hits_response.qtpl:15
-			qw422016.N().Q(c.Name)
-//line app/vlselect/logsql/hits_response.qtpl:15
-			qw422016.N().S(`:`)
-//line app/vlselect/logsql/hits_response.qtpl:15
-			qw422016.N().Q(c.Values[rowIdx])
-//line app/vlselect/logsql/hits_response.qtpl:16
-		}
-//line app/vlselect/logsql/hits_response.qtpl:17
-	}
-//line app/vlselect/logsql/hits_response.qtpl:17
-	qw422016.N().S(`}`)
-//line app/vlselect/logsql/hits_response.qtpl:19
-}
-
-//line app/vlselect/logsql/hits_response.qtpl:19
-func WriteFieldsForHits(qq422016 qtio422016.Writer, columns []logstorage.BlockColumn, rowIdx int) {
-//line app/vlselect/logsql/hits_response.qtpl:19
-	qw422016 := qt422016.AcquireWriter(qq422016)
-//line app/vlselect/logsql/hits_response.qtpl:19
-	StreamFieldsForHits(qw422016, columns, rowIdx)
-//line app/vlselect/logsql/hits_response.qtpl:19
-	qt422016.ReleaseWriter(qw422016)
-//line app/vlselect/logsql/hits_response.qtpl:19
-}
-
-//line app/vlselect/logsql/hits_response.qtpl:19
-func FieldsForHits(columns []logstorage.BlockColumn, rowIdx int) string {
-//line app/vlselect/logsql/hits_response.qtpl:19
-	qb422016 := qt422016.AcquireByteBuffer()
-//line app/vlselect/logsql/hits_response.qtpl:19
-	WriteFieldsForHits(qb422016, columns, rowIdx)
-//line app/vlselect/logsql/hits_response.qtpl:19
-	qs422016 := string(qb422016.B)
-//line app/vlselect/logsql/hits_response.qtpl:19
-	qt422016.ReleaseByteBuffer(qb422016)
-//line app/vlselect/logsql/hits_response.qtpl:19
-	return qs422016
-//line app/vlselect/logsql/hits_response.qtpl:19
-}
-
-//line app/vlselect/logsql/hits_response.qtpl:21
-func StreamHitsSeries(qw422016 *qt422016.Writer, m map[string]*hitsSeries) {
-//line app/vlselect/logsql/hits_response.qtpl:21
-	qw422016.N().S(`{`)
-//line app/vlselect/logsql/hits_response.qtpl:24
-	sortedKeys := make([]string, 0, len(m))
-	for k := range m {
-		sortedKeys = append(sortedKeys, k)
-	}
-	slices.Sort(sortedKeys)
-
-//line app/vlselect/logsql/hits_response.qtpl:29
-	qw422016.N().S(`"hits":[`)
-//line app/vlselect/logsql/hits_response.qtpl:31
-	if len(sortedKeys) > 0 {
-//line app/vlselect/logsql/hits_response.qtpl:32
-		streamhitsSeriesLine(qw422016, m, sortedKeys[0])
-//line app/vlselect/logsql/hits_response.qtpl:33
-		for _, k := range sortedKeys[1:] {
-//line app/vlselect/logsql/hits_response.qtpl:33
-			qw422016.N().S(`,`)
-//line app/vlselect/logsql/hits_response.qtpl:34
-			streamhitsSeriesLine(qw422016, m, k)
-//line app/vlselect/logsql/hits_response.qtpl:35
-		}
-//line app/vlselect/logsql/hits_response.qtpl:36
-	}
-//line app/vlselect/logsql/hits_response.qtpl:36
-	qw422016.N().S(`]}`)
-//line app/vlselect/logsql/hits_response.qtpl:39
-}
-
-//line app/vlselect/logsql/hits_response.qtpl:39
-func WriteHitsSeries(qq422016 qtio422016.Writer, m map[string]*hitsSeries) {
-//line app/vlselect/logsql/hits_response.qtpl:39
-	qw422016 := qt422016.AcquireWriter(qq422016)
-//line app/vlselect/logsql/hits_response.qtpl:39
-	StreamHitsSeries(qw422016, m)
-//line app/vlselect/logsql/hits_response.qtpl:39
-	qt422016.ReleaseWriter(qw422016)
-//line app/vlselect/logsql/hits_response.qtpl:39
-}
-
-//line app/vlselect/logsql/hits_response.qtpl:39
-func HitsSeries(m map[string]*hitsSeries) string {
-//line app/vlselect/logsql/hits_response.qtpl:39
-	qb422016 := qt422016.AcquireByteBuffer()
-//line app/vlselect/logsql/hits_response.qtpl:39
-	WriteHitsSeries(qb422016, m)
-//line app/vlselect/logsql/hits_response.qtpl:39
-	qs422016 := string(qb422016.B)
-//line app/vlselect/logsql/hits_response.qtpl:39
-	qt422016.ReleaseByteBuffer(qb422016)
-//line app/vlselect/logsql/hits_response.qtpl:39
-	return qs422016
-//line app/vlselect/logsql/hits_response.qtpl:39
-}
-
-//line app/vlselect/logsql/hits_response.qtpl:41
-func streamhitsSeriesLine(qw422016 *qt422016.Writer, m map[string]*hitsSeries, k string) {
-//line app/vlselect/logsql/hits_response.qtpl:41
-	qw422016.N().S(`{`)
-//line app/vlselect/logsql/hits_response.qtpl:44
-	hs := m[k]
-	hs.sort()
-	timestamps := hs.timestamps
-	hits := hs.hits
-
-//line app/vlselect/logsql/hits_response.qtpl:48
-	qw422016.N().S(`"fields":`)
-//line app/vlselect/logsql/hits_response.qtpl:49
-	qw422016.N().S(k)
-//line app/vlselect/logsql/hits_response.qtpl:49
-	qw422016.N().S(`,"timestamps":[`)
-//line app/vlselect/logsql/hits_response.qtpl:51
-	if len(timestamps) > 0 {
-//line app/vlselect/logsql/hits_response.qtpl:52
-		qw422016.N().Q(timestamps[0])
-//line app/vlselect/logsql/hits_response.qtpl:53
-		for _, ts := range timestamps[1:] {
-//line app/vlselect/logsql/hits_response.qtpl:53
-			qw422016.N().S(`,`)
-//line app/vlselect/logsql/hits_response.qtpl:54
-			qw422016.N().Q(ts)
-//line app/vlselect/logsql/hits_response.qtpl:55
-		}
-//line app/vlselect/logsql/hits_response.qtpl:56
-	}
-//line app/vlselect/logsql/hits_response.qtpl:56
-	qw422016.N().S(`],"values":[`)
-//line app/vlselect/logsql/hits_response.qtpl:59
-	if len(hits) > 0 {
-//line app/vlselect/logsql/hits_response.qtpl:60
-		qw422016.N().DUL(hits[0])
-//line app/vlselect/logsql/hits_response.qtpl:61
-		for _, v := range hits[1:] {
-//line app/vlselect/logsql/hits_response.qtpl:61
-			qw422016.N().S(`,`)
-//line app/vlselect/logsql/hits_response.qtpl:62
-			qw422016.N().DUL(v)
-//line app/vlselect/logsql/hits_response.qtpl:63
-		}
-//line app/vlselect/logsql/hits_response.qtpl:64
-	}
-//line app/vlselect/logsql/hits_response.qtpl:64
-	qw422016.N().S(`],"total":`)
-//line app/vlselect/logsql/hits_response.qtpl:66
-	qw422016.N().DUL(hs.hitsTotal)
-//line app/vlselect/logsql/hits_response.qtpl:66
-	qw422016.N().S(`}`)
-//line app/vlselect/logsql/hits_response.qtpl:68
-}
-
-//line app/vlselect/logsql/hits_response.qtpl:68
-func writehitsSeriesLine(qq422016 qtio422016.Writer, m map[string]*hitsSeries, k string) {
-//line app/vlselect/logsql/hits_response.qtpl:68
-	qw422016 := qt422016.AcquireWriter(qq422016)
-//line app/vlselect/logsql/hits_response.qtpl:68
-	streamhitsSeriesLine(qw422016, m, k)
-//line app/vlselect/logsql/hits_response.qtpl:68
-	qt422016.ReleaseWriter(qw422016)
-//line app/vlselect/logsql/hits_response.qtpl:68
-}
-
-//line app/vlselect/logsql/hits_response.qtpl:68
-func hitsSeriesLine(m map[string]*hitsSeries, k string) string {
-//line app/vlselect/logsql/hits_response.qtpl:68
-	qb422016 := qt422016.AcquireByteBuffer()
-//line app/vlselect/logsql/hits_response.qtpl:68
-	writehitsSeriesLine(qb422016, m, k)
-//line app/vlselect/logsql/hits_response.qtpl:68
-	qs422016 := string(qb422016.B)
-//line app/vlselect/logsql/hits_response.qtpl:68
-	qt422016.ReleaseByteBuffer(qb422016)
-//line app/vlselect/logsql/hits_response.qtpl:68
-	return qs422016
-//line app/vlselect/logsql/hits_response.qtpl:68
-}

app/vlselect/logsql/logsql.go

@@ -1,771 +0,0 @@
-package logsql
-
-import (
-	"context"
-	"fmt"
-	"math"
-	"net/http"
-	"sort"
-	"strconv"
-	"strings"
-	"sync"
-	"time"
-
-	"github.com/VictoriaMetrics/metrics"
-
-	"github.com/VictoriaMetrics/VictoriaMetrics/app/vlstorage"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/bytesutil"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/httputils"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logstorage"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/promutils"
-)
-
-// ProcessHitsRequest handles /select/logsql/hits request.
-//
-// See https://docs.victoriametrics.com/victorialogs/querying/#querying-hits-stats
-func ProcessHitsRequest(ctx context.Context, w http.ResponseWriter, r *http.Request) {
-	q, tenantIDs, err := parseCommonArgs(r)
-	if err != nil {
-		httpserver.Errorf(w, r, "%s", err)
-		return
-	}
-
-	// Obtain step
-	stepStr := r.FormValue("step")
-	if stepStr == "" {
-		stepStr = "1d"
-	}
-	step, err := promutils.ParseDuration(stepStr)
-	if err != nil {
-		httpserver.Errorf(w, r, "cannot parse 'step' arg: %s", err)
-		return
-	}
-	if step <= 0 {
-		httpserver.Errorf(w, r, "'step' must be bigger than zero")
-	}
-
-	// Obtain offset
-	offsetStr := r.FormValue("offset")
-	if offsetStr == "" {
-		offsetStr = "0s"
-	}
-	offset, err := promutils.ParseDuration(offsetStr)
-	if err != nil {
-		httpserver.Errorf(w, r, "cannot parse 'offset' arg: %s", err)
-		return
-	}
-
-	// Obtain field entries
-	fields := r.Form["field"]
-
-	// Obtain limit on the number of top fields entries.
-	fieldsLimit, err := httputils.GetInt(r, "fields_limit")
-	if err != nil {
-		httpserver.Errorf(w, r, "%s", err)
-		return
-	}
-	if fieldsLimit < 0 {
-		fieldsLimit = 0
-	}
-
-	// Prepare the query for hits count.
-	q.Optimize()
-	q.DropAllPipes()
-	q.AddCountByTimePipe(int64(step), int64(offset), fields)
-
-	var mLock sync.Mutex
-	m := make(map[string]*hitsSeries)
-	writeBlock := func(_ uint, timestamps []int64, columns []logstorage.BlockColumn) {
-		if len(columns) == 0 || len(columns[0].Values) == 0 {
-			return
-		}
-
-		timestampValues := columns[0].Values
-		hitsValues := columns[len(columns)-1].Values
-		columns = columns[1 : len(columns)-1]
-
-		bb := blockResultPool.Get()
-		for i := range timestamps {
-			timestampStr := strings.Clone(timestampValues[i])
-			hitsStr := strings.Clone(hitsValues[i])
-			hits, err := strconv.ParseUint(hitsStr, 10, 64)
-			if err != nil {
-				logger.Panicf("BUG: cannot parse hitsStr=%q: %s", hitsStr, err)
-			}
-
-			bb.Reset()
-			WriteFieldsForHits(bb, columns, i)
-
-			mLock.Lock()
-			hs, ok := m[string(bb.B)]
-			if !ok {
-				k := string(bb.B)
-				hs = &hitsSeries{}
-				m[k] = hs
-			}
-			hs.timestamps = append(hs.timestamps, timestampStr)
-			hs.hits = append(hs.hits, hits)
-			hs.hitsTotal += hits
-			mLock.Unlock()
-		}
-		blockResultPool.Put(bb)
-	}
-
-	// Execute the query
-	if err := vlstorage.RunQuery(ctx, tenantIDs, q, writeBlock); err != nil {
-		httpserver.Errorf(w, r, "cannot execute query [%s]: %s", q, err)
-		return
-	}
-
-	m = getTopHitsSeries(m, fieldsLimit)
-
-	// Write response
-	w.Header().Set("Content-Type", "application/json")
-	WriteHitsSeries(w, m)
-}
-
-func getTopHitsSeries(m map[string]*hitsSeries, fieldsLimit int) map[string]*hitsSeries {
-	if fieldsLimit <= 0 || fieldsLimit >= len(m) {
-		return m
-	}
-
-	type fieldsHits struct {
-		fieldsStr string
-		hs        *hitsSeries
-	}
-	a := make([]fieldsHits, 0, len(m))
-	for fieldsStr, hs := range m {
-		a = append(a, fieldsHits{
-			fieldsStr: fieldsStr,
-			hs:        hs,
-		})
-	}
-	sort.Slice(a, func(i, j int) bool {
-		return a[i].hs.hitsTotal > a[j].hs.hitsTotal
-	})
-
-	hitsOther := make(map[string]uint64)
-	for _, x := range a[fieldsLimit:] {
-		for i, timestampStr := range x.hs.timestamps {
-			hitsOther[timestampStr] += x.hs.hits[i]
-		}
-	}
-	var hsOther hitsSeries
-	for timestampStr, hits := range hitsOther {
-		hsOther.timestamps = append(hsOther.timestamps, timestampStr)
-		hsOther.hits = append(hsOther.hits, hits)
-		hsOther.hitsTotal += hits
-	}
-
-	mNew := make(map[string]*hitsSeries, fieldsLimit+1)
-	for _, x := range a[:fieldsLimit] {
-		mNew[x.fieldsStr] = x.hs
-	}
-	mNew["{}"] = &hsOther
-
-	return mNew
-}
-
-type hitsSeries struct {
-	hitsTotal  uint64
-	timestamps []string
-	hits       []uint64
-}
-
-func (hs *hitsSeries) sort() {
-	sort.Sort(hs)
-}
-
-func (hs *hitsSeries) Len() int {
-	return len(hs.timestamps)
-}
-
-func (hs *hitsSeries) Swap(i, j int) {
-	hs.timestamps[i], hs.timestamps[j] = hs.timestamps[j], hs.timestamps[i]
-	hs.hits[i], hs.hits[j] = hs.hits[j], hs.hits[i]
-}
-
-func (hs *hitsSeries) Less(i, j int) bool {
-	return hs.timestamps[i] < hs.timestamps[j]
-}
-
-// ProcessFieldNamesRequest handles /select/logsql/field_names request.
-//
-// See https://docs.victoriametrics.com/victorialogs/querying/#querying-field-names
-func ProcessFieldNamesRequest(ctx context.Context, w http.ResponseWriter, r *http.Request) {
-	q, tenantIDs, err := parseCommonArgs(r)
-	if err != nil {
-		httpserver.Errorf(w, r, "%s", err)
-		return
-	}
-
-	// Obtain field names for the given query
-	q.Optimize()
-	fieldNames, err := vlstorage.GetFieldNames(ctx, tenantIDs, q)
-	if err != nil {
-		httpserver.Errorf(w, r, "cannot obtain field names: %s", err)
-		return
-	}
-
-	// Write results
-	w.Header().Set("Content-Type", "application/json")
-	WriteValuesWithHitsJSON(w, fieldNames)
-}
-
-// ProcessFieldValuesRequest handles /select/logsql/field_values request.
-//
-// See https://docs.victoriametrics.com/victorialogs/querying/#querying-field-values
-func ProcessFieldValuesRequest(ctx context.Context, w http.ResponseWriter, r *http.Request) {
-	q, tenantIDs, err := parseCommonArgs(r)
-	if err != nil {
-		httpserver.Errorf(w, r, "%s", err)
-		return
-	}
-
-	// Parse fieldName query arg
-	fieldName := r.FormValue("field")
-	if fieldName == "" {
-		httpserver.Errorf(w, r, "missing 'field' query arg")
-		return
-	}
-
-	// Parse limit query arg
-	limit, err := httputils.GetInt(r, "limit")
-	if err != nil {
-		httpserver.Errorf(w, r, "%s", err)
-		return
-	}
-	if limit < 0 {
-		limit = 0
-	}
-
-	// Obtain unique values for the given field
-	q.Optimize()
-	values, err := vlstorage.GetFieldValues(ctx, tenantIDs, q, fieldName, uint64(limit))
-	if err != nil {
-		httpserver.Errorf(w, r, "cannot obtain values for field %q: %s", fieldName, err)
-		return
-	}
-
-	// Write results
-	w.Header().Set("Content-Type", "application/json")
-	WriteValuesWithHitsJSON(w, values)
-}
-
-// ProcessStreamFieldNamesRequest processes /select/logsql/stream_field_names request.
-//
-// See https://docs.victoriametrics.com/victorialogs/querying/#querying-stream-field-names
-func ProcessStreamFieldNamesRequest(ctx context.Context, w http.ResponseWriter, r *http.Request) {
-	q, tenantIDs, err := parseCommonArgs(r)
-	if err != nil {
-		httpserver.Errorf(w, r, "%s", err)
-		return
-	}
-
-	// Obtain stream field names for the given query
-	q.Optimize()
-	names, err := vlstorage.GetStreamFieldNames(ctx, tenantIDs, q)
-	if err != nil {
-		httpserver.Errorf(w, r, "cannot obtain stream field names: %s", err)
-	}
-
-	// Write results
-	w.Header().Set("Content-Type", "application/json")
-	WriteValuesWithHitsJSON(w, names)
-}
-
-// ProcessStreamFieldValuesRequest processes /select/logsql/stream_field_values request.
-//
-// See https://docs.victoriametrics.com/victorialogs/querying/#querying-stream-field-values
-func ProcessStreamFieldValuesRequest(ctx context.Context, w http.ResponseWriter, r *http.Request) {
-	q, tenantIDs, err := parseCommonArgs(r)
-	if err != nil {
-		httpserver.Errorf(w, r, "%s", err)
-		return
-	}
-
-	// Parse fieldName query arg
-	fieldName := r.FormValue("field")
-	if fieldName == "" {
-		httpserver.Errorf(w, r, "missing 'field' query arg")
-		return
-	}
-
-	// Parse limit query arg
-	limit, err := httputils.GetInt(r, "limit")
-	if err != nil {
-		httpserver.Errorf(w, r, "%s", err)
-		return
-	}
-	if limit < 0 {
-		limit = 0
-	}
-
-	// Obtain stream field values for the given query and the given fieldName
-	q.Optimize()
-	values, err := vlstorage.GetStreamFieldValues(ctx, tenantIDs, q, fieldName, uint64(limit))
-	if err != nil {
-		httpserver.Errorf(w, r, "cannot obtain stream field values: %s", err)
-	}
-
-	// Write results
-	w.Header().Set("Content-Type", "application/json")
-	WriteValuesWithHitsJSON(w, values)
-}
-
-// ProcessStreamIDsRequest processes /select/logsql/stream_ids request.
-//
-// See https://docs.victoriametrics.com/victorialogs/querying/#querying-stream_ids
-func ProcessStreamIDsRequest(ctx context.Context, w http.ResponseWriter, r *http.Request) {
-	q, tenantIDs, err := parseCommonArgs(r)
-	if err != nil {
-		httpserver.Errorf(w, r, "%s", err)
-		return
-	}
-
-	// Parse limit query arg
-	limit, err := httputils.GetInt(r, "limit")
-	if err != nil {
-		httpserver.Errorf(w, r, "%s", err)
-		return
-	}
-	if limit < 0 {
-		limit = 0
-	}
-
-	// Obtain streamIDs for the given query
-	q.Optimize()
-	streamIDs, err := vlstorage.GetStreamIDs(ctx, tenantIDs, q, uint64(limit))
-	if err != nil {
-		httpserver.Errorf(w, r, "cannot obtain stream_ids: %s", err)
-	}
-
-	// Write results
-	w.Header().Set("Content-Type", "application/json")
-	WriteValuesWithHitsJSON(w, streamIDs)
-}
-
-// ProcessStreamsRequest processes /select/logsql/streams request.
-//
-// See https://docs.victoriametrics.com/victorialogs/querying/#querying-streams
-func ProcessStreamsRequest(ctx context.Context, w http.ResponseWriter, r *http.Request) {
-	q, tenantIDs, err := parseCommonArgs(r)
-	if err != nil {
-		httpserver.Errorf(w, r, "%s", err)
-		return
-	}
-
-	// Parse limit query arg
-	limit, err := httputils.GetInt(r, "limit")
-	if err != nil {
-		httpserver.Errorf(w, r, "%s", err)
-		return
-	}
-	if limit < 0 {
-		limit = 0
-	}
-
-	// Obtain streams for the given query
-	q.Optimize()
-	streams, err := vlstorage.GetStreams(ctx, tenantIDs, q, uint64(limit))
-	if err != nil {
-		httpserver.Errorf(w, r, "cannot obtain streams: %s", err)
-	}
-
-	// Write results
-	w.Header().Set("Content-Type", "application/json")
-	WriteValuesWithHitsJSON(w, streams)
-}
-
-// ProcessLiveTailRequest processes live tailing request to /select/logsq/tail
-func ProcessLiveTailRequest(ctx context.Context, w http.ResponseWriter, r *http.Request) {
-	liveTailRequests.Inc()
-	defer liveTailRequests.Dec()
-
-	q, tenantIDs, err := parseCommonArgs(r)
-	if err != nil {
-		httpserver.Errorf(w, r, "%s", err)
-		return
-	}
-	if !q.CanLiveTail() {
-		httpserver.Errorf(w, r, "the query [%s] cannot be used in live tailing; see https://docs.victoriametrics.com/victorialogs/querying/#live-tailing for details", q)
-	}
-	q.Optimize()
-
-	refreshIntervalMsecs, err := httputils.GetDuration(r, "refresh_interval", 1000)
-	if err != nil {
-		httpserver.Errorf(w, r, "%s", err)
-		return
-	}
-	refreshInterval := time.Millisecond * time.Duration(refreshIntervalMsecs)
-
-	ctxWithCancel, cancel := context.WithCancel(ctx)
-	tp := newTailProcessor(cancel)
-
-	ticker := time.NewTicker(refreshInterval)
-	defer ticker.Stop()
-
-	end := time.Now().UnixNano()
-	doneCh := ctxWithCancel.Done()
-	flusher, ok := w.(http.Flusher)
-	if !ok {
-		logger.Panicf("BUG: it is expected that http.ResponseWriter (%T) supports http.Flusher interface", w)
-	}
-	for {
-		start := end - tailOffsetNsecs
-		end = time.Now().UnixNano()
-
-		qCopy := q.Clone()
-		qCopy.AddTimeFilter(start, end)
-		if err := vlstorage.RunQuery(ctxWithCancel, tenantIDs, qCopy, tp.writeBlock); err != nil {
-			httpserver.Errorf(w, r, "cannot execute tail query [%s]: %s", q, err)
-			return
-		}
-		resultRows, err := tp.getTailRows()
-		if err != nil {
-			httpserver.Errorf(w, r, "cannot get tail results for query [%q]: %s", q, err)
-			return
-		}
-		if len(resultRows) > 0 {
-			WriteJSONRows(w, resultRows)
-			flusher.Flush()
-		}
-
-		select {
-		case <-doneCh:
-			return
-		case <-ticker.C:
-		}
-	}
-}
-
-var liveTailRequests = metrics.NewCounter(`vl_live_tailing_requests`)
-
-const tailOffsetNsecs = 5e9
-
-type logRow struct {
-	timestamp int64
-	fields    []logstorage.Field
-}
-
-func sortLogRows(rows []logRow) {
-	sort.SliceStable(rows, func(i, j int) bool {
-		return rows[i].timestamp < rows[j].timestamp
-	})
-}
-
-type tailProcessor struct {
-	cancel func()
-
-	mu sync.Mutex
-
-	perStreamRows  map[string][]logRow
-	lastTimestamps map[string]int64
-
-	err error
-}
-
-func newTailProcessor(cancel func()) *tailProcessor {
-	return &tailProcessor{
-		cancel: cancel,
-
-		perStreamRows:  make(map[string][]logRow),
-		lastTimestamps: make(map[string]int64),
-	}
-}
-
-func (tp *tailProcessor) writeBlock(_ uint, timestamps []int64, columns []logstorage.BlockColumn) {
-	if len(timestamps) == 0 {
-		return
-	}
-
-	tp.mu.Lock()
-	defer tp.mu.Unlock()
-
-	if tp.err != nil {
-		return
-	}
-
-	// Make sure columns contain _time field, since it is needed for proper tail work.
-	hasTime := false
-	for _, c := range columns {
-		if c.Name == "_time" {
-			hasTime = true
-			break
-		}
-	}
-	if !hasTime {
-		tp.err = fmt.Errorf("missing _time field")
-		tp.cancel()
-		return
-	}
-
-	// Copy block rows to tp.perStreamRows
-	for i, timestamp := range timestamps {
-		streamID := ""
-		fields := make([]logstorage.Field, len(columns))
-		for j, c := range columns {
-			name := strings.Clone(c.Name)
-			value := strings.Clone(c.Values[i])
-
-			fields[j] = logstorage.Field{
-				Name:  name,
-				Value: value,
-			}
-
-			if name == "_stream_id" {
-				streamID = value
-			}
-		}
-
-		tp.perStreamRows[streamID] = append(tp.perStreamRows[streamID], logRow{
-			timestamp: timestamp,
-			fields:    fields,
-		})
-	}
-}
-
-func (tp *tailProcessor) getTailRows() ([][]logstorage.Field, error) {
-	if tp.err != nil {
-		return nil, tp.err
-	}
-
-	var resultRows []logRow
-	for streamID, rows := range tp.perStreamRows {
-		sortLogRows(rows)
-
-		lastTimestamp, ok := tp.lastTimestamps[streamID]
-		if ok {
-			// Skip already written rows
-			for len(rows) > 0 && rows[0].timestamp <= lastTimestamp {
-				rows = rows[1:]
-			}
-		}
-		if len(rows) > 0 {
-			resultRows = append(resultRows, rows...)
-			tp.lastTimestamps[streamID] = rows[len(rows)-1].timestamp
-		}
-	}
-	clear(tp.perStreamRows)
-
-	sortLogRows(resultRows)
-
-	tailRows := make([][]logstorage.Field, len(resultRows))
-	for i, row := range resultRows {
-		tailRows[i] = row.fields
-	}
-
-	return tailRows, nil
-}
-
-// ProcessQueryRequest handles /select/logsql/query request.
-//
-// See https://docs.victoriametrics.com/victorialogs/querying/#http-api
-func ProcessQueryRequest(ctx context.Context, w http.ResponseWriter, r *http.Request) {
-	q, tenantIDs, err := parseCommonArgs(r)
-	if err != nil {
-		httpserver.Errorf(w, r, "%s", err)
-		return
-	}
-
-	// Parse limit query arg
-	limit, err := httputils.GetInt(r, "limit")
-	if err != nil {
-		httpserver.Errorf(w, r, "%s", err)
-		return
-	}
-
-	bw := getBufferedWriter(w)
-	defer func() {
-		bw.FlushIgnoreErrors()
-		putBufferedWriter(bw)
-	}()
-	w.Header().Set("Content-Type", "application/stream+json")
-
-	if limit > 0 {
-		if q.CanReturnLastNResults() {
-			rows, err := getLastNQueryResults(ctx, tenantIDs, q, limit)
-			if err != nil {
-				httpserver.Errorf(w, r, "%s", err)
-				return
-			}
-			bb := blockResultPool.Get()
-			b := bb.B
-			for i := range rows {
-				b = logstorage.MarshalFieldsToJSON(b[:0], rows[i].fields)
-				b = append(b, '\n')
-				bw.WriteIgnoreErrors(b)
-			}
-			bb.B = b
-			blockResultPool.Put(bb)
-			return
-		}
-
-		q.AddPipeLimit(uint64(limit))
-	}
-	q.Optimize()
-
-	writeBlock := func(_ uint, timestamps []int64, columns []logstorage.BlockColumn) {
-		if len(columns) == 0 || len(columns[0].Values) == 0 {
-			return
-		}
-
-		bb := blockResultPool.Get()
-		for i := range timestamps {
-			WriteJSONRow(bb, columns, i)
-		}
-		bw.WriteIgnoreErrors(bb.B)
-		blockResultPool.Put(bb)
-	}
-
-	if err := vlstorage.RunQuery(ctx, tenantIDs, q, writeBlock); err != nil {
-		httpserver.Errorf(w, r, "cannot execute query [%s]: %s", q, err)
-		return
-	}
-}
-
-var blockResultPool bytesutil.ByteBufferPool
-
-type row struct {
-	timestamp int64
-	fields    []logstorage.Field
-}
-
-func getLastNQueryResults(ctx context.Context, tenantIDs []logstorage.TenantID, q *logstorage.Query, limit int) ([]row, error) {
-	limitUpper := 2 * limit
-	q.AddPipeLimit(uint64(limitUpper))
-	q.Optimize()
-	rows, err := getQueryResultsWithLimit(ctx, tenantIDs, q, limitUpper)
-	if err != nil {
-		return nil, err
-	}
-	if len(rows) < limitUpper {
-		// Fast path - the requested time range contains up to limitUpper rows.
-		rows = getLastNRows(rows, limit)
-		return rows, nil
-	}
-
-	// Slow path - search for the time range containing up to limitUpper rows.
-	start, end := q.GetFilterTimeRange()
-	d := end/2 - start/2
-	start += d
-
-	qOrig := q
-	for {
-		q = qOrig.Clone()
-		q.AddTimeFilter(start, end)
-		rows, err := getQueryResultsWithLimit(ctx, tenantIDs, q, limitUpper)
-		if err != nil {
-			return nil, err
-		}
-
-		if len(rows) >= limit && len(rows) < limitUpper || d == 0 {
-			rows = getLastNRows(rows, limit)
-			return rows, nil
-		}
-
-		lastBit := d & 1
-		d /= 2
-		if len(rows) > limit {
-			start += d
-		} else {
-			start -= d + lastBit
-		}
-	}
-}
-
-func getLastNRows(rows []row, limit int) []row {
-	sort.Slice(rows, func(i, j int) bool {
-		return rows[i].timestamp < rows[j].timestamp
-	})
-	if len(rows) > limit {
-		rows = rows[len(rows)-limit:]
-	}
-	return rows
-}
-
-func getQueryResultsWithLimit(ctx context.Context, tenantIDs []logstorage.TenantID, q *logstorage.Query, limit int) ([]row, error) {
-	ctxWithCancel, cancel := context.WithCancel(ctx)
-	defer cancel()
-
-	var rows []row
-	var rowsLock sync.Mutex
-	writeBlock := func(_ uint, timestamps []int64, columns []logstorage.BlockColumn) {
-		rowsLock.Lock()
-		defer rowsLock.Unlock()
-
-		for i, timestamp := range timestamps {
-			fields := make([]logstorage.Field, len(columns))
-			for j := range columns {
-				f := &fields[j]
-				f.Name = strings.Clone(columns[j].Name)
-				f.Value = strings.Clone(columns[j].Values[i])
-			}
-			rows = append(rows, row{
-				timestamp: timestamp,
-				fields:    fields,
-			})
-		}
-
-		if len(rows) >= limit {
-			cancel()
-		}
-	}
-	if err := vlstorage.RunQuery(ctxWithCancel, tenantIDs, q, writeBlock); err != nil {
-		return nil, err
-	}
-
-	return rows, nil
-}
-
-func parseCommonArgs(r *http.Request) (*logstorage.Query, []logstorage.TenantID, error) {
-	// Extract tenantID
-	tenantID, err := logstorage.GetTenantIDFromRequest(r)
-	if err != nil {
-		return nil, nil, fmt.Errorf("cannot obtain tenanID: %w", err)
-	}
-	tenantIDs := []logstorage.TenantID{tenantID}
-
-	// Parse query
-	qStr := r.FormValue("query")
-	q, err := logstorage.ParseQuery(qStr)
-	if err != nil {
-		return nil, nil, fmt.Errorf("cannot parse query [%s]: %s", qStr, err)
-	}
-
-	// Parse optional start and end args
-	start, okStart, err := getTimeNsec(r, "start")
-	if err != nil {
-		return nil, nil, err
-	}
-	end, okEnd, err := getTimeNsec(r, "end")
-	if err != nil {
-		return nil, nil, err
-	}
-	if okStart || okEnd {
-		if !okStart {
-			start = math.MinInt64
-		}
-		if !okEnd {
-			end = math.MaxInt64
-		}
-		q.AddTimeFilter(start, end)
-	}
-
-	return q, tenantIDs, nil
-}
-
-func getTimeNsec(r *http.Request, argName string) (int64, bool, error) {
-	s := r.FormValue(argName)
-	if s == "" {
-		return 0, false, nil
-	}
-	currentTimestamp := time.Now().UnixNano()
-	nsecs, err := promutils.ParseTimeAt(s, currentTimestamp)
-	if err != nil {
-		return 0, false, fmt.Errorf("cannot parse %s=%s: %w", argName, s, err)
-	}
-	return nsecs, true, nil
-}

app/vlselect/logsql/logsql.qtpl

@@ -1,32 +0,0 @@
-{% import (
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logstorage"
-) %}
-
-{% stripspace %}
-
-// ValuesWithHitsJSON generates JSON from the given values.
-{% func ValuesWithHitsJSON(values []logstorage.ValueWithHits) %}
-{
-	"values":{%= valuesWithHitsJSONArray(values) %}
-}
-{% endfunc %}
-
-{% func valuesWithHitsJSONArray(values []logstorage.ValueWithHits) %}
-[
-	{% if len(values) > 0 %}
-		{%= valueWithHitsJSON(values[0]) %}
-		{% for _, v := range values[1:] %}
-			,{%= valueWithHitsJSON(v) %}
-		{% endfor %}
-	{% endif %}
-]
-{% endfunc %}
-
-{% func valueWithHitsJSON(v logstorage.ValueWithHits) %}
-{
-	"value":{%q= v.Value %},
-	"hits":{%dul= v.Hits %}
-}
-{% endfunc %}
-
-{% endstripspace %}

app/vlselect/logsql/logsql.qtpl.go

@@ -1,152 +0,0 @@
-// Code generated by qtc from "logsql.qtpl". DO NOT EDIT.
-// See https://github.com/valyala/quicktemplate for details.
-
-//line app/vlselect/logsql/logsql.qtpl:1
-package logsql
-
-//line app/vlselect/logsql/logsql.qtpl:1
-import (
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logstorage"
-)
-
-// ValuesWithHitsJSON generates JSON from the given values.
-
-//line app/vlselect/logsql/logsql.qtpl:8
-import (
-	qtio422016 "io"
-
-	qt422016 "github.com/valyala/quicktemplate"
-)
-
-//line app/vlselect/logsql/logsql.qtpl:8
-var (
-	_ = qtio422016.Copy
-	_ = qt422016.AcquireByteBuffer
-)
-
-//line app/vlselect/logsql/logsql.qtpl:8
-func StreamValuesWithHitsJSON(qw422016 *qt422016.Writer, values []logstorage.ValueWithHits) {
-//line app/vlselect/logsql/logsql.qtpl:8
-	qw422016.N().S(`{"values":`)
-//line app/vlselect/logsql/logsql.qtpl:10
-	streamvaluesWithHitsJSONArray(qw422016, values)
-//line app/vlselect/logsql/logsql.qtpl:10
-	qw422016.N().S(`}`)
-//line app/vlselect/logsql/logsql.qtpl:12
-}
-
-//line app/vlselect/logsql/logsql.qtpl:12
-func WriteValuesWithHitsJSON(qq422016 qtio422016.Writer, values []logstorage.ValueWithHits) {
-//line app/vlselect/logsql/logsql.qtpl:12
-	qw422016 := qt422016.AcquireWriter(qq422016)
-//line app/vlselect/logsql/logsql.qtpl:12
-	StreamValuesWithHitsJSON(qw422016, values)
-//line app/vlselect/logsql/logsql.qtpl:12
-	qt422016.ReleaseWriter(qw422016)
-//line app/vlselect/logsql/logsql.qtpl:12
-}
-
-//line app/vlselect/logsql/logsql.qtpl:12
-func ValuesWithHitsJSON(values []logstorage.ValueWithHits) string {
-//line app/vlselect/logsql/logsql.qtpl:12
-	qb422016 := qt422016.AcquireByteBuffer()
-//line app/vlselect/logsql/logsql.qtpl:12
-	WriteValuesWithHitsJSON(qb422016, values)
-//line app/vlselect/logsql/logsql.qtpl:12
-	qs422016 := string(qb422016.B)
-//line app/vlselect/logsql/logsql.qtpl:12
-	qt422016.ReleaseByteBuffer(qb422016)
-//line app/vlselect/logsql/logsql.qtpl:12
-	return qs422016
-//line app/vlselect/logsql/logsql.qtpl:12
-}
-
-//line app/vlselect/logsql/logsql.qtpl:14
-func streamvaluesWithHitsJSONArray(qw422016 *qt422016.Writer, values []logstorage.ValueWithHits) {
-//line app/vlselect/logsql/logsql.qtpl:14
-	qw422016.N().S(`[`)
-//line app/vlselect/logsql/logsql.qtpl:16
-	if len(values) > 0 {
-//line app/vlselect/logsql/logsql.qtpl:17
-		streamvalueWithHitsJSON(qw422016, values[0])
-//line app/vlselect/logsql/logsql.qtpl:18
-		for _, v := range values[1:] {
-//line app/vlselect/logsql/logsql.qtpl:18
-			qw422016.N().S(`,`)
-//line app/vlselect/logsql/logsql.qtpl:19
-			streamvalueWithHitsJSON(qw422016, v)
-//line app/vlselect/logsql/logsql.qtpl:20
-		}
-//line app/vlselect/logsql/logsql.qtpl:21
-	}
-//line app/vlselect/logsql/logsql.qtpl:21
-	qw422016.N().S(`]`)
-//line app/vlselect/logsql/logsql.qtpl:23
-}
-
-//line app/vlselect/logsql/logsql.qtpl:23
-func writevaluesWithHitsJSONArray(qq422016 qtio422016.Writer, values []logstorage.ValueWithHits) {
-//line app/vlselect/logsql/logsql.qtpl:23
-	qw422016 := qt422016.AcquireWriter(qq422016)
-//line app/vlselect/logsql/logsql.qtpl:23
-	streamvaluesWithHitsJSONArray(qw422016, values)
-//line app/vlselect/logsql/logsql.qtpl:23
-	qt422016.ReleaseWriter(qw422016)
-//line app/vlselect/logsql/logsql.qtpl:23
-}
-
-//line app/vlselect/logsql/logsql.qtpl:23
-func valuesWithHitsJSONArray(values []logstorage.ValueWithHits) string {
-//line app/vlselect/logsql/logsql.qtpl:23
-	qb422016 := qt422016.AcquireByteBuffer()
-//line app/vlselect/logsql/logsql.qtpl:23
-	writevaluesWithHitsJSONArray(qb422016, values)
-//line app/vlselect/logsql/logsql.qtpl:23
-	qs422016 := string(qb422016.B)
-//line app/vlselect/logsql/logsql.qtpl:23
-	qt422016.ReleaseByteBuffer(qb422016)
-//line app/vlselect/logsql/logsql.qtpl:23
-	return qs422016
-//line app/vlselect/logsql/logsql.qtpl:23
-}
-
-//line app/vlselect/logsql/logsql.qtpl:25
-func streamvalueWithHitsJSON(qw422016 *qt422016.Writer, v logstorage.ValueWithHits) {
-//line app/vlselect/logsql/logsql.qtpl:25
-	qw422016.N().S(`{"value":`)
-//line app/vlselect/logsql/logsql.qtpl:27
-	qw422016.N().Q(v.Value)
-//line app/vlselect/logsql/logsql.qtpl:27
-	qw422016.N().S(`,"hits":`)
-//line app/vlselect/logsql/logsql.qtpl:28
-	qw422016.N().DUL(v.Hits)
-//line app/vlselect/logsql/logsql.qtpl:28
-	qw422016.N().S(`}`)
-//line app/vlselect/logsql/logsql.qtpl:30
-}
-
-//line app/vlselect/logsql/logsql.qtpl:30
-func writevalueWithHitsJSON(qq422016 qtio422016.Writer, v logstorage.ValueWithHits) {
-//line app/vlselect/logsql/logsql.qtpl:30
-	qw422016 := qt422016.AcquireWriter(qq422016)
-//line app/vlselect/logsql/logsql.qtpl:30
-	streamvalueWithHitsJSON(qw422016, v)
-//line app/vlselect/logsql/logsql.qtpl:30
-	qt422016.ReleaseWriter(qw422016)
-//line app/vlselect/logsql/logsql.qtpl:30
-}
-
-//line app/vlselect/logsql/logsql.qtpl:30
-func valueWithHitsJSON(v logstorage.ValueWithHits) string {
-//line app/vlselect/logsql/logsql.qtpl:30
-	qb422016 := qt422016.AcquireByteBuffer()
-//line app/vlselect/logsql/logsql.qtpl:30
-	writevalueWithHitsJSON(qb422016, v)
-//line app/vlselect/logsql/logsql.qtpl:30
-	qs422016 := string(qb422016.B)
-//line app/vlselect/logsql/logsql.qtpl:30
-	qt422016.ReleaseByteBuffer(qb422016)
-//line app/vlselect/logsql/logsql.qtpl:30
-	return qs422016
-//line app/vlselect/logsql/logsql.qtpl:30
-}

app/vlselect/logsql/query_response.qtpl

@@ -1,41 +0,0 @@
-{% import (
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logstorage"
-) %}
-
-{% stripspace %}
-
-// JSONRow creates JSON row from the given fields.
-{% func JSONRow(columns []logstorage.BlockColumn, rowIdx int) %}
-{
-	{% code c := &columns[0] %}
-	{%q= c.Name %}:{%q= c.Values[rowIdx] %}
-	{% code columns = columns[1:] %}
-	{% for colIdx := range columns %}
-		{% code c := &columns[colIdx] %}
-		,{%q= c.Name %}:{%q= c.Values[rowIdx] %}
-	{% endfor %}
-}{% newline %}
-{% endfunc %}
-
-// JSONRows prints formatted rows
-{% func JSONRows(rows [][]logstorage.Field) %}
-	{% if len(rows) == 0 %}
-		{% return %}
-	{% endif %}
-	{% for _, fields := range rows %}
-	{
-		{% if len(fields) > 0 %}
-			{% code
-				f := fields[0]
-				fields = fields[1:]
-			%}
-			{%q= f.Name %}:{%q= f.Value %}
-			{% for _, f := range fields %}
-				,{%q= f.Name %}:{%q= f.Value %}
-			{% endfor %}
-		{% endif %}
-	}{% newline %}
-	{% endfor %}
-{% endfunc %}
-
-{% endstripspace %}

app/vlselect/logsql/query_response.qtpl.go

@@ -1,166 +0,0 @@
-// Code generated by qtc from "query_response.qtpl". DO NOT EDIT.
-// See https://github.com/valyala/quicktemplate for details.
-
-//line app/vlselect/logsql/query_response.qtpl:1
-package logsql
-
-//line app/vlselect/logsql/query_response.qtpl:1
-import (
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logstorage"
-)
-
-// JSONRow creates JSON row from the given fields.
-
-//line app/vlselect/logsql/query_response.qtpl:8
-import (
-	qtio422016 "io"
-
-	qt422016 "github.com/valyala/quicktemplate"
-)
-
-//line app/vlselect/logsql/query_response.qtpl:8
-var (
-	_ = qtio422016.Copy
-	_ = qt422016.AcquireByteBuffer
-)
-
-//line app/vlselect/logsql/query_response.qtpl:8
-func StreamJSONRow(qw422016 *qt422016.Writer, columns []logstorage.BlockColumn, rowIdx int) {
-//line app/vlselect/logsql/query_response.qtpl:8
-	qw422016.N().S(`{`)
-//line app/vlselect/logsql/query_response.qtpl:10
-	c := &columns[0]
-
-//line app/vlselect/logsql/query_response.qtpl:11
-	qw422016.N().Q(c.Name)
-//line app/vlselect/logsql/query_response.qtpl:11
-	qw422016.N().S(`:`)
-//line app/vlselect/logsql/query_response.qtpl:11
-	qw422016.N().Q(c.Values[rowIdx])
-//line app/vlselect/logsql/query_response.qtpl:12
-	columns = columns[1:]
-
-//line app/vlselect/logsql/query_response.qtpl:13
-	for colIdx := range columns {
-//line app/vlselect/logsql/query_response.qtpl:14
-		c := &columns[colIdx]
-
-//line app/vlselect/logsql/query_response.qtpl:14
-		qw422016.N().S(`,`)
-//line app/vlselect/logsql/query_response.qtpl:15
-		qw422016.N().Q(c.Name)
-//line app/vlselect/logsql/query_response.qtpl:15
-		qw422016.N().S(`:`)
-//line app/vlselect/logsql/query_response.qtpl:15
-		qw422016.N().Q(c.Values[rowIdx])
-//line app/vlselect/logsql/query_response.qtpl:16
-	}
-//line app/vlselect/logsql/query_response.qtpl:16
-	qw422016.N().S(`}`)
-//line app/vlselect/logsql/query_response.qtpl:17
-	qw422016.N().S(`
-`)
-//line app/vlselect/logsql/query_response.qtpl:18
-}
-
-//line app/vlselect/logsql/query_response.qtpl:18
-func WriteJSONRow(qq422016 qtio422016.Writer, columns []logstorage.BlockColumn, rowIdx int) {
-//line app/vlselect/logsql/query_response.qtpl:18
-	qw422016 := qt422016.AcquireWriter(qq422016)
-//line app/vlselect/logsql/query_response.qtpl:18
-	StreamJSONRow(qw422016, columns, rowIdx)
-//line app/vlselect/logsql/query_response.qtpl:18
-	qt422016.ReleaseWriter(qw422016)
-//line app/vlselect/logsql/query_response.qtpl:18
-}
-
-//line app/vlselect/logsql/query_response.qtpl:18
-func JSONRow(columns []logstorage.BlockColumn, rowIdx int) string {
-//line app/vlselect/logsql/query_response.qtpl:18
-	qb422016 := qt422016.AcquireByteBuffer()
-//line app/vlselect/logsql/query_response.qtpl:18
-	WriteJSONRow(qb422016, columns, rowIdx)
-//line app/vlselect/logsql/query_response.qtpl:18
-	qs422016 := string(qb422016.B)
-//line app/vlselect/logsql/query_response.qtpl:18
-	qt422016.ReleaseByteBuffer(qb422016)
-//line app/vlselect/logsql/query_response.qtpl:18
-	return qs422016
-//line app/vlselect/logsql/query_response.qtpl:18
-}
-
-// JSONRows prints formatted rows
-
-//line app/vlselect/logsql/query_response.qtpl:21
-func StreamJSONRows(qw422016 *qt422016.Writer, rows [][]logstorage.Field) {
-//line app/vlselect/logsql/query_response.qtpl:22
-	if len(rows) == 0 {
-//line app/vlselect/logsql/query_response.qtpl:23
-		return
-//line app/vlselect/logsql/query_response.qtpl:24
-	}
-//line app/vlselect/logsql/query_response.qtpl:25
-	for _, fields := range rows {
-//line app/vlselect/logsql/query_response.qtpl:25
-		qw422016.N().S(`{`)
-//line app/vlselect/logsql/query_response.qtpl:27
-		if len(fields) > 0 {
-//line app/vlselect/logsql/query_response.qtpl:29
-			f := fields[0]
-			fields = fields[1:]
-
-//line app/vlselect/logsql/query_response.qtpl:32
-			qw422016.N().Q(f.Name)
-//line app/vlselect/logsql/query_response.qtpl:32
-			qw422016.N().S(`:`)
-//line app/vlselect/logsql/query_response.qtpl:32
-			qw422016.N().Q(f.Value)
-//line app/vlselect/logsql/query_response.qtpl:33
-			for _, f := range fields {
-//line app/vlselect/logsql/query_response.qtpl:33
-				qw422016.N().S(`,`)
-//line app/vlselect/logsql/query_response.qtpl:34
-				qw422016.N().Q(f.Name)
-//line app/vlselect/logsql/query_response.qtpl:34
-				qw422016.N().S(`:`)
-//line app/vlselect/logsql/query_response.qtpl:34
-				qw422016.N().Q(f.Value)
-//line app/vlselect/logsql/query_response.qtpl:35
-			}
-//line app/vlselect/logsql/query_response.qtpl:36
-		}
-//line app/vlselect/logsql/query_response.qtpl:36
-		qw422016.N().S(`}`)
-//line app/vlselect/logsql/query_response.qtpl:37
-		qw422016.N().S(`
-`)
-//line app/vlselect/logsql/query_response.qtpl:38
-	}
-//line app/vlselect/logsql/query_response.qtpl:39
-}
-
-//line app/vlselect/logsql/query_response.qtpl:39
-func WriteJSONRows(qq422016 qtio422016.Writer, rows [][]logstorage.Field) {
-//line app/vlselect/logsql/query_response.qtpl:39
-	qw422016 := qt422016.AcquireWriter(qq422016)
-//line app/vlselect/logsql/query_response.qtpl:39
-	StreamJSONRows(qw422016, rows)
-//line app/vlselect/logsql/query_response.qtpl:39
-	qt422016.ReleaseWriter(qw422016)
-//line app/vlselect/logsql/query_response.qtpl:39
-}
-
-//line app/vlselect/logsql/query_response.qtpl:39
-func JSONRows(rows [][]logstorage.Field) string {
-//line app/vlselect/logsql/query_response.qtpl:39
-	qb422016 := qt422016.AcquireByteBuffer()
-//line app/vlselect/logsql/query_response.qtpl:39
-	WriteJSONRows(qb422016, rows)
-//line app/vlselect/logsql/query_response.qtpl:39
-	qs422016 := string(qb422016.B)
-//line app/vlselect/logsql/query_response.qtpl:39
-	qt422016.ReleaseByteBuffer(qb422016)
-//line app/vlselect/logsql/query_response.qtpl:39
-	return qs422016
-//line app/vlselect/logsql/query_response.qtpl:39
-}

app/vlselect/main.go

@@ -1,240 +0,0 @@
-package vlselect
-
-import (
-	"context"
-	"embed"
-	"flag"
-	"fmt"
-	"net/http"
-	"strings"
-	"time"
-
-	"github.com/VictoriaMetrics/VictoriaMetrics/app/vlselect/logsql"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/cgroup"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/httputils"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
-	"github.com/VictoriaMetrics/metrics"
-)
-
-var (
-	maxConcurrentRequests = flag.Int("search.maxConcurrentRequests", getDefaultMaxConcurrentRequests(), "The maximum number of concurrent search requests. "+
-		"It shouldn't be high, since a single request can saturate all the CPU cores, while many concurrently executed requests may require high amounts of memory. "+
-		"See also -search.maxQueueDuration")
-	maxQueueDuration = flag.Duration("search.maxQueueDuration", 10*time.Second, "The maximum time the search request waits for execution when -search.maxConcurrentRequests "+
-		"limit is reached; see also -search.maxQueryDuration")
-	maxQueryDuration = flag.Duration("search.maxQueryDuration", time.Second*30, "The maximum duration for query execution. It can be overridden on a per-query basis via 'timeout' query arg")
-)
-
-func getDefaultMaxConcurrentRequests() int {
-	n := cgroup.AvailableCPUs()
-	if n <= 4 {
-		n *= 2
-	}
-	if n > 16 {
-		// A single request can saturate all the CPU cores, so there is no sense
-		// in allowing higher number of concurrent requests - they will just contend
-		// for unavailable CPU time.
-		n = 16
-	}
-	return n
-}
-
-// Init initializes vlselect
-func Init() {
-	concurrencyLimitCh = make(chan struct{}, *maxConcurrentRequests)
-}
-
-// Stop stops vlselect
-func Stop() {
-}
-
-var concurrencyLimitCh chan struct{}
-
-var (
-	concurrencyLimitReached = metrics.NewCounter(`vl_concurrent_select_limit_reached_total`)
-	concurrencyLimitTimeout = metrics.NewCounter(`vl_concurrent_select_limit_timeout_total`)
-
-	_ = metrics.NewGauge(`vl_concurrent_select_capacity`, func() float64 {
-		return float64(cap(concurrencyLimitCh))
-	})
-	_ = metrics.NewGauge(`vl_concurrent_select_current`, func() float64 {
-		return float64(len(concurrencyLimitCh))
-	})
-)
-
-//go:embed vmui
-var vmuiFiles embed.FS
-
-var vmuiFileServer = http.FileServer(http.FS(vmuiFiles))
-
-// RequestHandler handles select requests for VictoriaLogs
-func RequestHandler(w http.ResponseWriter, r *http.Request) bool {
-	path := r.URL.Path
-	if !strings.HasPrefix(path, "/select/") {
-		// Skip requests, which do not start with /select/, since these aren't our requests.
-		return false
-	}
-	path = strings.ReplaceAll(path, "//", "/")
-
-	if path == "/select/vmui" {
-		// VMUI access via incomplete url without `/` in the end. Redirect to complete url.
-		// Use relative redirect, since the hostname and path prefix may be incorrect if VictoriaMetrics
-		// is hidden behind vmauth or similar proxy.
-		_ = r.ParseForm()
-		newURL := "vmui/?" + r.Form.Encode()
-		httpserver.Redirect(w, newURL)
-		return true
-	}
-	if strings.HasPrefix(path, "/select/vmui/") {
-		if strings.HasPrefix(path, "/select/vmui/static/") {
-			// Allow clients caching static contents for long period of time, since it shouldn't change over time.
-			// Path to static contents (such as js and css) must be changed whenever its contents is changed.
-			// See https://developer.chrome.com/docs/lighthouse/performance/uses-long-cache-ttl/
-			w.Header().Set("Cache-Control", "max-age=31536000")
-		}
-		r.URL.Path = strings.TrimPrefix(path, "/select")
-		vmuiFileServer.ServeHTTP(w, r)
-		return true
-	}
-
-	// Limit the number of concurrent queries, which can consume big amounts of CPU time.
-	startTime := time.Now()
-	ctx := r.Context()
-	d := getMaxQueryDuration(r)
-	ctxWithTimeout, cancel := context.WithTimeout(ctx, d)
-	defer cancel()
-
-	stopCh := ctxWithTimeout.Done()
-	select {
-	case concurrencyLimitCh <- struct{}{}:
-		defer func() { <-concurrencyLimitCh }()
-	default:
-		// Sleep for a while until giving up. This should resolve short bursts in requests.
-		concurrencyLimitReached.Inc()
-		select {
-		case concurrencyLimitCh <- struct{}{}:
-			defer func() { <-concurrencyLimitCh }()
-		case <-stopCh:
-			switch ctxWithTimeout.Err() {
-			case context.Canceled:
-				remoteAddr := httpserver.GetQuotedRemoteAddr(r)
-				requestURI := httpserver.GetRequestURI(r)
-				logger.Infof("client has canceled the pending request after %.3f seconds: remoteAddr=%s, requestURI: %q",
-					time.Since(startTime).Seconds(), remoteAddr, requestURI)
-			case context.DeadlineExceeded:
-				concurrencyLimitTimeout.Inc()
-				err := &httpserver.ErrorWithStatusCode{
-					Err: fmt.Errorf("couldn't start executing the request in %.3f seconds, since -search.maxConcurrentRequests=%d concurrent requests "+
-						"are executed. Possible solutions: to reduce query load; to add more compute resources to the server; "+
-						"to increase -search.maxQueueDuration=%s; to increase -search.maxQueryDuration=%s; to increase -search.maxConcurrentRequests; "+
-						"to pass bigger value to 'timeout' query arg",
-						d.Seconds(), *maxConcurrentRequests, maxQueueDuration, maxQueryDuration),
-					StatusCode: http.StatusServiceUnavailable,
-				}
-				httpserver.Errorf(w, r, "%s", err)
-			}
-			return true
-		}
-	}
-
-	if path == "/select/logsql/tail" {
-		logsqlTailRequests.Inc()
-		// Process live tailing request without timeout (e.g. use ctx instead of ctxWithTimeout),
-		// since it is OK to run live tailing requests for very long time.
-		logsql.ProcessLiveTailRequest(ctx, w, r)
-		return true
-	}
-
-	ok := processSelectRequest(ctxWithTimeout, w, r, path)
-	if !ok {
-		return false
-	}
-
-	err := ctxWithTimeout.Err()
-	switch err {
-	case nil:
-		// nothing to do
-	case context.Canceled:
-		remoteAddr := httpserver.GetQuotedRemoteAddr(r)
-		requestURI := httpserver.GetRequestURI(r)
-		logger.Infof("client has canceled the request after %.3f seconds: remoteAddr=%s, requestURI: %q",
-			time.Since(startTime).Seconds(), remoteAddr, requestURI)
-	case context.DeadlineExceeded:
-		err = &httpserver.ErrorWithStatusCode{
-			Err: fmt.Errorf("the request couldn't be executed in %.3f seconds; possible solutions: "+
-				"to increase -search.maxQueryDuration=%s; to pass bigger value to 'timeout' query arg", d.Seconds(), maxQueryDuration),
-			StatusCode: http.StatusServiceUnavailable,
-		}
-		httpserver.Errorf(w, r, "%s", err)
-	default:
-		httpserver.Errorf(w, r, "unexpected error: %s", err)
-	}
-
-	return true
-}
-
-func processSelectRequest(ctx context.Context, w http.ResponseWriter, r *http.Request, path string) bool {
-	httpserver.EnableCORS(w, r)
-	switch path {
-	case "/select/logsql/field_names":
-		logsqlFieldNamesRequests.Inc()
-		logsql.ProcessFieldNamesRequest(ctx, w, r)
-		return true
-	case "/select/logsql/field_values":
-		logsqlFieldValuesRequests.Inc()
-		logsql.ProcessFieldValuesRequest(ctx, w, r)
-		return true
-	case "/select/logsql/hits":
-		logsqlHitsRequests.Inc()
-		logsql.ProcessHitsRequest(ctx, w, r)
-		return true
-	case "/select/logsql/query":
-		logsqlQueryRequests.Inc()
-		logsql.ProcessQueryRequest(ctx, w, r)
-		return true
-	case "/select/logsql/stream_field_names":
-		logsqlStreamFieldNamesRequests.Inc()
-		logsql.ProcessStreamFieldNamesRequest(ctx, w, r)
-		return true
-	case "/select/logsql/stream_field_values":
-		logsqlStreamFieldValuesRequests.Inc()
-		logsql.ProcessStreamFieldValuesRequest(ctx, w, r)
-		return true
-	case "/select/logsql/stream_ids":
-		logsqlStreamIDsRequests.Inc()
-		logsql.ProcessStreamIDsRequest(ctx, w, r)
-		return true
-	case "/select/logsql/streams":
-		logsqlStreamsRequests.Inc()
-		logsql.ProcessStreamsRequest(ctx, w, r)
-		return true
-	default:
-		return false
-	}
-}
-
-// getMaxQueryDuration returns the maximum duration for query from r.
-func getMaxQueryDuration(r *http.Request) time.Duration {
-	dms, err := httputils.GetDuration(r, "timeout", 0)
-	if err != nil {
-		dms = 0
-	}
-	d := time.Duration(dms) * time.Millisecond
-	if d <= 0 || d > *maxQueryDuration {
-		d = *maxQueryDuration
-	}
-	return d
-}
-
-var (
-	logsqlFieldNamesRequests        = metrics.NewCounter(`vl_http_requests_total{path="/select/logsql/field_names"}`)
-	logsqlFieldValuesRequests       = metrics.NewCounter(`vl_http_requests_total{path="/select/logsql/field_values"}`)
-	logsqlHitsRequests              = metrics.NewCounter(`vl_http_requests_total{path="/select/logsql/hits"}`)
-	logsqlQueryRequests             = metrics.NewCounter(`vl_http_requests_total{path="/select/logsql/query"}`)
-	logsqlStreamFieldNamesRequests  = metrics.NewCounter(`vl_http_requests_total{path="/select/logsql/stream_field_names"}`)
-	logsqlStreamFieldValuesRequests = metrics.NewCounter(`vl_http_requests_total{path="/select/logsql/stream_field_values"}`)
-	logsqlStreamIDsRequests         = metrics.NewCounter(`vl_http_requests_total{path="/select/logsql/stream_ids"}`)
-	logsqlStreamsRequests           = metrics.NewCounter(`vl_http_requests_total{path="/select/logsql/streams"}`)
-	logsqlTailRequests              = metrics.NewCounter(`vl_http_requests_total{path="/select/logsql/tail"}`)
-)

app/vlselect/vmui/asset-manifest.json

@@ -1,13 +0,0 @@
-{
-  "files": {
-    "main.css": "./static/css/main.c9cc37dd.css",
-    "main.js": "./static/js/main.7ae2e2c4.js",
-    "static/js/685.bebe1265.chunk.js": "./static/js/685.bebe1265.chunk.js",
-    "static/media/MetricsQL.md": "./static/media/MetricsQL.8c2e588d62b87f90dbf0.md",
-    "index.html": "./index.html"
-  },
-  "entrypoints": [
-    "static/css/main.c9cc37dd.css",
-    "static/js/main.7ae2e2c4.js"
-  ]
-}

app/vlselect/vmui/index.html

@@ -1 +0,0 @@
-<!doctype html><html lang="en"><head><meta charset="utf-8"/><link rel="icon" href="./favicon.ico"/><meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=5"/><meta name="theme-color" content="#000000"/><meta name="description" content="UI for VictoriaMetrics"/><link rel="apple-touch-icon" href="./apple-touch-icon.png"/><link rel="icon" type="image/png" sizes="32x32" href="./favicon-32x32.png"><link rel="manifest" href="./manifest.json"/><title>VM UI</title><meta name="twitter:card" content="summary_large_image"><meta name="twitter:image" content="./preview.jpg"><meta name="twitter:title" content="UI for VictoriaMetrics"><meta name="twitter:description" content="Explore and troubleshoot your VictoriaMetrics data"><meta name="twitter:site" content="@VictoriaMetrics"><meta property="og:title" content="Metric explorer for VictoriaMetrics"><meta property="og:description" content="Explore and troubleshoot your VictoriaMetrics data"><meta property="og:image" content="./preview.jpg"><meta property="og:type" content="website"><script defer="defer" src="./static/js/main.7ae2e2c4.js"></script><link href="./static/css/main.c9cc37dd.css" rel="stylesheet"></head><body><noscript>You need to enable JavaScript to run this app.</noscript><div id="root"></div></body></html>

app/vlselect/vmui/manifest.json

@@ -1,20 +0,0 @@
-{
-  "short_name": "Victoria Metrics UI",
-  "name": "Victoria Metrics UI is a metric explorer for Victoria Metrics",
-  "icons": [
-    {
-      "src": "favicon-32x32.png",
-      "sizes": "32x32",
-      "type": "image/png"
-    },
-    {
-      "src": "apple-touch-icon.png",
-      "type": "image/png",
-      "sizes": "192x192"
-    }
-  ],
-  "start_url": ".",
-  "display": "standalone",
-  "theme_color": "#000000",
-  "background_color": "#ffffff"
-}

app/vlselect/vmui/robots.txt

@@ -1,3 +0,0 @@
-# https://www.robotstxt.org/robotstxt.html
-User-agent: *
-Disallow:

app/vlselect/vmui/static/js/main.7ae2e2c4.js.LICENSE.txt

@@ -1,38 +0,0 @@
-/*!
-	Copyright (c) 2018 Jed Watson.
-	Licensed under the MIT License (MIT), see
-	http://jedwatson.github.io/classnames
-*/
-
-/**
- * @remix-run/router v1.17.0
- *
- * Copyright (c) Remix Software Inc.
- *
- * This source code is licensed under the MIT license found in the
- * LICENSE.md file in the root directory of this source tree.
- *
- * @license MIT
- */
-
-/**
- * React Router DOM v6.24.0
- *
- * Copyright (c) Remix Software Inc.
- *
- * This source code is licensed under the MIT license found in the
- * LICENSE.md file in the root directory of this source tree.
- *
- * @license MIT
- */
-
-/**
- * React Router v6.24.0
- *
- * Copyright (c) Remix Software Inc.
- *
- * This source code is licensed under the MIT license found in the
- * LICENSE.md file in the root directory of this source tree.
- *
- * @license MIT
- */

app/vlselect/vmui/victorialogs.html

@@ -1,55 +0,0 @@
-<!DOCTYPE html>
-<html lang="en">
-<head>
-  <meta charset="utf-8"/>
-  <link rel="icon" href="%PUBLIC_URL%/favicon.ico"/>
-  <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=5"/>
-  <meta name="theme-color" content="#000000"/>
-  <meta
-    name="description"
-    content="UI for VictoriaMetrics"
-  />
-  <link rel="apple-touch-icon" href="%PUBLIC_URL%/apple-touch-icon.png"/>
-  <link rel="icon" type="image/png" sizes="32x32" href="%PUBLIC_URL%/favicon-32x32.png">
-  <!--
-    manifest.json provides metadata used when your web app is installed on a
-    user's mobile device or desktop. See https://developers.google.com/web/fundamentals/web-app-manifest/
-  -->
-  <link rel="manifest" href="%PUBLIC_URL%/manifest.json"/>
-  <!--
-    Notice the use of %PUBLIC_URL% in the tags above.
-    It will be replaced with the URL of the `public` folder during the build.
-    Only files inside the `public` folder can be referenced from the HTML.
-
-    Unlike "/favicon.ico" or "favicon.ico", "%PUBLIC_URL%/favicon.ico" will
-    work correctly both with client-side routing and a non-root public URL.
-    Learn how to configure a non-root public URL by running `npm run build`.
-  -->
-  <title>VM UI</title>
-
-  <meta name="twitter:card" content="summary_large_image">
-  <meta name="twitter:image" content="%PUBLIC_URL%/preview.jpg">
-  <meta name="twitter:title" content="UI for VictoriaMetrics">
-  <meta name="twitter:description" content="Explore and troubleshoot your VictoriaMetrics data">
-  <meta name="twitter:site" content="@VictoriaMetrics">
-
-  <meta property="og:title" content="Metric explorer for VictoriaMetrics">
-  <meta property="og:description" content="Explore and troubleshoot your VictoriaMetrics data">
-  <meta property="og:image" content="%PUBLIC_URL%/preview.jpg">
-  <meta property="og:type" content="website">
-</head>
-<body>
-<noscript>You need to enable JavaScript to run this app.</noscript>
-<div id="root"></div>
-<!--
-  This HTML file is a template.
-  If you open it directly in the browser, you will see an empty page.
-
-  You can add webfonts, meta tags, or analytics to this file.
-  The build step will place the bundled scripts into the <body> tag.
-
-  To begin the development, run `npm start` or `yarn start`.
-  To create a production bundle, use `npm run build` or `yarn build`.
--->
-</body>
-</html>