Compare commits

...

19 Commits

Author SHA1 Message Date
Aliaksandr Valialkin
d0ce874b13 docs/CHANGELOG.md: document v1.79.9 release 2023-02-24 15:11:27 -08:00
Aliaksandr Valialkin
619da7bfa5 docs/CHANGELOG.md: cut v1.87.2 release 2023-02-24 15:09:30 -08:00
Aliaksandr Valialkin
dd6e7089a8 docs/CHANGELOG.md: document d8eaa511b0 2023-02-24 12:44:24 -08:00
Zakhar Bessarab
1dbf7a204c lib/{fs,mergeset,storage}: skip .must-remove. dirs when creating snapshot (#3858) (#3867) 2023-02-24 12:44:22 -08:00
Aliaksandr Valialkin
4c82081f57 docs/CHANGELOG.md: typo fix: scrape scrape -> scrape 2023-02-24 12:34:01 -08:00
Aliaksandr Valialkin
69e7621ad5 lib/promscrape: follow-up for 43e104a83f
- Return immediately on context cancel during the backoff sleep.
  This should help with https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3747

- Add a comment describing why the second attempt to obtain the response from remote side
  is perfromed immediately after the first attempt.

- Remove fasthttp dependency from lib/promscrape/discoveryutils

- Set context deadline before calling doRequestWithPossibleRetry().
  This simplifies the doRequestWithPossibleRetry() a bit.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3293
2023-02-24 12:26:13 -08:00
Zakhar Bessarab
00bc28626d fix: do not use exponential backoff for first retry of scrape request (#3824)
* fix: do not use exponential backoff for first retry of scrape request (#3293)

* lib/promscrape: refactor `doRequestWithPossibleRetry` backoff to simplify logic

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* Update lib/promscrape/client.go

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

* lib/promscrape: refactor `doRequestWithPossibleRetry` to make it more straightforward

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2023-02-24 12:26:06 -08:00
Alexander Marshalov
b443b7e2ca fix interpolate function for filling only intermediate gaps (#3816) (#3857)
* fix interpolate function for filling only intermediate gaps (#3816)

* Update docs/CHANGELOG.md

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-02-23 19:29:54 -08:00
Aliaksandr Valialkin
e0a16874f6 docs/CHANGELOG.md: document 6d019a3c37
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3830
2023-02-23 19:28:56 -08:00
Mattias Ängehov
62f40cb33b Azure Service Discovery - Fix token fetch for Container Apps/App Services (#3832)
* Modify API version when running in Container App

* Handle expires on from token response

Response from IMDS does not always contain expires in value which is
currently used to get the token expiry time. An example resources that
doesn't provide it are Container Apps and App Service.

Signed-off-by: Mattias Ängehov <mattias.angehov@castoredc.com>

* Fix client id parameter for user assigned identity

* Apply suggestions from code review

---------

Signed-off-by: Mattias Ängehov <mattias.angehov@castoredc.com>
Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
2023-02-23 19:28:49 -08:00
Aliaksandr Valialkin
905f1839ef docs/CHANGELOG.md: document d2b92d3264
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3747
2023-02-22 18:29:20 -08:00
Zakhar Bessarab
05ca0c16c7 lib/promscrape: fix cancelling in-flight scrape requests during configuration reload (#3853)
* lib/promscrape: fix cancelling in-flight scrape requests during configuration reload (see #3747)

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* lib/promscrape: fix order of params for `doRequestWithPossibleRetry` to follow codestyle

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* lib/promscrape: accept deadline explicitly and extend passed context for local use

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-02-22 18:28:19 -08:00
Aliaksandr Valialkin
0476f2a7ca vendor: update github.com/VictoriaMetrics/fasthttp from v1.1.0 to v1.2.0
The v1.2.0 adds HostClient.DoCtx() function, which is needed by https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3747
for implementing fast canceling of pending requests to scrape targets on config update
2023-02-22 18:27:37 -08:00
Aliaksandr Valialkin
d26b9d89e0 vendor: make vendor-update 2023-02-18 23:16:22 -08:00
Aliaksandr Valialkin
29f0a33500 all: update Go builder from Go1.20.0 to Go1.20.1
See https://github.com/golang/go/issues?q=milestone%3AGo1.20.1+label%3ACherryPickApproved
2023-02-14 23:13:01 -08:00
Aliaksandr Valialkin
af68892e68 docs/CHANGELOG.md: improve the docs for 8ea02eaa8e 2023-02-14 22:45:31 -08:00
Droxenator
0b91514a8f fixed opentsdbListenAddr timestamp conversion (#3810)
Co-authored-by: Andrei Ivanov <a.ivanov@corp.mail.ru>
2023-02-14 22:44:37 -08:00
Aliaksandr Valialkin
a96f0df64a lib/{mergeset,storage}: allow at least 3 concurrent flushes during background merges on systems with 1 or 2 CPU cores
This should prevent from data ingestion slowdown and query performance degradation
on systems with small number of CPU cores (1 or 2), when big merge is performed.

This should help https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3790

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3337
2023-02-11 12:17:06 -08:00
Aliaksandr Valialkin
af83ce33f0 all: update alpine base docker image from 1.17.1 to 1.17.2
See https://alpinelinux.org/posts/Alpine-3.17.2-released.html
2023-02-11 12:16:04 -08:00
90 changed files with 1572 additions and 642 deletions

View File

@@ -17,7 +17,7 @@ jobs:
- name: Setup Go
uses: actions/setup-go@main
with:
go-version: 1.20.0
go-version: 1.21.0
id: go
- name: Code checkout
uses: actions/checkout@master

View File

@@ -57,7 +57,7 @@ jobs:
- name: Set up Go
uses: actions/setup-go@v3
with:
go-version: 1.20.0
go-version: 1.20.1
check-latest: true
cache: true
if: ${{ matrix.language == 'go' }}

View File

@@ -32,7 +32,7 @@ jobs:
- name: Setup Go
uses: actions/setup-go@v3
with:
go-version: 1.20.0
go-version: 1.20.1
check-latest: true
cache: true
@@ -56,7 +56,7 @@ jobs:
- name: Setup Go
uses: actions/setup-go@v3
with:
go-version: 1.20.0
go-version: 1.20.1
check-latest: true
cache: true
@@ -81,7 +81,7 @@ jobs:
id: go
uses: actions/setup-go@v3
with:
go-version: 1.20.0
go-version: 1.20.1
check-latest: true
cache: true

View File

@@ -20,7 +20,7 @@ jobs:
- name: Setup Go
uses: actions/setup-go@main
with:
go-version: 1.20.0
go-version: 1.20.1
id: go
- name: Setup docker scan

View File

@@ -6122,7 +6122,7 @@ func TestExecSuccess(t *testing.T) {
q := `interpolate(time() < 1300)`
r1 := netstorage.Result{
MetricName: metricNameExpected,
Values: []float64{1000, 1200, 1200, 1200, 1200, 1200},
Values: []float64{1000, 1200, nan, nan, nan, nan},
Timestamps: timestampsExpected,
}
resultExpected := []netstorage.Result{r1}
@@ -6133,7 +6133,18 @@ func TestExecSuccess(t *testing.T) {
q := `interpolate(time() > 1500)`
r1 := netstorage.Result{
MetricName: metricNameExpected,
Values: []float64{1600, 1600, 1600, 1600, 1800, 2000},
Values: []float64{nan, nan, nan, 1600, 1800, 2000},
Timestamps: timestampsExpected,
}
resultExpected := []netstorage.Result{r1}
f(q, resultExpected)
})
t.Run(`interpolate(tail_head_and_middle)`, func(t *testing.T) {
t.Parallel()
q := `interpolate(time() > 1100 and time() < 1300 default time() > 1700 and time() < 1900)`
r1 := netstorage.Result{
MetricName: metricNameExpected,
Values: []float64{nan, 1200, 1400, 1600, 1800, nan},
Timestamps: timestampsExpected,
}
resultExpected := []netstorage.Result{r1}

View File

@@ -1165,7 +1165,8 @@ func transformInterpolate(tfa *transformFuncArg) ([]*timeseries, error) {
}
rvs := args[0]
for _, ts := range rvs {
values := ts.Values
values := skipLeadingNaNs(ts.Values)
values = skipTrailingNaNs(values)
if len(values) == 0 {
continue
}

View File

@@ -1,4 +1,4 @@
FROM node:18-alpine3.15
FROM node:18-alpine3.17
RUN apk update && apk upgrade
RUN apk add --no-cache bash bash-doc bash-completion libtool autoconf automake nasm pkgconfig libpng gcc make g++ zlib-dev gawk

View File

@@ -1,4 +1,4 @@
FROM golang:1.19.5 as build-web-stage
FROM golang:1.20.1 as build-web-stage
COPY build /build
WORKDIR /build
@@ -6,7 +6,7 @@ COPY web/ /build/
RUN GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -o web-amd64 github.com/VictoriMetrics/vmui/ && \
GOOS=windows GOARCH=amd64 CGO_ENABLED=0 go build -o web-windows github.com/VictoriMetrics/vmui/
FROM alpine:3.17.1
FROM alpine:3.17.2
USER root
COPY --from=build-web-stage /build/web-amd64 /app/web

View File

@@ -2,9 +2,9 @@
DOCKER_NAMESPACE := victoriametrics
ROOT_IMAGE ?= alpine:3.17.1
CERTS_IMAGE := alpine:3.17.1
GO_BUILDER_IMAGE := golang:1.20.0-alpine
ROOT_IMAGE ?= alpine:3.17.2
CERTS_IMAGE := alpine:3.17.2
GO_BUILDER_IMAGE := golang:1.20.1-alpine
BUILDER_IMAGE := local/builder:2.0.0-$(shell echo $(GO_BUILDER_IMAGE) | tr :/ __)-1
BASE_IMAGE := local/base:1.1.4-$(shell echo $(ROOT_IMAGE) | tr :/ __)-$(shell echo $(CERTS_IMAGE) | tr :/ __)

View File

@@ -13,7 +13,22 @@ The following tip changes can be tested by building VictoriaMetrics components f
* [How to build vmauth](https://docs.victoriametrics.com/vmauth.html#how-to-build-from-sources)
* [How to build vmctl](https://docs.victoriametrics.com/vmctl.html#how-to-build)
## tip
## v1.87.x long-time support release (LTS)
## [v1.87.2](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.87.2)
Released at 2023-02-24
* SECURITY: upgrade base docker image (alpine) from 3.17.1 to 3.17.2. See [alpine 3.17.2 release notes](https://alpinelinux.org/posts/Alpine-3.17.2-released.html).
* SECURITY: upgrade Go builder from Go1.20.0 to Go1.20.1. See [the list of issues addressed in Go1.20.1](https://github.com/golang/go/issues?q=milestone%3AGo1.20.1+label%3ACherryPickApproved).
* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent.html): immediately cancel in-flight scrape requests during configuration reload when [stream parsing mode](https://docs.victoriametrics.com/vmagent.html#stream-parsing-mode) is disabled. Previously `vmagent` could wait for long time until all the in-flight requests are completed before reloading the configuration. This could significantly slow down configuration reload. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3747).
* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent.html): do not wait for 2 seconds after the first unsuccessful attempt to scrape the target before performing the next attempt. This should improve scrape speed when the target closes [http keep-alive connection](https://en.wikipedia.org/wiki/HTTP_persistent_connection) between scrapes. See [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3293) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3747) issues.
* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent.html): fix [Azure service discovery](https://docs.victoriametrics.com/sd_configs.html#azure_sd_configs) inside [Azure Container App](https://learn.microsoft.com/en-us/azure/container-apps/overview). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3830). Thanks to @MattiasAng for the fix!
* BUGFIX: do not put auxiliary directories scheduled for removal into snapshots. This should prevent from `cannot create hard links from ...must-remove...` errors when making snapshots / backups. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3858).
* BUGFIX: prevent from possible data ingestion slowdown and query performance slowdown during [background merges of big parts](https://docs.victoriametrics.com/#storage) on systems with small number of CPU cores (1 or 2 CPU cores). The issue has been introduced in [v1.85.0](https://docs.victoriametrics.com/CHANGELOG.html#v1850) when implementing [this feature](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3337). See also [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3790).
* BUGFIX: properly parse timestamps in milliseconds when [ingesting data via OpenTSDB telnet put protocol](https://docs.victoriametrics.com/#sending-data-via-telnet-put-protocol). Previously timestamps in milliseconds were mistakenly multiplied by 1000. Thanks to @Droxenator for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3810).
* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/MetricsQL.html): do not add extrapolated points outside the real points when using [interpolate()](https://docs.victoriametrics.com/MetricsQL.html#interpolate) function. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3816).
## [v1.87.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.87.1)
@@ -520,6 +535,18 @@ Released at 2022-08-08
* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): properly show date picker at `Table` tab. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2874).
* BUGFIX: properly generate http redirects if `-http.pathPrefix` command-line flag is set. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2918).
## [v1.79.9](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.79.9)
Released at 2023-02-24
**v1.79.x is a line of LTS releases (e.g. long-time support). It contains important up-to-date bugfixes.
The v1.79.x line will be supported for at least 12 months since [v1.79.0](https://docs.victoriametrics.com/CHANGELOG.html#v1790) release**
* SECURITY: upgrade base docker image (alpine) from 3.17.1 to 3.17.2. See [alpine 3.17.2 release notes](https://alpinelinux.org/posts/Alpine-3.17.2-released.html).
* SECURITY: upgrade Go builder from Go1.20.0 to Go1.20.1. See [the list of issues addressed in Go1.20.1](https://github.com/golang/go/issues?q=milestone%3AGo1.20.1+label%3ACherryPickApproved).
* BUGFIX: properly parse timestamps in milliseconds when [ingesting data via OpenTSDB telnet put protocol](https://docs.victoriametrics.com/#sending-data-via-telnet-put-protocol). Previously timestamps in milliseconds were mistakenly multiplied by 1000. Thanks to @Droxenator for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3810).
## [v1.79.8](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.79.8)
Released at 2023-02-03

40
go.mod
View File

@@ -10,13 +10,13 @@ require (
// Do not use the original github.com/valyala/fasthttp because of issues
// like https://github.com/valyala/fasthttp/commit/996610f021ff45fdc98c2ce7884d5fa4e7f9199b
github.com/VictoriaMetrics/fasthttp v1.1.0
github.com/VictoriaMetrics/fasthttp v1.2.0
github.com/VictoriaMetrics/metrics v1.23.1
github.com/VictoriaMetrics/metricsql v0.51.2
github.com/VictoriaMetrics/metricsql v0.56.1
github.com/aws/aws-sdk-go-v2 v1.17.4
github.com/aws/aws-sdk-go-v2/config v1.18.12
github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.11.51
github.com/aws/aws-sdk-go-v2/service/s3 v1.30.2
github.com/aws/aws-sdk-go-v2/config v1.18.13
github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.11.53
github.com/aws/aws-sdk-go-v2/service/s3 v1.30.3
github.com/cespare/xxhash/v2 v2.2.0
github.com/cheggaaa/pb/v3 v3.1.0
github.com/gogo/protobuf v1.3.2
@@ -25,42 +25,42 @@ require (
github.com/influxdata/influxdb v1.11.0
github.com/klauspost/compress v1.15.15
github.com/prometheus/prometheus v0.42.0
github.com/urfave/cli/v2 v2.24.3
github.com/urfave/cli/v2 v2.24.4
github.com/valyala/fastjson v1.6.4
github.com/valyala/fastrand v1.1.0
github.com/valyala/fasttemplate v1.2.2
github.com/valyala/gozstd v1.17.0
github.com/valyala/gozstd v1.18.0
github.com/valyala/histogram v1.2.0
github.com/valyala/quicktemplate v1.7.0
golang.org/x/net v0.6.0
golang.org/x/net v0.7.0
golang.org/x/oauth2 v0.5.0
golang.org/x/sys v0.5.0
google.golang.org/api v0.109.0
google.golang.org/api v0.110.0
gopkg.in/yaml.v2 v2.4.0
)
require (
cloud.google.com/go v0.109.0 // indirect
cloud.google.com/go v0.110.0 // indirect
cloud.google.com/go/compute v1.18.0 // indirect
cloud.google.com/go/compute/metadata v0.2.3 // indirect
cloud.google.com/go/iam v0.10.0 // indirect
cloud.google.com/go/iam v0.12.0 // indirect
github.com/Azure/azure-sdk-for-go/sdk/internal v1.1.2 // indirect
github.com/VividCortex/ewma v1.2.0 // indirect
github.com/alecthomas/units v0.0.0-20211218093645-b94a6e3cc137 // indirect
github.com/aws/aws-sdk-go v1.44.198 // indirect
github.com/aws/aws-sdk-go v1.44.204 // indirect
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.4.10 // indirect
github.com/aws/aws-sdk-go-v2/credentials v1.13.12 // indirect
github.com/aws/aws-sdk-go-v2/credentials v1.13.13 // indirect
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.12.22 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.28 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.22 // indirect
github.com/aws/aws-sdk-go-v2/internal/ini v1.3.29 // indirect
github.com/aws/aws-sdk-go-v2/internal/v4a v1.0.19 // indirect
github.com/aws/aws-sdk-go-v2/internal/v4a v1.0.20 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.9.11 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.1.23 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.9.22 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.13.22 // indirect
github.com/aws/aws-sdk-go-v2/service/sso v1.12.1 // indirect
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.14.1 // indirect
github.com/aws/aws-sdk-go-v2/service/sso v1.12.2 // indirect
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.14.2 // indirect
github.com/aws/aws-sdk-go-v2/service/sts v1.18.3 // indirect
github.com/aws/smithy-go v1.13.5 // indirect
github.com/beorn7/perks v1.0.1 // indirect
@@ -77,7 +77,7 @@ require (
github.com/golang/protobuf v1.5.2 // indirect
github.com/google/go-cmp v0.5.9 // indirect
github.com/google/uuid v1.3.0 // indirect
github.com/googleapis/enterprise-certificate-proxy v0.2.2 // indirect
github.com/googleapis/enterprise-certificate-proxy v0.2.3 // indirect
github.com/grafana/regexp v0.0.0-20221122212121-6b5c0a4cb7fd // indirect
github.com/jmespath/go-jmespath v0.4.0 // indirect
github.com/jpillora/backoff v1.0.0 // indirect
@@ -105,14 +105,14 @@ require (
go.opentelemetry.io/otel/metric v0.36.0 // indirect
go.opentelemetry.io/otel/trace v1.13.0 // indirect
go.uber.org/atomic v1.10.0 // indirect
go.uber.org/goleak v1.2.0 // indirect
golang.org/x/exp v0.0.0-20230206171751-46f607a40771 // indirect
go.uber.org/goleak v1.2.1 // indirect
golang.org/x/exp v0.0.0-20230213192124-5e25df0256eb // indirect
golang.org/x/sync v0.1.0 // indirect
golang.org/x/text v0.7.0 // indirect
golang.org/x/time v0.3.0 // indirect
golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2 // indirect
google.golang.org/appengine v1.6.7 // indirect
google.golang.org/genproto v0.0.0-20230209215440-0dfe4f8abfcc // indirect
google.golang.org/genproto v0.0.0-20230216225411-c8e22ba71e44 // indirect
google.golang.org/grpc v1.53.0 // indirect
google.golang.org/protobuf v1.28.1 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect

83
go.sum
View File

@@ -13,8 +13,8 @@ cloud.google.com/go v0.56.0/go.mod h1:jr7tqZxxKOVYizybht9+26Z/gUq7tiRzu+ACVAMbKV
cloud.google.com/go v0.57.0/go.mod h1:oXiQ6Rzq3RAkkY7N6t3TcE6jE+CIBBbA36lwQ1JyzZs=
cloud.google.com/go v0.62.0/go.mod h1:jmCYTdRCQuc1PHIIJ/maLInMho30T/Y0M4hTdTShOYc=
cloud.google.com/go v0.65.0/go.mod h1:O5N8zS7uWy9vkA9vayVHs65eM1ubvY4h553ofrNHObY=
cloud.google.com/go v0.109.0 h1:38CZoKGlCnPZjGdyj0ZfpoGae0/wgNfy5F0byyxg0Gk=
cloud.google.com/go v0.109.0/go.mod h1:2sYycXt75t/CSB5R9M2wPU1tJmire7AQZTPtITcGBVE=
cloud.google.com/go v0.110.0 h1:Zc8gqp3+a9/Eyph2KDmcGaPtbKRIoqq4YTlL4NMD0Ys=
cloud.google.com/go v0.110.0/go.mod h1:SJnCLqQ0FCFGSZMUNUf84MV3Aia54kn7pi8st7tMzaY=
cloud.google.com/go/bigquery v1.0.1/go.mod h1:i/xbL2UlR5RvWAURpBYZTtm/cXjCha9lbfbpx4poX+o=
cloud.google.com/go/bigquery v1.3.0/go.mod h1:PjpwJnslEMmckchkHFfq+HTD2DmtT67aNFKH1/VBDHE=
cloud.google.com/go/bigquery v1.4.0/go.mod h1:S8dzgnTigyfTmLBfrtrhyYhwRxG72rYxvftPBK2Dvzc=
@@ -27,8 +27,8 @@ cloud.google.com/go/compute/metadata v0.2.3 h1:mg4jlk7mCAj6xXp9UJ4fjI9VUI5rubuGB
cloud.google.com/go/compute/metadata v0.2.3/go.mod h1:VAV5nSsACxMJvgaAuX6Pk2AawlZn8kiOGuCv6gTkwuA=
cloud.google.com/go/datastore v1.0.0/go.mod h1:LXYbyblFSglQ5pkeyhO+Qmw7ukd3C+pD7TKLgZqpHYE=
cloud.google.com/go/datastore v1.1.0/go.mod h1:umbIZjpQpHh4hmRpGhH4tLFup+FVzqBi1b3c64qFpCk=
cloud.google.com/go/iam v0.10.0 h1:fpP/gByFs6US1ma53v7VxhvbJpO2Aapng6wabJ99MuI=
cloud.google.com/go/iam v0.10.0/go.mod h1:nXAECrMt2qHpF6RZUZseteD6QyanL68reN4OXPw0UWM=
cloud.google.com/go/iam v0.12.0 h1:DRtTY29b75ciH6Ov1PHb4/iat2CLCvrOm40Q0a6DFpE=
cloud.google.com/go/iam v0.12.0/go.mod h1:knyHGviacl11zrtZUoDuYpDgLjvr28sLQaG0YB2GYAY=
cloud.google.com/go/longrunning v0.3.0 h1:NjljC+FYPV3uh5/OwWT6pVU+doBqMg2x/rZlE+CamDs=
cloud.google.com/go/pubsub v1.0.1/go.mod h1:R0Gpsv3s54REJCy4fxDixWD93lHJMoZTyQ2kNxGRt3I=
cloud.google.com/go/pubsub v1.1.0/go.mod h1:EwwdRX2sKPjnvnqCa270oGRyludottCI76h+R3AArQw=
@@ -64,13 +64,13 @@ github.com/BurntSushi/xgb v0.0.0-20160522181843-27f122750802/go.mod h1:IVnqGOEym
github.com/Microsoft/go-winio v0.5.1 h1:aPJp2QD7OOrhO5tQXqQoGSJc+DjDtWTGLOmNyAm6FgY=
github.com/VictoriaMetrics/fastcache v1.12.0 h1:vnVi/y9yKDcD9akmc4NqAoqgQhJrOwUF+j9LTgn4QDE=
github.com/VictoriaMetrics/fastcache v1.12.0/go.mod h1:tjiYeEfYXCqacuvYw/7UoDIeJaNxq6132xHICNP77w8=
github.com/VictoriaMetrics/fasthttp v1.1.0 h1:3crd4YWHsMwu60GUXRH6OstowiFvqrwS4a/ueoLdLL0=
github.com/VictoriaMetrics/fasthttp v1.1.0/go.mod h1:/7DMcogqd+aaD3G3Hg5kFgoFwlR2uydjiWvoLp5ZTqQ=
github.com/VictoriaMetrics/fasthttp v1.2.0 h1:nd9Wng4DlNtaI27WlYh5mGXCJOmee/2c2blTJwfyU9I=
github.com/VictoriaMetrics/fasthttp v1.2.0/go.mod h1:zv5YSmasAoSyv8sBVexfArzFDIGGTN4TfCKAtAw7IfE=
github.com/VictoriaMetrics/metrics v1.18.1/go.mod h1:ArjwVz7WpgpegX/JpB0zpNF2h2232kErkEnzH1sxMmA=
github.com/VictoriaMetrics/metrics v1.23.1 h1:/j8DzeJBxSpL2qSIdqnRFLvQQhbJyJbbEi22yMm7oL0=
github.com/VictoriaMetrics/metrics v1.23.1/go.mod h1:rAr/llLpEnAdTehiNlUxKgnjcOuROSzpw0GvjpEbvFc=
github.com/VictoriaMetrics/metricsql v0.51.2 h1:GCbxti0I46x3Ld/WhcUyawvLr6J0x90IaMftkjosHJI=
github.com/VictoriaMetrics/metricsql v0.51.2/go.mod h1:6pP1ZeLVJHqJrHlF6Ij3gmpQIznSsgktEcZgsAWYel0=
github.com/VictoriaMetrics/metricsql v0.56.1 h1:j+W4fA/gtozsZb4PlKDU0Ma2VOgl88xla4FEf29w94g=
github.com/VictoriaMetrics/metricsql v0.56.1/go.mod h1:6pP1ZeLVJHqJrHlF6Ij3gmpQIznSsgktEcZgsAWYel0=
github.com/VividCortex/ewma v1.1.1/go.mod h1:2Tkkvm3sRDVXaiyucHiACn4cqf7DpdyLvmxzcbUokwA=
github.com/VividCortex/ewma v1.2.0 h1:f58SaIzcDXrSy3kWaHNvuJgJ3Nmz59Zji6XoJR/q1ow=
github.com/VividCortex/ewma v1.2.0/go.mod h1:nz4BbCtbLyFDeC9SUHbtcT5644juEuWfUAUnGx7j5l4=
@@ -87,28 +87,28 @@ github.com/andybalholm/brotli v1.0.2/go.mod h1:loMXtMfwqflxFJPmdbJO0a3KNoPuLBgiu
github.com/andybalholm/brotli v1.0.3/go.mod h1:fO7iG3H7G2nSZ7m0zPUDn85XEX2GTukHGRSepvi9Eig=
github.com/armon/go-metrics v0.3.10 h1:FR+drcQStOe+32sYyJYyZ7FIdgoGGBnwLl+flodp8Uo=
github.com/aws/aws-sdk-go v1.38.35/go.mod h1:hcU610XS61/+aQV88ixoOzUoG7v3b31pl2zKMmprdro=
github.com/aws/aws-sdk-go v1.44.198 h1:kgnvxQv4/kP5M0nbxBx0Ac0so9ndr9f8Ti0g+NmPQF8=
github.com/aws/aws-sdk-go v1.44.198/go.mod h1:aVsgQcEevwlmQ7qHE9I3h+dtQgpqhFB+i8Phjh7fkwI=
github.com/aws/aws-sdk-go v1.44.204 h1:7/tPUXfNOHB390A63t6fJIwmlwVQAkAwcbzKsU2/6OQ=
github.com/aws/aws-sdk-go v1.44.204/go.mod h1:aVsgQcEevwlmQ7qHE9I3h+dtQgpqhFB+i8Phjh7fkwI=
github.com/aws/aws-sdk-go-v2 v1.17.4 h1:wyC6p9Yfq6V2y98wfDsj6OnNQa4w2BLGCLIxzNhwOGY=
github.com/aws/aws-sdk-go-v2 v1.17.4/go.mod h1:uzbQtefpm44goOPmdKyAlXSNcwlRgF3ePWVW6EtJvvw=
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.4.10 h1:dK82zF6kkPeCo8J1e+tGx4JdvDIQzj7ygIoLg8WMuGs=
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.4.10/go.mod h1:VeTZetY5KRJLuD/7fkQXMU6Mw7H5m/KP2J5Iy9osMno=
github.com/aws/aws-sdk-go-v2/config v1.18.12 h1:fKs/I4wccmfrNRO9rdrbMO1NgLxct6H9rNMiPdBxHWw=
github.com/aws/aws-sdk-go-v2/config v1.18.12/go.mod h1:J36fOhj1LQBr+O4hJCiT8FwVvieeoSGOtPuvhKlsNu8=
github.com/aws/aws-sdk-go-v2/credentials v1.13.12 h1:Cb+HhuEnV19zHRaYYVglwvdHGMJWbdsyP4oHhw04xws=
github.com/aws/aws-sdk-go-v2/credentials v1.13.12/go.mod h1:37HG2MBroXK3jXfxVGtbM2J48ra2+Ltu+tmwr/jO0KA=
github.com/aws/aws-sdk-go-v2/config v1.18.13 h1:v0xlYqbO6/EVlM8tUn2QEOA7btQxcgidEq2JRDBPTho=
github.com/aws/aws-sdk-go-v2/config v1.18.13/go.mod h1:r39wGSZB7wPDW1i54JyQXUpc5KsWjh5z/3S5D9eCqDg=
github.com/aws/aws-sdk-go-v2/credentials v1.13.13 h1:zw1KAc1kl00NYd3ofVmFrb09qnYlSQMeh+fmlQRAihI=
github.com/aws/aws-sdk-go-v2/credentials v1.13.13/go.mod h1:DW9nbIIF9MrIja0cBQrUpeWYQMSlNmP8fevLUyF9W38=
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.12.22 h1:3aMfcTmoXtTZnaT86QlVaYh+BRMbvrrmZwIQ5jWqCZQ=
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.12.22/go.mod h1:YGSIJyQ6D6FjKMQh16hVFSIUD54L4F7zTGePqYMYYJU=
github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.11.51 h1:iTFYCAdKzSAjGnVIUe88Hxvix0uaBqr0Rv7qJEOX5hE=
github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.11.51/go.mod h1:7Grl2gV+dx9SWrUIgwwlUvU40t7+lOSbx34XwfmsTkY=
github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.11.53 h1:h1MmqGtYgkf49DhG2BSjGukpm8c+BJ9CL+bBbdFGzlk=
github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.11.53/go.mod h1:mlWLxwKZNeEwE+3Pko07lSr1NvHZwUtdzmo9AiGn7QU=
github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.28 h1:r+XwaCLpIvCKjBIYy/HVZujQS9tsz5ohHG3ZIe0wKoE=
github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.28/go.mod h1:3lwChorpIM/BhImY/hy+Z6jekmN92cXGPI1QJasVPYY=
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.22 h1:7AwGYXDdqRQYsluvKFmWoqpcOQJ4bH634SkYf3FNj/A=
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.22/go.mod h1:EqK7gVrIGAHyZItrD1D8B0ilgwMD1GiWAmbU4u/JHNk=
github.com/aws/aws-sdk-go-v2/internal/ini v1.3.29 h1:J4xhFd6zHhdF9jPP0FQJ6WknzBboGMBNjKOv4iTuw4A=
github.com/aws/aws-sdk-go-v2/internal/ini v1.3.29/go.mod h1:TwuqRBGzxjQJIwH16/fOZodwXt2Zxa9/cwJC5ke4j7s=
github.com/aws/aws-sdk-go-v2/internal/v4a v1.0.19 h1:FGvpyTg2LKEmMrLlpjOgkoNp9XF5CGeyAyo33LdqZW8=
github.com/aws/aws-sdk-go-v2/internal/v4a v1.0.19/go.mod h1:8W88sW3PjamQpKFUQvHWWKay6ARsNvZnzU7+a4apubw=
github.com/aws/aws-sdk-go-v2/internal/v4a v1.0.20 h1:YIvKIfPXQVp0EhXUV644kmQo6cQPPSRmC44A1HSoJeg=
github.com/aws/aws-sdk-go-v2/internal/v4a v1.0.20/go.mod h1:8W88sW3PjamQpKFUQvHWWKay6ARsNvZnzU7+a4apubw=
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.9.11 h1:y2+VQzC6Zh2ojtV2LoC0MNwHWc6qXv/j2vrQtlftkdA=
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.9.11/go.mod h1:iV4q2hsqtNECrfmlXyord9u4zyuFEJX9eLgLpSPzWA8=
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.1.23 h1:c5+bNdV8E4fIPteWx4HZSkqI07oY9exbfQ7JH7Yx4PI=
@@ -117,12 +117,12 @@ github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.9.22 h1:LjFQf8hFu
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.9.22/go.mod h1:xt0Au8yPIwYXf/GYPy/vl4K3CgwhfQMYbrH7DlUUIws=
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.13.22 h1:ISLJ2BKXe4zzyZ7mp5ewKECiw0U7KpLgS3S6OxY9Cm0=
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.13.22/go.mod h1:QFVbqK54XArazLvn2wvWMRBi/jGrWii46qbr5DyPGjc=
github.com/aws/aws-sdk-go-v2/service/s3 v1.30.2 h1:5EQWIFO+Hc8E2hFcXQJ1vm6ufl/PMt/6RVRDZRju2vM=
github.com/aws/aws-sdk-go-v2/service/s3 v1.30.2/go.mod h1:SXDHd6fI2RhqB7vmAzyYQCTQnpZrIprVJvYxpzW3JAM=
github.com/aws/aws-sdk-go-v2/service/sso v1.12.1 h1:lQKN/LNa3qqu2cDOQZybP7oL4nMGGiFqob0jZJaR8/4=
github.com/aws/aws-sdk-go-v2/service/sso v1.12.1/go.mod h1:IgV8l3sj22nQDd5qcAGY0WenwCzCphqdbFOpfktZPrI=
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.14.1 h1:0bLhH6DRAqox+g0LatcjGKjjhU6Eudyys6HB6DJVPj8=
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.14.1/go.mod h1:O1YSOg3aekZibh2SngvCRRG+cRHKKlYgxf/JBF/Kr/k=
github.com/aws/aws-sdk-go-v2/service/s3 v1.30.3 h1:PVieHTwugdlHedlxLpYLQsOZAq736RScuEb/m4zhzc4=
github.com/aws/aws-sdk-go-v2/service/s3 v1.30.3/go.mod h1:XN3YcdmnWYZ3Hrnojvo5p2mc/wfF973nkq3ClXPDMHk=
github.com/aws/aws-sdk-go-v2/service/sso v1.12.2 h1:EN102fWY7hI5u/2FPheTrwwMHkSXfl49RYkeEnJsrCU=
github.com/aws/aws-sdk-go-v2/service/sso v1.12.2/go.mod h1:IgV8l3sj22nQDd5qcAGY0WenwCzCphqdbFOpfktZPrI=
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.14.2 h1:f1lmlce7r13CX1BPyPqt9oh/H+uqOWc9367lDoGGwNQ=
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.14.2/go.mod h1:O1YSOg3aekZibh2SngvCRRG+cRHKKlYgxf/JBF/Kr/k=
github.com/aws/aws-sdk-go-v2/service/sts v1.18.3 h1:s49mSnsBZEXjfGBkRfmK+nPqzT7Lt3+t2SmAKNyHblw=
github.com/aws/aws-sdk-go-v2/service/sts v1.18.3/go.mod h1:b+psTJn33Q4qGoDaM7ZiOVVG8uVjGI6HaZ8WBHdgDgU=
github.com/aws/smithy-go v1.13.5 h1:hgz0X/DX0dGqTYpGALqXJoRKRj5oQ7150i5FdTePzO8=
@@ -255,7 +255,7 @@ github.com/google/gofuzz v1.2.0 h1:xRy4A+RhZaiKjJ1bPfwQ8sedCA+YS2YcCHW6ec7JMi0=
github.com/google/martian v2.1.0+incompatible h1:/CP5g8u/VJHijgedC/Legn3BAbAaWPgecwXBIDzw5no=
github.com/google/martian v2.1.0+incompatible/go.mod h1:9I4somxYTbIHy5NJKHRl3wXiIaQGbYVAs8BPL6v8lEs=
github.com/google/martian/v3 v3.0.0/go.mod h1:y5Zk1BBys9G+gd6Jrk0W3cC1+ELVxBWuIGO+w/tUAp0=
github.com/google/martian/v3 v3.2.1 h1:d8MncMlErDFTwQGBK1xhv026j9kqhvw1Qv9IbWT1VLQ=
github.com/google/martian/v3 v3.3.2 h1:IqNFLAmvJOgVlpdEBiQbDc2EwKW77amAycfTuWKdfvw=
github.com/google/pprof v0.0.0-20181206194817-3ea8567a2e57/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc=
github.com/google/pprof v0.0.0-20190515194954-54271f7e092f/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc=
github.com/google/pprof v0.0.0-20191218002539-d4f498aebedc/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM=
@@ -267,8 +267,8 @@ github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm4
github.com/google/uuid v1.1.2/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/google/uuid v1.3.0 h1:t6JiXgmwXMjEs8VusXIJk2BXHsn+wx8BZdTaoZ5fu7I=
github.com/google/uuid v1.3.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/googleapis/enterprise-certificate-proxy v0.2.2 h1:jUqbmxlR+gGPQq/uvQviKpS1bSQecfs2t7o6F14sk9s=
github.com/googleapis/enterprise-certificate-proxy v0.2.2/go.mod h1:AwSRAtLfXpU5Nm3pW+v7rGDHp09LsPtGY9MduiEsR9k=
github.com/googleapis/enterprise-certificate-proxy v0.2.3 h1:yk9/cqRKtT9wXZSsRH9aurXEpJX+U6FLtpYTdC3R06k=
github.com/googleapis/enterprise-certificate-proxy v0.2.3/go.mod h1:AwSRAtLfXpU5Nm3pW+v7rGDHp09LsPtGY9MduiEsR9k=
github.com/googleapis/gax-go/v2 v2.0.4/go.mod h1:0Wqv26UfaUD9n4G6kQubkQ+KchISgw+vpHVxEJEs9eg=
github.com/googleapis/gax-go/v2 v2.0.5/go.mod h1:DWXyrwAJ9X0FpwwEdw+IPEYBICEFu5mhpdKc/us6bOk=
github.com/googleapis/gax-go/v2 v2.7.0 h1:IcsPKeInNvYi7eqSaDjiZqDDKu5rsmunY0Y1YupQSSQ=
@@ -421,8 +421,8 @@ github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/
github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
github.com/stretchr/testify v1.8.1 h1:w7B6lhMri9wdJUVmEZPGGhZzrYTPvgJArz7wNPgYKsk=
github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
github.com/urfave/cli/v2 v2.24.3 h1:7Q1w8VN8yE0MJEHP06bv89PjYsN4IHWED2s1v/Zlfm0=
github.com/urfave/cli/v2 v2.24.3/go.mod h1:GHupkWPMM0M/sj1a2b4wUrWBPzazNrIjouW6fmdJLxc=
github.com/urfave/cli/v2 v2.24.4 h1:0gyJJEBYtCV87zI/x2nZCPyDxD51K6xM8SkwjHFCNEU=
github.com/urfave/cli/v2 v2.24.4/go.mod h1:GHupkWPMM0M/sj1a2b4wUrWBPzazNrIjouW6fmdJLxc=
github.com/valyala/bytebufferpool v1.0.0 h1:GqA5TC/0021Y/b9FG4Oi9Mr3q7XYx6KllzawFIhcdPw=
github.com/valyala/bytebufferpool v1.0.0/go.mod h1:6bBcMArwyJ5K/AmCkWv1jt77kVWyCJ6HpOuEn7z0Csc=
github.com/valyala/fasthttp v1.30.0/go.mod h1:2rsYD01CKFrjjsvFxx75KlEUNpWNBY9JWD3K/7o2Cus=
@@ -432,8 +432,8 @@ github.com/valyala/fastrand v1.1.0 h1:f+5HkLW4rsgzdNoleUOB69hyT9IlD2ZQh9GyDMfb5G
github.com/valyala/fastrand v1.1.0/go.mod h1:HWqCzkrkg6QXT8V2EXWvXCoow7vLwOFN002oeRzjapQ=
github.com/valyala/fasttemplate v1.2.2 h1:lxLXG0uE3Qnshl9QyaK6XJxMXlQZELvChBOCmQD0Loo=
github.com/valyala/fasttemplate v1.2.2/go.mod h1:KHLXt3tVN2HBp8eijSv/kGJopbvo7S+qRAEEKiv+SiQ=
github.com/valyala/gozstd v1.17.0 h1:M4Ds4MIrw+pD+s6vYtuFZ8D3iEw9htzfdytOV3C3iQU=
github.com/valyala/gozstd v1.17.0/go.mod h1:y5Ew47GLlP37EkTB+B4s7r6A5rdaeB7ftbl9zoYiIPQ=
github.com/valyala/gozstd v1.18.0 h1:f4BskcUZBnDrEJ2F+lVbNCMGOFBoGHEw71RBkCNR4IM=
github.com/valyala/gozstd v1.18.0/go.mod h1:y5Ew47GLlP37EkTB+B4s7r6A5rdaeB7ftbl9zoYiIPQ=
github.com/valyala/histogram v1.2.0 h1:wyYGAZZt3CpwUiIb9AU/Zbllg1llXyrtApRS815OLoQ=
github.com/valyala/histogram v1.2.0/go.mod h1:Hb4kBwb4UxsaNbbbh+RRz8ZR6pdodR57tzWUS3BUzXY=
github.com/valyala/quicktemplate v1.7.0 h1:LUPTJmlVcb46OOUY3IeD9DojFpAVbsG+5WFTcjMJzCM=
@@ -464,8 +464,8 @@ go.opentelemetry.io/otel/trace v1.13.0 h1:CBgRZ6ntv+Amuj1jDsMhZtlAPT6gbyIRdaIzFh
go.opentelemetry.io/otel/trace v1.13.0/go.mod h1:muCvmmO9KKpvuXSf3KKAXXB2ygNYHQ+ZfI5X08d3tds=
go.uber.org/atomic v1.10.0 h1:9qC72Qh0+3MqyJbAn8YU5xVq1frD8bn3JtD2oXtafVQ=
go.uber.org/atomic v1.10.0/go.mod h1:LUxbIzbOniOlMKjJjyPfpl4v+PKK2cNJn91OQbhoJI0=
go.uber.org/goleak v1.2.0 h1:xqgm/S+aQvhWFTtR0XK3Jvg7z8kGV8P4X14IzwN3Eqk=
go.uber.org/goleak v1.2.0/go.mod h1:XJYK+MuIchqpmGmUSAzotztawfKvYLUIgg7guXrwVUo=
go.uber.org/goleak v1.2.1 h1:NBol2c7O1ZokfZ0LEU9K6Whx/KnwvepVetCUhtKja4A=
go.uber.org/goleak v1.2.1/go.mod h1:qlT2yGI9QafXHhZZLxlSuNsMw3FFLxBr+tBRlmO1xH4=
golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20190510104115-cbcb75029529/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
@@ -485,8 +485,8 @@ golang.org/x/exp v0.0.0-20191227195350-da58074b4299/go.mod h1:2RIsYlXP63K8oxa1u0
golang.org/x/exp v0.0.0-20200119233911-0405dc783f0a/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4=
golang.org/x/exp v0.0.0-20200207192155-f17229e696bd/go.mod h1:J/WKrq2StrnmMY6+EHIKF9dgMWnmCNThgcyBT1FY9mM=
golang.org/x/exp v0.0.0-20200224162631-6cc2880d07d6/go.mod h1:3jZMyOhIsHpP37uCMkUooju7aAi5cS1Q23tOzKc+0MU=
golang.org/x/exp v0.0.0-20230206171751-46f607a40771 h1:xP7rWLUr1e1n2xkK5YB4LI0hPEy3LJC6Wk+D4pGlOJg=
golang.org/x/exp v0.0.0-20230206171751-46f607a40771/go.mod h1:CxIveKay+FTh1D0yPZemJVgC/95VzuuOLq5Qi4xnoYc=
golang.org/x/exp v0.0.0-20230213192124-5e25df0256eb h1:PaBZQdo+iSDyHT053FjUCgZQ/9uqVwPOcl7KSWhKn6w=
golang.org/x/exp v0.0.0-20230213192124-5e25df0256eb/go.mod h1:CxIveKay+FTh1D0yPZemJVgC/95VzuuOLq5Qi4xnoYc=
golang.org/x/image v0.0.0-20190227222117-0694c2d4d067/go.mod h1:kZ7UVZpmo3dzQBMxlp+ypCbDeSB+sBbTgSJuh5dn5js=
golang.org/x/image v0.0.0-20190802002840-cff245a6509b/go.mod h1:FeLwcggjj3mMvU+oOTbSwawSJRM1uh48EjtB4UJZlP0=
golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
@@ -499,7 +499,6 @@ golang.org/x/lint v0.0.0-20190930215403-16217165b5de/go.mod h1:6SW0HCj/g11FgYtHl
golang.org/x/lint v0.0.0-20191125180803-fdd1cda4f05f/go.mod h1:5qLYkcX4OjUUV8bRuDixDT3tpyyb+LUpUlRWLxfhWrs=
golang.org/x/lint v0.0.0-20200130185559-910be7a94367/go.mod h1:3xt1FjdF8hUf6vQPIChWIBhFzV8gjjsPE/fR3IyQdNY=
golang.org/x/lint v0.0.0-20200302205851-738671d3881b/go.mod h1:3xt1FjdF8hUf6vQPIChWIBhFzV8gjjsPE/fR3IyQdNY=
golang.org/x/lint v0.0.0-20210508222113-6edffad5e616 h1:VLliZ0d+/avPrXXH+OakdXhpJuEoBZuwh1m2j7U6Iug=
golang.org/x/mobile v0.0.0-20190312151609-d3739f865fa6/go.mod h1:z+o9i4GpDbdi3rU15maQ/Ox0txvL9dWGYEHz965HBQE=
golang.org/x/mobile v0.0.0-20190719004257-d2bd2a29d028/go.mod h1:E/iHnbuqvinMTCcRqshq8CkpyQDoeVncDDYHnLhea+o=
golang.org/x/mod v0.0.0-20190513183733-4bf6d317e70e/go.mod h1:mXi4GBBbnImb6dmsKGUJ2LatrhH/nqhxcFungHvyanc=
@@ -545,8 +544,8 @@ golang.org/x/net v0.0.0-20210510120150-4163338589ed/go.mod h1:9nx3DQGgdP8bBQD5qx
golang.org/x/net v0.0.0-20210525063256-abc453219eb5/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
golang.org/x/net v0.1.0/go.mod h1:Cx3nUiGt4eDBEyega/BKRp+/AlGL8hYe7U9odMt2Cco=
golang.org/x/net v0.6.0 h1:L4ZwwTvKW9gr0ZMS1yrHD9GZhIuVjOBBnaKH+SPQK0Q=
golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
golang.org/x/net v0.7.0 h1:rJrUqqhjsgNp7KqAIc25s9pZnjU7TUcSY7HcVZjdn1g=
golang.org/x/net v0.7.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
@@ -700,8 +699,8 @@ google.golang.org/api v0.24.0/go.mod h1:lIXQywCXRcnZPGlsd8NbLnOjtAoL6em04bJ9+z0M
google.golang.org/api v0.28.0/go.mod h1:lIXQywCXRcnZPGlsd8NbLnOjtAoL6em04bJ9+z0MncE=
google.golang.org/api v0.29.0/go.mod h1:Lcubydp8VUV7KeIHD9z2Bys/sm/vGKnG1UHuDBSrHWM=
google.golang.org/api v0.30.0/go.mod h1:QGmEvQ87FHZNiUVJkT14jQNYJ4ZJjdRF23ZXz5138Fc=
google.golang.org/api v0.109.0 h1:sW9hgHyX497PP5//NUM7nqfV8D0iDfBApqq7sOh1XR8=
google.golang.org/api v0.109.0/go.mod h1:2Ts0XTHNVWxypznxWOYUeI4g3WdP9Pk2Qk58+a/O9MY=
google.golang.org/api v0.110.0 h1:l+rh0KYUooe9JGbGVx71tbFo4SMbMTXK3I3ia2QSEeU=
google.golang.org/api v0.110.0/go.mod h1:7FC4Vvx1Mooxh8C5HWjzZHcavuS2f6pmJpZx60ca7iI=
google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM=
google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
google.golang.org/appengine v1.5.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
@@ -739,8 +738,8 @@ google.golang.org/genproto v0.0.0-20200618031413-b414f8b61790/go.mod h1:jDfRM7Fc
google.golang.org/genproto v0.0.0-20200729003335-053ba62fc06f/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
google.golang.org/genproto v0.0.0-20200804131852-c06518451d9c/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
google.golang.org/genproto v0.0.0-20200825200019-8632dd797987/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
google.golang.org/genproto v0.0.0-20230209215440-0dfe4f8abfcc h1:ijGwO+0vL2hJt5gaygqP2j6PfflOBrRot0IczKbmtio=
google.golang.org/genproto v0.0.0-20230209215440-0dfe4f8abfcc/go.mod h1:RGgjbofJ8xD9Sq1VVhDM1Vok1vRONV+rg+CjzG4SZKM=
google.golang.org/genproto v0.0.0-20230216225411-c8e22ba71e44 h1:EfLuoKW5WfkgVdDy7dTK8qSbH37AX5mj/MFh+bGPz14=
google.golang.org/genproto v0.0.0-20230216225411-c8e22ba71e44/go.mod h1:8B0gmkoRebU8ukX6HP+4wrVQUY1+6PkQ44BSyIlflHA=
google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
google.golang.org/grpc v1.20.1/go.mod h1:10oTOabMzJvdu6/UiuZezV6QK5dSlG84ov/aaiqXj38=
google.golang.org/grpc v1.21.1/go.mod h1:oYelfM1adQP15Ek0mdvEgi9Df8B9CZIaU1084ijfRaM=

View File

@@ -262,7 +262,7 @@ func MustRemoveTemporaryDirs(dir string) {
continue
}
dirName := fi.Name()
if strings.Contains(dirName, ".must-remove.") {
if IsScheduledForRemoval(dirName) {
fullPath := dir + "/" + dirName
MustRemoveAll(fullPath)
}
@@ -467,3 +467,7 @@ func isHTTPURL(targetURL string) bool {
return err == nil && (parsed.Scheme == "http" || parsed.Scheme == "https") && parsed.Host != ""
}
func IsScheduledForRemoval(name string) bool {
return strings.Contains(name, ".must-remove.")
}

View File

@@ -788,7 +788,18 @@ func (tb *Table) notifyBackgroundMergers() bool {
}
}
var flushConcurrencyCh = make(chan struct{}, cgroup.AvailableCPUs())
var flushConcurrencyLimit = func() int {
n := cgroup.AvailableCPUs()
if n < 2 {
// Allow at least 2 concurrent flushers on systems with a single CPU core
// in order to guarantee that in-memory data flushes and background merges can be continued
// when a single flusher is busy with the long merge.
n = 2
}
return n
}()
var flushConcurrencyCh = make(chan struct{}, flushConcurrencyLimit)
func needAssistedMerge(pws []*partWrapper, maxParts int) bool {
if len(pws) < maxParts {
@@ -1824,5 +1835,5 @@ func removeParts(pws []*partWrapper, partsToRemove map[*partWrapper]bool) ([]*pa
func isSpecialDir(name string) bool {
// Snapshots and cache dirs aren't used anymore.
// Keep them here for backwards compatibility.
return name == "tmp" || name == "txn" || name == "snapshots" || name == "cache"
return name == "tmp" || name == "txn" || name == "snapshots" || name == "cache" || fs.IsScheduledForRemoval(name)
}

View File

@@ -14,6 +14,7 @@ import (
"github.com/VictoriaMetrics/VictoriaMetrics/lib/bytesutil"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/flagutil"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/promscrape/discoveryutils"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/proxy"
"github.com/VictoriaMetrics/fasthttp"
"github.com/VictoriaMetrics/metrics"
@@ -262,7 +263,11 @@ func (c *client) ReadData(dst []byte) ([]byte, error) {
// This should reduce memory uage when scraping big targets.
dst = resp.SwapBody(dst)
}
err := doRequestWithPossibleRetry(c.hc, req, resp, deadline)
ctx, cancel := context.WithDeadline(c.ctx, deadline)
defer cancel()
err := doRequestWithPossibleRetry(ctx, c.hc, req, resp)
statusCode := resp.StatusCode()
redirectsCount := 0
for err == nil && isStatusRedirect(statusCode) {
@@ -282,7 +287,7 @@ func (c *client) ReadData(dst []byte) ([]byte, error) {
break
}
req.URI().UpdateBytes(location)
err = doRequestWithPossibleRetry(c.hc, req, resp, deadline)
err = doRequestWithPossibleRetry(ctx, c.hc, req, resp)
statusCode = resp.StatusCode()
redirectsCount++
}
@@ -349,32 +354,45 @@ var (
scrapeRetries = metrics.NewCounter(`vm_promscrape_scrape_retries_total`)
)
func doRequestWithPossibleRetry(hc *fasthttp.HostClient, req *fasthttp.Request, resp *fasthttp.Response, deadline time.Time) error {
sleepTime := time.Second
func doRequestWithPossibleRetry(ctx context.Context, hc *fasthttp.HostClient, req *fasthttp.Request, resp *fasthttp.Response) error {
scrapeRequests.Inc()
for {
// Use DoDeadline instead of Do even if hc.ReadTimeout is already set in order to guarantee the given deadline
// across multiple retries.
err := hc.DoDeadline(req, resp, deadline)
if err == nil {
var reqErr error
// Return true if the request execution is completed and retry is not required
attempt := func() bool {
// Use DoCtx instead of Do in order to support context cancellation
reqErr = hc.DoCtx(ctx, req, resp)
if reqErr == nil {
statusCode := resp.StatusCode()
if statusCode != fasthttp.StatusTooManyRequests {
return nil
return true
}
} else if err != fasthttp.ErrConnectionClosed && !strings.Contains(err.Error(), "broken pipe") {
return err
} else if reqErr != fasthttp.ErrConnectionClosed && !strings.Contains(reqErr.Error(), "broken pipe") {
return true
}
// Retry request after exponentially increased sleep.
maxSleepTime := time.Until(deadline)
if sleepTime > maxSleepTime {
return fmt.Errorf("the server closes all the connection attempts: %w", err)
return false
}
if attempt() {
return reqErr
}
// The first attempt was unsuccessful. Use exponential backoff for further attempts.
// Perform the second attempt immediately after the first attempt - this should help
// in cases when the remote side closes the keep-alive connection before the first attempt.
// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3293
sleepTime := time.Second
// It is expected that the deadline is already set to ctx, so the loop below
// should eventually finish if all the attempt() calls are unsuccessful.
for {
scrapeRetries.Inc()
if attempt() {
return reqErr
}
sleepTime += sleepTime
if sleepTime > maxSleepTime {
sleepTime = maxSleepTime
if !discoveryutils.SleepCtx(ctx, sleepTime) {
return reqErr
}
time.Sleep(sleepTime)
scrapeRetries.Inc()
}
}

View File

@@ -199,12 +199,17 @@ func getRefreshTokenFunc(sdc *SDConfig, ac, proxyAC *promauth.Config, env *cloud
q := endpointURL.Query()
msiSecret := os.Getenv("MSI_SECRET")
identityHeader := os.Getenv("IDENTITY_HEADER")
clientIDParam := "client_id"
apiVersion := "2018-02-01"
if msiSecret != "" {
clientIDParam = "clientid"
apiVersion = "2017-09-01"
}
if identityHeader != "" {
clientIDParam = "client_id"
apiVersion = "2019-08-01"
}
q.Set("api-version", apiVersion)
q.Set(clientIDParam, sdc.ClientID)
q.Set("resource", env.ResourceManagerEndpoint)
@@ -214,6 +219,9 @@ func getRefreshTokenFunc(sdc *SDConfig, ac, proxyAC *promauth.Config, env *cloud
modifyRequest = func(request *http.Request) {
if msiSecret != "" {
request.Header.Set("secret", msiSecret)
if identityHeader != "" {
request.Header.Set("X-IDENTITY-HEADER", msiSecret)
}
} else {
request.Header.Set("Metadata", "true")
}
@@ -235,15 +243,38 @@ func getRefreshTokenFunc(sdc *SDConfig, ac, proxyAC *promauth.Config, env *cloud
if err := json.Unmarshal(data, &tr); err != nil {
return "", 0, fmt.Errorf("cannot parse token auth response %q: %w", data, err)
}
expiresInSeconds, err := strconv.ParseInt(tr.ExpiresIn, 10, 64)
expiresInSeconds, err := parseTokenExpiry(tr)
if err != nil {
return "", 0, fmt.Errorf("cannot parse expiresIn param in token auth %q: %w", tr.ExpiresIn, err)
return "", 0, err
}
return tr.AccessToken, time.Second * time.Duration(expiresInSeconds), nil
}
return refreshToken, nil
}
// parseTokenExpiry returns token expiry in seconds
func parseTokenExpiry(tr tokenResponse) (int64, error) {
var expiresInSeconds int64
var err error
if tr.ExpiresIn == "" {
var expiresOnSeconds int64
expiresOnSeconds, err = strconv.ParseInt(tr.ExpiresOn, 10, 64)
if err != nil {
return 0, fmt.Errorf("cannot parse expiresOn=%q in auth token response: %w", tr.ExpiresOn, err)
}
expiresInSeconds = expiresOnSeconds - time.Now().Unix()
} else {
expiresInSeconds, err = strconv.ParseInt(tr.ExpiresIn, 10, 64)
if err != nil {
return 0, fmt.Errorf("cannot parse expiresIn=%q auth token response: %w", tr.ExpiresIn, err)
}
}
return expiresInSeconds, nil
}
// mustGetAuthToken returns auth token
// in case of error, logs error and return empty token
func (ac *apiConfig) mustGetAuthToken() string {
@@ -270,4 +301,5 @@ func (ac *apiConfig) mustGetAuthToken() string {
type tokenResponse struct {
AccessToken string `json:"access_token"`
ExpiresIn string `json:"expires_in"`
ExpiresOn string `json:"expires_on"`
}

View File

@@ -225,7 +225,7 @@ func (c *Client) getAPIResponseWithParamsAndClientCtx(ctx context.Context, clien
modifyRequest(req)
}
resp, err := doRequestWithPossibleRetry(client, req, deadline)
resp, err := doRequestWithPossibleRetry(client, req)
if err != nil {
return nil, fmt.Errorf("cannot fetch %q: %w", requestURL, err)
}
@@ -256,31 +256,48 @@ func (c *Client) Stop() {
c.clientCancel()
}
func doRequestWithPossibleRetry(hc *HTTPClient, req *http.Request, deadline time.Time) (*http.Response, error) {
sleepTime := time.Second
func doRequestWithPossibleRetry(hc *HTTPClient, req *http.Request) (*http.Response, error) {
discoveryRequests.Inc()
for {
resp, err := hc.client.Do(req)
if err == nil {
var (
reqErr error
resp *http.Response
)
// Return true if the request execution is completed and retry is not required
attempt := func() bool {
resp, reqErr = hc.client.Do(req)
if reqErr == nil {
statusCode := resp.StatusCode
if statusCode != http.StatusTooManyRequests {
return resp, nil
return true
}
} else if err != net.ErrClosed && !strings.Contains(err.Error(), "broken pipe") {
return nil, err
} else if reqErr != net.ErrClosed && !strings.Contains(reqErr.Error(), "broken pipe") {
return true
}
// Retry request after exponentially increased sleep.
maxSleepTime := time.Until(deadline)
if sleepTime > maxSleepTime {
return nil, fmt.Errorf("the server closes all the connection attempts: %w", err)
return false
}
if attempt() {
return resp, reqErr
}
// The first attempt was unsuccessful. Use exponential backoff for further attempts.
// Perform the second attempt immediately after the first attempt - this should help
// in cases when the remote side closes the keep-alive connection before the first attempt.
// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3293
sleepTime := time.Second
// It is expected that the deadline is already set to req.Context(), so the loop below
// should eventually finish if all the attempt() calls are unsuccessful.
ctx := req.Context()
for {
discoveryRetries.Inc()
if attempt() {
return resp, reqErr
}
sleepTime += sleepTime
if sleepTime > maxSleepTime {
sleepTime = maxSleepTime
if !SleepCtx(ctx, sleepTime) {
return resp, reqErr
}
time.Sleep(sleepTime)
discoveryRetries.Inc()
}
}
@@ -288,3 +305,17 @@ var (
discoveryRequests = metrics.NewCounter(`vm_promscrape_discovery_requests_total`)
discoveryRetries = metrics.NewCounter(`vm_promscrape_discovery_retries_total`)
)
// SleepCtx sleeps for sleepDuration.
//
// It immediately returns false on ctx cancel or deadline, without waiting for sleepDuration.
func SleepCtx(ctx context.Context, sleepDuration time.Duration) bool {
t := timerpool.Get(sleepDuration)
defer timerpool.Put(t)
select {
case <-ctx.Done():
return false
case <-t.C:
return true
}
}

View File

@@ -174,9 +174,13 @@ func (uw *unmarshalWork) Unmarshal() {
}
}
// Convert timestamps from seconds to milliseconds
// Convert timestamps in seconds to milliseconds if needed.
// See http://opentsdb.net/docs/javadoc/net/opentsdb/core/Const.html#SECOND_MASK
for i := range rows {
rows[i].Timestamp *= 1e3
r := &rows[i]
if r.Timestamp&secondMask == 0 {
r.Timestamp *= 1e3
}
}
// Trim timestamps if required.
@@ -191,6 +195,8 @@ func (uw *unmarshalWork) Unmarshal() {
putUnmarshalWork(uw)
}
const secondMask int64 = 0x7FFFFFFF00000000
func getUnmarshalWork() *unmarshalWork {
v := unmarshalWorkPool.Get()
if v == nil {

View File

@@ -615,7 +615,19 @@ func (pt *partition) notifyBackgroundMergers() bool {
}
}
var flushConcurrencyCh = make(chan struct{}, cgroup.AvailableCPUs())
var flushConcurrencyLimit = func() int {
n := cgroup.AvailableCPUs()
if n < 3 {
// Allow at least 3 concurrent flushers on systems with a single CPU core
// in order to guarantee that in-memory data flushes and background merges can be continued
// when a single flusher is busy with the long merge of big parts,
// while another flusher is busy with the long merge of small parts.
n = 3
}
return n
}()
var flushConcurrencyCh = make(chan struct{}, flushConcurrencyLimit)
func needAssistedMerge(pws []*partWrapper, maxParts int) bool {
if len(pws) < maxParts {
@@ -1007,7 +1019,7 @@ func hasActiveMerges(pws []*partWrapper) bool {
return false
}
var mergeWorkersLimitCh = make(chan struct{}, getDefaultMergeConcurrency(16))
var mergeWorkersLimitCh = make(chan struct{}, adjustMergeWorkersLimit(getDefaultMergeConcurrency(16)))
var bigMergeWorkersLimitCh = make(chan struct{}, getDefaultMergeConcurrency(4))
@@ -1038,9 +1050,20 @@ func SetMergeWorkersCount(n int) {
// Do nothing
return
}
n = adjustMergeWorkersLimit(n)
mergeWorkersLimitCh = make(chan struct{}, n)
}
func adjustMergeWorkersLimit(n int) int {
if n < 2 {
// Allow at least 2 merge workers on systems with a single CPU core
// in order to guarantee that background merges can be continued
// when a single worker is busy with the long merge of big parts.
return 2
}
return n
}
func (pt *partition) startMergeWorkers() {
// Start a merge worker per available CPU core.
// The actual number of concurrent merges is limited inside mergeWorker() below.
@@ -1999,7 +2022,7 @@ func (pt *partition) createSnapshot(srcDir, dstDir string) error {
// Skip non-directories.
continue
}
if fn == "tmp" || fn == "txn" {
if fn == "tmp" || fn == "txn" || fs.IsScheduledForRemoval(fn) {
// Skip special dirs.
continue
}

View File

@@ -1,4 +1,4 @@
GO_VERSION ?=1.20.0
GO_VERSION ?=1.20.1
SNAP_BUILDER_IMAGE := local/snap-builder:2.0.0-$(shell echo $(GO_VERSION) | tr :/ __)

View File

@@ -1,5 +1,19 @@
# Changes
## [0.12.0](https://github.com/googleapis/google-cloud-go/compare/iam/v0.11.0...iam/v0.12.0) (2023-02-17)
### Features
* **iam:** Migrate to new stubs ([a61ddcd](https://github.com/googleapis/google-cloud-go/commit/a61ddcd3041c7af4a15109dc4431f9b327c497fb))
## [0.11.0](https://github.com/googleapis/google-cloud-go/compare/iam/v0.10.0...iam/v0.11.0) (2023-02-16)
### Features
* **iam:** Start generating proto stubs ([970d763](https://github.com/googleapis/google-cloud-go/commit/970d763531b54b2bc75d7ff26a20b6e05150cab8))
## [0.10.0](https://github.com/googleapis/google-cloud-go/compare/iam/v0.9.0...iam/v0.10.0) (2023-01-04)

View File

@@ -15,7 +15,7 @@
// Code generated by protoc-gen-go. DO NOT EDIT.
// versions:
// protoc-gen-go v1.26.0
// protoc v3.21.9
// protoc v3.21.12
// source: google/iam/v1/iam_policy.proto
package iampb

View File

@@ -15,7 +15,7 @@
// Code generated by protoc-gen-go. DO NOT EDIT.
// versions:
// protoc-gen-go v1.26.0
// protoc v3.21.9
// protoc v3.21.12
// source: google/iam/v1/options.proto
package iampb

View File

@@ -15,7 +15,7 @@
// Code generated by protoc-gen-go. DO NOT EDIT.
// versions:
// protoc-gen-go v1.26.0
// protoc v3.21.9
// protoc v3.21.12
// source: google/iam/v1/policy.proto
package iampb

View File

@@ -26,8 +26,8 @@ import (
"fmt"
"time"
pb "cloud.google.com/go/iam/apiv1/iampb"
gax "github.com/googleapis/gax-go/v2"
pb "google.golang.org/genproto/googleapis/iam/v1"
"google.golang.org/grpc"
"google.golang.org/grpc/codes"
"google.golang.org/grpc/metadata"

View File

@@ -719,6 +719,15 @@
"release_level": "beta",
"library_type": "GAPIC_AUTO"
},
"cloud.google.com/go/discoveryengine/apiv1beta": {
"distribution_name": "cloud.google.com/go/discoveryengine/apiv1beta",
"description": "Discovery Engine API",
"language": "Go",
"client_library_type": "generated",
"docs_url": "https://cloud.google.com/go/docs/reference/cloud.google.com/go/discoveryengine/latest/apiv1beta",
"release_level": "beta",
"library_type": "GAPIC_AUTO"
},
"cloud.google.com/go/dlp/apiv2": {
"distribution_name": "cloud.google.com/go/dlp/apiv2",
"description": "Cloud Data Loss Prevention (DLP) API",
@@ -1088,6 +1097,15 @@
"release_level": "beta",
"library_type": "GAPIC_AUTO"
},
"cloud.google.com/go/maps/mapsplatformdatasets/apiv1alpha": {
"distribution_name": "cloud.google.com/go/maps/mapsplatformdatasets/apiv1alpha",
"description": "Maps Platform Datasets API",
"language": "Go",
"client_library_type": "generated",
"docs_url": "https://cloud.google.com/go/docs/reference/cloud.google.com/go/maps/latest/mapsplatformdatasets/apiv1alpha",
"release_level": "alpha",
"library_type": "GAPIC_AUTO"
},
"cloud.google.com/go/maps/routing/apiv2": {
"distribution_name": "cloud.google.com/go/maps/routing/apiv2",
"description": "Routes API",

View File

@@ -24,3 +24,20 @@ To kick off a build locally run from the repo root:
```bash
gcloud builds submit --project=cloud-devrel-kokoro-resources --config=internal/cloudbuild.yaml
```
### Updating OwlBot SHA
You may want to manually update the which version of the post processor will be
used -- to do this you need to update the SHA in the OwlBot lock file. Start by
running the following commands:
```bash
docker pull gcr.io/cloud-devrel-public-resources/owlbot-go:latest
docker inspect --format='{{index .RepoDigests 0}}' gcr.io/cloud-devrel-public-resources/owlbot-go:latest
```
This will give you a SHA. You can use this value to update the value in
`.github/.OwlBot.lock.yaml`.
*Note*: OwlBot will eventually open a pull request to update this value if it
discovers a new version of the container.

View File

@@ -1,3 +1,4 @@
//go:build !amd64 && !arm64 && !ppc64
// +build !amd64,!arm64,!ppc64
package fasthttp

View File

@@ -1,3 +1,4 @@
//go:build amd64 || arm64 || ppc64
// +build amd64 arm64 ppc64
package fasthttp

View File

@@ -3,6 +3,7 @@ package fasthttp
import (
"bufio"
"bytes"
"context"
"crypto/tls"
"errors"
"fmt"
@@ -872,17 +873,40 @@ func (c *HostClient) DoDeadline(req *Request, resp *Response, deadline time.Time
return clientDoDeadline(req, resp, deadline, c)
}
// DoCtx performs the given request and waits for response until
// the given context is cancelled or deadline is reached.
//
// Request must contain at least non-zero RequestURI with full url (including
// scheme and host) or non-zero Host header + RequestURI.
//
// The function doesn't follow redirects. Use Get* for following redirects.
//
// Response is ignored if resp is nil.
//
// ErrTimeout is returned if the response wasn't returned until
// the deadline provided by the given context.
//
// ErrNoFreeConns is returned if all HostClient.MaxConns connections
// to the host are busy.
//
// It is recommended obtaining req and resp via AcquireRequest
// and AcquireResponse in performance-critical code.
func (c *HostClient) DoCtx(ctx context.Context, req *Request, resp *Response) error {
return clientDoCtx(ctx, req, resp, c)
}
func clientDoTimeout(req *Request, resp *Response, timeout time.Duration, c clientDoer) error {
deadline := time.Now().Add(timeout)
return clientDoDeadline(req, resp, deadline, c)
}
func clientDoDeadline(req *Request, resp *Response, deadline time.Time, c clientDoer) error {
timeout := -time.Since(deadline)
if timeout <= 0 {
return ErrTimeout
}
ctx, cancel := context.WithDeadline(context.Background(), deadline)
defer cancel()
return clientDoCtx(ctx, req, resp, c)
}
func clientDoCtx(ctx context.Context, req *Request, resp *Response, c clientDoer) error {
var ch chan error
chv := errorChPool.Get()
if chv == nil {
@@ -910,7 +934,6 @@ func clientDoDeadline(req *Request, resp *Response, deadline time.Time, c client
ch <- c.Do(reqCopy, respCopy)
}()
tc := acquireTimer(timeout)
var err error
select {
case err = <-ch:
@@ -921,10 +944,12 @@ func clientDoDeadline(req *Request, resp *Response, deadline time.Time, c client
ReleaseResponse(respCopy)
ReleaseRequest(reqCopy)
errorChPool.Put(chv)
case <-tc.C:
err = ErrTimeout
case <-ctx.Done():
err = ctx.Err()
if errors.Is(err, context.DeadlineExceeded) {
err = ErrTimeout
}
}
releaseTimer(tc)
return err
}

View File

@@ -127,11 +127,11 @@ var (
//
// Supported compression levels are:
//
// * CompressNoCompression
// * CompressBestSpeed
// * CompressBestCompression
// * CompressDefaultCompression
// * CompressHuffmanOnly
// - CompressNoCompression
// - CompressBestSpeed
// - CompressBestCompression
// - CompressDefaultCompression
// - CompressHuffmanOnly
func AppendGzipBytesLevel(dst, src []byte, level int) []byte {
w := &byteSliceWriter{dst}
WriteGzipLevel(w, src, level)
@@ -143,11 +143,11 @@ func AppendGzipBytesLevel(dst, src []byte, level int) []byte {
//
// Supported compression levels are:
//
// * CompressNoCompression
// * CompressBestSpeed
// * CompressBestCompression
// * CompressDefaultCompression
// * CompressHuffmanOnly
// - CompressNoCompression
// - CompressBestSpeed
// - CompressBestCompression
// - CompressDefaultCompression
// - CompressHuffmanOnly
func WriteGzipLevel(w io.Writer, p []byte, level int) (int, error) {
switch w.(type) {
case *byteSliceWriter,
@@ -224,11 +224,11 @@ func AppendGunzipBytes(dst, src []byte) ([]byte, error) {
//
// Supported compression levels are:
//
// * CompressNoCompression
// * CompressBestSpeed
// * CompressBestCompression
// * CompressDefaultCompression
// * CompressHuffmanOnly
// - CompressNoCompression
// - CompressBestSpeed
// - CompressBestCompression
// - CompressDefaultCompression
// - CompressHuffmanOnly
func AppendDeflateBytesLevel(dst, src []byte, level int) []byte {
w := &byteSliceWriter{dst}
WriteDeflateLevel(w, src, level)
@@ -240,11 +240,11 @@ func AppendDeflateBytesLevel(dst, src []byte, level int) []byte {
//
// Supported compression levels are:
//
// * CompressNoCompression
// * CompressBestSpeed
// * CompressBestCompression
// * CompressDefaultCompression
// * CompressHuffmanOnly
// - CompressNoCompression
// - CompressBestSpeed
// - CompressBestCompression
// - CompressDefaultCompression
// - CompressHuffmanOnly
func WriteDeflateLevel(w io.Writer, p []byte, level int) (int, error) {
switch w.(type) {
case *byteSliceWriter,

View File

@@ -3,38 +3,57 @@ Package fasthttp provides fast HTTP server and client API.
Fasthttp provides the following features:
* Optimized for speed. Easily handles more than 100K qps and more than 1M
concurrent keep-alive connections on modern hardware.
* Optimized for low memory usage.
* Easy 'Connection: Upgrade' support via RequestCtx.Hijack.
* Server supports requests' pipelining. Multiple requests may be read from
a single network packet and multiple responses may be sent in a single
network packet. This may be useful for highly loaded REST services.
* Server provides the following anti-DoS limits:
- Optimized for speed. Easily handles more than 100K qps and more than 1M
concurrent keep-alive connections on modern hardware.
* The number of concurrent connections.
* The number of concurrent connections per client IP.
* The number of requests per connection.
* Request read timeout.
* Response write timeout.
* Maximum request header size.
* Maximum request body size.
* Maximum request execution time.
* Maximum keep-alive connection lifetime.
* Early filtering out non-GET requests.
- Optimized for low memory usage.
* A lot of additional useful info is exposed to request handler:
- Easy 'Connection: Upgrade' support via RequestCtx.Hijack.
* Server and client address.
* Per-request logger.
* Unique request id.
* Request start time.
* Connection start time.
* Request sequence number for the current connection.
- Server supports requests' pipelining. Multiple requests may be read from
a single network packet and multiple responses may be sent in a single
network packet. This may be useful for highly loaded REST services.
* Client supports automatic retry on idempotent requests' failure.
* Fasthttp API is designed with the ability to extend existing client
and server implementations or to write custom client and server
implementations from scratch.
- Server provides the following anti-DoS limits:
- The number of concurrent connections.
- The number of concurrent connections per client IP.
- The number of requests per connection.
- Request read timeout.
- Response write timeout.
- Maximum request header size.
- Maximum request body size.
- Maximum request execution time.
- Maximum keep-alive connection lifetime.
- Early filtering out non-GET requests.
- A lot of additional useful info is exposed to request handler:
- Server and client address.
- Per-request logger.
- Unique request id.
- Request start time.
- Connection start time.
- Request sequence number for the current connection.
- Client supports automatic retry on idempotent requests' failure.
- Fasthttp API is designed with the ability to extend existing client
and server implementations or to write custom client and server
implementations from scratch.
*/
package fasthttp

View File

@@ -51,8 +51,8 @@ func ServeFileUncompressed(ctx *RequestCtx, path string) {
//
// HTTP response may contain uncompressed file contents in the following cases:
//
// * Missing 'Accept-Encoding: gzip' request header.
// * No write access to directory containing the file.
// - Missing 'Accept-Encoding: gzip' request header.
// - No write access to directory containing the file.
//
// Directory contents is returned if path points to directory.
//
@@ -69,8 +69,8 @@ func ServeFileBytes(ctx *RequestCtx, path []byte) {
//
// HTTP response may contain uncompressed file contents in the following cases:
//
// * Missing 'Accept-Encoding: gzip' request header.
// * No write access to directory containing the file.
// - Missing 'Accept-Encoding: gzip' request header.
// - No write access to directory containing the file.
//
// Directory contents is returned if path points to directory.
//
@@ -123,12 +123,11 @@ type PathRewriteFunc func(ctx *RequestCtx) []byte
//
// Examples:
//
// * host=foobar.com, slashesCount=0, original path="/foo/bar".
// - host=foobar.com, slashesCount=0, original path="/foo/bar".
// Resulting path: "/foobar.com/foo/bar"
//
// * host=img.aaa.com, slashesCount=1, original path="/images/123/456.jpg"
// - host=img.aaa.com, slashesCount=1, original path="/images/123/456.jpg"
// Resulting path: "/img.aaa.com/123/456.jpg"
//
func NewVHostPathRewriter(slashesCount int) PathRewriteFunc {
return func(ctx *RequestCtx) []byte {
path := stripLeadingSlashes(ctx.Path(), slashesCount)
@@ -157,9 +156,9 @@ var strInvalidHost = []byte("invalid-host")
//
// Examples:
//
// * slashesCount = 0, original path: "/foo/bar", result: "/foo/bar"
// * slashesCount = 1, original path: "/foo/bar", result: "/bar"
// * slashesCount = 2, original path: "/foo/bar", result: ""
// - slashesCount = 0, original path: "/foo/bar", result: "/foo/bar"
// - slashesCount = 1, original path: "/foo/bar", result: "/bar"
// - slashesCount = 2, original path: "/foo/bar", result: ""
//
// The returned path rewriter may be used as FS.PathRewrite .
func NewPathSlashesStripper(slashesCount int) PathRewriteFunc {
@@ -173,9 +172,9 @@ func NewPathSlashesStripper(slashesCount int) PathRewriteFunc {
//
// Examples:
//
// * prefixSize = 0, original path: "/foo/bar", result: "/foo/bar"
// * prefixSize = 3, original path: "/foo/bar", result: "o/bar"
// * prefixSize = 7, original path: "/foo/bar", result: "r"
// - prefixSize = 0, original path: "/foo/bar", result: "/foo/bar"
// - prefixSize = 3, original path: "/foo/bar", result: "o/bar"
// - prefixSize = 7, original path: "/foo/bar", result: "r"
//
// The returned path rewriter may be used as FS.PathRewrite .
func NewPathPrefixStripper(prefixSize int) PathRewriteFunc {
@@ -273,9 +272,9 @@ const FSHandlerCacheDuration = 10 * time.Second
// from requested path before searching requested file in the root folder.
// Examples:
//
// * stripSlashes = 0, original path: "/foo/bar", result: "/foo/bar"
// * stripSlashes = 1, original path: "/foo/bar", result: "/bar"
// * stripSlashes = 2, original path: "/foo/bar", result: ""
// - stripSlashes = 0, original path: "/foo/bar", result: "/foo/bar"
// - stripSlashes = 1, original path: "/foo/bar", result: "/bar"
// - stripSlashes = 2, original path: "/foo/bar", result: ""
//
// The returned request handler automatically generates index pages
// for directories without index.html.

View File

@@ -90,8 +90,8 @@ func (h *ResponseHeader) SetContentRange(startPos, endPos, contentLength int) {
// SetByteRange sets 'Range: bytes=startPos-endPos' header.
//
// * If startPos is negative, then 'bytes=-startPos' value is set.
// * If endPos is negative, then 'bytes=startPos-' value is set.
// - If startPos is negative, then 'bytes=-startPos' value is set.
// - If endPos is negative, then 'bytes=startPos-' value is set.
func (h *RequestHeader) SetByteRange(startPos, endPos int) {
h.parseRawHeaders()
@@ -1985,9 +1985,9 @@ func normalizeHeaderKey(b []byte) {
// after dashes are also uppercased. All the other letters are lowercased.
// Examples:
//
// * coNTENT-TYPe -> Content-Type
// * HOST -> Host
// * foo-bar-baz -> Foo-Bar-Baz
// - coNTENT-TYPe -> Content-Type
// - HOST -> Host
// - foo-bar-baz -> Foo-Bar-Baz
func AppendNormalizedHeaderKey(dst []byte, key string) []byte {
dst = append(dst, key...)
normalizeHeaderKey(dst[len(dst)-len(key):])
@@ -2001,9 +2001,9 @@ func AppendNormalizedHeaderKey(dst []byte, key string) []byte {
// after dashes are also uppercased. All the other letters are lowercased.
// Examples:
//
// * coNTENT-TYPe -> Content-Type
// * HOST -> Host
// * foo-bar-baz -> Foo-Bar-Baz
// - coNTENT-TYPe -> Content-Type
// - HOST -> Host
// - foo-bar-baz -> Foo-Bar-Baz
func AppendNormalizedHeaderKeyBytes(dst, key []byte) []byte {
return AppendNormalizedHeaderKey(dst, b2s(key))
}

View File

@@ -216,14 +216,14 @@ func (resp *Response) IsBodyStream() bool {
//
// This function may be used in the following cases:
//
// * if request body is too big (more than 10MB).
// * if request body is streamed from slow external sources.
// * if request body must be streamed to the server in chunks
// - if request body is too big (more than 10MB).
// - if request body is streamed from slow external sources.
// - if request body must be streamed to the server in chunks
// (aka `http client push` or `chunked transfer-encoding`).
//
// Note that GET and HEAD requests cannot have body.
//
/// See also SetBodyStream.
// / See also SetBodyStream.
func (req *Request) SetBodyStreamWriter(sw StreamWriter) {
sr := NewStreamReader(sw)
req.SetBodyStream(sr, -1)
@@ -233,9 +233,9 @@ func (req *Request) SetBodyStreamWriter(sw StreamWriter) {
//
// This function may be used in the following cases:
//
// * if response body is too big (more than 10MB).
// * if response body is streamed from slow external sources.
// * if response body must be streamed to the client in chunks
// - if response body is too big (more than 10MB).
// - if response body is streamed from slow external sources.
// - if response body must be streamed to the client in chunks
// (aka `http server push` or `chunked transfer-encoding`).
//
// See also SetBodyStream.
@@ -831,11 +831,11 @@ func (resp *Response) resetSkipHeader() {
//
// If MayContinue returns true, the caller must:
//
// - Either send StatusExpectationFailed response if request headers don't
// satisfy the caller.
// - Or send StatusContinue response before reading request body
// with ContinueReadBody.
// - Or close the connection.
// - Either send StatusExpectationFailed response if request headers don't
// satisfy the caller.
// - Or send StatusContinue response before reading request body
// with ContinueReadBody.
// - Or close the connection.
//
// io.EOF is returned if r is closed before reading the first header byte.
func (req *Request) Read(r *bufio.Reader) error {
@@ -857,11 +857,11 @@ var errGetOnly = errors.New("non-GET request received")
//
// If MayContinue returns true, the caller must:
//
// - Either send StatusExpectationFailed response if request headers don't
// satisfy the caller.
// - Or send StatusContinue response before reading request body
// with ContinueReadBody.
// - Or close the connection.
// - Either send StatusExpectationFailed response if request headers don't
// satisfy the caller.
// - Or send StatusContinue response before reading request body
// with ContinueReadBody.
// - Or close the connection.
//
// io.EOF is returned if r is closed before reading the first header byte.
func (req *Request) ReadLimitBody(r *bufio.Reader, maxBodySize int) error {
@@ -900,11 +900,11 @@ func (req *Request) readLimitBody(r *bufio.Reader, maxBodySize int, getOnly bool
//
// The caller must do one of the following actions if MayContinue returns true:
//
// - Either send StatusExpectationFailed response if request headers don't
// satisfy the caller.
// - Or send StatusContinue response before reading request body
// with ContinueReadBody.
// - Or close the connection.
// - Either send StatusExpectationFailed response if request headers don't
// satisfy the caller.
// - Or send StatusContinue response before reading request body
// with ContinueReadBody.
// - Or close the connection.
func (req *Request) MayContinue() bool {
return bytes.Equal(req.Header.peek(strExpect), str100Continue)
}
@@ -1138,11 +1138,11 @@ func (resp *Response) WriteGzip(w *bufio.Writer) error {
//
// Level is the desired compression level:
//
// * CompressNoCompression
// * CompressBestSpeed
// * CompressBestCompression
// * CompressDefaultCompression
// * CompressHuffmanOnly
// - CompressNoCompression
// - CompressBestSpeed
// - CompressBestCompression
// - CompressDefaultCompression
// - CompressHuffmanOnly
//
// The method gzips response body and sets 'Content-Encoding: gzip'
// header before writing response to w.
@@ -1169,11 +1169,11 @@ func (resp *Response) WriteDeflate(w *bufio.Writer) error {
//
// Level is the desired compression level:
//
// * CompressNoCompression
// * CompressBestSpeed
// * CompressBestCompression
// * CompressDefaultCompression
// * CompressHuffmanOnly
// - CompressNoCompression
// - CompressBestSpeed
// - CompressBestCompression
// - CompressDefaultCompression
// - CompressHuffmanOnly
//
// The method deflates response body and sets 'Content-Encoding: deflate'
// header before writing response to w.

View File

@@ -318,11 +318,11 @@ func CompressHandler(h RequestHandler) RequestHandler {
//
// Level is the desired compression level:
//
// * CompressNoCompression
// * CompressBestSpeed
// * CompressBestCompression
// * CompressDefaultCompression
// * CompressHuffmanOnly
// - CompressNoCompression
// - CompressBestSpeed
// - CompressBestCompression
// - CompressDefaultCompression
// - CompressHuffmanOnly
func CompressHandlerLevel(h RequestHandler, level int) RequestHandler {
return func(ctx *RequestCtx) {
h(ctx)
@@ -405,8 +405,8 @@ type HijackHandler func(c net.Conn)
//
// The server skips calling the handler in the following cases:
//
// * 'Connection: close' header exists in either request or response.
// * Unexpected error during response writing to the connection.
// - 'Connection: close' header exists in either request or response.
// - Unexpected error during response writing to the connection.
//
// The server stops processing requests from hijacked connections.
// Server limits such as Concurrency, ReadTimeout, WriteTimeout, etc.
@@ -417,9 +417,8 @@ type HijackHandler func(c net.Conn)
// Arbitrary 'Connection: Upgrade' protocols may be implemented
// with HijackHandler. For instance,
//
// * WebSocket ( https://en.wikipedia.org/wiki/WebSocket )
// * HTTP/2.0 ( https://en.wikipedia.org/wiki/HTTP/2 )
//
// - WebSocket ( https://en.wikipedia.org/wiki/WebSocket )
// - HTTP/2.0 ( https://en.wikipedia.org/wiki/HTTP/2 )
func (ctx *RequestCtx) Hijack(handler HijackHandler) {
ctx.hijackHandler = handler
}
@@ -759,15 +758,15 @@ func SaveMultipartFile(fh *multipart.FileHeader, path string) error {
//
// The value is searched in the following places:
//
// * Query string.
// * POST or PUT body.
// - Query string.
// - POST or PUT body.
//
// There are more fine-grained methods for obtaining form values:
//
// * QueryArgs for obtaining values from query string.
// * PostArgs for obtaining values from POST or PUT body.
// * MultipartForm for obtaining values from multipart form.
// * FormFile for obtaining uploaded files.
// - QueryArgs for obtaining values from query string.
// - PostArgs for obtaining values from POST or PUT body.
// - MultipartForm for obtaining values from multipart form.
// - FormFile for obtaining uploaded files.
//
// The returned value is valid until returning from RequestHandler.
func (ctx *RequestCtx) FormValue(key string) []byte {
@@ -896,10 +895,10 @@ func (ctx *RequestCtx) SuccessString(contentType, body string) {
//
// statusCode must have one of the following values:
//
// * StatusMovedPermanently (301)
// * StatusFound (302)
// * StatusSeeOther (303)
// * StatusTemporaryRedirect (307)
// - StatusMovedPermanently (301)
// - StatusFound (302)
// - StatusSeeOther (303)
// - StatusTemporaryRedirect (307)
//
// All other statusCode values are replaced by StatusFound (302).
//
@@ -918,10 +917,10 @@ func (ctx *RequestCtx) Redirect(uri string, statusCode int) {
//
// statusCode must have one of the following values:
//
// * StatusMovedPermanently (301)
// * StatusFound (302)
// * StatusSeeOther (303)
// * StatusTemporaryRedirect (307)
// - StatusMovedPermanently (301)
// - StatusFound (302)
// - StatusSeeOther (303)
// - StatusTemporaryRedirect (307)
//
// All other statusCode values are replaced by StatusFound (302).
//
@@ -1056,9 +1055,9 @@ func (ctx *RequestCtx) SetBodyStream(bodyStream io.Reader, bodySize int) {
//
// This function may be used in the following cases:
//
// * if response body is too big (more than 10MB).
// * if response body is streamed from slow external sources.
// * if response body must be streamed to the client in chunks.
// - if response body is too big (more than 10MB).
// - if response body is streamed from slow external sources.
// - if response body must be streamed to the client in chunks.
// (aka `http server push`).
func (ctx *RequestCtx) SetBodyStreamWriter(sw StreamWriter) {
ctx.Response.SetBodyStreamWriter(sw)

View File

@@ -13,12 +13,12 @@ import (
//
// This function has the following additional features comparing to net.Dial:
//
// * It reduces load on DNS resolver by caching resolved TCP addressed
// - It reduces load on DNS resolver by caching resolved TCP addressed
// for DefaultDNSCacheDuration.
// * It dials all the resolved TCP addresses in round-robin manner until
// - It dials all the resolved TCP addresses in round-robin manner until
// connection is established. This may be useful if certain addresses
// are temporarily unreachable.
// * It returns ErrDialTimeout if connection cannot be established during
// - It returns ErrDialTimeout if connection cannot be established during
// DefaultDialTimeout seconds. Use DialTimeout for customizing dial timeout.
//
// This dialer is intended for custom code wrapping before passing
@@ -29,9 +29,9 @@ import (
//
// The addr passed to the function must contain port. Example addr values:
//
// * foobar.baz:443
// * foo.bar:80
// * aaa.com:8080
// - foobar.baz:443
// - foo.bar:80
// - aaa.com:8080
func Dial(addr string) (net.Conn, error) {
return getDialer(DefaultDialTimeout, false)(addr)
}
@@ -40,9 +40,9 @@ func Dial(addr string) (net.Conn, error) {
//
// This function has the following additional features comparing to net.Dial:
//
// * It reduces load on DNS resolver by caching resolved TCP addressed
// - It reduces load on DNS resolver by caching resolved TCP addressed
// for DefaultDNSCacheDuration.
// * It dials all the resolved TCP addresses in round-robin manner until
// - It dials all the resolved TCP addresses in round-robin manner until
// connection is established. This may be useful if certain addresses
// are temporarily unreachable.
//
@@ -54,9 +54,9 @@ func Dial(addr string) (net.Conn, error) {
//
// The addr passed to the function must contain port. Example addr values:
//
// * foobar.baz:443
// * foo.bar:80
// * aaa.com:8080
// - foobar.baz:443
// - foo.bar:80
// - aaa.com:8080
func DialTimeout(addr string, timeout time.Duration) (net.Conn, error) {
return getDialer(timeout, false)(addr)
}
@@ -65,12 +65,12 @@ func DialTimeout(addr string, timeout time.Duration) (net.Conn, error) {
//
// This function has the following additional features comparing to net.Dial:
//
// * It reduces load on DNS resolver by caching resolved TCP addressed
// - It reduces load on DNS resolver by caching resolved TCP addressed
// for DefaultDNSCacheDuration.
// * It dials all the resolved TCP addresses in round-robin manner until
// - It dials all the resolved TCP addresses in round-robin manner until
// connection is established. This may be useful if certain addresses
// are temporarily unreachable.
// * It returns ErrDialTimeout if connection cannot be established during
// - It returns ErrDialTimeout if connection cannot be established during
// DefaultDialTimeout seconds. Use DialDualStackTimeout for custom dial
// timeout.
//
@@ -82,9 +82,9 @@ func DialTimeout(addr string, timeout time.Duration) (net.Conn, error) {
//
// The addr passed to the function must contain port. Example addr values:
//
// * foobar.baz:443
// * foo.bar:80
// * aaa.com:8080
// - foobar.baz:443
// - foo.bar:80
// - aaa.com:8080
func DialDualStack(addr string) (net.Conn, error) {
return getDialer(DefaultDialTimeout, true)(addr)
}
@@ -94,9 +94,9 @@ func DialDualStack(addr string) (net.Conn, error) {
//
// This function has the following additional features comparing to net.Dial:
//
// * It reduces load on DNS resolver by caching resolved TCP addressed
// - It reduces load on DNS resolver by caching resolved TCP addressed
// for DefaultDNSCacheDuration.
// * It dials all the resolved TCP addresses in round-robin manner until
// - It dials all the resolved TCP addresses in round-robin manner until
// connection is established. This may be useful if certain addresses
// are temporarily unreachable.
//
@@ -108,9 +108,9 @@ func DialDualStack(addr string) (net.Conn, error) {
//
// The addr passed to the function must contain port. Example addr values:
//
// * foobar.baz:443
// * foo.bar:80
// * aaa.com:8080
// - foobar.baz:443
// - foo.bar:80
// - aaa.com:8080
func DialDualStackTimeout(addr string, timeout time.Duration) (net.Conn, error) {
return getDialer(timeout, true)(addr)
}

View File

@@ -356,9 +356,9 @@ func (u *URI) RequestURI() []byte {
//
// Examples:
//
// * For /foo/bar/baz.html path returns baz.html.
// * For /foo/bar/ returns empty byte slice.
// * For /foobar.js returns foobar.js.
// - For /foo/bar/baz.html path returns baz.html.
// - For /foo/bar/ returns empty byte slice.
// - For /foobar.js returns foobar.js.
func (u *URI) LastPathSegment() []byte {
path := u.Path()
n := bytes.LastIndexByte(path, '/')
@@ -372,14 +372,14 @@ func (u *URI) LastPathSegment() []byte {
//
// The following newURI types are accepted:
//
// * Absolute, i.e. http://foobar.com/aaa/bb?cc . In this case the original
// uri is replaced by newURI.
// * Absolute without scheme, i.e. //foobar.com/aaa/bb?cc. In this case
// the original scheme is preserved.
// * Missing host, i.e. /aaa/bb?cc . In this case only RequestURI part
// of the original uri is replaced.
// * Relative path, i.e. xx?yy=abc . In this case the original RequestURI
// is updated according to the new relative path.
// - Absolute, i.e. http://foobar.com/aaa/bb?cc . In this case the original
// uri is replaced by newURI.
// - Absolute without scheme, i.e. //foobar.com/aaa/bb?cc. In this case
// the original scheme is preserved.
// - Missing host, i.e. /aaa/bb?cc . In this case only RequestURI part
// of the original uri is replaced.
// - Relative path, i.e. xx?yy=abc . In this case the original RequestURI
// is updated according to the new relative path.
func (u *URI) Update(newURI string) {
u.UpdateBytes(s2b(newURI))
}
@@ -388,14 +388,14 @@ func (u *URI) Update(newURI string) {
//
// The following newURI types are accepted:
//
// * Absolute, i.e. http://foobar.com/aaa/bb?cc . In this case the original
// uri is replaced by newURI.
// * Absolute without scheme, i.e. //foobar.com/aaa/bb?cc. In this case
// the original scheme is preserved.
// * Missing host, i.e. /aaa/bb?cc . In this case only RequestURI part
// of the original uri is replaced.
// * Relative path, i.e. xx?yy=abc . In this case the original RequestURI
// is updated according to the new relative path.
// - Absolute, i.e. http://foobar.com/aaa/bb?cc . In this case the original
// uri is replaced by newURI.
// - Absolute without scheme, i.e. //foobar.com/aaa/bb?cc. In this case
// the original scheme is preserved.
// - Missing host, i.e. /aaa/bb?cc . In this case only RequestURI part
// of the original uri is replaced.
// - Relative path, i.e. xx?yy=abc . In this case the original RequestURI
// is updated according to the new relative path.
func (u *URI) UpdateBytes(newURI []byte) {
u.requestURI = u.updateBytes(newURI, u.requestURI)
}

View File

@@ -1,3 +1,4 @@
//go:build !windows
// +build !windows
package fasthttp

View File

@@ -1,3 +1,4 @@
//go:build windows
// +build windows
package fasthttp

View File

@@ -29,6 +29,7 @@ var aggrFuncs = map[string]bool{
"outliersk": true,
"quantile": true,
"quantiles": true,
"share": true,
"stddev": true,
"stdvar": true,
"sum": true,

View File

@@ -392,7 +392,8 @@ func getTransformArgIdxForOptimization(funcName string, args []Expr) int {
return -1
case "limit_offset":
return 2
case "buckets_limit", "histogram_quantile", "histogram_share", "range_quantile", "range_trim_spikes":
case "buckets_limit", "histogram_quantile", "histogram_share", "range_quantile",
"range_trim_outliers", "range_trim_spikes", "range_trim_zscore":
return 1
case "histogram_quantiles":
return len(args) - 1

View File

@@ -42,6 +42,7 @@ var rollupFuncs = map[string]bool{
"lag": true,
"last_over_time": true,
"lifetime": true,
"mad_over_time": true,
"max_over_time": true,
"min_over_time": true,
"mode_over_time": true,

View File

@@ -74,6 +74,7 @@ var transformFuncs = map[string]bool{
"range_first": true,
"range_last": true,
"range_linear_regression": true,
"range_mad": true,
"range_max": true,
"range_min": true,
"range_normalize": true,
@@ -81,7 +82,10 @@ var transformFuncs = map[string]bool{
"range_stddev": true,
"range_stdvar": true,
"range_sum": true,
"range_trim_outliers": true,
"range_trim_spikes": true,
"range_trim_zscore": true,
"range_zscore": true,
"remove_resets": true,
"round": true,
"running_avg": true,

View File

@@ -1,3 +1,7 @@
# v1.18.13 (2023-02-15)
* **Dependency Update**: Updated to the latest SDK module versions
# v1.18.12 (2023-02-03)
* **Dependency Update**: Updated to the latest SDK module versions

View File

@@ -3,4 +3,4 @@
package config
// goModuleVersion is the tagged release for this module
const goModuleVersion = "1.18.12"
const goModuleVersion = "1.18.13"

View File

@@ -1,3 +1,7 @@
# v1.13.13 (2023-02-15)
* **Dependency Update**: Updated to the latest SDK module versions
# v1.13.12 (2023-02-03)
* **Dependency Update**: Updated to the latest SDK module versions

View File

@@ -3,4 +3,4 @@
package credentials
// goModuleVersion is the tagged release for this module
const goModuleVersion = "1.13.12"
const goModuleVersion = "1.13.13"

View File

@@ -1,3 +1,11 @@
# v1.11.53 (2023-02-15)
* **Dependency Update**: Updated to the latest SDK module versions
# v1.11.52 (2023-02-14)
* **Dependency Update**: Updated to the latest SDK module versions
# v1.11.51 (2023-02-03)
* **Dependency Update**: Updated to the latest SDK module versions

View File

@@ -3,4 +3,4 @@
package manager
// goModuleVersion is the tagged release for this module
const goModuleVersion = "1.11.51"
const goModuleVersion = "1.11.53"

View File

@@ -1,3 +1,7 @@
# v1.0.20 (2023-02-14)
* No change notes available for this release.
# v1.0.19 (2023-02-03)
* **Dependency Update**: Updated to the latest SDK module versions

View File

@@ -3,4 +3,4 @@
package v4a
// goModuleVersion is the tagged release for this module
const goModuleVersion = "1.0.19"
const goModuleVersion = "1.0.20"

View File

@@ -1,3 +1,7 @@
# v1.30.3 (2023-02-14)
* **Dependency Update**: Updated to the latest SDK module versions
# v1.30.2 (2023-02-03)
* **Dependency Update**: Updated to the latest SDK module versions

View File

@@ -3,4 +3,4 @@
package s3
// goModuleVersion is the tagged release for this module
const goModuleVersion = "1.30.2"
const goModuleVersion = "1.30.3"

View File

@@ -1,3 +1,8 @@
# v1.12.2 (2023-02-15)
* **Announcement**: When receiving an error response in restJson-based services, an incorrect error type may have been returned based on the content of the response. This has been fixed via PR #2012 tracked in issue #1910.
* **Bug Fix**: Correct error type parsing for restJson services.
# v1.12.1 (2023-02-03)
* **Dependency Update**: Updated to the latest SDK module versions

View File

@@ -86,9 +86,9 @@ func awsRestjson1_deserializeOpErrorGetRoleCredentials(response *smithyhttp.Resp
errorCode := "UnknownError"
errorMessage := errorCode
code := response.Header.Get("X-Amzn-ErrorType")
if len(code) != 0 {
errorCode = restjson.SanitizeErrorCode(code)
headerCode := response.Header.Get("X-Amzn-ErrorType")
if len(headerCode) != 0 {
errorCode = restjson.SanitizeErrorCode(headerCode)
}
var buff [1024]byte
@@ -97,7 +97,7 @@ func awsRestjson1_deserializeOpErrorGetRoleCredentials(response *smithyhttp.Resp
body := io.TeeReader(errorBody, ringBuffer)
decoder := json.NewDecoder(body)
decoder.UseNumber()
code, message, err := restjson.GetErrorInfo(decoder)
jsonCode, message, err := restjson.GetErrorInfo(decoder)
if err != nil {
var snapshot bytes.Buffer
io.Copy(&snapshot, ringBuffer)
@@ -109,8 +109,8 @@ func awsRestjson1_deserializeOpErrorGetRoleCredentials(response *smithyhttp.Resp
}
errorBody.Seek(0, io.SeekStart)
if len(code) != 0 {
errorCode = restjson.SanitizeErrorCode(code)
if len(headerCode) == 0 && len(jsonCode) != 0 {
errorCode = restjson.SanitizeErrorCode(jsonCode)
}
if len(message) != 0 {
errorMessage = message
@@ -242,9 +242,9 @@ func awsRestjson1_deserializeOpErrorListAccountRoles(response *smithyhttp.Respon
errorCode := "UnknownError"
errorMessage := errorCode
code := response.Header.Get("X-Amzn-ErrorType")
if len(code) != 0 {
errorCode = restjson.SanitizeErrorCode(code)
headerCode := response.Header.Get("X-Amzn-ErrorType")
if len(headerCode) != 0 {
errorCode = restjson.SanitizeErrorCode(headerCode)
}
var buff [1024]byte
@@ -253,7 +253,7 @@ func awsRestjson1_deserializeOpErrorListAccountRoles(response *smithyhttp.Respon
body := io.TeeReader(errorBody, ringBuffer)
decoder := json.NewDecoder(body)
decoder.UseNumber()
code, message, err := restjson.GetErrorInfo(decoder)
jsonCode, message, err := restjson.GetErrorInfo(decoder)
if err != nil {
var snapshot bytes.Buffer
io.Copy(&snapshot, ringBuffer)
@@ -265,8 +265,8 @@ func awsRestjson1_deserializeOpErrorListAccountRoles(response *smithyhttp.Respon
}
errorBody.Seek(0, io.SeekStart)
if len(code) != 0 {
errorCode = restjson.SanitizeErrorCode(code)
if len(headerCode) == 0 && len(jsonCode) != 0 {
errorCode = restjson.SanitizeErrorCode(jsonCode)
}
if len(message) != 0 {
errorMessage = message
@@ -407,9 +407,9 @@ func awsRestjson1_deserializeOpErrorListAccounts(response *smithyhttp.Response,
errorCode := "UnknownError"
errorMessage := errorCode
code := response.Header.Get("X-Amzn-ErrorType")
if len(code) != 0 {
errorCode = restjson.SanitizeErrorCode(code)
headerCode := response.Header.Get("X-Amzn-ErrorType")
if len(headerCode) != 0 {
errorCode = restjson.SanitizeErrorCode(headerCode)
}
var buff [1024]byte
@@ -418,7 +418,7 @@ func awsRestjson1_deserializeOpErrorListAccounts(response *smithyhttp.Response,
body := io.TeeReader(errorBody, ringBuffer)
decoder := json.NewDecoder(body)
decoder.UseNumber()
code, message, err := restjson.GetErrorInfo(decoder)
jsonCode, message, err := restjson.GetErrorInfo(decoder)
if err != nil {
var snapshot bytes.Buffer
io.Copy(&snapshot, ringBuffer)
@@ -430,8 +430,8 @@ func awsRestjson1_deserializeOpErrorListAccounts(response *smithyhttp.Response,
}
errorBody.Seek(0, io.SeekStart)
if len(code) != 0 {
errorCode = restjson.SanitizeErrorCode(code)
if len(headerCode) == 0 && len(jsonCode) != 0 {
errorCode = restjson.SanitizeErrorCode(jsonCode)
}
if len(message) != 0 {
errorMessage = message
@@ -550,9 +550,9 @@ func awsRestjson1_deserializeOpErrorLogout(response *smithyhttp.Response, metada
errorCode := "UnknownError"
errorMessage := errorCode
code := response.Header.Get("X-Amzn-ErrorType")
if len(code) != 0 {
errorCode = restjson.SanitizeErrorCode(code)
headerCode := response.Header.Get("X-Amzn-ErrorType")
if len(headerCode) != 0 {
errorCode = restjson.SanitizeErrorCode(headerCode)
}
var buff [1024]byte
@@ -561,7 +561,7 @@ func awsRestjson1_deserializeOpErrorLogout(response *smithyhttp.Response, metada
body := io.TeeReader(errorBody, ringBuffer)
decoder := json.NewDecoder(body)
decoder.UseNumber()
code, message, err := restjson.GetErrorInfo(decoder)
jsonCode, message, err := restjson.GetErrorInfo(decoder)
if err != nil {
var snapshot bytes.Buffer
io.Copy(&snapshot, ringBuffer)
@@ -573,8 +573,8 @@ func awsRestjson1_deserializeOpErrorLogout(response *smithyhttp.Response, metada
}
errorBody.Seek(0, io.SeekStart)
if len(code) != 0 {
errorCode = restjson.SanitizeErrorCode(code)
if len(headerCode) == 0 && len(jsonCode) != 0 {
errorCode = restjson.SanitizeErrorCode(jsonCode)
}
if len(message) != 0 {
errorMessage = message

View File

@@ -3,4 +3,4 @@
package sso
// goModuleVersion is the tagged release for this module
const goModuleVersion = "1.12.1"
const goModuleVersion = "1.12.2"

View File

@@ -1,3 +1,8 @@
# v1.14.2 (2023-02-15)
* **Announcement**: When receiving an error response in restJson-based services, an incorrect error type may have been returned based on the content of the response. This has been fixed via PR #2012 tracked in issue #1910.
* **Bug Fix**: Correct error type parsing for restJson services.
# v1.14.1 (2023-02-03)
* **Dependency Update**: Updated to the latest SDK module versions

View File

@@ -85,9 +85,9 @@ func awsRestjson1_deserializeOpErrorCreateToken(response *smithyhttp.Response, m
errorCode := "UnknownError"
errorMessage := errorCode
code := response.Header.Get("X-Amzn-ErrorType")
if len(code) != 0 {
errorCode = restjson.SanitizeErrorCode(code)
headerCode := response.Header.Get("X-Amzn-ErrorType")
if len(headerCode) != 0 {
errorCode = restjson.SanitizeErrorCode(headerCode)
}
var buff [1024]byte
@@ -96,7 +96,7 @@ func awsRestjson1_deserializeOpErrorCreateToken(response *smithyhttp.Response, m
body := io.TeeReader(errorBody, ringBuffer)
decoder := json.NewDecoder(body)
decoder.UseNumber()
code, message, err := restjson.GetErrorInfo(decoder)
jsonCode, message, err := restjson.GetErrorInfo(decoder)
if err != nil {
var snapshot bytes.Buffer
io.Copy(&snapshot, ringBuffer)
@@ -108,8 +108,8 @@ func awsRestjson1_deserializeOpErrorCreateToken(response *smithyhttp.Response, m
}
errorBody.Seek(0, io.SeekStart)
if len(code) != 0 {
errorCode = restjson.SanitizeErrorCode(code)
if len(headerCode) == 0 && len(jsonCode) != 0 {
errorCode = restjson.SanitizeErrorCode(jsonCode)
}
if len(message) != 0 {
errorMessage = message
@@ -306,9 +306,9 @@ func awsRestjson1_deserializeOpErrorRegisterClient(response *smithyhttp.Response
errorCode := "UnknownError"
errorMessage := errorCode
code := response.Header.Get("X-Amzn-ErrorType")
if len(code) != 0 {
errorCode = restjson.SanitizeErrorCode(code)
headerCode := response.Header.Get("X-Amzn-ErrorType")
if len(headerCode) != 0 {
errorCode = restjson.SanitizeErrorCode(headerCode)
}
var buff [1024]byte
@@ -317,7 +317,7 @@ func awsRestjson1_deserializeOpErrorRegisterClient(response *smithyhttp.Response
body := io.TeeReader(errorBody, ringBuffer)
decoder := json.NewDecoder(body)
decoder.UseNumber()
code, message, err := restjson.GetErrorInfo(decoder)
jsonCode, message, err := restjson.GetErrorInfo(decoder)
if err != nil {
var snapshot bytes.Buffer
io.Copy(&snapshot, ringBuffer)
@@ -329,8 +329,8 @@ func awsRestjson1_deserializeOpErrorRegisterClient(response *smithyhttp.Response
}
errorBody.Seek(0, io.SeekStart)
if len(code) != 0 {
errorCode = restjson.SanitizeErrorCode(code)
if len(headerCode) == 0 && len(jsonCode) != 0 {
errorCode = restjson.SanitizeErrorCode(jsonCode)
}
if len(message) != 0 {
errorMessage = message
@@ -519,9 +519,9 @@ func awsRestjson1_deserializeOpErrorStartDeviceAuthorization(response *smithyhtt
errorCode := "UnknownError"
errorMessage := errorCode
code := response.Header.Get("X-Amzn-ErrorType")
if len(code) != 0 {
errorCode = restjson.SanitizeErrorCode(code)
headerCode := response.Header.Get("X-Amzn-ErrorType")
if len(headerCode) != 0 {
errorCode = restjson.SanitizeErrorCode(headerCode)
}
var buff [1024]byte
@@ -530,7 +530,7 @@ func awsRestjson1_deserializeOpErrorStartDeviceAuthorization(response *smithyhtt
body := io.TeeReader(errorBody, ringBuffer)
decoder := json.NewDecoder(body)
decoder.UseNumber()
code, message, err := restjson.GetErrorInfo(decoder)
jsonCode, message, err := restjson.GetErrorInfo(decoder)
if err != nil {
var snapshot bytes.Buffer
io.Copy(&snapshot, ringBuffer)
@@ -542,8 +542,8 @@ func awsRestjson1_deserializeOpErrorStartDeviceAuthorization(response *smithyhtt
}
errorBody.Seek(0, io.SeekStart)
if len(code) != 0 {
errorCode = restjson.SanitizeErrorCode(code)
if len(headerCode) == 0 && len(jsonCode) != 0 {
errorCode = restjson.SanitizeErrorCode(jsonCode)
}
if len(message) != 0 {
errorMessage = message

View File

@@ -3,4 +3,4 @@
package ssooidc
// goModuleVersion is the tagged release for this module
const goModuleVersion = "1.14.1"
const goModuleVersion = "1.14.2"

View File

@@ -226,12 +226,24 @@ func NewCredentialsCommand(command *exec.Cmd, options ...func(*ProcessProvider))
return credentials.NewCredentials(p)
}
type credentialProcessResponse struct {
Version int
AccessKeyID string `json:"AccessKeyId"`
// A CredentialProcessResponse is the AWS credentials format that must be
// returned when executing an external credential_process.
type CredentialProcessResponse struct {
// As of this writing, the Version key must be set to 1. This might
// increment over time as the structure evolves.
Version int
// The access key ID that identifies the temporary security credentials.
AccessKeyID string `json:"AccessKeyId"`
// The secret access key that can be used to sign requests.
SecretAccessKey string
SessionToken string
Expiration *time.Time
// The token that users must pass to the service API to use the temporary credentials.
SessionToken string
// The date on which the current credentials expire.
Expiration *time.Time
}
// Retrieve executes the 'credential_process' and returns the credentials.
@@ -242,7 +254,7 @@ func (p *ProcessProvider) Retrieve() (credentials.Value, error) {
}
// Serialize and validate response
resp := &credentialProcessResponse{}
resp := &CredentialProcessResponse{}
if err = json.Unmarshal(out, resp); err != nil {
return credentials.Value{ProviderName: ProviderName}, awserr.New(
ErrCodeProcessProviderParse,

View File

@@ -2605,6 +2605,9 @@ var awsPartition = partition{
endpointKey{
Region: "ap-southeast-2",
}: endpoint{},
endpointKey{
Region: "ap-southeast-3",
}: endpoint{},
endpointKey{
Region: "ca-central-1",
}: endpoint{},
@@ -4060,6 +4063,21 @@ var awsPartition = partition{
},
"cases": service{
Endpoints: serviceEndpoints{
endpointKey{
Region: "ap-southeast-1",
}: endpoint{},
endpointKey{
Region: "ap-southeast-2",
}: endpoint{},
endpointKey{
Region: "ca-central-1",
}: endpoint{},
endpointKey{
Region: "eu-central-1",
}: endpoint{},
endpointKey{
Region: "eu-west-2",
}: endpoint{},
endpointKey{
Region: "fips-us-east-1",
}: endpoint{
@@ -6588,6 +6606,9 @@ var awsPartition = partition{
endpointKey{
Region: "ap-southeast-2",
}: endpoint{},
endpointKey{
Region: "ca-central-1",
}: endpoint{},
endpointKey{
Region: "eu-west-2",
}: endpoint{},
@@ -9523,6 +9544,15 @@ var awsPartition = partition{
}: endpoint{
Hostname: "elasticfilesystem-fips.ap-south-1.amazonaws.com",
},
endpointKey{
Region: "ap-south-2",
}: endpoint{},
endpointKey{
Region: "ap-south-2",
Variant: fipsVariant,
}: endpoint{
Hostname: "elasticfilesystem-fips.ap-south-2.amazonaws.com",
},
endpointKey{
Region: "ap-southeast-1",
}: endpoint{},
@@ -9685,6 +9715,15 @@ var awsPartition = partition{
},
Deprecated: boxedTrue,
},
endpointKey{
Region: "fips-ap-south-2",
}: endpoint{
Hostname: "elasticfilesystem-fips.ap-south-2.amazonaws.com",
CredentialScope: credentialScope{
Region: "ap-south-2",
},
Deprecated: boxedTrue,
},
endpointKey{
Region: "fips-ap-southeast-1",
}: endpoint{
@@ -12633,6 +12672,9 @@ var awsPartition = partition{
endpointKey{
Region: "ap-south-1",
}: endpoint{},
endpointKey{
Region: "ap-south-2",
}: endpoint{},
endpointKey{
Region: "ap-southeast-1",
}: endpoint{},
@@ -14319,6 +14361,11 @@ var awsPartition = partition{
}: endpoint{
Hostname: "kendra-ranking.ap-east-1.api.aws",
},
endpointKey{
Region: "ap-northeast-1",
}: endpoint{
Hostname: "kendra-ranking.ap-northeast-1.api.aws",
},
endpointKey{
Region: "ap-northeast-2",
}: endpoint{
@@ -14329,6 +14376,11 @@ var awsPartition = partition{
}: endpoint{
Hostname: "kendra-ranking.ap-northeast-3.api.aws",
},
endpointKey{
Region: "ap-south-1",
}: endpoint{
Hostname: "kendra-ranking.ap-south-1.api.aws",
},
endpointKey{
Region: "ap-south-2",
}: endpoint{
@@ -19755,6 +19807,9 @@ var awsPartition = partition{
endpointKey{
Region: "ap-southeast-3",
}: endpoint{},
endpointKey{
Region: "ap-southeast-4",
}: endpoint{},
endpointKey{
Region: "ca-central-1",
}: endpoint{},
@@ -30306,6 +30361,21 @@ var awscnPartition = partition{
},
},
},
"servicequotas": service{
Defaults: endpointDefaults{
defaultKey{}: endpoint{
Protocols: []string{"https"},
},
},
Endpoints: serviceEndpoints{
endpointKey{
Region: "cn-north-1",
}: endpoint{},
endpointKey{
Region: "cn-northwest-1",
}: endpoint{},
},
},
"sms": service{
Endpoints: serviceEndpoints{
endpointKey{
@@ -32471,6 +32541,24 @@ var awsusgovPartition = partition{
Region: "us-gov-east-1",
},
},
endpointKey{
Region: "us-gov-east-1",
Variant: fipsVariant,
}: endpoint{
Hostname: "elasticbeanstalk.us-gov-east-1.amazonaws.com",
CredentialScope: credentialScope{
Region: "us-gov-east-1",
},
},
endpointKey{
Region: "us-gov-east-1-fips",
}: endpoint{
Hostname: "elasticbeanstalk.us-gov-east-1.amazonaws.com",
CredentialScope: credentialScope{
Region: "us-gov-east-1",
},
Deprecated: boxedTrue,
},
endpointKey{
Region: "us-gov-west-1",
}: endpoint{
@@ -32479,6 +32567,24 @@ var awsusgovPartition = partition{
Region: "us-gov-west-1",
},
},
endpointKey{
Region: "us-gov-west-1",
Variant: fipsVariant,
}: endpoint{
Hostname: "elasticbeanstalk.us-gov-west-1.amazonaws.com",
CredentialScope: credentialScope{
Region: "us-gov-west-1",
},
},
endpointKey{
Region: "us-gov-west-1-fips",
}: endpoint{
Hostname: "elasticbeanstalk.us-gov-west-1.amazonaws.com",
CredentialScope: credentialScope{
Region: "us-gov-west-1",
},
Deprecated: boxedTrue,
},
},
},
"elasticfilesystem": service{
@@ -32916,21 +33022,43 @@ var awsusgovPartition = partition{
"glacier": service{
Endpoints: serviceEndpoints{
endpointKey{
Region: "us-gov-east-1",
Region: "fips-us-gov-east-1",
}: endpoint{
Hostname: "glacier.us-gov-east-1.amazonaws.com",
CredentialScope: credentialScope{
Region: "us-gov-east-1",
},
Deprecated: boxedTrue,
},
endpointKey{
Region: "fips-us-gov-west-1",
}: endpoint{
Hostname: "glacier.us-gov-west-1.amazonaws.com",
CredentialScope: credentialScope{
Region: "us-gov-west-1",
},
Deprecated: boxedTrue,
},
endpointKey{
Region: "us-gov-east-1",
}: endpoint{},
endpointKey{
Region: "us-gov-east-1",
Variant: fipsVariant,
}: endpoint{
Hostname: "glacier.us-gov-east-1.amazonaws.com",
},
endpointKey{
Region: "us-gov-west-1",
}: endpoint{
Protocols: []string{"http", "https"},
},
endpointKey{
Region: "us-gov-west-1",
Variant: fipsVariant,
}: endpoint{
Hostname: "glacier.us-gov-west-1.amazonaws.com",
Protocols: []string{"http", "https"},
CredentialScope: credentialScope{
Region: "us-gov-west-1",
},
},
},
},
@@ -34151,20 +34279,40 @@ var awsusgovPartition = partition{
"outposts": service{
Endpoints: serviceEndpoints{
endpointKey{
Region: "us-gov-east-1",
Region: "fips-us-gov-east-1",
}: endpoint{
Hostname: "outposts.us-gov-east-1.amazonaws.com",
CredentialScope: credentialScope{
Region: "us-gov-east-1",
},
Deprecated: boxedTrue,
},
endpointKey{
Region: "us-gov-west-1",
Region: "fips-us-gov-west-1",
}: endpoint{
Hostname: "outposts.us-gov-west-1.amazonaws.com",
CredentialScope: credentialScope{
Region: "us-gov-west-1",
},
Deprecated: boxedTrue,
},
endpointKey{
Region: "us-gov-east-1",
}: endpoint{},
endpointKey{
Region: "us-gov-east-1",
Variant: fipsVariant,
}: endpoint{
Hostname: "outposts.us-gov-east-1.amazonaws.com",
},
endpointKey{
Region: "us-gov-west-1",
}: endpoint{},
endpointKey{
Region: "us-gov-west-1",
Variant: fipsVariant,
}: endpoint{
Hostname: "outposts.us-gov-west-1.amazonaws.com",
},
},
},
@@ -34679,6 +34827,9 @@ var awsusgovPartition = partition{
},
},
Endpoints: serviceEndpoints{
endpointKey{
Region: "us-gov-east-1",
}: endpoint{},
endpointKey{
Region: "us-gov-west-1",
}: endpoint{},

View File

@@ -5,4 +5,4 @@ package aws
const SDKName = "aws-sdk-go"
// SDKVersion is the version of this SDK
const SDKVersion = "1.44.198"
const SDKVersion = "1.44.204"

View File

@@ -25,8 +25,6 @@ import (
"errors"
"fmt"
"io"
"io/ioutil"
"log"
"net/rpc"
"os"
"os/exec"
@@ -54,17 +52,6 @@ func (c *Connection) Close() error {
return werr
}
// If ECP Logging is enabled return true
// Otherwise return false
func enableECPLogging() bool {
if os.Getenv("ENABLE_ENTERPRISE_CERTIFICATE_LOGS") != "" {
return true
}
log.SetOutput(ioutil.Discard)
return false
}
func init() {
gob.Register(crypto.SHA256)
gob.Register(&rsa.PSSOptions{})
@@ -97,7 +84,7 @@ func (k *Key) Close() error {
}
// Wait for cmd to exit and release resources. Since the process is forcefully killed, this
// will return a non-nil error (varies by OS), which we will ignore.
k.cmd.Wait()
_ = k.cmd.Wait()
// The Pipes connecting the RPC client should have been closed when the signer subprocess was killed.
// Calling `k.client.Close()` before `k.cmd.Process.Kill()` or `k.cmd.Wait()` _will_ cause a segfault.
if err := k.client.Close(); err.Error() != "close |0: file already closed" {
@@ -132,7 +119,6 @@ var ErrCredUnavailable = errors.New("Cred is unavailable")
//
// The config file also specifies which certificate the signer should use.
func Cred(configFilePath string) (*Key, error) {
enableECPLogging()
if configFilePath == "" {
configFilePath = util.GetDefaultConfigFilePath()
}

View File

@@ -17,7 +17,7 @@ package util
import (
"encoding/json"
"errors"
"io/ioutil"
"io"
"os"
"os/user"
"path/filepath"
@@ -50,7 +50,7 @@ func LoadSignerBinaryPath(configFilePath string) (path string, err error) {
return "", err
}
byteValue, err := ioutil.ReadAll(jsonFile)
byteValue, err := io.ReadAll(jsonFile)
if err != nil {
return "", err
}

View File

@@ -4,7 +4,7 @@ import (
"errors"
"flag"
"fmt"
"io/ioutil"
"io"
"os"
"regexp"
"runtime"
@@ -178,7 +178,7 @@ func flagSet(name string, flags []Flag, spec separatorSpec) (*flag.FlagSet, erro
return nil, err
}
}
set.SetOutput(ioutil.Discard)
set.SetOutput(io.Discard)
return set, nil
}
@@ -384,7 +384,7 @@ func flagFromEnvOrFile(envVars []string, filePath string) (value string, fromWhe
}
for _, fileVar := range strings.Split(filePath, ",") {
if fileVar != "" {
if data, err := ioutil.ReadFile(fileVar); err == nil {
if data, err := os.ReadFile(fileVar); err == nil {
return string(data), fmt.Sprintf("file %q", filePath), true
}
}

View File

@@ -3,8 +3,8 @@ GOARCH ?= $(shell go env GOARCH)
GOOS_GOARCH := $(GOOS)_$(GOARCH)
GOOS_GOARCH_NATIVE := $(shell go env GOHOSTOS)_$(shell go env GOHOSTARCH)
LIBZSTD_NAME := libzstd_$(GOOS_GOARCH).a
ZSTD_VERSION ?= v1.5.2
MUSL_BUILDER_IMAGE=golang:1.8.1-alpine
ZSTD_VERSION ?= v1.5.4
MUSL_BUILDER_IMAGE=golang:1.20.1-alpine
BUILDER_IMAGE := local/builder_musl:2.0.0-$(shell echo $(MUSL_BUILDER_IMAGE) | tr : _)-1
.PHONY: libzstd.a $(LIBZSTD_NAME)

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

View File

@@ -15,7 +15,12 @@ package gozstd
// See https://github.com/golang/go/issues/24450 .
static size_t ZSTD_initDStream_usingDDict_wrapper(uintptr_t ds, uintptr_t dict) {
return ZSTD_initDStream_usingDDict((ZSTD_DStream*)ds, (ZSTD_DDict*)dict);
ZSTD_DStream *zds = (ZSTD_DStream *)ds;
size_t rv = ZSTD_DCtx_reset(zds, ZSTD_reset_session_only);
if (rv != 0) {
return rv;
}
return ZSTD_DCtx_refDDict(zds, (ZSTD_DDict *)dict);
}
static size_t ZSTD_freeDStream_wrapper(uintptr_t ds) {

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) Yann Collet, Facebook, Inc.
* Copyright (c) Meta Platforms, Inc. and affiliates.
* All rights reserved.
*
* This source code is licensed under both the BSD-style license (found in the
@@ -8,32 +8,43 @@
* You may select, at your option, one of the above-listed licenses.
*/
#ifndef DICTBUILDER_H_001
#define DICTBUILDER_H_001
#if defined (__cplusplus)
extern "C" {
#endif
#ifndef ZSTD_ZDICT_H
#define ZSTD_ZDICT_H
/*====== Dependencies ======*/
#include <stddef.h> /* size_t */
/* ===== ZDICTLIB_API : control library symbols visibility ===== */
#ifndef ZDICTLIB_VISIBILITY
# if defined(__GNUC__) && (__GNUC__ >= 4)
# define ZDICTLIB_VISIBILITY __attribute__ ((visibility ("default")))
#ifndef ZDICTLIB_VISIBLE
/* Backwards compatibility with old macro name */
# ifdef ZDICTLIB_VISIBILITY
# define ZDICTLIB_VISIBLE ZDICTLIB_VISIBILITY
# elif defined(__GNUC__) && (__GNUC__ >= 4) && !defined(__MINGW32__)
# define ZDICTLIB_VISIBLE __attribute__ ((visibility ("default")))
# else
# define ZDICTLIB_VISIBILITY
# define ZDICTLIB_VISIBLE
# endif
#endif
#ifndef ZDICTLIB_HIDDEN
# if defined(__GNUC__) && (__GNUC__ >= 4) && !defined(__MINGW32__)
# define ZDICTLIB_HIDDEN __attribute__ ((visibility ("hidden")))
# else
# define ZDICTLIB_HIDDEN
# endif
#endif
#if defined(ZSTD_DLL_EXPORT) && (ZSTD_DLL_EXPORT==1)
# define ZDICTLIB_API __declspec(dllexport) ZDICTLIB_VISIBILITY
# define ZDICTLIB_API __declspec(dllexport) ZDICTLIB_VISIBLE
#elif defined(ZSTD_DLL_IMPORT) && (ZSTD_DLL_IMPORT==1)
# define ZDICTLIB_API __declspec(dllimport) ZDICTLIB_VISIBILITY /* It isn't required but allows to generate better code, saving a function pointer load from the IAT and an indirect jump.*/
# define ZDICTLIB_API __declspec(dllimport) ZDICTLIB_VISIBLE /* It isn't required but allows to generate better code, saving a function pointer load from the IAT and an indirect jump.*/
#else
# define ZDICTLIB_API ZDICTLIB_VISIBILITY
# define ZDICTLIB_API ZDICTLIB_VISIBLE
#endif
/*******************************************************************************
@@ -110,8 +121,8 @@ extern "C" {
* The zstd CLI defaults to a 110KB dictionary. You likely don't need a
* dictionary larger than that. But, most use cases can get away with a
* smaller dictionary. The advanced dictionary builders can automatically
* shrink the dictionary for you, and select a the smallest size that
* doesn't hurt compression ratio too much. See the `shrinkDict` parameter.
* shrink the dictionary for you, and select the smallest size that doesn't
* hurt compression ratio too much. See the `shrinkDict` parameter.
* A smaller dictionary can save memory, and potentially speed up
* compression.
*
@@ -201,9 +212,9 @@ ZDICTLIB_API size_t ZDICT_trainFromBuffer(void* dictBuffer, size_t dictBufferCap
const size_t* samplesSizes, unsigned nbSamples);
typedef struct {
int compressionLevel; /*< optimize for a specific zstd compression level; 0 means default */
unsigned notificationLevel; /*< Write log to stderr; 0 = none (default); 1 = errors; 2 = progression; 3 = details; 4 = debug; */
unsigned dictID; /*< force dictID value; 0 means auto mode (32-bits random value)
int compressionLevel; /**< optimize for a specific zstd compression level; 0 means default */
unsigned notificationLevel; /**< Write log to stderr; 0 = none (default); 1 = errors; 2 = progression; 3 = details; 4 = debug; */
unsigned dictID; /**< force dictID value; 0 means auto mode (32-bits random value)
* NOTE: The zstd format reserves some dictionary IDs for future use.
* You may use them in private settings, but be warned that they
* may be used by zstd in a public dictionary registry in the future.
@@ -260,9 +271,21 @@ ZDICTLIB_API size_t ZDICT_getDictHeaderSize(const void* dictBuffer, size_t dictS
ZDICTLIB_API unsigned ZDICT_isError(size_t errorCode);
ZDICTLIB_API const char* ZDICT_getErrorName(size_t errorCode);
#endif /* ZSTD_ZDICT_H */
#if defined(ZDICT_STATIC_LINKING_ONLY) && !defined(ZSTD_ZDICT_H_STATIC)
#define ZSTD_ZDICT_H_STATIC
#ifdef ZDICT_STATIC_LINKING_ONLY
/* This can be overridden externally to hide static symbols. */
#ifndef ZDICTLIB_STATIC_API
# if defined(ZSTD_DLL_EXPORT) && (ZSTD_DLL_EXPORT==1)
# define ZDICTLIB_STATIC_API __declspec(dllexport) ZDICTLIB_VISIBLE
# elif defined(ZSTD_DLL_IMPORT) && (ZSTD_DLL_IMPORT==1)
# define ZDICTLIB_STATIC_API __declspec(dllimport) ZDICTLIB_VISIBLE
# else
# define ZDICTLIB_STATIC_API ZDICTLIB_VISIBLE
# endif
#endif
/* ====================================================================================
* The definitions in this section are considered experimental.
@@ -318,7 +341,7 @@ typedef struct {
* In general, it's recommended to provide a few thousands samples, though this can vary a lot.
* It's recommended that total size of all samples be about ~x100 times the target size of dictionary.
*/
ZDICTLIB_API size_t ZDICT_trainFromBuffer_cover(
ZDICTLIB_STATIC_API size_t ZDICT_trainFromBuffer_cover(
void *dictBuffer, size_t dictBufferCapacity,
const void *samplesBuffer, const size_t *samplesSizes, unsigned nbSamples,
ZDICT_cover_params_t parameters);
@@ -340,7 +363,7 @@ ZDICTLIB_API size_t ZDICT_trainFromBuffer_cover(
* See ZDICT_trainFromBuffer() for details on failure modes.
* Note: ZDICT_optimizeTrainFromBuffer_cover() requires about 8 bytes of memory for each input byte and additionally another 5 bytes of memory for each byte of memory for each thread.
*/
ZDICTLIB_API size_t ZDICT_optimizeTrainFromBuffer_cover(
ZDICTLIB_STATIC_API size_t ZDICT_optimizeTrainFromBuffer_cover(
void* dictBuffer, size_t dictBufferCapacity,
const void* samplesBuffer, const size_t* samplesSizes, unsigned nbSamples,
ZDICT_cover_params_t* parameters);
@@ -361,7 +384,7 @@ ZDICTLIB_API size_t ZDICT_optimizeTrainFromBuffer_cover(
* In general, it's recommended to provide a few thousands samples, though this can vary a lot.
* It's recommended that total size of all samples be about ~x100 times the target size of dictionary.
*/
ZDICTLIB_API size_t ZDICT_trainFromBuffer_fastCover(void *dictBuffer,
ZDICTLIB_STATIC_API size_t ZDICT_trainFromBuffer_fastCover(void *dictBuffer,
size_t dictBufferCapacity, const void *samplesBuffer,
const size_t *samplesSizes, unsigned nbSamples,
ZDICT_fastCover_params_t parameters);
@@ -384,7 +407,7 @@ ZDICTLIB_API size_t ZDICT_trainFromBuffer_fastCover(void *dictBuffer,
* See ZDICT_trainFromBuffer() for details on failure modes.
* Note: ZDICT_optimizeTrainFromBuffer_fastCover() requires about 6 * 2^f bytes of memory for each thread.
*/
ZDICTLIB_API size_t ZDICT_optimizeTrainFromBuffer_fastCover(void* dictBuffer,
ZDICTLIB_STATIC_API size_t ZDICT_optimizeTrainFromBuffer_fastCover(void* dictBuffer,
size_t dictBufferCapacity, const void* samplesBuffer,
const size_t* samplesSizes, unsigned nbSamples,
ZDICT_fastCover_params_t* parameters);
@@ -409,7 +432,7 @@ typedef struct {
* It's recommended that total size of all samples be about ~x100 times the target size of dictionary.
* Note: ZDICT_trainFromBuffer_legacy() will send notifications into stderr if instructed to, using notificationLevel>0.
*/
ZDICTLIB_API size_t ZDICT_trainFromBuffer_legacy(
ZDICTLIB_STATIC_API size_t ZDICT_trainFromBuffer_legacy(
void* dictBuffer, size_t dictBufferCapacity,
const void* samplesBuffer, const size_t* samplesSizes, unsigned nbSamples,
ZDICT_legacy_params_t parameters);
@@ -421,32 +444,31 @@ ZDICTLIB_API size_t ZDICT_trainFromBuffer_legacy(
or _CRT_SECURE_NO_WARNINGS in Visual.
Otherwise, it's also possible to manually define ZDICT_DISABLE_DEPRECATE_WARNINGS */
#ifdef ZDICT_DISABLE_DEPRECATE_WARNINGS
# define ZDICT_DEPRECATED(message) ZDICTLIB_API /* disable deprecation warnings */
# define ZDICT_DEPRECATED(message) /* disable deprecation warnings */
#else
# define ZDICT_GCC_VERSION (__GNUC__ * 100 + __GNUC_MINOR__)
# if defined (__cplusplus) && (__cplusplus >= 201402) /* C++14 or greater */
# define ZDICT_DEPRECATED(message) [[deprecated(message)]] ZDICTLIB_API
# define ZDICT_DEPRECATED(message) [[deprecated(message)]]
# elif defined(__clang__) || (ZDICT_GCC_VERSION >= 405)
# define ZDICT_DEPRECATED(message) ZDICTLIB_API __attribute__((deprecated(message)))
# define ZDICT_DEPRECATED(message) __attribute__((deprecated(message)))
# elif (ZDICT_GCC_VERSION >= 301)
# define ZDICT_DEPRECATED(message) ZDICTLIB_API __attribute__((deprecated))
# define ZDICT_DEPRECATED(message) __attribute__((deprecated))
# elif defined(_MSC_VER)
# define ZDICT_DEPRECATED(message) ZDICTLIB_API __declspec(deprecated(message))
# define ZDICT_DEPRECATED(message) __declspec(deprecated(message))
# else
# pragma message("WARNING: You need to implement ZDICT_DEPRECATED for this compiler")
# define ZDICT_DEPRECATED(message) ZDICTLIB_API
# define ZDICT_DEPRECATED(message)
# endif
#endif /* ZDICT_DISABLE_DEPRECATE_WARNINGS */
ZDICT_DEPRECATED("use ZDICT_finalizeDictionary() instead")
ZDICTLIB_STATIC_API
size_t ZDICT_addEntropyTablesFromBuffer(void* dictBuffer, size_t dictContentSize, size_t dictBufferCapacity,
const void* samplesBuffer, const size_t* samplesSizes, unsigned nbSamples);
#endif /* ZDICT_STATIC_LINKING_ONLY */
#endif /* ZSTD_ZDICT_H_STATIC */
#if defined (__cplusplus)
}
#endif
#endif /* DICTBUILDER_H_001 */

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) Yann Collet, Facebook, Inc.
* Copyright (c) Meta Platforms, Inc. and affiliates.
* All rights reserved.
*
* This source code is licensed under both the BSD-style license (found in the
@@ -14,21 +14,31 @@ extern "C" {
#ifndef ZSTD_H_235446
#define ZSTD_H_235446
/* ====== Dependency ======*/
/* ====== Dependencies ======*/
#include <limits.h> /* INT_MAX */
#include <stddef.h> /* size_t */
/* ===== ZSTDLIB_API : control library symbols visibility ===== */
#ifndef ZSTDLIB_VISIBLE
# if defined(__GNUC__) && (__GNUC__ >= 4) && !defined(__MINGW32__)
/* Backwards compatibility with old macro name */
# ifdef ZSTDLIB_VISIBILITY
# define ZSTDLIB_VISIBLE ZSTDLIB_VISIBILITY
# elif defined(__GNUC__) && (__GNUC__ >= 4) && !defined(__MINGW32__)
# define ZSTDLIB_VISIBLE __attribute__ ((visibility ("default")))
# define ZSTDLIB_HIDDEN __attribute__ ((visibility ("hidden")))
# else
# define ZSTDLIB_VISIBLE
# endif
#endif
#ifndef ZSTDLIB_HIDDEN
# if defined(__GNUC__) && (__GNUC__ >= 4) && !defined(__MINGW32__)
# define ZSTDLIB_HIDDEN __attribute__ ((visibility ("hidden")))
# else
# define ZSTDLIB_HIDDEN
# endif
#endif
#if defined(ZSTD_DLL_EXPORT) && (ZSTD_DLL_EXPORT==1)
# define ZSTDLIB_API __declspec(dllexport) ZSTDLIB_VISIBLE
#elif defined(ZSTD_DLL_IMPORT) && (ZSTD_DLL_IMPORT==1)
@@ -37,6 +47,28 @@ extern "C" {
# define ZSTDLIB_API ZSTDLIB_VISIBLE
#endif
/* Deprecation warnings :
* Should these warnings be a problem, it is generally possible to disable them,
* typically with -Wno-deprecated-declarations for gcc or _CRT_SECURE_NO_WARNINGS in Visual.
* Otherwise, it's also possible to define ZSTD_DISABLE_DEPRECATE_WARNINGS.
*/
#ifdef ZSTD_DISABLE_DEPRECATE_WARNINGS
# define ZSTD_DEPRECATED(message) /* disable deprecation warnings */
#else
# if defined (__cplusplus) && (__cplusplus >= 201402) /* C++14 or greater */
# define ZSTD_DEPRECATED(message) [[deprecated(message)]]
# elif (defined(GNUC) && (GNUC > 4 || (GNUC == 4 && GNUC_MINOR >= 5))) || defined(__clang__)
# define ZSTD_DEPRECATED(message) __attribute__((deprecated(message)))
# elif defined(__GNUC__) && (__GNUC__ >= 3)
# define ZSTD_DEPRECATED(message) __attribute__((deprecated))
# elif defined(_MSC_VER)
# define ZSTD_DEPRECATED(message) __declspec(deprecated(message))
# else
# pragma message("WARNING: You need to implement ZSTD_DEPRECATED for this compiler")
# define ZSTD_DEPRECATED(message)
# endif
#endif /* ZSTD_DISABLE_DEPRECATE_WARNINGS */
/*******************************************************************************
Introduction
@@ -74,7 +106,7 @@ extern "C" {
/*------ Version ------*/
#define ZSTD_VERSION_MAJOR 1
#define ZSTD_VERSION_MINOR 5
#define ZSTD_VERSION_RELEASE 2
#define ZSTD_VERSION_RELEASE 4
#define ZSTD_VERSION_NUMBER (ZSTD_VERSION_MAJOR *100*100 + ZSTD_VERSION_MINOR *100 + ZSTD_VERSION_RELEASE)
/*! ZSTD_versionNumber() :
@@ -165,7 +197,9 @@ ZSTDLIB_API unsigned long long ZSTD_getFrameContentSize(const void *src, size_t
* "empty", "unknown" and "error" results to the same return value (0),
* while ZSTD_getFrameContentSize() gives them separate return values.
* @return : decompressed size of `src` frame content _if known and not empty_, 0 otherwise. */
ZSTDLIB_API unsigned long long ZSTD_getDecompressedSize(const void* src, size_t srcSize);
ZSTD_DEPRECATED("Replaced by ZSTD_getFrameContentSize")
ZSTDLIB_API
unsigned long long ZSTD_getDecompressedSize(const void* src, size_t srcSize);
/*! ZSTD_findFrameCompressedSize() : Requires v1.4.0+
* `src` should point to the start of a ZSTD frame or skippable frame.
@@ -177,8 +211,30 @@ ZSTDLIB_API size_t ZSTD_findFrameCompressedSize(const void* src, size_t srcSize)
/*====== Helper functions ======*/
#define ZSTD_COMPRESSBOUND(srcSize) ((srcSize) + ((srcSize)>>8) + (((srcSize) < (128<<10)) ? (((128<<10) - (srcSize)) >> 11) /* margin, from 64 to 0 */ : 0)) /* this formula ensures that bound(A) + bound(B) <= bound(A+B) as long as A and B >= 128 KB */
ZSTDLIB_API size_t ZSTD_compressBound(size_t srcSize); /*!< maximum compressed size in worst case single-pass scenario */
/* ZSTD_compressBound() :
* maximum compressed size in worst case single-pass scenario.
* When invoking `ZSTD_compress()` or any other one-pass compression function,
* it's recommended to provide @dstCapacity >= ZSTD_compressBound(srcSize)
* as it eliminates one potential failure scenario,
* aka not enough room in dst buffer to write the compressed frame.
* Note : ZSTD_compressBound() itself can fail, if @srcSize > ZSTD_MAX_INPUT_SIZE .
* In which case, ZSTD_compressBound() will return an error code
* which can be tested using ZSTD_isError().
*
* ZSTD_COMPRESSBOUND() :
* same as ZSTD_compressBound(), but as a macro.
* It can be used to produce constants, which can be useful for static allocation,
* for example to size a static array on stack.
* Will produce constant value 0 if srcSize too large.
*/
#define ZSTD_MAX_INPUT_SIZE ((sizeof(size_t)==8) ? 0xFF00FF00FF00FF00LLU : 0xFF00FF00U)
#define ZSTD_COMPRESSBOUND(srcSize) (((size_t)(srcSize) >= ZSTD_MAX_INPUT_SIZE) ? 0 : (srcSize) + ((srcSize)>>8) + (((srcSize) < (128<<10)) ? (((128<<10) - (srcSize)) >> 11) /* margin, from 64 to 0 */ : 0)) /* this formula ensures that bound(A) + bound(B) <= bound(A+B) as long as A and B >= 128 KB */
ZSTDLIB_API size_t ZSTD_compressBound(size_t srcSize); /*!< maximum compressed size in worst case single-pass scenario */
/* ZSTD_isError() :
* Most ZSTD_* functions returning a size_t value can be tested for error,
* using ZSTD_isError().
* @return 1 if error, 0 otherwise
*/
ZSTDLIB_API unsigned ZSTD_isError(size_t code); /*!< tells if a `size_t` function result is an error code */
ZSTDLIB_API const char* ZSTD_getErrorName(size_t code); /*!< provides readable string from an error code */
ZSTDLIB_API int ZSTD_minCLevel(void); /*!< minimum negative compression level allowed, requires v1.4.0+ */
@@ -421,6 +477,9 @@ typedef enum {
* ZSTD_c_validateSequences
* ZSTD_c_useBlockSplitter
* ZSTD_c_useRowMatchFinder
* ZSTD_c_prefetchCDictTables
* ZSTD_c_enableSeqProducerFallback
* ZSTD_c_maxBlockSize
* Because they are not stable, it's necessary to define ZSTD_STATIC_LINKING_ONLY to access them.
* note : never ever use experimentalParam? names directly;
* also, the enums values themselves are unstable and can still change.
@@ -439,7 +498,11 @@ typedef enum {
ZSTD_c_experimentalParam12=1009,
ZSTD_c_experimentalParam13=1010,
ZSTD_c_experimentalParam14=1011,
ZSTD_c_experimentalParam15=1012
ZSTD_c_experimentalParam15=1012,
ZSTD_c_experimentalParam16=1013,
ZSTD_c_experimentalParam17=1014,
ZSTD_c_experimentalParam18=1015,
ZSTD_c_experimentalParam19=1016
} ZSTD_cParameter;
typedef struct {
@@ -502,7 +565,7 @@ typedef enum {
* They will be used to compress next frame.
* Resetting session never fails.
* - The parameters : changes all parameters back to "default".
* This removes any reference to any dictionary too.
* This also removes any reference to any dictionary or external sequence producer.
* Parameters can only be changed between 2 sessions (i.e. no compression is currently ongoing)
* otherwise the reset fails, and function returns an error value (which can be tested using ZSTD_isError())
* - Both : similar to resetting the session, followed by resetting parameters.
@@ -552,13 +615,15 @@ typedef enum {
* ZSTD_d_stableOutBuffer
* ZSTD_d_forceIgnoreChecksum
* ZSTD_d_refMultipleDDicts
* ZSTD_d_disableHuffmanAssembly
* Because they are not stable, it's necessary to define ZSTD_STATIC_LINKING_ONLY to access them.
* note : never ever use experimentalParam? names directly
*/
ZSTD_d_experimentalParam1=1000,
ZSTD_d_experimentalParam2=1001,
ZSTD_d_experimentalParam3=1002,
ZSTD_d_experimentalParam4=1003
ZSTD_d_experimentalParam4=1003,
ZSTD_d_experimentalParam5=1004
} ZSTD_dParameter;
@@ -737,8 +802,6 @@ ZSTDLIB_API size_t ZSTD_CStreamOutSize(void); /**< recommended size for output
* This following is a legacy streaming API, available since v1.0+ .
* It can be replaced by ZSTD_CCtx_reset() and ZSTD_compressStream2().
* It is redundant, but remains fully supported.
* Streaming in combination with advanced parameters and dictionary compression
* can only be used through the new API.
******************************************************************************/
/*!
@@ -747,6 +810,9 @@ ZSTDLIB_API size_t ZSTD_CStreamOutSize(void); /**< recommended size for output
* ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only);
* ZSTD_CCtx_refCDict(zcs, NULL); // clear the dictionary (if any)
* ZSTD_CCtx_setParameter(zcs, ZSTD_c_compressionLevel, compressionLevel);
*
* Note that ZSTD_initCStream() clears any previously set dictionary. Use the new API
* to compress with a dictionary.
*/
ZSTDLIB_API size_t ZSTD_initCStream(ZSTD_CStream* zcs, int compressionLevel);
/*!
@@ -797,13 +863,31 @@ ZSTDLIB_API size_t ZSTD_freeDStream(ZSTD_DStream* zds); /* accept NULL pointer
/*===== Streaming decompression functions =====*/
/* This function is redundant with the advanced API and equivalent to:
/*! ZSTD_initDStream() :
* Initialize/reset DStream state for new decompression operation.
* Call before new decompression operation using same DStream.
*
* Note : This function is redundant with the advanced API and equivalent to:
* ZSTD_DCtx_reset(zds, ZSTD_reset_session_only);
* ZSTD_DCtx_refDDict(zds, NULL);
*/
ZSTDLIB_API size_t ZSTD_initDStream(ZSTD_DStream* zds);
/*! ZSTD_decompressStream() :
* Streaming decompression function.
* Call repetitively to consume full input updating it as necessary.
* Function will update both input and output `pos` fields exposing current state via these fields:
* - `input.pos < input.size`, some input remaining and caller should provide remaining input
* on the next call.
* - `output.pos < output.size`, decoder finished and flushed all remaining buffers.
* - `output.pos == output.size`, potentially uncflushed data present in the internal buffers,
* call ZSTD_decompressStream() again to flush remaining data to output.
* Note : with no additional input, amount of data flushed <= ZSTD_BLOCKSIZE_MAX.
*
* @return : 0 when a frame is completely decoded and fully flushed,
* or an error code, which can be tested using ZSTD_isError(),
* or any other value > 0, which means there is some decoding or flushing to do to complete current frame.
*/
ZSTDLIB_API size_t ZSTD_decompressStream(ZSTD_DStream* zds, ZSTD_outBuffer* output, ZSTD_inBuffer* input);
ZSTDLIB_API size_t ZSTD_DStreamInSize(void); /*!< recommended size for input buffer */
@@ -922,7 +1006,7 @@ ZSTDLIB_API unsigned ZSTD_getDictID_fromDDict(const ZSTD_DDict* ddict);
* If @return == 0, the dictID could not be decoded.
* This could for one of the following reasons :
* - The frame does not require a dictionary to be decoded (most common case).
* - The frame was built with dictID intentionally removed. Whatever dictionary is necessary is a hidden information.
* - The frame was built with dictID intentionally removed. Whatever dictionary is necessary is a hidden piece of information.
* Note : this use case also happens when using a non-conformant dictionary.
* - `srcSize` is too small, and as a result, the frame header could not be decoded (only possible if `srcSize < ZSTD_FRAMEHEADERSIZE_MAX`).
* - This is not a Zstandard frame.
@@ -946,8 +1030,9 @@ ZSTDLIB_API unsigned ZSTD_getDictID_fromFrame(const void* src, size_t srcSize);
* @result : 0, or an error code (which can be tested with ZSTD_isError()).
* Special: Loading a NULL (or 0-size) dictionary invalidates previous dictionary,
* meaning "return to no-dictionary mode".
* Note 1 : Dictionary is sticky, it will be used for all future compressed frames.
* To return to "no-dictionary" situation, load a NULL dictionary (or reset parameters).
* Note 1 : Dictionary is sticky, it will be used for all future compressed frames,
* until parameters are reset, a new dictionary is loaded, or the dictionary
* is explicitly invalidated by loading a NULL dictionary.
* Note 2 : Loading a dictionary involves building tables.
* It's also a CPU consuming operation, with non-negligible impact on latency.
* Tables are dependent on compression parameters, and for this reason,
@@ -960,7 +1045,7 @@ ZSTDLIB_API unsigned ZSTD_getDictID_fromFrame(const void* src, size_t srcSize);
ZSTDLIB_API size_t ZSTD_CCtx_loadDictionary(ZSTD_CCtx* cctx, const void* dict, size_t dictSize);
/*! ZSTD_CCtx_refCDict() : Requires v1.4.0+
* Reference a prepared dictionary, to be used for all next compressed frames.
* Reference a prepared dictionary, to be used for all future compressed frames.
* Note that compression parameters are enforced from within CDict,
* and supersede any compression parameter previously set within CCtx.
* The parameters ignored are labelled as "superseded-by-cdict" in the ZSTD_cParameter enum docs.
@@ -995,9 +1080,9 @@ ZSTDLIB_API size_t ZSTD_CCtx_refPrefix(ZSTD_CCtx* cctx,
const void* prefix, size_t prefixSize);
/*! ZSTD_DCtx_loadDictionary() : Requires v1.4.0+
* Create an internal DDict from dict buffer,
* to be used to decompress next frames.
* The dictionary remains valid for all future frames, until explicitly invalidated.
* Create an internal DDict from dict buffer, to be used to decompress all future frames.
* The dictionary remains valid for all future frames, until explicitly invalidated, or
* a new dictionary is loaded.
* @result : 0, or an error code (which can be tested with ZSTD_isError()).
* Special : Adding a NULL (or 0-size) dictionary invalidates any previous dictionary,
* meaning "return to no-dictionary mode".
@@ -1021,9 +1106,10 @@ ZSTDLIB_API size_t ZSTD_DCtx_loadDictionary(ZSTD_DCtx* dctx, const void* dict, s
* The memory for the table is allocated on the first call to refDDict, and can be
* freed with ZSTD_freeDCtx().
*
* If called with ZSTD_d_refMultipleDDicts disabled (the default), only one dictionary
* will be managed, and referencing a dictionary effectively "discards" any previous one.
*
* @result : 0, or an error code (which can be tested with ZSTD_isError()).
* Note 1 : Currently, only one dictionary can be managed.
* Referencing a new dictionary effectively "discards" any previous one.
* Special: referencing a NULL DDict means "return to no-dictionary mode".
* Note 2 : DDict is just referenced, its lifetime must outlive its usage from DCtx.
*/
@@ -1086,28 +1172,6 @@ ZSTDLIB_API size_t ZSTD_sizeof_DDict(const ZSTD_DDict* ddict);
# endif
#endif
/* Deprecation warnings :
* Should these warnings be a problem, it is generally possible to disable them,
* typically with -Wno-deprecated-declarations for gcc or _CRT_SECURE_NO_WARNINGS in Visual.
* Otherwise, it's also possible to define ZSTD_DISABLE_DEPRECATE_WARNINGS.
*/
#ifdef ZSTD_DISABLE_DEPRECATE_WARNINGS
# define ZSTD_DEPRECATED(message) ZSTDLIB_STATIC_API /* disable deprecation warnings */
#else
# if defined (__cplusplus) && (__cplusplus >= 201402) /* C++14 or greater */
# define ZSTD_DEPRECATED(message) [[deprecated(message)]] ZSTDLIB_STATIC_API
# elif (defined(GNUC) && (GNUC > 4 || (GNUC == 4 && GNUC_MINOR >= 5))) || defined(__clang__)
# define ZSTD_DEPRECATED(message) ZSTDLIB_STATIC_API __attribute__((deprecated(message)))
# elif defined(__GNUC__) && (__GNUC__ >= 3)
# define ZSTD_DEPRECATED(message) ZSTDLIB_STATIC_API __attribute__((deprecated))
# elif defined(_MSC_VER)
# define ZSTD_DEPRECATED(message) ZSTDLIB_STATIC_API __declspec(deprecated(message))
# else
# pragma message("WARNING: You need to implement ZSTD_DEPRECATED for this compiler")
# define ZSTD_DEPRECATED(message) ZSTDLIB_STATIC_API
# endif
#endif /* ZSTD_DISABLE_DEPRECATE_WARNINGS */
/****************************************************************************************
* experimental API (static linking only)
****************************************************************************************
@@ -1142,6 +1206,7 @@ ZSTDLIB_API size_t ZSTD_sizeof_DDict(const ZSTD_DDict* ddict);
#define ZSTD_TARGETLENGTH_MIN 0 /* note : comparing this constant to an unsigned results in a tautological test */
#define ZSTD_STRATEGY_MIN ZSTD_fast
#define ZSTD_STRATEGY_MAX ZSTD_btultra2
#define ZSTD_BLOCKSIZE_MAX_MIN (1 << 10) /* The minimum valid max blocksize. Maximum blocksizes smaller than this make compressBound() inaccurate. */
#define ZSTD_OVERLAPLOG_MIN 0
@@ -1369,33 +1434,89 @@ ZSTDLIB_STATIC_API unsigned long long ZSTD_decompressBound(const void* src, size
* or an error code (if srcSize is too small) */
ZSTDLIB_STATIC_API size_t ZSTD_frameHeaderSize(const void* src, size_t srcSize);
/*! ZSTD_decompressionMargin() :
* Zstd supports in-place decompression, where the input and output buffers overlap.
* In this case, the output buffer must be at least (Margin + Output_Size) bytes large,
* and the input buffer must be at the end of the output buffer.
*
* _______________________ Output Buffer ________________________
* | |
* | ____ Input Buffer ____|
* | | |
* v v v
* |---------------------------------------|-----------|----------|
* ^ ^ ^
* |___________________ Output_Size ___________________|_ Margin _|
*
* NOTE: See also ZSTD_DECOMPRESSION_MARGIN().
* NOTE: This applies only to single-pass decompression through ZSTD_decompress() or
* ZSTD_decompressDCtx().
* NOTE: This function supports multi-frame input.
*
* @param src The compressed frame(s)
* @param srcSize The size of the compressed frame(s)
* @returns The decompression margin or an error that can be checked with ZSTD_isError().
*/
ZSTDLIB_STATIC_API size_t ZSTD_decompressionMargin(const void* src, size_t srcSize);
/*! ZSTD_DECOMPRESS_MARGIN() :
* Similar to ZSTD_decompressionMargin(), but instead of computing the margin from
* the compressed frame, compute it from the original size and the blockSizeLog.
* See ZSTD_decompressionMargin() for details.
*
* WARNING: This macro does not support multi-frame input, the input must be a single
* zstd frame. If you need that support use the function, or implement it yourself.
*
* @param originalSize The original uncompressed size of the data.
* @param blockSize The block size == MIN(windowSize, ZSTD_BLOCKSIZE_MAX).
* Unless you explicitly set the windowLog smaller than
* ZSTD_BLOCKSIZELOG_MAX you can just use ZSTD_BLOCKSIZE_MAX.
*/
#define ZSTD_DECOMPRESSION_MARGIN(originalSize, blockSize) ((size_t)( \
ZSTD_FRAMEHEADERSIZE_MAX /* Frame header */ + \
4 /* checksum */ + \
((originalSize) == 0 ? 0 : 3 * (((originalSize) + (blockSize) - 1) / blockSize)) /* 3 bytes per block */ + \
(blockSize) /* One block of margin */ \
))
typedef enum {
ZSTD_sf_noBlockDelimiters = 0, /* Representation of ZSTD_Sequence has no block delimiters, sequences only */
ZSTD_sf_explicitBlockDelimiters = 1 /* Representation of ZSTD_Sequence contains explicit block delimiters */
} ZSTD_sequenceFormat_e;
/*! ZSTD_sequenceBound() :
* `srcSize` : size of the input buffer
* @return : upper-bound for the number of sequences that can be generated
* from a buffer of srcSize bytes
*
* note : returns number of sequences - to get bytes, multiply by sizeof(ZSTD_Sequence).
*/
ZSTDLIB_STATIC_API size_t ZSTD_sequenceBound(size_t srcSize);
/*! ZSTD_generateSequences() :
* Generate sequences using ZSTD_compress2, given a source buffer.
* Generate sequences using ZSTD_compress2(), given a source buffer.
*
* Each block will end with a dummy sequence
* with offset == 0, matchLength == 0, and litLength == length of last literals.
* litLength may be == 0, and if so, then the sequence of (of: 0 ml: 0 ll: 0)
* simply acts as a block delimiter.
*
* zc can be used to insert custom compression params.
* This function invokes ZSTD_compress2
* @zc can be used to insert custom compression params.
* This function invokes ZSTD_compress2().
*
* The output of this function can be fed into ZSTD_compressSequences() with CCtx
* setting of ZSTD_c_blockDelimiters as ZSTD_sf_explicitBlockDelimiters
* @return : number of sequences generated
*/
ZSTDLIB_STATIC_API size_t ZSTD_generateSequences(ZSTD_CCtx* zc, ZSTD_Sequence* outSeqs,
size_t outSeqsSize, const void* src, size_t srcSize);
ZSTDLIB_STATIC_API size_t
ZSTD_generateSequences( ZSTD_CCtx* zc,
ZSTD_Sequence* outSeqs, size_t outSeqsSize,
const void* src, size_t srcSize);
/*! ZSTD_mergeBlockDelimiters() :
* Given an array of ZSTD_Sequence, remove all sequences that represent block delimiters/last literals
* by merging them into into the literals of the next sequence.
* by merging them into the literals of the next sequence.
*
* As such, the final generated result has no explicit representation of block boundaries,
* and the final last literals segment is not represented in the sequences.
@@ -1407,7 +1528,9 @@ ZSTDLIB_STATIC_API size_t ZSTD_generateSequences(ZSTD_CCtx* zc, ZSTD_Sequence* o
ZSTDLIB_STATIC_API size_t ZSTD_mergeBlockDelimiters(ZSTD_Sequence* sequences, size_t seqsSize);
/*! ZSTD_compressSequences() :
* Compress an array of ZSTD_Sequence, generated from the original source buffer, into dst.
* Compress an array of ZSTD_Sequence, associated with @src buffer, into dst.
* @src contains the entire input (not just the literals).
* If @srcSize > sum(sequence.length), the remaining bytes are considered all literals
* If a dictionary is included, then the cctx should reference the dict. (see: ZSTD_CCtx_refCDict(), ZSTD_CCtx_loadDictionary(), etc.)
* The entire source is compressed into a single frame.
*
@@ -1432,17 +1555,18 @@ ZSTDLIB_STATIC_API size_t ZSTD_mergeBlockDelimiters(ZSTD_Sequence* sequences, si
* Note: Repcodes are, as of now, always re-calculated within this function, so ZSTD_Sequence::rep is unused.
* Note 2: Once we integrate ability to ingest repcodes, the explicit block delims mode must respect those repcodes exactly,
* and cannot emit an RLE block that disagrees with the repcode history
* @return : final compressed size or a ZSTD error.
* @return : final compressed size, or a ZSTD error code.
*/
ZSTDLIB_STATIC_API size_t ZSTD_compressSequences(ZSTD_CCtx* const cctx, void* dst, size_t dstSize,
const ZSTD_Sequence* inSeqs, size_t inSeqsSize,
const void* src, size_t srcSize);
ZSTDLIB_STATIC_API size_t
ZSTD_compressSequences( ZSTD_CCtx* cctx, void* dst, size_t dstSize,
const ZSTD_Sequence* inSeqs, size_t inSeqsSize,
const void* src, size_t srcSize);
/*! ZSTD_writeSkippableFrame() :
* Generates a zstd skippable frame containing data given by src, and writes it to dst buffer.
*
* Skippable frames begin with a a 4-byte magic number. There are 16 possible choices of magic number,
* Skippable frames begin with a 4-byte magic number. There are 16 possible choices of magic number,
* ranging from ZSTD_MAGIC_SKIPPABLE_START to ZSTD_MAGIC_SKIPPABLE_START+15.
* As such, the parameter magicVariant controls the exact skippable frame magic number variant used, so
* the magic number used will be ZSTD_MAGIC_SKIPPABLE_START + magicVariant.
@@ -1500,8 +1624,11 @@ ZSTDLIB_API unsigned ZSTD_isSkippableFrame(const void* buffer, size_t size);
* and ZSTD_estimateCCtxSize_usingCCtxParams(), which can be used in tandem with ZSTD_CCtxParams_setParameter().
* Both can be used to estimate memory using custom compression parameters and arbitrary srcSize limits.
*
* Note 2 : only single-threaded compression is supported.
* Note : only single-threaded compression is supported.
* ZSTD_estimateCCtxSize_usingCCtxParams() will return an error code if ZSTD_c_nbWorkers is >= 1.
*
* Note 2 : ZSTD_estimateCCtxSize* functions are not compatible with the Block-Level Sequence Producer API at this time.
* Size estimates assume that no external sequence producer is registered.
*/
ZSTDLIB_STATIC_API size_t ZSTD_estimateCCtxSize(int compressionLevel);
ZSTDLIB_STATIC_API size_t ZSTD_estimateCCtxSize_usingCParams(ZSTD_compressionParameters cParams);
@@ -1520,7 +1647,12 @@ ZSTDLIB_STATIC_API size_t ZSTD_estimateDCtxSize(void);
* or deducted from a valid frame Header, using ZSTD_estimateDStreamSize_fromFrame();
* Note : if streaming is init with function ZSTD_init?Stream_usingDict(),
* an internal ?Dict will be created, which additional size is not estimated here.
* In this case, get total size by adding ZSTD_estimate?DictSize */
* In this case, get total size by adding ZSTD_estimate?DictSize
* Note 2 : only single-threaded compression is supported.
* ZSTD_estimateCStreamSize_usingCCtxParams() will return an error code if ZSTD_c_nbWorkers is >= 1.
* Note 3 : ZSTD_estimateCStreamSize* functions are not compatible with the Block-Level Sequence Producer API at this time.
* Size estimates assume that no external sequence producer is registered.
*/
ZSTDLIB_STATIC_API size_t ZSTD_estimateCStreamSize(int compressionLevel);
ZSTDLIB_STATIC_API size_t ZSTD_estimateCStreamSize_usingCParams(ZSTD_compressionParameters cParams);
ZSTDLIB_STATIC_API size_t ZSTD_estimateCStreamSize_usingCCtxParams(const ZSTD_CCtx_params* params);
@@ -1670,22 +1802,31 @@ ZSTDLIB_STATIC_API size_t ZSTD_checkCParams(ZSTD_compressionParameters params);
* This function never fails (wide contract) */
ZSTDLIB_STATIC_API ZSTD_compressionParameters ZSTD_adjustCParams(ZSTD_compressionParameters cPar, unsigned long long srcSize, size_t dictSize);
/*! ZSTD_CCtx_setCParams() :
* Set all parameters provided within @cparams into the working @cctx.
* Note : if modifying parameters during compression (MT mode only),
* note that changes to the .windowLog parameter will be ignored.
* @return 0 on success, or an error code (can be checked with ZSTD_isError()) */
ZSTDLIB_STATIC_API size_t ZSTD_CCtx_setCParams(ZSTD_CCtx* cctx, ZSTD_compressionParameters cparams);
/*! ZSTD_compress_advanced() :
* Note : this function is now DEPRECATED.
* It can be replaced by ZSTD_compress2(), in combination with ZSTD_CCtx_setParameter() and other parameter setters.
* This prototype will generate compilation warnings. */
ZSTD_DEPRECATED("use ZSTD_compress2")
ZSTDLIB_STATIC_API
size_t ZSTD_compress_advanced(ZSTD_CCtx* cctx,
void* dst, size_t dstCapacity,
const void* src, size_t srcSize,
const void* dict,size_t dictSize,
ZSTD_parameters params);
void* dst, size_t dstCapacity,
const void* src, size_t srcSize,
const void* dict,size_t dictSize,
ZSTD_parameters params);
/*! ZSTD_compress_usingCDict_advanced() :
* Note : this function is now DEPRECATED.
* It can be replaced by ZSTD_compress2(), in combination with ZSTD_CCtx_loadDictionary() and other parameter setters.
* This prototype will generate compilation warnings. */
ZSTD_DEPRECATED("use ZSTD_compress2 with ZSTD_CCtx_loadDictionary")
ZSTDLIB_STATIC_API
size_t ZSTD_compress_usingCDict_advanced(ZSTD_CCtx* cctx,
void* dst, size_t dstCapacity,
const void* src, size_t srcSize,
@@ -1829,13 +1970,16 @@ ZSTDLIB_STATIC_API size_t ZSTD_CCtx_refPrefix_advanced(ZSTD_CCtx* cctx, const vo
* Experimental parameter.
* Default is 0 == disabled. Set to 1 to enable.
*
* Tells the compressor that the ZSTD_inBuffer will ALWAYS be the same
* between calls, except for the modifications that zstd makes to pos (the
* caller must not modify pos). This is checked by the compressor, and
* compression will fail if it ever changes. This means the only flush
* mode that makes sense is ZSTD_e_end, so zstd will error if ZSTD_e_end
* is not used. The data in the ZSTD_inBuffer in the range [src, src + pos)
* MUST not be modified during compression or you will get data corruption.
* Tells the compressor that input data presented with ZSTD_inBuffer
* will ALWAYS be the same between calls.
* Technically, the @src pointer must never be changed,
* and the @pos field can only be updated by zstd.
* However, it's possible to increase the @size field,
* allowing scenarios where more data can be appended after compressions starts.
* These conditions are checked by the compressor,
* and compression will fail if they are not respected.
* Also, data in the ZSTD_inBuffer within the range [src, src + pos)
* MUST not be modified during compression or it will result in data corruption.
*
* When this flag is enabled zstd won't allocate an input window buffer,
* because the user guarantees it can reference the ZSTD_inBuffer until
@@ -1843,18 +1987,15 @@ ZSTDLIB_STATIC_API size_t ZSTD_CCtx_refPrefix_advanced(ZSTD_CCtx* cctx, const vo
* large enough to fit a block (see ZSTD_c_stableOutBuffer). This will also
* avoid the memcpy() from the input buffer to the input window buffer.
*
* NOTE: ZSTD_compressStream2() will error if ZSTD_e_end is not used.
* That means this flag cannot be used with ZSTD_compressStream().
*
* NOTE: So long as the ZSTD_inBuffer always points to valid memory, using
* this flag is ALWAYS memory safe, and will never access out-of-bounds
* memory. However, compression WILL fail if you violate the preconditions.
* memory. However, compression WILL fail if conditions are not respected.
*
* WARNING: The data in the ZSTD_inBuffer in the range [dst, dst + pos) MUST
* not be modified during compression or you will get data corruption. This
* is because zstd needs to reference data in the ZSTD_inBuffer to find
* WARNING: The data in the ZSTD_inBuffer in the range [src, src + pos) MUST
* not be modified during compression or it will result in data corruption.
* This is because zstd needs to reference data in the ZSTD_inBuffer to find
* matches. Normally zstd maintains its own window buffer for this purpose,
* but passing this flag tells zstd to use the user provided buffer.
* but passing this flag tells zstd to rely on user provided buffer instead.
*/
#define ZSTD_c_stableInBuffer ZSTD_c_experimentalParam9
@@ -1899,7 +2040,7 @@ ZSTDLIB_STATIC_API size_t ZSTD_CCtx_refPrefix_advanced(ZSTD_CCtx* cctx, const vo
* Without validation, providing a sequence that does not conform to the zstd spec will cause
* undefined behavior, and may produce a corrupted block.
*
* With validation enabled, a if sequence is invalid (see doc/zstd_compression_format.md for
* With validation enabled, if sequence is invalid (see doc/zstd_compression_format.md for
* specifics regarding offset/matchlength requirements) then the function will bail out and
* return an error.
*
@@ -1949,6 +2090,79 @@ ZSTDLIB_STATIC_API size_t ZSTD_CCtx_refPrefix_advanced(ZSTD_CCtx* cctx, const vo
*/
#define ZSTD_c_deterministicRefPrefix ZSTD_c_experimentalParam15
/* ZSTD_c_prefetchCDictTables
* Controlled with ZSTD_paramSwitch_e enum. Default is ZSTD_ps_auto.
*
* In some situations, zstd uses CDict tables in-place rather than copying them
* into the working context. (See docs on ZSTD_dictAttachPref_e above for details).
* In such situations, compression speed is seriously impacted when CDict tables are
* "cold" (outside CPU cache). This parameter instructs zstd to prefetch CDict tables
* when they are used in-place.
*
* For sufficiently small inputs, the cost of the prefetch will outweigh the benefit.
* For sufficiently large inputs, zstd will by default memcpy() CDict tables
* into the working context, so there is no need to prefetch. This parameter is
* targeted at a middle range of input sizes, where a prefetch is cheap enough to be
* useful but memcpy() is too expensive. The exact range of input sizes where this
* makes sense is best determined by careful experimentation.
*
* Note: for this parameter, ZSTD_ps_auto is currently equivalent to ZSTD_ps_disable,
* but in the future zstd may conditionally enable this feature via an auto-detection
* heuristic for cold CDicts.
* Use ZSTD_ps_disable to opt out of prefetching under any circumstances.
*/
#define ZSTD_c_prefetchCDictTables ZSTD_c_experimentalParam16
/* ZSTD_c_enableSeqProducerFallback
* Allowed values are 0 (disable) and 1 (enable). The default setting is 0.
*
* Controls whether zstd will fall back to an internal sequence producer if an
* external sequence producer is registered and returns an error code. This fallback
* is block-by-block: the internal sequence producer will only be called for blocks
* where the external sequence producer returns an error code. Fallback parsing will
* follow any other cParam settings, such as compression level, the same as in a
* normal (fully-internal) compression operation.
*
* The user is strongly encouraged to read the full Block-Level Sequence Producer API
* documentation (below) before setting this parameter. */
#define ZSTD_c_enableSeqProducerFallback ZSTD_c_experimentalParam17
/* ZSTD_c_maxBlockSize
* Allowed values are between 1KB and ZSTD_BLOCKSIZE_MAX (128KB).
* The default is ZSTD_BLOCKSIZE_MAX, and setting to 0 will set to the default.
*
* This parameter can be used to set an upper bound on the blocksize
* that overrides the default ZSTD_BLOCKSIZE_MAX. It cannot be used to set upper
* bounds greater than ZSTD_BLOCKSIZE_MAX or bounds lower than 1KB (will make
* compressBound() innacurate). Only currently meant to be used for testing.
*
*/
#define ZSTD_c_maxBlockSize ZSTD_c_experimentalParam18
/* ZSTD_c_searchForExternalRepcodes
* This parameter affects how zstd parses external sequences, such as sequences
* provided through the compressSequences() API or from an external block-level
* sequence producer.
*
* If set to ZSTD_ps_enable, the library will check for repeated offsets in
* external sequences, even if those repcodes are not explicitly indicated in
* the "rep" field. Note that this is the only way to exploit repcode matches
* while using compressSequences() or an external sequence producer, since zstd
* currently ignores the "rep" field of external sequences.
*
* If set to ZSTD_ps_disable, the library will not exploit repeated offsets in
* external sequences, regardless of whether the "rep" field has been set. This
* reduces sequence compression overhead by about 25% while sacrificing some
* compression ratio.
*
* The default value is ZSTD_ps_auto, for which the library will enable/disable
* based on compression level.
*
* Note: for now, this param only has an effect if ZSTD_c_blockDelimiters is
* set to ZSTD_sf_explicitBlockDelimiters. That may change in the future.
*/
#define ZSTD_c_searchForExternalRepcodes ZSTD_c_experimentalParam19
/*! ZSTD_CCtx_getParameter() :
* Get the requested compression parameter value, selected by enum ZSTD_cParameter,
* and store it into int* value.
@@ -2105,7 +2319,7 @@ ZSTDLIB_STATIC_API size_t ZSTD_DCtx_getParameter(ZSTD_DCtx* dctx, ZSTD_dParamete
* in the range [dst, dst + pos) MUST not be modified during decompression
* or you will get data corruption.
*
* When this flags is enabled zstd won't allocate an output buffer, because
* When this flag is enabled zstd won't allocate an output buffer, because
* it can write directly to the ZSTD_outBuffer, but it will still allocate
* an input buffer large enough to fit any compressed block. This will also
* avoid the memcpy() from the internal output buffer to the ZSTD_outBuffer.
@@ -2158,6 +2372,17 @@ ZSTDLIB_STATIC_API size_t ZSTD_DCtx_getParameter(ZSTD_DCtx* dctx, ZSTD_dParamete
*/
#define ZSTD_d_refMultipleDDicts ZSTD_d_experimentalParam4
/* ZSTD_d_disableHuffmanAssembly
* Set to 1 to disable the Huffman assembly implementation.
* The default value is 0, which allows zstd to use the Huffman assembly
* implementation if available.
*
* This parameter can be used to disable Huffman assembly at runtime.
* If you want to disable it at compile time you can define the macro
* ZSTD_DISABLE_ASM.
*/
#define ZSTD_d_disableHuffmanAssembly ZSTD_d_experimentalParam5
/*! ZSTD_DCtx_setFormat() :
* This function is REDUNDANT. Prefer ZSTD_DCtx_setParameter().
@@ -2166,6 +2391,7 @@ ZSTDLIB_STATIC_API size_t ZSTD_DCtx_getParameter(ZSTD_DCtx* dctx, ZSTD_dParamete
* such ZSTD_f_zstd1_magicless for example.
* @return : 0, or an error code (which can be tested using ZSTD_isError()). */
ZSTD_DEPRECATED("use ZSTD_DCtx_setParameter() instead")
ZSTDLIB_STATIC_API
size_t ZSTD_DCtx_setFormat(ZSTD_DCtx* dctx, ZSTD_format_e format);
/*! ZSTD_decompressStream_simpleArgs() :
@@ -2202,6 +2428,7 @@ ZSTDLIB_STATIC_API size_t ZSTD_decompressStream_simpleArgs (
* This prototype will generate compilation warnings.
*/
ZSTD_DEPRECATED("use ZSTD_CCtx_reset, see zstd.h for detailed instructions")
ZSTDLIB_STATIC_API
size_t ZSTD_initCStream_srcSize(ZSTD_CStream* zcs,
int compressionLevel,
unsigned long long pledgedSrcSize);
@@ -2219,6 +2446,7 @@ size_t ZSTD_initCStream_srcSize(ZSTD_CStream* zcs,
* This prototype will generate compilation warnings.
*/
ZSTD_DEPRECATED("use ZSTD_CCtx_reset, see zstd.h for detailed instructions")
ZSTDLIB_STATIC_API
size_t ZSTD_initCStream_usingDict(ZSTD_CStream* zcs,
const void* dict, size_t dictSize,
int compressionLevel);
@@ -2239,6 +2467,7 @@ size_t ZSTD_initCStream_usingDict(ZSTD_CStream* zcs,
* This prototype will generate compilation warnings.
*/
ZSTD_DEPRECATED("use ZSTD_CCtx_reset, see zstd.h for detailed instructions")
ZSTDLIB_STATIC_API
size_t ZSTD_initCStream_advanced(ZSTD_CStream* zcs,
const void* dict, size_t dictSize,
ZSTD_parameters params,
@@ -2253,6 +2482,7 @@ size_t ZSTD_initCStream_advanced(ZSTD_CStream* zcs,
* This prototype will generate compilation warnings.
*/
ZSTD_DEPRECATED("use ZSTD_CCtx_reset and ZSTD_CCtx_refCDict, see zstd.h for detailed instructions")
ZSTDLIB_STATIC_API
size_t ZSTD_initCStream_usingCDict(ZSTD_CStream* zcs, const ZSTD_CDict* cdict);
/*! ZSTD_initCStream_usingCDict_advanced() :
@@ -2271,6 +2501,7 @@ size_t ZSTD_initCStream_usingCDict(ZSTD_CStream* zcs, const ZSTD_CDict* cdict);
* This prototype will generate compilation warnings.
*/
ZSTD_DEPRECATED("use ZSTD_CCtx_reset and ZSTD_CCtx_refCDict, see zstd.h for detailed instructions")
ZSTDLIB_STATIC_API
size_t ZSTD_initCStream_usingCDict_advanced(ZSTD_CStream* zcs,
const ZSTD_CDict* cdict,
ZSTD_frameParameters fParams,
@@ -2295,6 +2526,7 @@ size_t ZSTD_initCStream_usingCDict_advanced(ZSTD_CStream* zcs,
* This prototype will generate compilation warnings.
*/
ZSTD_DEPRECATED("use ZSTD_CCtx_reset, see zstd.h for detailed instructions")
ZSTDLIB_STATIC_API
size_t ZSTD_resetCStream(ZSTD_CStream* zcs, unsigned long long pledgedSrcSize);
@@ -2340,8 +2572,8 @@ ZSTDLIB_STATIC_API size_t ZSTD_toFlushNow(ZSTD_CCtx* cctx);
* ZSTD_DCtx_loadDictionary(zds, dict, dictSize);
*
* note: no dictionary will be used if dict == NULL or dictSize < 8
* Note : this prototype will be marked as deprecated and generate compilation warnings on reaching v1.5.x
*/
ZSTD_DEPRECATED("use ZSTD_DCtx_reset + ZSTD_DCtx_loadDictionary, see zstd.h for detailed instructions")
ZSTDLIB_STATIC_API size_t ZSTD_initDStream_usingDict(ZSTD_DStream* zds, const void* dict, size_t dictSize);
/*!
@@ -2351,8 +2583,8 @@ ZSTDLIB_STATIC_API size_t ZSTD_initDStream_usingDict(ZSTD_DStream* zds, const vo
* ZSTD_DCtx_refDDict(zds, ddict);
*
* note : ddict is referenced, it must outlive decompression session
* Note : this prototype will be marked as deprecated and generate compilation warnings on reaching v1.5.x
*/
ZSTD_DEPRECATED("use ZSTD_DCtx_reset + ZSTD_DCtx_refDDict, see zstd.h for detailed instructions")
ZSTDLIB_STATIC_API size_t ZSTD_initDStream_usingDDict(ZSTD_DStream* zds, const ZSTD_DDict* ddict);
/*!
@@ -2361,8 +2593,8 @@ ZSTDLIB_STATIC_API size_t ZSTD_initDStream_usingDDict(ZSTD_DStream* zds, const Z
* ZSTD_DCtx_reset(zds, ZSTD_reset_session_only);
*
* re-use decompression parameters from previous init; saves dictionary loading
* Note : this prototype will be marked as deprecated and generate compilation warnings on reaching v1.5.x
*/
ZSTD_DEPRECATED("use ZSTD_DCtx_reset, see zstd.h for detailed instructions")
ZSTDLIB_STATIC_API size_t ZSTD_resetDStream(ZSTD_DStream* zds);
@@ -2383,7 +2615,6 @@ ZSTDLIB_STATIC_API size_t ZSTD_resetDStream(ZSTD_DStream* zds);
Start by initializing a context.
Use ZSTD_compressBegin(), or ZSTD_compressBegin_usingDict() for dictionary compression.
It's also possible to duplicate a reference context which has already been initialized, using ZSTD_copyCCtx()
Then, consume your input using ZSTD_compressContinue().
There are some important considerations to keep in mind when using this advanced function :
@@ -2408,15 +2639,20 @@ ZSTDLIB_STATIC_API size_t ZSTD_resetDStream(ZSTD_DStream* zds);
ZSTDLIB_STATIC_API size_t ZSTD_compressBegin(ZSTD_CCtx* cctx, int compressionLevel);
ZSTDLIB_STATIC_API size_t ZSTD_compressBegin_usingDict(ZSTD_CCtx* cctx, const void* dict, size_t dictSize, int compressionLevel);
ZSTDLIB_STATIC_API size_t ZSTD_compressBegin_usingCDict(ZSTD_CCtx* cctx, const ZSTD_CDict* cdict); /**< note: fails if cdict==NULL */
ZSTDLIB_STATIC_API size_t ZSTD_copyCCtx(ZSTD_CCtx* cctx, const ZSTD_CCtx* preparedCCtx, unsigned long long pledgedSrcSize); /**< note: if pledgedSrcSize is not known, use ZSTD_CONTENTSIZE_UNKNOWN */
ZSTD_DEPRECATED("This function will likely be removed in a future release. It is misleading and has very limited utility.")
ZSTDLIB_STATIC_API
size_t ZSTD_copyCCtx(ZSTD_CCtx* cctx, const ZSTD_CCtx* preparedCCtx, unsigned long long pledgedSrcSize); /**< note: if pledgedSrcSize is not known, use ZSTD_CONTENTSIZE_UNKNOWN */
ZSTDLIB_STATIC_API size_t ZSTD_compressContinue(ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize);
ZSTDLIB_STATIC_API size_t ZSTD_compressEnd(ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize);
/* The ZSTD_compressBegin_advanced() and ZSTD_compressBegin_usingCDict_advanced() are now DEPRECATED and will generate a compiler warning */
ZSTD_DEPRECATED("use advanced API to access custom parameters")
ZSTDLIB_STATIC_API
size_t ZSTD_compressBegin_advanced(ZSTD_CCtx* cctx, const void* dict, size_t dictSize, ZSTD_parameters params, unsigned long long pledgedSrcSize); /**< pledgedSrcSize : If srcSize is not known at init time, use ZSTD_CONTENTSIZE_UNKNOWN */
ZSTD_DEPRECATED("use advanced API to access custom parameters")
ZSTDLIB_STATIC_API
size_t ZSTD_compressBegin_usingCDict_advanced(ZSTD_CCtx* const cctx, const ZSTD_CDict* const cdict, ZSTD_frameParameters const fParams, unsigned long long const pledgedSrcSize); /* compression parameters are already set within cdict. pledgedSrcSize must be correct. If srcSize is not known, use macro ZSTD_CONTENTSIZE_UNKNOWN */
/**
Buffer-less streaming decompression (synchronous mode)
@@ -2429,8 +2665,8 @@ size_t ZSTD_compressBegin_usingCDict_advanced(ZSTD_CCtx* const cctx, const ZSTD_
Frame header is extracted from the beginning of compressed frame, so providing only the frame's beginning is enough.
Data fragment must be large enough to ensure successful decoding.
`ZSTD_frameHeaderSize_max` bytes is guaranteed to always be large enough.
@result : 0 : successful decoding, the `ZSTD_frameHeader` structure is correctly filled.
>0 : `srcSize` is too small, please provide at least @result bytes on next attempt.
result : 0 : successful decoding, the `ZSTD_frameHeader` structure is correctly filled.
>0 : `srcSize` is too small, please provide at least result bytes on next attempt.
errorCode, which can be tested using ZSTD_isError().
It fills a ZSTD_frameHeader structure with important information to correctly decode the frame,
@@ -2449,7 +2685,7 @@ size_t ZSTD_compressBegin_usingCDict_advanced(ZSTD_CCtx* const cctx, const ZSTD_
The most memory efficient way is to use a round buffer of sufficient size.
Sufficient size is determined by invoking ZSTD_decodingBufferSize_min(),
which can @return an error code if required value is too large for current system (in 32-bits mode).
which can return an error code if required value is too large for current system (in 32-bits mode).
In a round buffer methodology, ZSTD_decompressContinue() decompresses each block next to previous one,
up to the moment there is not enough room left in the buffer to guarantee decoding another full block,
which maximum size is provided in `ZSTD_frameHeader` structure, field `blockSizeMax`.
@@ -2469,7 +2705,7 @@ size_t ZSTD_compressBegin_usingCDict_advanced(ZSTD_CCtx* const cctx, const ZSTD_
ZSTD_nextSrcSizeToDecompress() tells how many bytes to provide as 'srcSize' to ZSTD_decompressContinue().
ZSTD_decompressContinue() requires this _exact_ amount of bytes, or it will fail.
@result of ZSTD_decompressContinue() is the number of bytes regenerated within 'dst' (necessarily <= dstCapacity).
result of ZSTD_decompressContinue() is the number of bytes regenerated within 'dst' (necessarily <= dstCapacity).
It can be zero : it just means ZSTD_decompressContinue() has decoded some metadata item.
It can also be an error code, which can be tested with ZSTD_isError().
@@ -2501,6 +2737,8 @@ typedef struct {
unsigned headerSize;
unsigned dictID;
unsigned checksumFlag;
unsigned _reserved1;
unsigned _reserved2;
} ZSTD_frameHeader;
/*! ZSTD_getFrameHeader() :
@@ -2523,6 +2761,7 @@ ZSTDLIB_STATIC_API size_t ZSTD_nextSrcSizeToDecompress(ZSTD_DCtx* dctx);
ZSTDLIB_STATIC_API size_t ZSTD_decompressContinue(ZSTD_DCtx* dctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize);
/* misc */
ZSTD_DEPRECATED("This function will likely be removed in the next minor release. It is misleading and has very limited utility.")
ZSTDLIB_STATIC_API void ZSTD_copyDCtx(ZSTD_DCtx* dctx, const ZSTD_DCtx* preparedDCtx);
typedef enum { ZSTDnit_frameHeader, ZSTDnit_blockHeader, ZSTDnit_block, ZSTDnit_lastBlock, ZSTDnit_checksum, ZSTDnit_skippableFrame } ZSTD_nextInputType_e;
ZSTDLIB_STATIC_API ZSTD_nextInputType_e ZSTD_nextInputType(ZSTD_DCtx* dctx);
@@ -2545,7 +2784,6 @@ ZSTDLIB_STATIC_API ZSTD_nextInputType_e ZSTD_nextInputType(ZSTD_DCtx* dctx);
- It is necessary to init context before starting
+ compression : any ZSTD_compressBegin*() variant, including with dictionary
+ decompression : any ZSTD_decompressBegin*() variant, including with dictionary
+ copyCCtx() and copyDCtx() can be used too
- Block size is limited, it must be <= ZSTD_getBlockSize() <= ZSTD_BLOCKSIZE_MAX == 128 KB
+ If input is larger than a block size, it's necessary to split input data into multiple blocks
+ For inputs larger than a single block, consider using regular ZSTD_compress() instead.
@@ -2568,6 +2806,167 @@ ZSTDLIB_STATIC_API size_t ZSTD_decompressBlock(ZSTD_DCtx* dctx, void* dst, size_
ZSTDLIB_STATIC_API size_t ZSTD_insertBlock (ZSTD_DCtx* dctx, const void* blockStart, size_t blockSize); /**< insert uncompressed block into `dctx` history. Useful for multi-blocks decompression. */
/* ********************* BLOCK-LEVEL SEQUENCE PRODUCER API *********************
*
* *** OVERVIEW ***
* The Block-Level Sequence Producer API allows users to provide their own custom
* sequence producer which libzstd invokes to process each block. The produced list
* of sequences (literals and matches) is then post-processed by libzstd to produce
* valid compressed blocks.
*
* This block-level offload API is a more granular complement of the existing
* frame-level offload API compressSequences() (introduced in v1.5.1). It offers
* an easier migration story for applications already integrated with libzstd: the
* user application continues to invoke the same compression functions
* ZSTD_compress2() or ZSTD_compressStream2() as usual, and transparently benefits
* from the specific advantages of the external sequence producer. For example,
* the sequence producer could be tuned to take advantage of known characteristics
* of the input, to offer better speed / ratio, or could leverage hardware
* acceleration not available within libzstd itself.
*
* See contrib/externalSequenceProducer for an example program employing the
* Block-Level Sequence Producer API.
*
* *** USAGE ***
* The user is responsible for implementing a function of type
* ZSTD_sequenceProducer_F. For each block, zstd will pass the following
* arguments to the user-provided function:
*
* - sequenceProducerState: a pointer to a user-managed state for the sequence
* producer.
*
* - outSeqs, outSeqsCapacity: an output buffer for the sequence producer.
* outSeqsCapacity is guaranteed >= ZSTD_sequenceBound(srcSize). The memory
* backing outSeqs is managed by the CCtx.
*
* - src, srcSize: an input buffer for the sequence producer to parse.
* srcSize is guaranteed to be <= ZSTD_BLOCKSIZE_MAX.
*
* - dict, dictSize: a history buffer, which may be empty, which the sequence
* producer may reference as it parses the src buffer. Currently, zstd will
* always pass dictSize == 0 into external sequence producers, but this will
* change in the future.
*
* - compressionLevel: a signed integer representing the zstd compression level
* set by the user for the current operation. The sequence producer may choose
* to use this information to change its compression strategy and speed/ratio
* tradeoff. Note: the compression level does not reflect zstd parameters set
* through the advanced API.
*
* - windowSize: a size_t representing the maximum allowed offset for external
* sequences. Note that sequence offsets are sometimes allowed to exceed the
* windowSize if a dictionary is present, see doc/zstd_compression_format.md
* for details.
*
* The user-provided function shall return a size_t representing the number of
* sequences written to outSeqs. This return value will be treated as an error
* code if it is greater than outSeqsCapacity. The return value must be non-zero
* if srcSize is non-zero. The ZSTD_SEQUENCE_PRODUCER_ERROR macro is provided
* for convenience, but any value greater than outSeqsCapacity will be treated as
* an error code.
*
* If the user-provided function does not return an error code, the sequences
* written to outSeqs must be a valid parse of the src buffer. Data corruption may
* occur if the parse is not valid. A parse is defined to be valid if the
* following conditions hold:
* - The sum of matchLengths and literalLengths must equal srcSize.
* - All sequences in the parse, except for the final sequence, must have
* matchLength >= ZSTD_MINMATCH_MIN. The final sequence must have
* matchLength >= ZSTD_MINMATCH_MIN or matchLength == 0.
* - All offsets must respect the windowSize parameter as specified in
* doc/zstd_compression_format.md.
* - If the final sequence has matchLength == 0, it must also have offset == 0.
*
* zstd will only validate these conditions (and fail compression if they do not
* hold) if the ZSTD_c_validateSequences cParam is enabled. Note that sequence
* validation has a performance cost.
*
* If the user-provided function returns an error, zstd will either fall back
* to an internal sequence producer or fail the compression operation. The user can
* choose between the two behaviors by setting the ZSTD_c_enableSeqProducerFallback
* cParam. Fallback compression will follow any other cParam settings, such as
* compression level, the same as in a normal compression operation.
*
* The user shall instruct zstd to use a particular ZSTD_sequenceProducer_F
* function by calling
* ZSTD_registerSequenceProducer(cctx,
* sequenceProducerState,
* sequenceProducer)
* This setting will persist until the next parameter reset of the CCtx.
*
* The sequenceProducerState must be initialized by the user before calling
* ZSTD_registerSequenceProducer(). The user is responsible for destroying the
* sequenceProducerState.
*
* *** LIMITATIONS ***
* This API is compatible with all zstd compression APIs which respect advanced parameters.
* However, there are three limitations:
*
* First, the ZSTD_c_enableLongDistanceMatching cParam is not currently supported.
* COMPRESSION WILL FAIL if it is enabled and the user tries to compress with a block-level
* external sequence producer.
* - Note that ZSTD_c_enableLongDistanceMatching is auto-enabled by default in some
* cases (see its documentation for details). Users must explicitly set
* ZSTD_c_enableLongDistanceMatching to ZSTD_ps_disable in such cases if an external
* sequence producer is registered.
* - As of this writing, ZSTD_c_enableLongDistanceMatching is disabled by default
* whenever ZSTD_c_windowLog < 128MB, but that's subject to change. Users should
* check the docs on ZSTD_c_enableLongDistanceMatching whenever the Block-Level Sequence
* Producer API is used in conjunction with advanced settings (like ZSTD_c_windowLog).
*
* Second, history buffers are not currently supported. Concretely, zstd will always pass
* dictSize == 0 to the external sequence producer (for now). This has two implications:
* - Dictionaries are not currently supported. Compression will *not* fail if the user
* references a dictionary, but the dictionary won't have any effect.
* - Stream history is not currently supported. All advanced compression APIs, including
* streaming APIs, work with external sequence producers, but each block is treated as
* an independent chunk without history from previous blocks.
*
* Third, multi-threading within a single compression is not currently supported. In other words,
* COMPRESSION WILL FAIL if ZSTD_c_nbWorkers > 0 and an external sequence producer is registered.
* Multi-threading across compressions is fine: simply create one CCtx per thread.
*
* Long-term, we plan to overcome all three limitations. There is no technical blocker to
* overcoming them. It is purely a question of engineering effort.
*/
#define ZSTD_SEQUENCE_PRODUCER_ERROR ((size_t)(-1))
typedef size_t ZSTD_sequenceProducer_F (
void* sequenceProducerState,
ZSTD_Sequence* outSeqs, size_t outSeqsCapacity,
const void* src, size_t srcSize,
const void* dict, size_t dictSize,
int compressionLevel,
size_t windowSize
);
/*! ZSTD_registerSequenceProducer() :
* Instruct zstd to use a block-level external sequence producer function.
*
* The sequenceProducerState must be initialized by the caller, and the caller is
* responsible for managing its lifetime. This parameter is sticky across
* compressions. It will remain set until the user explicitly resets compression
* parameters.
*
* Sequence producer registration is considered to be an "advanced parameter",
* part of the "advanced API". This means it will only have an effect on compression
* APIs which respect advanced parameters, such as compress2() and compressStream2().
* Older compression APIs such as compressCCtx(), which predate the introduction of
* "advanced parameters", will ignore any external sequence producer setting.
*
* The sequence producer can be "cleared" by registering a NULL function pointer. This
* removes all limitations described above in the "LIMITATIONS" section of the API docs.
*
* The user is strongly encouraged to read the full API documentation (above) before
* calling this function. */
ZSTDLIB_STATIC_API void
ZSTD_registerSequenceProducer(
ZSTD_CCtx* cctx,
void* sequenceProducerState,
ZSTD_sequenceProducer_F* sequenceProducer
);
#endif /* ZSTD_H_ZSTD_STATIC_LINKING_ONLY */
#if defined (__cplusplus)

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) Yann Collet, Facebook, Inc.
* Copyright (c) Meta Platforms, Inc. and affiliates.
* All rights reserved.
*
* This source code is licensed under both the BSD-style license (found in the
@@ -20,19 +20,31 @@ extern "C" {
/* ===== ZSTDERRORLIB_API : control library symbols visibility ===== */
#ifndef ZSTDERRORLIB_VISIBILITY
# if defined(__GNUC__) && (__GNUC__ >= 4)
# define ZSTDERRORLIB_VISIBILITY __attribute__ ((visibility ("default")))
#ifndef ZSTDERRORLIB_VISIBLE
/* Backwards compatibility with old macro name */
# ifdef ZSTDERRORLIB_VISIBILITY
# define ZSTDERRORLIB_VISIBLE ZSTDERRORLIB_VISIBILITY
# elif defined(__GNUC__) && (__GNUC__ >= 4) && !defined(__MINGW32__)
# define ZSTDERRORLIB_VISIBLE __attribute__ ((visibility ("default")))
# else
# define ZSTDERRORLIB_VISIBILITY
# define ZSTDERRORLIB_VISIBLE
# endif
#endif
#ifndef ZSTDERRORLIB_HIDDEN
# if defined(__GNUC__) && (__GNUC__ >= 4) && !defined(__MINGW32__)
# define ZSTDERRORLIB_HIDDEN __attribute__ ((visibility ("hidden")))
# else
# define ZSTDERRORLIB_HIDDEN
# endif
#endif
#if defined(ZSTD_DLL_EXPORT) && (ZSTD_DLL_EXPORT==1)
# define ZSTDERRORLIB_API __declspec(dllexport) ZSTDERRORLIB_VISIBILITY
# define ZSTDERRORLIB_API __declspec(dllexport) ZSTDERRORLIB_VISIBLE
#elif defined(ZSTD_DLL_IMPORT) && (ZSTD_DLL_IMPORT==1)
# define ZSTDERRORLIB_API __declspec(dllimport) ZSTDERRORLIB_VISIBILITY /* It isn't required but allows to generate better code, saving a function pointer load from the IAT and an indirect jump.*/
# define ZSTDERRORLIB_API __declspec(dllimport) ZSTDERRORLIB_VISIBLE /* It isn't required but allows to generate better code, saving a function pointer load from the IAT and an indirect jump.*/
#else
# define ZSTDERRORLIB_API ZSTDERRORLIB_VISIBILITY
# define ZSTDERRORLIB_API ZSTDERRORLIB_VISIBLE
#endif
/*-*********************************************
@@ -58,14 +70,17 @@ typedef enum {
ZSTD_error_frameParameter_windowTooLarge = 16,
ZSTD_error_corruption_detected = 20,
ZSTD_error_checksum_wrong = 22,
ZSTD_error_literals_headerWrong = 24,
ZSTD_error_dictionary_corrupted = 30,
ZSTD_error_dictionary_wrong = 32,
ZSTD_error_dictionaryCreation_failed = 34,
ZSTD_error_parameter_unsupported = 40,
ZSTD_error_parameter_combination_unsupported = 41,
ZSTD_error_parameter_outOfBound = 42,
ZSTD_error_tableLog_tooLarge = 44,
ZSTD_error_maxSymbolValue_tooLarge = 46,
ZSTD_error_maxSymbolValue_tooSmall = 48,
ZSTD_error_stabilityCondition_notRespected = 50,
ZSTD_error_stage_wrong = 60,
ZSTD_error_init_missing = 62,
ZSTD_error_memory_allocation = 64,
@@ -73,11 +88,15 @@ typedef enum {
ZSTD_error_dstSize_tooSmall = 70,
ZSTD_error_srcSize_wrong = 72,
ZSTD_error_dstBuffer_null = 74,
ZSTD_error_noForwardProgress_destFull = 80,
ZSTD_error_noForwardProgress_inputEmpty = 82,
/* following error codes are __NOT STABLE__, they can be removed or changed in future versions */
ZSTD_error_frameIndex_tooLarge = 100,
ZSTD_error_seekableIO = 102,
ZSTD_error_dstBuffer_wrong = 104,
ZSTD_error_srcBuffer_wrong = 105,
ZSTD_error_sequenceProducer_failed = 106,
ZSTD_error_externalSequences_invalid = 107,
ZSTD_error_maxCode = 120 /* never EVER use this value directly, it can change in future versions! Use ZSTD_isError() instead */
} ZSTD_ErrorCode;

View File

@@ -4,6 +4,12 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).
## [1.2.1]
### Changed
- Drop golang/x/lint dependency.
[1.2.1]: https://github.com/uber-go/goleak/compare/v1.2.0...v1.2.1
## [1.2.0]
### Added
- Add Cleanup option that can be used for registering cleanup callbacks. (#78)
@@ -47,5 +53,7 @@ Thanks to @denis-tingajkin for their contributions to this release.
[1.0.0]: https://github.com/uber-go/goleak/compare/v0.10.0...v1.0.0
## 0.10.0
## [0.10.0]
- Initial release.
[0.10.0]: https://github.com/uber-go/goleak/compare/v0.10.0...HEAD

10
vendor/go.uber.org/goleak/Makefile generated vendored
View File

@@ -1,6 +1,6 @@
export GOBIN ?= $(shell pwd)/bin
GOLINT = $(GOBIN)/golint
REVIVE = $(GOBIN)/revive
GO_FILES := $(shell \
find . '(' -path '*/.*' -o -path './vendor' ')' -prune \
@@ -24,18 +24,18 @@ cover:
go test -race -coverprofile=cover.out -coverpkg=./... ./...
go tool cover -html=cover.out -o cover.html
$(GOLINT):
go install golang.org/x/lint/golint
$(REVIVE):
cd tools && go install github.com/mgechev/revive
.PHONY: lint
lint: $(GOLINT)
lint: $(REVIVE)
@rm -rf lint.log
@echo "Checking formatting..."
@gofmt -d -s $(GO_FILES) 2>&1 | tee lint.log
@echo "Checking vet..."
@go vet ./... 2>&1 | tee -a lint.log
@echo "Checking lint..."
@$(GOLINT) ./... 2>&1 | tee -a lint.log
@$(REVIVE) -set_exit_status ./... 2>&1 | tee -a lint.log
@echo "Checking for unresolved FIXMEs..."
@git grep -i fixme | grep -v -e '^vendor/' -e '^Makefile' | tee -a lint.log
@[ ! -s lint.log ]

View File

@@ -8,7 +8,9 @@ You can use `go get` to get the latest version:
`go get -u go.uber.org/goleak`
`goleak` also supports semver releases. It is compatible with Go 1.5+.
`goleak` also supports semver releases.
Note that go-leak only [supports][release] the two most recent minor versions of Go.
## Quick Start
@@ -69,3 +71,4 @@ No breaking changes will be made to exported APIs before 2.0.
[ci]: https://github.com/uber-go/goleak/actions/workflows/go.yml
[cov-img]: https://codecov.io/gh/uber-go/goleak/branch/master/graph/badge.svg
[cov]: https://codecov.io/gh/uber-go/goleak
[release]: https://go.dev/doc/devel/release#policy

22
vendor/go.uber.org/goleak/internal/stack/doc.go generated vendored Normal file
View File

@@ -0,0 +1,22 @@
// Copyright (c) 2017-2023 Uber Technologies, Inc.
//
// Permission is hereby granted, free of charge, to any person obtaining a copy
// of this software and associated documentation files (the "Software"), to deal
// in the Software without restriction, including without limitation the rights
// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
// copies of the Software, and to permit persons to whom the Software is
// furnished to do so, subject to the following conditions:
//
// The above copyright notice and this permission notice shall be included in
// all copies or substantial portions of the Software.
//
// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
// THE SOFTWARE.
// Package stack is used for parsing stacks from `runtime.Stack`.
package stack

View File

@@ -662,6 +662,15 @@ func (f *Framer) WriteData(streamID uint32, endStream bool, data []byte) error {
// It is the caller's responsibility not to violate the maximum frame size
// and to not call other Write methods concurrently.
func (f *Framer) WriteDataPadded(streamID uint32, endStream bool, data, pad []byte) error {
if err := f.startWriteDataPadded(streamID, endStream, data, pad); err != nil {
return err
}
return f.endWrite()
}
// startWriteDataPadded is WriteDataPadded, but only writes the frame to the Framer's internal buffer.
// The caller should call endWrite to flush the frame to the underlying writer.
func (f *Framer) startWriteDataPadded(streamID uint32, endStream bool, data, pad []byte) error {
if !validStreamID(streamID) && !f.AllowIllegalWrites {
return errStreamID
}
@@ -691,7 +700,7 @@ func (f *Framer) WriteDataPadded(streamID uint32, endStream bool, data, pad []by
}
f.wbuf = append(f.wbuf, data...)
f.wbuf = append(f.wbuf, pad...)
return f.endWrite()
return nil
}
// A SettingsFrame conveys configuration parameters that affect how

View File

@@ -359,6 +359,7 @@ func (d *Decoder) parseFieldLiteral(n uint8, it indexType) error {
var hf HeaderField
wantStr := d.emitEnabled || it.indexed()
var undecodedName undecodedString
if nameIdx > 0 {
ihf, ok := d.at(nameIdx)
if !ok {
@@ -366,15 +367,27 @@ func (d *Decoder) parseFieldLiteral(n uint8, it indexType) error {
}
hf.Name = ihf.Name
} else {
hf.Name, buf, err = d.readString(buf, wantStr)
undecodedName, buf, err = d.readString(buf)
if err != nil {
return err
}
}
hf.Value, buf, err = d.readString(buf, wantStr)
undecodedValue, buf, err := d.readString(buf)
if err != nil {
return err
}
if wantStr {
if nameIdx <= 0 {
hf.Name, err = d.decodeString(undecodedName)
if err != nil {
return err
}
}
hf.Value, err = d.decodeString(undecodedValue)
if err != nil {
return err
}
}
d.buf = buf
if it.indexed() {
d.dynTab.add(hf)
@@ -459,46 +472,52 @@ func readVarInt(n byte, p []byte) (i uint64, remain []byte, err error) {
return 0, origP, errNeedMore
}
// readString decodes an hpack string from p.
// readString reads an hpack string from p.
//
// wantStr is whether s will be used. If false, decompression and
// []byte->string garbage are skipped if s will be ignored
// anyway. This does mean that huffman decoding errors for non-indexed
// strings past the MAX_HEADER_LIST_SIZE are ignored, but the server
// is returning an error anyway, and because they're not indexed, the error
// won't affect the decoding state.
func (d *Decoder) readString(p []byte, wantStr bool) (s string, remain []byte, err error) {
// It returns a reference to the encoded string data to permit deferring decode costs
// until after the caller verifies all data is present.
func (d *Decoder) readString(p []byte) (u undecodedString, remain []byte, err error) {
if len(p) == 0 {
return "", p, errNeedMore
return u, p, errNeedMore
}
isHuff := p[0]&128 != 0
strLen, p, err := readVarInt(7, p)
if err != nil {
return "", p, err
return u, p, err
}
if d.maxStrLen != 0 && strLen > uint64(d.maxStrLen) {
return "", nil, ErrStringLength
// Returning an error here means Huffman decoding errors
// for non-indexed strings past the maximum string length
// are ignored, but the server is returning an error anyway
// and because the string is not indexed the error will not
// affect the decoding state.
return u, nil, ErrStringLength
}
if uint64(len(p)) < strLen {
return "", p, errNeedMore
return u, p, errNeedMore
}
if !isHuff {
if wantStr {
s = string(p[:strLen])
}
return s, p[strLen:], nil
}
if wantStr {
buf := bufPool.Get().(*bytes.Buffer)
buf.Reset() // don't trust others
defer bufPool.Put(buf)
if err := huffmanDecode(buf, d.maxStrLen, p[:strLen]); err != nil {
buf.Reset()
return "", nil, err
}
s = buf.String()
buf.Reset() // be nice to GC
}
return s, p[strLen:], nil
u.isHuff = isHuff
u.b = p[:strLen]
return u, p[strLen:], nil
}
type undecodedString struct {
isHuff bool
b []byte
}
func (d *Decoder) decodeString(u undecodedString) (string, error) {
if !u.isHuff {
return string(u.b), nil
}
buf := bufPool.Get().(*bytes.Buffer)
buf.Reset() // don't trust others
var s string
err := huffmanDecode(buf, d.maxStrLen, u.b)
if err == nil {
s = buf.String()
}
buf.Reset() // be nice to GC
bufPool.Put(buf)
return s, err
}

View File

@@ -843,8 +843,13 @@ type frameWriteResult struct {
// and then reports when it's done.
// At most one goroutine can be running writeFrameAsync at a time per
// serverConn.
func (sc *serverConn) writeFrameAsync(wr FrameWriteRequest) {
err := wr.write.writeFrame(sc)
func (sc *serverConn) writeFrameAsync(wr FrameWriteRequest, wd *writeData) {
var err error
if wd == nil {
err = wr.write.writeFrame(sc)
} else {
err = sc.framer.endWrite()
}
sc.wroteFrameCh <- frameWriteResult{wr: wr, err: err}
}
@@ -1251,9 +1256,16 @@ func (sc *serverConn) startFrameWrite(wr FrameWriteRequest) {
sc.writingFrameAsync = false
err := wr.write.writeFrame(sc)
sc.wroteFrame(frameWriteResult{wr: wr, err: err})
} else if wd, ok := wr.write.(*writeData); ok {
// Encode the frame in the serve goroutine, to ensure we don't have
// any lingering asynchronous references to data passed to Write.
// See https://go.dev/issue/58446.
sc.framer.startWriteDataPadded(wd.streamID, wd.endStream, wd.p, nil)
sc.writingFrameAsync = true
go sc.writeFrameAsync(wr, wd)
} else {
sc.writingFrameAsync = true
go sc.writeFrameAsync(wr)
go sc.writeFrameAsync(wr, nil)
}
}

View File

@@ -209,7 +209,6 @@ func (rx *ResumableUpload) Upload(ctx context.Context) (resp *http.Response, err
}
return prepareReturn(resp, err)
case <-pauseTimer.C:
quitAfterTimer.Stop()
case <-quitAfterTimer.C:
pauseTimer.Stop()
return prepareReturn(resp, err)
@@ -231,7 +230,6 @@ func (rx *ResumableUpload) Upload(ctx context.Context) (resp *http.Response, err
case <-quitAfterTimer.C:
return prepareReturn(resp, err)
default:
quitAfterTimer.Stop()
}
resp, err = rx.transferChunk(ctx)
@@ -243,6 +241,7 @@ func (rx *ResumableUpload) Upload(ctx context.Context) (resp *http.Response, err
// Check if we should retry the request.
if !errorFunc(status, err) {
quitAfterTimer.Stop()
break
}

View File

@@ -5,4 +5,4 @@
package internal
// Version is the current tagged release of the library.
const Version = "0.109.0"
const Version = "0.110.0"

View File

@@ -15,7 +15,6 @@ package cert
import (
"crypto/tls"
"errors"
"os"
"github.com/googleapis/enterprise-certificate-proxy/client"
)
@@ -36,8 +35,7 @@ type ecpSource struct {
func NewEnterpriseCertificateProxySource(configFilePath string) (Source, error) {
key, err := client.Cred(configFilePath)
if err != nil {
if errors.Is(err, os.ErrNotExist) {
// Config file missing means Enterprise Certificate Proxy is not supported.
if errors.Is(err, client.ErrCredUnavailable) {
return nil, errSourceUnavailable
}
return nil, err

View File

@@ -1,26 +0,0 @@
// Copyright 2021 Google LLC.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
//go:build go1.16
// +build go1.16
package http
import (
"net/http"
"time"
"golang.org/x/net/http2"
)
// configureHTTP2 configures the ReadIdleTimeout HTTP/2 option for the
// transport. This allows broken idle connections to be pruned more quickly,
// preventing the client from attempting to re-use connections that will no
// longer work.
func configureHTTP2(trans *http.Transport) {
http2Trans, err := http2.ConfigureTransports(trans)
if err == nil {
http2Trans.ReadIdleTimeout = time.Second * 31
}
}

View File

@@ -1,17 +0,0 @@
// Copyright 2021 Google LLC.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
//go:build !go1.16
// +build !go1.16
package http
import (
"net/http"
)
// configureHTTP2 configures the ReadIdleTimeout HTTP/2 option for the
// transport. The interface to do this is only available in Go 1.16 and up, so
// this performs a no-op.
func configureHTTP2(trans *http.Transport) {}

View File

@@ -16,6 +16,7 @@ import (
"time"
"go.opencensus.io/plugin/ochttp"
"golang.org/x/net/http2"
"golang.org/x/oauth2"
"google.golang.org/api/googleapi/transport"
"google.golang.org/api/internal"
@@ -175,13 +176,22 @@ func defaultBaseTransport(ctx context.Context, clientCertSource cert.Source) htt
}
}
// If possible, configure http2 transport in order to use ReadIdleTimeout
// setting. This can only be done in Go 1.16 and up.
configureHTTP2(trans)
return trans
}
// configureHTTP2 configures the ReadIdleTimeout HTTP/2 option for the
// transport. This allows broken idle connections to be pruned more quickly,
// preventing the client from attempting to re-use connections that will no
// longer work.
func configureHTTP2(trans *http.Transport) {
http2Trans, err := http2.ConfigureTransports(trans)
if err == nil {
http2Trans.ReadIdleTimeout = time.Second * 31
}
}
// fallbackBaseTransport is used in <go1.13 as well as in the rare case if
// http.DefaultTransport has been reassigned something that's not a
// *http.Transport.

42
vendor/modules.txt vendored
View File

@@ -1,4 +1,4 @@
# cloud.google.com/go v0.109.0
# cloud.google.com/go v0.110.0
## explicit; go 1.19
cloud.google.com/go/internal
cloud.google.com/go/internal/optional
@@ -10,7 +10,7 @@ cloud.google.com/go/compute/internal
# cloud.google.com/go/compute/metadata v0.2.3
## explicit; go 1.19
cloud.google.com/go/compute/metadata
# cloud.google.com/go/iam v0.10.0
# cloud.google.com/go/iam v0.12.0
## explicit; go 1.19
cloud.google.com/go/iam
cloud.google.com/go/iam/apiv1/iampb
@@ -63,15 +63,15 @@ github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/service
# github.com/VictoriaMetrics/fastcache v1.12.0
## explicit; go 1.13
github.com/VictoriaMetrics/fastcache
# github.com/VictoriaMetrics/fasthttp v1.1.0
## explicit; go 1.13
# github.com/VictoriaMetrics/fasthttp v1.2.0
## explicit; go 1.19
github.com/VictoriaMetrics/fasthttp
github.com/VictoriaMetrics/fasthttp/fasthttputil
github.com/VictoriaMetrics/fasthttp/stackless
# github.com/VictoriaMetrics/metrics v1.23.1
## explicit; go 1.15
github.com/VictoriaMetrics/metrics
# github.com/VictoriaMetrics/metricsql v0.51.2
# github.com/VictoriaMetrics/metricsql v0.56.1
## explicit; go 1.13
github.com/VictoriaMetrics/metricsql
github.com/VictoriaMetrics/metricsql/binaryop
@@ -81,7 +81,7 @@ github.com/VividCortex/ewma
# github.com/alecthomas/units v0.0.0-20211218093645-b94a6e3cc137
## explicit; go 1.15
github.com/alecthomas/units
# github.com/aws/aws-sdk-go v1.44.198
# github.com/aws/aws-sdk-go v1.44.204
## explicit; go 1.11
github.com/aws/aws-sdk-go/aws
github.com/aws/aws-sdk-go/aws/awserr
@@ -150,10 +150,10 @@ github.com/aws/aws-sdk-go-v2/internal/timeconv
## explicit; go 1.15
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream/eventstreamapi
# github.com/aws/aws-sdk-go-v2/config v1.18.12
# github.com/aws/aws-sdk-go-v2/config v1.18.13
## explicit; go 1.15
github.com/aws/aws-sdk-go-v2/config
# github.com/aws/aws-sdk-go-v2/credentials v1.13.12
# github.com/aws/aws-sdk-go-v2/credentials v1.13.13
## explicit; go 1.15
github.com/aws/aws-sdk-go-v2/credentials
github.com/aws/aws-sdk-go-v2/credentials/ec2rolecreds
@@ -166,7 +166,7 @@ github.com/aws/aws-sdk-go-v2/credentials/stscreds
## explicit; go 1.15
github.com/aws/aws-sdk-go-v2/feature/ec2/imds
github.com/aws/aws-sdk-go-v2/feature/ec2/imds/internal/config
# github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.11.51
# github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.11.53
## explicit; go 1.15
github.com/aws/aws-sdk-go-v2/feature/s3/manager
# github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.28
@@ -178,7 +178,7 @@ github.com/aws/aws-sdk-go-v2/internal/endpoints/v2
# github.com/aws/aws-sdk-go-v2/internal/ini v1.3.29
## explicit; go 1.15
github.com/aws/aws-sdk-go-v2/internal/ini
# github.com/aws/aws-sdk-go-v2/internal/v4a v1.0.19
# github.com/aws/aws-sdk-go-v2/internal/v4a v1.0.20
## explicit; go 1.15
github.com/aws/aws-sdk-go-v2/internal/v4a
github.com/aws/aws-sdk-go-v2/internal/v4a/internal/crypto
@@ -197,19 +197,19 @@ github.com/aws/aws-sdk-go-v2/service/internal/presigned-url
github.com/aws/aws-sdk-go-v2/service/internal/s3shared
github.com/aws/aws-sdk-go-v2/service/internal/s3shared/arn
github.com/aws/aws-sdk-go-v2/service/internal/s3shared/config
# github.com/aws/aws-sdk-go-v2/service/s3 v1.30.2
# github.com/aws/aws-sdk-go-v2/service/s3 v1.30.3
## explicit; go 1.15
github.com/aws/aws-sdk-go-v2/service/s3
github.com/aws/aws-sdk-go-v2/service/s3/internal/arn
github.com/aws/aws-sdk-go-v2/service/s3/internal/customizations
github.com/aws/aws-sdk-go-v2/service/s3/internal/endpoints
github.com/aws/aws-sdk-go-v2/service/s3/types
# github.com/aws/aws-sdk-go-v2/service/sso v1.12.1
# github.com/aws/aws-sdk-go-v2/service/sso v1.12.2
## explicit; go 1.15
github.com/aws/aws-sdk-go-v2/service/sso
github.com/aws/aws-sdk-go-v2/service/sso/internal/endpoints
github.com/aws/aws-sdk-go-v2/service/sso/types
# github.com/aws/aws-sdk-go-v2/service/ssooidc v1.14.1
# github.com/aws/aws-sdk-go-v2/service/ssooidc v1.14.2
## explicit; go 1.15
github.com/aws/aws-sdk-go-v2/service/ssooidc
github.com/aws/aws-sdk-go-v2/service/ssooidc/internal/endpoints
@@ -310,7 +310,7 @@ github.com/google/go-cmp/cmp/internal/value
# github.com/google/uuid v1.3.0
## explicit
github.com/google/uuid
# github.com/googleapis/enterprise-certificate-proxy v0.2.2
# github.com/googleapis/enterprise-certificate-proxy v0.2.3
## explicit; go 1.19
github.com/googleapis/enterprise-certificate-proxy/client
github.com/googleapis/enterprise-certificate-proxy/client/util
@@ -442,7 +442,7 @@ github.com/russross/blackfriday/v2
## explicit; go 1.13
github.com/stretchr/testify/assert
github.com/stretchr/testify/require
# github.com/urfave/cli/v2 v2.24.3
# github.com/urfave/cli/v2 v2.24.4
## explicit; go 1.18
github.com/urfave/cli/v2
# github.com/valyala/bytebufferpool v1.0.0
@@ -458,7 +458,7 @@ github.com/valyala/fastrand
# github.com/valyala/fasttemplate v1.2.2
## explicit; go 1.12
github.com/valyala/fasttemplate
# github.com/valyala/gozstd v1.17.0
# github.com/valyala/gozstd v1.18.0
## explicit; go 1.12
github.com/valyala/gozstd
# github.com/valyala/histogram v1.2.0
@@ -519,15 +519,15 @@ go.opentelemetry.io/otel/trace
# go.uber.org/atomic v1.10.0
## explicit; go 1.18
go.uber.org/atomic
# go.uber.org/goleak v1.2.0
# go.uber.org/goleak v1.2.1
## explicit; go 1.18
go.uber.org/goleak
go.uber.org/goleak/internal/stack
# golang.org/x/exp v0.0.0-20230206171751-46f607a40771
# golang.org/x/exp v0.0.0-20230213192124-5e25df0256eb
## explicit; go 1.18
golang.org/x/exp/constraints
golang.org/x/exp/slices
# golang.org/x/net v0.6.0
# golang.org/x/net v0.7.0
## explicit; go 1.17
golang.org/x/net/context
golang.org/x/net/http/httpguts
@@ -569,7 +569,7 @@ golang.org/x/time/rate
## explicit; go 1.17
golang.org/x/xerrors
golang.org/x/xerrors/internal
# google.golang.org/api v0.109.0
# google.golang.org/api v0.110.0
## explicit; go 1.19
google.golang.org/api/googleapi
google.golang.org/api/googleapi/transport
@@ -602,7 +602,7 @@ google.golang.org/appengine/internal/socket
google.golang.org/appengine/internal/urlfetch
google.golang.org/appengine/socket
google.golang.org/appengine/urlfetch
# google.golang.org/genproto v0.0.0-20230209215440-0dfe4f8abfcc
# google.golang.org/genproto v0.0.0-20230216225411-c8e22ba71e44
## explicit; go 1.19
google.golang.org/genproto/googleapis/api
google.golang.org/genproto/googleapis/api/annotations